Year: 2021

Abstract:
The idea of using the recurrent neural network for visual attention has gained popularity in computer vision community. Although the recurrent attention model (RAM) leverages the glimpses with more large patch size to increasing its scope, it may result in high variance and instability. For example, we need the Gaussian policy with high variance to explore object of interests in a large image, which may cause randomized search and unstable learning. In this paper, we propose to unify the top-down and bottom-up attention together for recurrent visual attention. Our model exploits the image pyramids and Q-learning to select regions of interests in the top-down attention mechanism, which in turn to guide the policy search in the bottom-up approach. In addition, we add another two constraints over the bottom-up recurrent neural networks for better exploration. We train our model in an end-to-end reinforcement learning framework, and evaluate our method on visual classification tasks. The experimental results outperform convolutional neural networks (CNNs) baseline and the bottom-up recurrent attention models on visual classification tasks.

heading: 1 INTRODUCTION
text: Recurrent visual attention model Mnih et al. (2014) (abbreviated as RAM) leverages reinforcement learning Sutton & Barto (1998) and recurrent neural networks Schuster et al. (1997); LeCun et al. (2015) to recognize the objects of interests in a sequential manner. Specifically, RAM models object recognition as a sequential decision problem based on reinforcement learning. The agent can adaptively select a series of regions based on its state and make decision to maximize the expected return. The advantages of RAM are follows: firstly, its computation time scales linearly with the complexity of the patch size and the glimpse length rather than the whole image, through a sequence of glimpses and interactions locally. Secondly, this model is very flexible and effective. For example, it considers both local and global features by using more number of glimpses, scales and large patch sizes to enhance its performance.
Unfortunately, the idea of using glimpse in the local region limits its understanding to the holistic view and restricts its power to capture the semantic regions in the image. For example, its initial location and patch extraction is totally random, which will take more steps to localize the target objects. Although we can increase the patch size to cover larger region, it will increase the computation time correspondently. In the same time, we need to set a high variance (if we use Gaussian policy) to explore unseen regions. Thinking a situation that you were in desert and you were looking for water, but you did not have GPS/Map over the whole scene. What would you do to find water? You might need to search each direction (east, south, north, and west), that is random searching and time-consuming. In this paper, we want to provide a global view to localize where to search. For example, the process of human perception is to scan the whole scene, and then may be attracted by specific regions of interest. In a high level view, it can help us to ignore non-important parts and focus information on where to look, and further to guide us to different fixations for better understanding the internal representation of the scene.
To solve ‘where to look’ using reinforcement learning, the desired algorithm needs to satisfy the following criterions: (1) it is stable and effective; (2) it is computationally efficient; (3) it must be more goal-oriented with global information; (4) instead of random searching, it attends regions of interest even over large image; (5) it has better exploration strategy on ‘where to look’. For a large
image, the weakness of RAM is exposed throughly, because it relies on local fixations to search the next location, which increasing the policy uncertainty as well as the learning time to optimizing the best in the interaction sequence. Inspired by the hierarchical image pyramids Adelson et al. (1984); Lin et al. (2017) and reinforcement learning Watkins & Dayan (1992); Kulkarni et al. (2016), we propose to unify the top-down and bottom-up mechanism to overcome the weakness of recurrent visual attention model.
In the high-level, we take a top-down mechanism to extract information at multiple scales and levels of abstraction, and learn to where to attend regions of interests via reinforcement learning. While in the low-level, we use the similar recurrent visual attention model to localize objects. In particular, we add another two constraints over the bottom-up recurrent neural networks for better exploration. Specifically, we add the entropy of image patch in the trajectory to enhance the policy search. In addition, we constrain the region that attend to should be more related to target objects. By combining the sequential information together, we can understand the big picture and make better decision while interacting with the environment.
We train our model in an end-to-end reinforcement learning framework, and evaluate our method on visual classification tasks including MNIST, CIFAR 10 classes dataset and street view house number (SVHN) dataset. The experimental results outperform convolutional neural networks (CNNs) baseline and the bottom-up recurrent attention models (RAM).

heading: 2 RELATED WORK
text: Since the recurrent models with visual attention (RAM) was proposed, many works have been inspired to either use RNNs or reinforcement learning to improve performance on computer vision problems, such as object recognition, location and question answering problems. One direction is the top-down attention mechanism. For example, Caicedo and Lazebnik extends the recurrent attention model and designs an agent with actions to deform a bounding box to determine the most specific location of target objects Caicedo & Lazebnik (2015). Xu et al. (2015) introduces an attention based model to generate words over an given image. Basically, it partitions image into regions, and then models the importance of the attention locations as latent variables, that can be automatically learned to attend to the salient regions of image for corresponding words. Wang et al. Wang et al. (2018) leverages the hierarchical reinforcement learning to capture multiple fine-grained actions in sub-goals and shows promising results on video captioning.
Another trend is to build the interpretation of the scene from bottom-up. Butko & Movellan (2009) builds a Bayesian model to decide the location to attend while interacting with local patches in a sequential process. Its idea is based on reinforcement learning, such as partially observed Markov decision processes (POMDP) to integrate patches into high-level understanding of the whole image. The design of RAM Mnih et al. (2014) takes a similar approach as Butko & Movellan (2008; 2009), where reinforcement learning is used to learn ‘where to attend’. Specifically, at each point, the agent only senses locally with limited bandwidth, not observe the full environment. Compared Butko & Movellan (2009), RAM leverages RNNs with hidden states to summarize observations for better understanding the environment. Similarly, drl-RPN Pirinen & Sminchisescu (2018) proposes a sequential attention model for object detection, which generates object proposal using deep reinforcement learning, as well as automatically determines when to stop the search process.
Other related work combine local and global information to improve performance on computer vision problems. Attention to Context Convolution Neural Network (AC-CNN) exploits both global and local contextual information and incorporates them effectively into the region-based CNN to improve object detection performance Li et al. (2017). Anderson et al. Anderson et al. (2018) proposes to combine bottom-up and top-down attention mechanism for image captioning and visual question answering. They uses the recurrent neural networks (specifically, LSTM) to predict an attention distribution over image regions from bottom-up. However, no reinforcement learning technique used to adaptively focus objects and other salient image regions of interests.
RAM learns to attend local regions while interacting with environment in a sequential decision process with the purpose to maximum the cumulative return. However, if the scope of local glimpse is small, then we lose the high level context information in the image when making decision in an sequential manner. In contrast, if we increase the patch size to the scale as large as the original
image, then we lose the attention mechanism, as well as its efficiency of this model. Moreover, when we increase the scale with the fixed patch size, we will deprive distinguished features after resizing back to original patch size. Also it is a challenge to balance the Gaussian policy variance and patch size. Since the initial location is randomly generated, we need to set a large variance in Gaussian policy to increase its policy exploration. However, it will result in high instability in the learning process. In this paper, we propose to unify bottom-up and top-down mechanism to address these issues mentioned above. In addition, we introduce entropy into policy gradient in order to attend regions with high information content.

heading: 3 BACKGROUND

heading: 3.1 REINFORCEMENT LEARNING
text: The objective of reinforcement learning is to maximize a cumulative return with sequential interactions between an agent and its environment Sutton & Barto (1998). At every time step t, the agent selects an action at in the state st according its policy and receives a scalar reward rt(st, at), and then transit to the next state st+1 with probability p(st+1|st, at). We model the agent’s behavior with πθ(a|s), which is a parametric distribution from a neural network. Suppose we have the finite trajectory length while the agent interacting with the environment. The return under the policy π for a trajectory τ = (st, at) T t=0
J(θ) = Eτ∼πθ(τ)[ T∑ t=0 γtr(st, at)] = Eτ∼πθ(τ)[R T 0 ] (1)
where γ is the return discount factor, which is necessary to decay the future rewards ensuring bounded returns. πθ(τ) denotes the distribution of trajectories below
ρ(τ) = π(s0, a0, s1, ..., sT , aT )
= p(s0) T∏ t=0 πθ(at|st)p(st+1|st, at) (2)
The goal of reinforcement learning is to learn a policy π which can maximize the expected returns.
θ = arg max J(θ) = arg maxEτ∼πθ(τ)[R T 0 ] (3)
Policy gradient: Take the derivative w.r.t. θ
∇θJ(θ) = ∇θEτ∼πθ(τ)[R T 0 ] = ∇θ ∫ ρ(τ)RT0 dτ
= ∫ ∇θρ(τ)RT0 dτ = ∫ ρ(τ) ∇θρ(τ) ρ(τ) RT0 dτ
= Eτ∼πθ(τ)[∇θlogπθ(τ)R T 0 ]
= Eτ∼πθ(τ)[∇θlogπθ(τ)(R T 0 − b)] (4)
Q-learning: The action-value function describes what the expected return of the agent is in state s and action a under the policy π. The advantage of action value function is to make actions explicit, so we can select actions even in the model-free environment. After taking an action at in state st and thereafter following policy π, the action value function is formatted as:
Qπ(st, at) = Eτ∼πθ(τ)[Rt|st, at]
= Eτ∼πθ(τ)[ T∑ i=t γ(i−t)r(si, ai)|st, at] (5)
To get the optimal value function, we can use the maximum over actions, denoted as Q∗(st, at) = maxπ Q
π(st, at), and the corresponding optimal policy π can be easily derived by π∗(s) ∈ arg maxat Q ∗(st, at).

heading: 3.2 RECURRENT ATTENTION MODEL
text: Instead of processing the whole image X , recurrent attention model (RAM) Mnih et al. (2014) uses the recurrent neural networks (RNN) to guide the agent to make a sequential decision process while interacting with a visual environment. At the time step t, the agent is at location Lt, observing image patch xt with different patch size, and then it makes decision to next location Lt+1. Finally, the agent executes action and receives a scalar reward (which will be used to correct its decision). In classification task, the agent will get positive reward if it predicts right. The goal of the agent is to maximize the total sum of such rewards.
Since RAM extracts one patch at each time step, the number of pixels it processes in the glimpse xt is much smaller than that in the original image X . Compared to deep neural networks, it is much efficient, i.e. the computational cost of a single glimpse is independent of the size of the image. The model structure of RAM is listed below with paraphrase:
Glimpse network fg: it extracts features around position Lt at different scales via its sensor, and then combines visual features and position features together via glimpse network fg to generate the glimpse feature vector Gt.
RNN fh: it takes the glimpse vector Gt and Ht−1 to output the next the hidden state Ht, which summarizes its history knowledge over the environment.
Action network fl: it uses the output from fh, and then decides the agent next location Lt based on the internal hidden state Ht.
Classification network fy: it integrates all information to the current state in the sequence and decides the class of the sequence or image (which further decides the reward the agent will receive).

heading: 4 UNIFIED VISUAL ATTENTION MODEL
text: In this section, we present an unified framework, which combines top-down and bottom-up mechanism for visual attention.

heading: 4.1 TOP-DOWN ATTENTION MODEL
text: The top-down attention mechanism is implemented to attend significant regions by following coarseto-fine image pyramid. In the coarse level, we divide the image into regions, and then select the region based on its importance. Further, we map the region into the next level pyramid with higher resolution and divide it into subregions and repeat the process until to the final level (original image). In Figure 1(a), we show an example image, which is divided into 2× 2 regions in the coarsest level.
In the next level, each region repeats this like quad tree image representation. How to select one from four regions is based on the reward backups from the bottom-up attention model. Since we divide the current level image into 2 × 2 regions, we need a strategy to select one region to look, which tries to solve “where to look” in our top-down attention model.
The action that learns which region to look is based on Q-learning, which is parametrized by 2-layer networks in our work. As for the agent, the state is the image (representation) at the current level `, and its actions are to choose one of its 4 regions, which can map to subregions in the level ` + 1. The reward is from the bottom-up model, which will be introduced in the next part. For the object recognition problem, if we classify the image correctly from bottom-up mechanism, then we get positive reward, otherwise negative reward backward to top-down attention model.
As for the network structure, we can use full connected neural networks (or CNN) to extract a set of features from the image in each level in the hierarchy. For instance, if we partition the low resolution image intoA regions (where we useA to denote the action space), then we get the |A| vectors, each of which represents as N dimensional vector corresponding to a part of the image in that level
` = {v1,v2, ...,v|A|},vi ∈ RN (6) where each region is represented as N -dimensional vector vi. Then we concatenate all vectors together to form the (sub)image representations at the current level
s` = [v1, ...,v|A|], s` ∈ R|A|N (7) To select one region from the total A regions in each level, we leverage Q-learning, which as an off-policy RL algorithm and has been extensively studied since it was proposed Watkins & Dayan (1992). The Q-value function q(s, a) should output |A|-dimensional values to indicate the importances of the A-regions. correspondently. Thus, we add another layer of full connected networks which outputs |A|-dimensional values to decide the probability (softmax) to decide the next region to look at in the next level. In our case, the agent needs to select regions according to its q-value.
Suppose we use neural network parametrized by θq to approximate Q-value in the interactive environment. To update Q-value function, we minimize the follow loss:
y` = r(s`, a`) + γmax a`+1 Q(s`+1, a`+1; θq) (8)
L(θq) = Es`∼pπ,a`∼π[(Q(s`, a`; θq)− y`)2] (9) where s`+1 is the state (or image region) in the level ` + 1, a`+1 is the action to pick one region from the total A regions. y` is from Bellman equation, and we update it from its action at+1 ∈ {1, 2, .., |A|} which is taken from frozen policy network (actor) to stabilizing the learning. Since we don not know which region to focus until the last layer reward from the bottom-up model, so we design r(s`, a`) = 1 if ` is the last pyramid layer (original image) as well as we recognize object correctly from bottom-up model, otherwise r(s`, a`) = 0. Q(s`, a`; θq) is approximated with a two-layer neural network (one convolution layer followed with fully connected layer).
In the current image level `, the agent takes step to select one region to focus on according to qvalue, and then further repeat to select subregions based on q-value in the next level ` + 1. How to learn the q-value function will be dependent on the final reward in the bottom-up model in the next section. We can minimize Eq. 9 to learn parameter θq .

heading: 4.2 BOTTOM-UP ATTENTION MECHANISM
text: Our bottom-up attention model takes the similar approach as RAM Mnih et al. (2014), to use the recurrent neural networks (RNN) to guide the agent to make a sequential decision process while interacting with a visual environment.
At each location, the agent observes the low-level scene properties in a local region and build up bottom-up processes over time to integrate information from visual environment in order to determine how to act and how to deploy its sensor most effectively.
Gt = fg(xt, Lt; θg) (10a) Ht = fh(Ht−1, Gt; θh) (10b) Lt = fl(Ht; θl) (10c) αt = fy(Ht; θy) (10d)
where xt is the local patch or observation of the visual environment (image), with location specified by Lt. fg combines visual feature Gimg(xt) and location feature Gloc(Lt) together, where visual feature Gimg(xt) consists of three convolutional hidden layers with normalization and pooling layers followed by a fully connected layer, and Gloc(Lt) is from full connected hidden layer. And we use Gt = Gimg(xt) + Gloc(Lt) to get glimpse feature at Lt. Ht integrates the previous hidden state and Gt to generate next hidden state, in order to capture the whole sequential information. fy is the classifier, which integrates all information to the current state in the sequence and decides the class of the image (which further decides the reward the agent will receive). it is formulated using a softmax output and for dynamic environments, its exact formulation depends on the action set defined for that particular environment. For classification with K classes, we have ŷ = arg maxk∈K αtk,αt ∈ R1×K . Compared to RAM, we add anther two constraints which balances the trade-off between exploration and exploitation. On the one hand, we want to take action to select high entropy regions to better explore the whole environment. On the other hand, we hope the regions selected have high confidence on classification task. To sum up, our contributions are followings:
(1) Context constraint: the basic idea is that for the regions selected in policy search, we should assign them more weights. αt is the softmax output at each step t, which in fact is the weight to measure the importance of the current hidden state, or whether we should attend to current location. Once the weight αt (which sums to one) are computed, then the global context vector C and its softmax output ẑ is computed as
C = T∑ t=1 αtyHt
ẑ = fc(C) (11)
where fc is the two-layer networks, with a fully connected layer and a softmax output layer. We hope ẑ matches its groundtruth ẑ = y.
(2) Entropy constraint with better exploration: Entropy has been widely use to measure information, where a higher entropy value indicates a relatively richer information content. As a bottom-up algorithm, we want our model gives more weight on regions with higher entropy. Specifically, we want to skip all black or white regions while searching objects. Thus, for each region we extracted, we threshold it and binarialize it into 2 bins and then compute its entropy. In order to take action towards high entropy regions, we modify the reward function as
J(θ) = Eτ∼πθ(τ)logπθ(τ)(entropy(xt) + λ)(R T 0 − b) (12)
where entropy(xt) is the entropy over the patch located by Lt, λ is the constant to scale the importance at the current time step, R0 is the reward over sequence and b is the baseline which can be approximated by neural network. Since the action to localize the next region is backward from reward, then we decide reward R0 as: if the model makes the right decision, then R0 = 1, otherwise R0 = 0. The location network fl decides action at Lt, and the policy gradient can be formalized as
∇θJ(θ) = Eτ∼πθ(τ)[∇θlogπθ(τ)(entropy(xt) + λ)(R T 0 − b)] (13)
To learn action in Lt, we use Gaussian policy with variance σ which measures the exploration scale.
The final objective function in the bottom-up algorithm is hybrid supervised loss as follow:
loss = −ylogαT − β1ylogẑ + β2J(θ) (14)
where β1 and β2 are the weights to balance the importance of each term.

heading: 4.3 ALGORITHM
text: We outline the sketch of our training process in Algorithm 1. Note that we use another frozen Q-learning network to stabilize the learning procedure.
Algorithm 1 Unified visual attention model 1: Initialize model parameters in Eq. 9 and Eq. 10, 2: for each epoch do 3: for each image, label(y) do 4: Decide the action ai = maxa` Q(s`, a`); 5: Get the corresponding region vi via ai ∈ A; 6: Randomly sample location L0 from image region vi; 7: Randomly generate initial hidden state H0; 8: for each glimpse t = 0 to T do 9: Run the RNN model to decide next location to focus via Eq. 10a, 10b and 10c; 10: if condition is ok then 11: Run classifier via Eq. 10d; 12: ŷt = argmaxk∈K αtk; 13: end if 14: end for 15: Evaluate the sequential reward R = [y == ŷ] 16: if R is correct in the whole sequence then 17: r(s`, a`) = 1; 18: else 19: r(s`, a`) = 0; 20: end if 21: Update Q-learning (top-down) model parameter θq by minimizing loss in Eq. 9; 22: Update bottom-up model parameters {θg, θh, θl, θy} by minimizing loss in Eq. 14; 23: end for 24: end for

heading: 5 EXPERIMENTS
text: We evaluated our approach on MNIST, CIFAR10 and SVHN datasets. As for parameter setting, we use 2-level pyramid representation in the top-down q-learning: the coarse and the original image level. Basically, the coarse level image is the half resolution (size) of the original image, and we divide the coarse image into 2 by 2 subregions, then we can learn the location in the coarse level, which in turn can be mapped back to the original image. In the bottom-up attention model, we use Adam algorithm with learning rate 3e − 4, batch size 128, λ = 0.5, β1 = 1, β2 = 0.01 and σ = 0.15. Without other specification, we use the same parameter setting above in all the following experiments. MNIST classification: MNIST dataset is handwritten images (28×28) with digital numbers ranging from 0 to 9. In this experiment, we tested our method by varying MNIST dataset including original images and cluttered translated images with different size. The MNIST training set has total 60000 images, in which we use 54000 images for training and the rest 6000 for validation. Then we evaluate the performance on test dataset with 10,000 examples. As for the experimental settings, we use the similar RNN architecture (in Appendix) as RAM to make a fair comparison.
Patch feature encoding: at the location Lt, we extract m square patches with different scales. Then we concatenate features at different scales together to get the final patch representation xt centered at location Lt. Note that Lt is a two dimensional vector normalized in the range [-1, 1], with the image center (0, 0) and the right bottom (1, 1).
Since the digits in MNIST are centered, there is no need to use top-down mechanism “where to look” to initialize the search. Thus, we only leverage the bottom-up strategy as RAM, but we introduce new constrains in Eqs. 11 and 12 and want to test whether these new features get performance gain or not. The evaluation result on the MNIST test dataset is shown in Table 1. Using 6 glimpses, patch size 8 × 8 and 1 scale (where m = 1), our method with entropy beats RAM under the same parameter setting. It demonstrates that the entropy constraint in Eq. 12 is helpful. In addition, we tested whether the context constraint in Eq. 11 contributes or not, and it also shows that it can improve the classification task with more gain. Cluttered Translated MNIST classification: we use the same network architecture given in the MNIST classification experiment, but evaluate on the cluttered and translated digital images. In this experiment, we use the same protocol as RAM to create the cluttered and translated MNIST data. First, we place an MNIST digit in a random location of a larger blank image 100×100 and then add 8×8 subpatches which are sampled randomly from other random MNIST digits to random locations of the image.
Table 2 shows the classification results for the models we trained on 100 by 100 Cluttered Translated MNIST with 10 pieces of clutter. The presence of clutter is kind of random noise, which makes the task much more difficult. Since our model unifies top-down and bottom-up attention, it can find the meaningful regions and avoid the negative impact from noising subpatches. In Table 2, we can see the performance of our attention model is affected less than the performance of the other models. Using the same parameters as RAM with 6 glimpses, 12 × 12 and 4 scales, our approach gains almost 3%, which demonstrates that our top-down and bottom-up attention unification is more powerful than RAM. By adding the entropy and global context constraints, our method gains another 3%. We can observe the similar results while using 8 glimpses.
Sequential multi-digit recognition: since the core network is based on recurrent neural networks, we can easily extend it to handle multi-digit recognition Goodfellow et al. (2014); Ba et al. (2015). Suppose that we have training images with the target {y1, y2, ..., yT }, where we can use y1 marks the length of sequence or use yT to indicates the end of sequence. The task is to predict the digits as it explores the local image glimpses.
We take the same protocol as Goodfellow et al. (2014) on street view house number recognition task. Specifically, we crop 64 x 64 images with multi-digits at the center and then randomly sample
to create 54x54 jittered images with similar data augmentation. We have 212243 images for training (73257 digits from training dataset and the rest from 531131 additional image), 23511 digits for validation and 26032 digits for testing. Same as DRAM Ba et al. (2015), our attention model observes 3 glimpses for each digit before making a prediction. The recurrent model keeps running until it predicts a terminal label or until the longest digit length in the dataset is reached. In the SVHN dataset, up to 5 digits can appear in an image, so it means the recurrent model will run up to 18 glimpses per image, that is 5 x 3 plus 3 glimpses for a terminal label.
To conduct a fair comparison, we use the same architecture as DRAM (refer in Appendix). The experiment results is shown in Table 3. Compared to DRAM, our model gains better result with an error rate 4.03%. And our result is better than 10 layer CNN, and it is also comparable to the 11 layer CNN Goodfellow et al. (2014), which uses more complex network architecture and more model parameters.
Robust to PGD attack: in this experiment, we analyzed and answered whether our patch-based attention model is robust to adversarial attack or not. Specifically, we will leverage projected gradient descent (PGD) to generate adversarial examples because it has been widely used to attack deep neural networks Madry et al. (2018); Pintor et al. (2021). PGD is a very powerful adversary approach, which has the following projected gradient descent on the negative loss function:
xt+1 = Πx+S ( xt + sign(∇x`(x, y; θp)) ) (15)
where `(x, y; θp) is the loss function defined in Eq. 14, S ⊆ Rd formalizes the variance of manipulative space of the adversary attack, and θp is the set of model parameters. In order to test our model’s robustness to PGD, we change equation equation 10d as follows:
feat = VGG(x) (16) αt = fy(cat(Ht, feat); θy) (17)
where VGG is the 16-layer neural networks defined in Simonyan & Zisserman (2015), and cat indicates feature concatenation. Our attention model use 6 glimpses with patch size 12×12 to make prediction on each image. As for feature extraction, we concatenate VGG16’s output with Gt as input in the RNN network. In the backpropagation stage, we can minimize loss in Eq. 14 and then compute gradient w.r.t. feat in Eq. 16, so we can further calculate gradient w.r.t. x based on chain rule. After we get∇x`(x, y; θp), we can update x to generate adversarial examples according to Eq. 15. In CIFAR10 classification task, we use step = 30 and = 0.3 in Eq. 15.
Our attention model takes a patch-based approach and then sequentially decides the next location/patch to minimize the loss in Eq. 14. The result in Table 4 shows that our model is more robust to PGD attack. The intuition behind this is that our model can select better patch in each glimpse, and further it can learn some useful structure information to do better prediction.

heading: 6 CONCLUSION
text: In this paper, we propose a visual attention approach, which unifies the top-down attention mechanism with the bottom-up recurrent attention model. Considering bottom-up RAM randomly initializes the location to search target object, our top-down attention model is to answer ‘where to look’ the best initial location in an image. In addition, we introduce global context and entropy constraints, in order to focus attention on edge, instead of pure black/while regions. To sum, our model leverages reinforcement learning to combine top-down and bottom-up mechanism, both of which depends the reward from classification to decide next locations to attend in a sequential manner. We conduct a bunch of experiments on MNIST, CIFAR10 and SVHN datasets, and our model shows better result than RAM, DRAM and CNN (complex architecture). We also show our model is more robust than other neural network models on recognizing noise images.

heading: A APPENDIX
text: MNIST network architecture:
Glimpse network Gt = fg(Xt, Lt; θg): fg is two-layer fully connected (FC) neural networks:
hg = Rect(Linear(Xt)), hl = Rect(Linear(Lt))
Gt = Rect(Linear(hg + ht))
where hg and hl both have dimensionality 128, while the dimension ofGt is 256 for all experiments.
Recurrent network: Ht = fh(Ht−1, Gt; θh) to remember the sequence of glimpses in order to make action to get best return. Ht = Rect(Linear(Ht−1) + Linear(Gt)))
Action network: Lt = fl(Ht; θl) is the policy network, where its mean is the linear network over Ht and its variance σ is fixed.
SVHN network architecture:
the glimpse network is composed of three convolutional layers where the first layer is with filter size 5 × 5 and the later two layers with filter size 3 × 3. The number of filters in those layers was {64, 64, 128} respectively. The recurrent model was 2-layer LSTM, with 512 hidden units in each layer.

Review summaries:
This submission proposes to offer a better initialization by using image pyramids and Q-learning (top-down manner) for the original recurrent visual attention (RAM) model (bottom-up manner). Two new constraints are also proposed for better exploration for RAM. The proposed method has been tested on several image classification datasets, including MNIST, cluttered translated MNIST, SNHN (sequential multi-digit recognition). They also test the robustness to adversarial attack (PGD attack) on CIFAR10.

This paper extends the recurrent attention model(RAM) with another extra top-down attention. Specifically, they exploit image pyramids and Q-learning to select regions-of-interest first in the top-down attention mechanism, and then follow RAM to use policy gradient to find the patch in the bottom-up attention. Meanwhile, they also propose two loss constraints to further boost the performance of bottom-up recurrent neural networks. The proposed framework is an end-to-end framework. Experiments on three datasets (MNIST, CIFAR 10, and SVHN) have demonstrated the effectiveness of the proposed model.