Unifying Top-down and Bottom-up for Recurrent Visual Attention
Authors: GANG CHEN
Venue: ICLR 2022 Submitted (2022)
Keywords: visual attention model
reinforcement learning

Abstract: The idea of using the recurrent neural network for visual attention has gained popularity in computer vision community. Although the recurrent visual attention model (RAM) leverages the glimpses with more large patch size to increasing its scope, it may result in high variance and instability. For example, we need the Gaussian policy with high variance to explore object of interests in a large image, which may cause randomized search and unstable learning. In this paper, we propose to unify the top-down and bottom-up attention together for recurrent visual attention. Our model exploits the image pyramids and Q-learning to select regions of interests in the top-down attention mechanism, which in turn to guide the policy search in the bottom-up approach. In addition, we add another two constraints over the bottom-up recurrent neural networks for better exploration. We train our model in an end-to-end reinforcement learning framework, and evaluate our method on visual classification tasks. The experimental results outperform convolutional neural networks (CNNs) baseline and the bottom-up recurrent models with visual attention.

Under review as a conference paper at ICLR 2022
UNIFYING TOP-DOWN AND BOTTOM-UP RECURRENT ATTENTION MODEL FOR VISUAL RECOGNITION
Anonymous authors Paper under double-blind review
ABSTRACT
The idea of using the recurrent neural network for visual attention has gained popularity in computer vision community. Although the recurrent attention model (RAM) leverages the glimpses with more large patch size to increasing its scope, it may result high variance and instability. For example, we need the Gaussian policy with high variance to explore object of interests in a large image, which may cause randomized search and unstable learning. In this paper, we propose to unify the top-down and bottom-up attention together for recurrent visual attention. Our model exploits the image pyramids and Q-learning to select regions of inter- ests in the top-down attention mechanism, which in turn to guide the policy search in the bottom-up approach. In addition, we add another two constraints over the bottom-up recurrent neural networks for better exploration. We train our model in an end-to-end reinforcement learning framework, and evaluate our method on vi- sual classification tasks. The experimental results outperform convolutional neural networks (CNNs) baseline and the bottom-u up recurrent attention models on visual classification tasks.
INTRODUCTION
Recurrent visual attention model Mnih al. (2014) (abbreviated as RAM) leverages reinforcement learning Sutton & Barto (9998) and recurrent neural networks Schuster et al. (1997): LeCun et al. (2015) to recognize the objects of interests in sequential manner. Specifically. ly. RAM models ob- ject recognition as a sequential decision problem based on reinforcement learning. The agent can adaptively select series of regions based on its state and make decision maximize the expected return. The advantages of RAM are follows: firstly, its computation time scales linearly with the complexity of the patch size and the glimpse length rather than the whole image, through a sequence of glimpses and interactions locally. Secondly, this model is very flexible and effective. For exam- ple, it considers both local and global features by using more number of glimpses, seales and large patch sizes enhance its performance.
Unfortunately, the idea of using glimpse in the local region limits its understanding to the holistic view and restricts its power to capture the semantic regions in the image. For example, its initial location and patch extraction is totally random, which will take more steps to localize the target objects. Although can increase the patch size cover larger region, it will increase the compu- lation time correspondently. In the same time, we need to set high variance (if we use Gaussian policy) explore unseen regions. Thinking a situation that you were in desert and you were looking for water, but you did not have GPS/Map over the whole scene. What would you do to find water? You might need to search each direction (east, south, north, and west), that is random searching and time-consuming. In this paper, we want to provide a global view to localize where to search. For example, the process of human perception by specific regions of interest. In high level view, it can help us to ignore non-important parts and focus information on where to look, and further to guide us to different fixations for better understanding the internal representation of the scene.
to scan the whole scene, and then may be attracted
To solve 'where to look' using reinforcement learning. the desired algorithm needs to satisfy the following criterions: (1) it is stable and effective; (2) is computationally efficient; (3) it must be more
with global information; (4) instead of random searching, it attends regions of interest even over large image; (5) it has better exploration strategy 'where look'. For large

---

Under review as a conference paper at ICLR 2022
image, the weakness of RAM is exposed throughly, because it relies local fixations search the next location, which increasing the policy uncertainty as well as the learning time to optimizing the best in the interaction sequence. Inspired by the hierarchical image pyramids Adelson et al. (1984); Lin et al. 2017) and reinforcement learning Watkins & Dayan 1992); Kulkarni et al. (2016), we propose to unify the top-down and bottom-up mechanism to overcome the weakness of recurrent visual attention model.
In the high-level, we take top-down mechanism extract information at multiple scales and levels of abstraction, and learn to where to attend regions of interests via reinforcement learning. While in the low-level, we use the similar recurrent visual attention model localize objects. In particular, we add another two constraints over the bottom-up recurent neural networks for better exploration. Specifically, we add the entropy of image patch the trajectory to enhance the policy search. In addition, we constrain the region that attend to should be more related to target objects. By combin- ing the sequential information together, we can understand the big picture and make better decision while interacting with the environment.
We train our model in an end-to-end reinforcement learning framework, and evaluate our method on visual classification tasks including MNIST, CIFAR 10 classes dataset and street view house num- ber (SVHN) dataset. The experimental results outperform convolutional neural networks (CNNs) baseline and the bottom-up recurrent attention models (RAM).
2 RELATED WORK
Since the recurrent models with visual attention (RAM) was proposed, many works have been in- spired to either use RNNs reinforcement learning improve performance computer vision problems, such as object recogmition, location and question answering problems. One direction is the top-down attention mechanism. For example, Caicedo and Lazebnik extends the recurrent at- tention model and designs an agent with actions to deform a bounding box to determine the most specific location of target objects Caicedo & Lazebnik (2015). Xu al.] (2015) introduces an at- tention based model to generate words over an given image. Basically, it partitions image into regions, and then models the importance of the attention locations as latent variables, that can be automatically learned to attend to the salient regions of image for corresponding words. Wang et al. Wang et al. (018)) leverages the hierarchical reinforcement learning to capture multiple fine- -raiira actions in gob-goals and shows promising results on video captioning.
Another trend to build the interpretation of the scene from bottom-up. Butko Movellan (2009) builds a Bayesian model to decide the location to attend while interacting with local patches in sequential process. Its idea is based on reinforcement learning, such as partially observed Markov decision processes (POMDP) to integrate patches into high-level understanding of the whole im- age. The design of RAM Mnih et al. (2014) takes a similar approach as Butko & Movellan (2008) 2009]. where reinforcement learning is used to learn 'where to attend'. Specifically, at each point, the agent only senses locally with limited bandwidth, not observe the full environment. Compared Butko & Movellan (2009), RAM leverages RNNs with hidden states summarize observations for better understanding the environment. Similarly, drI-RPN Pirinenn & Sminchisescu (2018) pro- poses a sequential attention model for object detection, which generates object proposal using deep reinforcement learning, as well as automatically determines when to stop the search process.
Other related work combine local and global information to improve performance on computer vi- sion problems. Attention to Context Convolution Neural Network (AC-CNN) exploits both global and local contextual information and incorporates them effectively into the region- based CNN to im- prove object detection performance [i et al. (2017). Anderson et al. Anderson et al. (2018) proposes to combine bottom-up and top-down attention mechanism for image captioning and visual question answering. They uses the recurrent neural networks (specifically, LSTM) to predict attention dis- tribution over image regions from bottom-up. However, no reinforcement learning technique used to adaptively focus objects and other salient image regions of interests.
RAM learns to attend local regions while interacting with environment in a sequential decision process with the purpose to maximum the cumulative return. However, if the scope of local glimpse is small, then we lose the high level context information in the image when making decision in an sequential manner. In contrast, if we increase the patch size to the scale large as the original
2

---

Under review as a conference paper at ICLR 2022
image, then we lose the attention mechanism, well its efficiency of this model. Moreover, when we increase the scale with the fixed patch size, we will deprive distinguished features after resizing back to original patch size. Also it is challenge to balance the Gaussian policy variance and patch size. Since the initial location is randomly generated, we need to set a large variance in Gaussian policy increase its policy exploration. However, it will result in high instability in the learning process. In this paper, we propose to unify bottom-up and top-down mechanism to address these issues mentioned above. In addition, we introduce entropy into policy gradient in order to attend regions with high information content.
3 BACKGROUND
3.1 REINFORCEMEN LEARNING
The objective of reinforcement learning to maximize a cumulative return with sequential interac- tions between an agent and its environment /Sutonn & Barto (1998). At every time step the agent selects an action a, in the state 8; according its policy and receives a scalar reward (8 ae), and then transit to the next state 8(+1 with probability p(*++1 1 az). We model the agent's behavior with Th (s)s,, which is a parametric distribution from a neural network.
Suppose we have the finite trajectory length while the agent interacting with the environment. The return under the policy for trajectory T (82, (:)=o
J(0) E (TT Σ (rr(s), at)) E [R))
where is the return discount factor, which is necessary to decay the future rewards ensuring bounded returns. no (T) denotes the distribution of trajectories below
p(T) T((8 do 81, 8T, aT)
T
= p(00) II
no po
The goal of reinforcement learning is learn policy which maximize the expected returns. 0 arg max [(0) arg max Er (R)
(3)
Policy gradient: Take the derivative w.r.t.
VoJ(0)-
[11 Vo R( dr
/ Vop(T) Rf dr= p(T) Vop(T) p(T)
Rig dr
V Velogmo (T) = E
Er Volog (olog=o(r)) (T) b)]
(4)
Q-learning: The action-value function describes what the expected return of the agent is in state and action under the policy æ. The advantage of action value function is make actions explicit, so we can select actions even in the model-free environment. After taking an action a, in state 8 and thereafter following policy æ, the action value function is formatted as:
Qat (st, ar) E ( ozl
(
(ss,
(5)
To get the optimal value function, max, (=(ss, az), and the corresponding optimal policy æ can easily derived by x xs(s) E arg max,, (8( at).
the maximum over actions, denoted as Q*(ss,

---

Under review as a conference paper at ICLR 2022
6
Top-dou atsention model
Clasiitattion
Regions of Interects
New location
madel
Location
(a) Top-down model
(b) Bottom-up model
Figure (a) The top-down attention model: divides the region (marked as red) into sub- regions in the current resolution, and then concatenates features of each region (extracted from neural networks) as in the input to action network to decide which sub-region should choose to attend; This process repeats select sub-region in the next level of image pyramid (b) the bottom- up attention model: based on the location output from the top-down attention model, extracts features at different scales the current location and then combines location feature input to recurrent neural networks and action network to decide next location.
3.2 RECURRENT ATTENTION MODEL
Instead of processing the whole image recurrent attention model (RAM) Mnih al. (2014) uses the recurrent neural networks (RNN) to guide the agent to make a sequential decision process while interacting with a visual environment. At the time step t, the agent is location L,, observing image patch with different patch size, and then it makes decision to next location L++1. Finally, the agent executes action and receives a scalar reward (which will be used to correct its decision). In classification task, the agent will get positive reward if it predicts right. The goal of the agent is to maximize the total sum of such rewards.
Since RAM extracts patch each time step, the number of pixels it processes in the glimpse zt much smaller than that in the original image X. Compared deep neural networks, it is much efficient, iee. the computational cost of single glimpse is independent of the size of the image. The model structure of RAM is listed below with paraphrase:
Glimpse network fy: it extracts features around position L, different scales via its sensor, and then combines visual features and position features together via glimpse network fy to generate the glimpse feature vector Gt.
RNN f:: it takes the glimpse vector G, and H to output the next the hidden state H,, which summarizes its history knowledge over the environment.
Action network fi: it uses the output from fr. and then decides the agent next location L, based on the internal hidden state Hr.
Classification network fy: it integrates all information to the current state in the sequence and de- cides the class of the sequence image (which further decides the reward the agent will receive).
UNIFIED VISUAL ATTENTION MODEL
In this section, we present an unified framework, which combines top-down and bottom-up mecha- nism for visual attention.
4.1 TOP-DOWN ATTENTION MODEL
The top-down attention mechanism is implemented to attend significant regions by following coarse- to-fine image pyramid. the coarse level, we divide the image into regions, and then select the region based on its importance. Further, map the region into the next level pyramid with higher resolution and divide it into subregions and repeat the process until to the final level (original image). In Figure I a), show example image, which divided into 2 x regions in the coarsest level.
4

---

Under review as a conference paper at ICLR 2022
In the next level, each region repeats this like quad tree image representation. How to select one from four regions is based on the reward backups from the bottom-up attention model. Since we divide the current level image into 2 x 2 regions, we need strategy to select one region to look, which tries to solve "where to look in our top-down attention model.
The action that learns which region look is based Q-learming, which is parametrized by 2-layer networks in our work. As for the agent, the state is the image (representation) at the current level e, and its actions are choose one of its 4 regions, which can map to subregions in the level + 1. The reward is from the bottom-up model, which will be introduced in the next part. For the object recognition problem. if we classify the image correctly from bottom-up mechanism, then we get positive reward, otherwise negative reward backward to top-down attention model.
As for the network structure, we can use full connected neural networks (or CNN) to extract a set of features from the image in each level in the hierarchy. For instance, if we partition the low resolution image into A regions (where we use A to denote the action space), then we get the |A] vectors, each of which represents as N dimensional vector corresponding
part of the image in that level [V1, V..... VI V; E R-V
(6)
where each region is represented as N-dimensional vector V;. Then we concatenate all vectors together to form the (sub)image representations at the current level
St [V1, .... VIAl se E BRAIIN
To select one region from the total A regions in each level, we leverage Q-learning, which as an off-policy RL algorithm and has been extensively studied since it was proposed Watkins & Dayan (1992). The Q-value function g(s, a) should output |A -Alddimessional values to indicate the impor- tances of the A-regions. correspondently. Thus, we add another layer of full connected networks which outputs (Al-dimmnsional values to decide the probability (softmax) to decide the next region to look in the next level. In our case, the agent needs to select regions according its q-value.
Suppose we use neural network parametrized by 0g to approximate Q-value in the interactive envi- ronment. To update Q-value function, we minimize the follow loss:
Vt r r(se, (1) + y max Q(#++1,(( 0e41;
(8)
a211
T(OO,) E. S.p- [[Q(se, az; 0g) - =
where se+1 is the state (or image region) in the level + 1, a&+1 is the action to pick one region from the total A regions. Ye is from Bellman equation, and we update it from its action a++1 {1, 2.. [A|} which is taken from frozen policy network (actor) to stabilizing the learning. Since don not know which region to focus until the last layer reward from the bottom-up model, design r(se, ag) = 1 if & is the last pyramid layer (original image) as well as we recognize object correctly from bottom-up model, otherwise r(se, az) O. ( de; 02)) is approximated with two-layer neural network (one convolution layer followed with fully connected layer).
In the current image level f, the agent takes step to select one region to focus on according q- value, and then further repeat to select subregions based on q-value in the next level f + 1. How to learn the value function will be dependent on the final reward in the bottom-up model in the next section. We can minimize Eq. to learn parameter 0g-
4.2 BOTTOM-UP ATTENTION MECHANISM
Our bottom-uy up attention model takes the similar approach as RAM Mnih et al. (2014). to use the recurrent neural networks (RNN) to guide the agent to make sequential decision process while interacting with a visual environment.
At each location, the agent observes the low-level scene properties in local region and build up bottom-up processes over time to integrate information from visual environment in order to deter- mine how to act and how to deploy its sensor most effectively.
G. = fy( Le; 0g)
(10a)
He = fr TH Ge: 0%)
(10b)
L = fi 01)
(10c)
ae (o(:: By)
(10d)
5

---

Under review as a conference paper at ICLR 2022
where 2, is the local patch observation of the visual environment (image), with location spec- ified by Lr- fg combines visual feature Gsmg(2i (2t) and location feature (Lz) together, where visual feature (img((zz) consists of three convolutional hidden layers with normalization and pool- ing layers followed by a fully connected layer, and Gioe(Tr is from full connected hidden layer. And we use G2 Gamg(I:)
Gloc(T4) to get glimpse feature at Tu. H, integrates the previous hidden state and Gr to generate next hidden state, in order to capture the whole sequential infor- mation. fy the classifier, which integrates all information to the current state in the sequence and decides the class of the image (which further decides the reward the agent will receive). it is formulated using a softmax output and for dynamic environments, its exact formulation depends on the action set defined for that particular environment. For classification with K classes, we have ÿ = arg ma Olth: Ot2 E R RixK
Compared to RAM, we add anther two constraints which balances the trade-off between exploration and exploitation. On the one hand, we want to take action to select high entropy regions to better ex- plore the whole environment. On the other hand, we hope the regions selected have high confidence on classification task. To sum up, our contributions are followings:
(I) Context constraint: the basic idea is that for the regions selected in policy search, we should assign them more weights. a is the softmax output at each step t. which in fact is the weight to measure the importance of the current hidden state, whether we should attend to current location. Once the weight a₁ (which sums to one) are computed, then the global context vector C' and its softmax output is computed as
T
C Σ
aight
1=1
= Sc(C)
(11)
where fo is the two-layer networks, with a fully connected layer and a softmax output layer. We hope matches its groundtruth : y.
Entropy has been widely use to measure information, where higher entropy value indicates relatively richer information content. As a bottom-up algorithm, we want our model gives more weight on regions with higher entropy. Specifically, we want to skip all black or white regions while searching objects. Thus, for each region we extracted, we threshold it and binarialize it into 2 bins and then compute its entropy. In order to take action towards high entropy regions, modify the reward function
(2) Entropy constraint with better exploration:
(0] = E (T
+ 2) -
(12)
where entropy(2:) is the entropy over the patch located by I,, À is the constant to scale the im- portance at the current time step, Ro is the reward over sequence and is the baseline which can be approximated by neural network. Since the action to localize the next region is backward from reward, then we decide reward Ro as: the model makes the right decision, then Ro 1, otherwise Ru 0. The location network fi decides action at L,, and the policy gradient can be formalized
JoJ = ( ( (f
+ X) - bb]
(13)
To learn action L,,, we use Gaussian policy with variance a which measures the exploration scale.
The final objective function in the bottom-up algorithm is hybrid supervised loss as follow:
loss -ylogar- -iylogg + (0)
(14)
where 31 and B2 are the weights to balance the importance of each term.
4.3 ALGORITIMM
We outline the sketch of our training process in Algorithm Note that we use another frozen Q-learning network to stabilize the learning procedure.

---

Under review as a conference paper at ICLR 2022
Algorithm Unified visual attention model
1: Initialize model parameters in Eq. 9 and Eq. 10
2: for each epoch do
for each image, (y) do
4: Decide the action o; max,, Q(st, ar);
5: Get the corresponding region via A:
6:
Randomly sample location Lo from image region vę Randomly generate initial hidden state Hu: for each glimpse t 0
8:
do Run the RNN model to decide next location to focus via Eq. if condition is ok then Run classifier via Eq. 10d
9:
10a
10b and
10c
10:
11:
12:
VP = arg maXKeK Oleki end end for
13:
14:
15:
Evaluate the sequential reward == 5] if correct in the whole sequence then
17:
r(8e, =
18:
else
19:
r(s,,0c) 0;
20: end if
Update Q-learning (top-down) model parameter Og by minimizing loss Eq.
22:
Update bottom-up model parameters {0g. On. Ot, 0v3 by minimizing loss in Eq.
23:
end for
24:
end for
5 EXPERIMENTS
We evaluated our approach on MNIST, CIFARIO and SVHN datasets. As for parameter setting, we use 2-level pyramid representation in the top-down q-learning: the coarse and the original image level. Basically, the coarse level image is the half resolution (size) of the original image, and we divide the coarse image into by 2 subregions, then we can learn the location in the coarse level, which in turn can be mapped back to the original image. In the bottom-up attention model, we use Adam algorithm with learning rate 3e 4, batch size 128, À 0.5, 31 = 1, 32 0.01 and the same parameter setting above in all the following
= 0.15. Without other specification, experiments.
MNIST classification: MNIST dataset is handwritten images (28 28) with digital numbers ranging from to In this experiment, we tested our method by varying MNIST dataset including original images and cluttered translated images with different size. The MNIST training set has total 60000 images, in which we use 54000 images for training and the rest 6000 for validation. Then we evaluate the performance on dataset with 10,000 examples. As for the experimental settings, use the similar RNN architecture (in Appendix) as RAM to make a fair comparison.
Patch feature encoding: at the location Lt, we extract m square patches with different scales. Then we concatenate features at different scales together to get the final patch representation 2; centered location L, Note that L, a two dimensional vector normalized in the range [-1, 1], with the image center (0, 0) and the right bottom (1, l).
Table 28 x 28 MNIST classification
Model
Error rate
FC, 2 layers (256 hiddens each) Convolutional networks, 2 layers
1.69%
1.21 %
RAM, glimpses, 8 x 8, scale
1.12%
RAM. glimpses, 8
scale
1.07%
Our method (Entropy), 6 glimpses, 8 x 8, 1 scale
1.05%
Our method (Entropy+Contex)), glimpses, 8 x 8, scale
1.01%
7

---

Under review as a conference paper at ICLR 2022
Since the digits in MNIST are centered, there is no need to use top-down mechanism "where to look" to initialize the search. Thus, we only leverage the bottom-up strategy as RAM, but we introduce new constrains in Eqs. II and 12 and want to test whether these features get performance gain or not. The evaluation result on the MNIST test dataset is shown in Table 1 Using 6 glimpses, patch size 8 x 8 and 1 scale (where m = 1), our method with entropy beats RAM under the same parameter setting. It demonstrates that the entropy constraint in Eq. [2] is helpful. In addition, tested whether the context constraint in Eq. [] contributes or not, and also shows that it can improve the classification task with more gain. Cluttered Translated MNIST classification: we use the same network architecture given in the MNIST classification experiment, but evaluate on the cluttered and translated digital images. In this experiment, we use the same protocol as RAM to create the cluttered and translated MNIST data. First, we place an MNIST digit in a random location of a larger blank image 100 x 100 and then add 8 subpatches which are sampled randomly from other random MNIST digits to random locations of the image.
Table shows shows the classification results for the models we trained on 100 by 100 Cluttered Translated MNIST with 10 pieces of clutter. The presence of clutter is kind of random noise, which makes the task much more difficult. Since our model unifies top-down and bottom-up attention, it can find the meaningful regions and avoid the negative impact from noising subpatches. In Table 2 can see the performance of our attention model is affected less than the performance of the other models. Using the same parameters as RAM with glimpses, 12 12 and 4 scales, our approach gains almost 3%. which demonstrates that our top-down and bottom-up attention unification is more powerful than RAM. By adding the entropy and global context constraints, our method gains another 3%. We can observe the similar results while using 8 glimpses.
Table 2: 100 100 Cluttered Translated MNIST classification
Model
Error rate
FC, layers (256 hiddens each)
57.3%
Convolutional, 5 layers
49.5%
RAM,1 glimpses, x 8, 1 scale
88.6%
RAM.1 glimpses, 12 x 12, 1 scale
87.3%
RAM, glimpses, 8 x 8, I scale
79.3%
RAM, 6 glimpses, 12 12, 4 scales
29.8%
RAM, 8 glimpses, 12 x 12, 4 scales
24.7%
Our method, 6 glimpses, 12 12, 4 scale
26.3%
Our method (with Entropy constraint), 6 glimpses, 12 x 12, 4 scale
23.6%
Our method (with Enntopy)(Conte)), glimpses, 12 12, 4 scale
20.3%
Our method, 8 glimpses, 12 x 12, 4 scales
20.7%
Our method (with Entropy Context), glimpses, 12 12, 4 scales
17.3%
Sequential multi-digit recognition: since the core network is based on recurrent neural networks, we can easily extend it to handle multi-digit recognition Goodfellow et al. (2014); : Ba et al. (2015). Suppose that we have training images with the target {31, 191,9/2-0 .... Mr J. where we can use y1 marks the length of sequence use yr to indicates the end of sequence. The task is to predict the digits as explores the local image glimpses.
Table 3: Whole sequence recognition error rates on 54 54 MNIST multi-digit SVHN dataset
Model
Error rate
11 layer CNN Goodfellow et al. (2014)
3.96%
10 layer CNN
Ba et al. (2015)
4.11%
Single DRAM Ba ct al.
2015 18 glimpses, 12 x 12, 2 scales
5.1%
Our method (Entropy), 18 glimpses, 12 12, 2 scales
4.58%
Our method (Entropy+Context), 6 glimpses, 18 glimpses, 12 x 12, 2 scales
4.03%
We take the same protocol as Goodfflllo et al. (2014) on street view house number recognition task. Specifically, we crop 64 x 64 images with multi-digits at the center and then randomly sample
8

---

Under review as a conference paper at ICLR 2022
Table 4: The recognition accuracy on CIFARIO dataset with PGD attack.
Model
Accuracy
VGGI6 + PGD
47.76%
Our method + VGGI6 PGD
54.3%
create 54x54 jittered images with similar data augmentation. We have 212243 images for train- ing (73257 digits from training dataset and the rest from 531131 additional image), 23511 digits for validation and 26032 digits for testing, Same as DRAM Ba et al. (2015), our attention model observes 3 glimpses for each digit before making a prediction. The recurrent model keeps running until it predicts terminal label or until the longest digit length in the dataset is reached. In the SVHN dataset, up to 5 digits can appear in an image, so it means the recurrent model will run up to 18 glimpses per image, that is x plus 3 glimpses for a terminal label.
To conduct fair comparison, we use the same architecture as DRAM (refer in Appendix). The experiment results shown in Table 3 Compared to DRAM, our model gains better result with an error rate 4.03%. And our result better than 10 layer CNN, and it is also comparable to the 11 layer CNN Goodfellow et al. (2014), which uses more complex network architecture and more model parameters.
Robust to PGD attack: in this experiment, we analyzed and answered whether our patch- at- tention model is robust to adversarial attack not. Specifically, we will leverage projected gradient descent (PGD) to generate adversarial examples because it has been widely used to attack deep neu- ral networks Madry et al. (2018); Pintor al. (202T). PGD is a very powerful adversary approach, which has the following projected gradient descent on the negative loss function:
r++ (1' + esign( V Vel( y; 0p)))
(15)
C Rd formalizes the variance of ma- nipulative space of the adversary attack, and 8, is the set of model parameters. In order test our model's 's robustness to PGD, we change equation equation [0d as follows:
where ((x, yr 0p) is the loss function defined Eq. 14
feat = VGG(z)
(16)
a₂ (c(ttt(H,, 0.)
(17)
where VGG is the 16-layer neural networks defined in Simonyya & Zisserman (2015). and cat
indicates feature concatenation. Our attention model use 6 glimpses with patch size 12 x I2 to make prediction on each image. As for feature extraction, we concatenate VGG16's output with G2 as input in the RNN network. In the backpropagation stage, we can minimize loss in Eq. 14 and then compute gradient w.r.t. feat Eq. 16 so we can further calculate gradient w.r.t. xr based on chain rule. After we get vz((x, y; 9p), we can update x to generate adversarial examples according to Eq.
15
In CIFARIO 10 classification task, we use step = 30 and = 0.3 in Eq, 15
Our attention model takes patch- approach and then sequentially decides the next loca- tion/patch to minimize the loss in Eq. [4] The result in Table 4 shows that our model more robust to PGD attack. The intuition behind this is that our model can select better patch in each glimpse, and further it can learn some useful structure information to do better prediction.
CONCLUSION
In this paper, we propose a visual attention approach, which unifies the top-down attention mecha- nism with the bottom-up recurrent attention model. Considering bottom-up RAM randomly initial- izes the location to search target object, our top-down attention model is to answer 'where look' the best initial location in an image. In addition, we introduce global context and entropy constraints, in order to focus attention on edge, instead of pure black/while regions. To sum, our model leverages reinforcement learning to combine top-down and bottom-up mechanism, both of which depends the reward from classification to decide next locations to attend in a sequential manner. We conduct a bunch of experiments MNIST, CIFAR1O and SVHN datasets, and model shows better result than RAM, DRAM and CNN (complex architecture). We also show our model is more robust than other neural network models recognizing noise images.

---

Under review as a conference paper at ICLR 2022
REFERENCES
E. H. Adelson, C. H. Anderson, J. R. Bergen, P. Burt, and J. M. Ogden. 1984, Pyramid methods in image processing. RCA Engineer, 29,6):33-4,, 1984.
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question an- swering. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6077-6086. IEEE Computer Society, 2018.
Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. Multiple object recognition with visual attention, 2015.
Nicholas J. Butko and Javier R. Movellan. 1-pomdp: An infomax model of eye movement. In IEEE International Conference on Development and Learning (ICDL.), 2008
Nicholas J. Butko and Javier R. Movellan. Optimal scanning for faster object detection. In IEEE Conference on Computer Vision and Puttern Recognition, 2009. 2009..
C. Caicedo and S. Lazebnik. Active object localization with deep reinforcement learning. In 2015 IEEE International Conference Computer Vision (ICCV), pp. 2488-24966 2015.
Ian J. Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, and Vinay Shet. Multi-digit num- ber recognition from street view imagery using deep convolutional neural networks. 2014.
Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Daniel D. Lee. Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Gamett (eds.), Ad- vances Neural Information Processing Systems 29: Anmual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 3675-36833 2016
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521((553):436 444, 2015.
Jianan Li, Yunchao Wei, Xiaodan Liang, Jian Dong, Tingfa Xu, Jiashi Feng, and Shuicheng Yan. Attentive contexts for object detection. IEEE Trans. Multim., 195)):94-954, 2017.
Tsung- Yi Lin, Piotr Dollr, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. Feature pyramid networks for object detection. In CVPR, pp. 936-944. IEEE Computer Society, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver. BC, Canada, April 30 May 3, 2018, Confer- ence Track Proceedings. OpenReview.net, 2018.
Volodymyr Mnih, Nicolas Heess, Alex Graves, and koray kavukcuoglu. Recurrent mod- els of visual attention. In 7. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 2204-2212. Curran Associates, Inc., 2014. URL http:://pppe: nips ccpps.c/pppprr/ 5542 2e0ureecurrett models -or--of-vviiis visual attention.pdi
Maura Pintor, F. Roli, Wieland Brendel, and B. Biggio. Fast minimum-norm adversarial attacks through adaptive norm constraints. ArXiv, abs/2102.12827.. 2021.
Aleksis Pirinen and Cristian Sminchisescu. Deep reinforcement learning of region proposal net- works for object detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Mike Schuster, Kuldip K. Paliwal, and A. General. Bidirectional recurrent neural networks. JEEE Transactions Signal Processing, 1997.
10

---

Under review as a conference paper at ICLR 2022
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceed- ings, 2015.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning an introduction. Adaptive com- putation and machine learning. MIT Press, 1998.
X. Wang, W. Chen, Wu, Y. Wang, and W. Y. Wang. Video captioning via hierarchical reinforce- ment learning. In 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 4213-4222, 2018
Christopher J. C. H. Watkins and Peter Dayan. Q-learning. In Machine Learning, pp- 279-292, 1992.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, pp. 2048-2051 2015. URL http:::/proccee.ingg.. press/v37/ x1777/xccl11 xuc15.html
A APPENDIX
MNIST network architecture:
Glimpse network G, = [g(,,, 0g):
fg is two-layer fully connected (FC) neural networks:
hg Rect (eintaLnnerrM h4 Rect(Linear(LL))
G. Rect( (hg hz))
where hy and h both have dimensionality 128, while the dimension of G, is 256 for all experiments.
Recurrent network: H, = ( G1; 0n) to remember the sequence of glimpses order to make action to get best return. H, Rect Linear + Linear(G.)))
Action network: L. = fi HH; 01) is the policy network, where mean is the linear network over H, and its variance a is fixed.
SVHN network architecture:
the glimpse network is composed of three convolutional layers where the first layer is with filter
size 5 x 5 and the later two layers with filter size The number of filters in those layers was {64, 64, 128} respectively. The recurrent model was 2-layer LSTM, with 512 hidden units in each layer.


Reviews:
[{"review_id": "ju-3L2l7iXO", "reviewer": "Reviewer_ixZU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Weaknesses\n\n- Any direct evidence to show that the initialization and attention trajectory are better than the original RAM method other than final accuracy? Can you propose some quantitative metrics and offer more visualizations? Now I do not know where the improvement comes from. Is there any possibility that the improvement simply comes from larger model capacity?\n- Tab. 2: On this task, the proposed method brings significant improvement, which is amazing. But there is not any in-depth analysis about it. BTW, I think a fair comparison to CNN-based methods would be a spatial transformer network, or any other CNN with spatial attention.\n- Tab. 3: Seems to me that RAM-based methods are very suitable for sequential digit recognition since they have the concept of glimpse. But it performs worse than a CNN method proposed in 2014. Why is that?\n\n- better report run-time speed and computation cost for all the comparisons. Accuracy is not one major advantage for RAM-based methods compared to CNN-based methods (see Tab. 3)\n- Tab. 2 is too difficult to read. Maybe you can report RAM and the proposed method under different settings pair by pair.", "summary_of_the_paper": "This submission proposes to offer a better initialization by using image pyramids and Q-learning (top-down manner) for the original recurrent visual attention (RAM) model (bottom-up manner). Two new constraints are also proposed for better exploration for RAM. The proposed method has been tested on several image classification datasets, including MNIST, cluttered translated MNIST, SNHN (sequential multi-digit recognition). They also test the robustness to adversarial attack (PGD attack) on CIFAR10.", "summary_of_the_review": "The motivation looks OK but there still remain many questions. A bunch of experiments have been conducted and some of them look very promising, but more in-depth analysis are required to address my concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "c12ZLjsRz5z", "reviewer": "Reviewer_ARgD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: The idea is interesting and new. The writing is good and easy to understand.  Sufficient experiments also demonstrate the effectiveness of the proposed model.\n\nWeakness: The authors should give some visual examples to show the difference of the searched regions between the proposed method and other compared methods, especially the RAM model. Such visualization can also serve as qualitative evidence to support the effectiveness of the proposed method.", "summary_of_the_paper": "In this paper, the authors propose a novel method to unify the top-down and bottom-up attention together for recurrent visual attention. They also propose two constraints in the bottom-up recurrent neural networks for better balancing the trade-off between exploration and exploitation when searching local regions.", "summary_of_the_review": "Overall, this paper proposed an interesting idea and successfully demonstrated its effectiveness. Only a few visualizations should be done to better justify the proposed ideas.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "HalC1D_Tp13", "reviewer": "Reviewer_ooVw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The whole paper is well-written and well-organized.\n2. The idea of combing top-down attention and bottom-up attention in the recurrent attention model is interesting and reasonable.\n\nWeaknesses:\n1. From Algorithm 1 and implementation details, the top-down model is only a one-step prediction, which makes the  whole model less convincing. In contrast, I suggest the authors can discuss multiple-step top-down predictions, and how to (dynamically) control the step of top-down prediction or trade-off the steps in two types of attention.\n\n2. The comparisons with existing works are unfair. Since the proposed model has extra top-down steps, it is unfair to compare with the baselines at some fixed steps of glimpses. Instead, I suggest including the number of steps of the top-down steps into evaluation.\n\n3. Motivations of the proposed context constraint are not clear. Based on my understanding, \\alpha_{ty} is the normalized action probability distribution in y-th class at t-th step. Thus, based on Eq.(11), the meaning of C is the averaged hidden state of all T steps, and I am confused about the exact meaning of C.\n\n4. The proposed entropy constraint is limited. For the MNIST-like dataset, it is reasonable to skip all black or white regions. However, for other more complex datasets or visual scenes, image patches with higher entropy seem to have no direct relation with the final prediction.\n\n\nMinor:\n1. Eq.(10)c seems wrong. It should be L_{t+1} instead of L_t?\n\nWriting Suggestions:\n1. It would be better to change the sequence of \"context constraint\" (Eq. (11)) and \"entropy constraint with better exploration\" (Eq. (12) - (14)) in Page 6 to make it consistent with the introduction section and article part.", "summary_of_the_paper": "This paper extends the recurrent attention model(RAM) with another extra top-down attention. Specifically, they exploit image pyramids and Q-learning to select regions-of-interest first in the top-down attention mechanism, and then follow RAM to use policy gradient to find the patch in the bottom-up attention. Meanwhile, they also propose two loss constraints to further boost the performance of bottom-up recurrent neural networks. The proposed framework is an end-to-end framework. Experiments on three datasets (MNIST, CIFAR 10, and SVHN) have demonstrated the effectiveness of the proposed model.", "summary_of_the_review": "The idea itself is interesting, and but I think these are several limitations of existing versions (cf. the weaknesses part), and the submission can be further improved by solving my concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission receives mixed reviews. One reviewer leans positively while two reviewers are negative. They raise several issues upon improper evaluations, insufficient experimental analysis, baseline and sota network comparisons, presentation unclarity, and technical motivations. In the rebuttal and discussion phases, the authors do not make any response to these reviews. After checking the whole submission, the AC agrees with these two reviewers that there are several drawbacks to the aspects of the technical presentation and experimental configurations. The authors shall take these suggestions into consideration and make further improvements upon the current submission.
Decision: Reject