Unifying Top-down and Bottom-up for Recurrent Visual Attention

1 INTRODUCTION . Recurrent visual attention model Mnih et al . ( 2014 ) ( abbreviated as RAM ) leverages reinforcement learning Sutton & Barto ( 1998 ) and recurrent neural networks Schuster et al . ( 1997 ) ; LeCun et al . ( 2015 ) to recognize the objects of interests in a sequential manner . Specifically , RAM models object recognition as a sequential decision problem based on reinforcement learning . The agent can adaptively select a series of regions based on its state and make decision to maximize the expected return . The advantages of RAM are follows : firstly , its computation time scales linearly with the complexity of the patch size and the glimpse length rather than the whole image , through a sequence of glimpses and interactions locally . Secondly , this model is very flexible and effective . For example , it considers both local and global features by using more number of glimpses , scales and large patch sizes to enhance its performance . Unfortunately , the idea of using glimpse in the local region limits its understanding to the holistic view and restricts its power to capture the semantic regions in the image . For example , its initial location and patch extraction is totally random , which will take more steps to localize the target objects . Although we can increase the patch size to cover larger region , it will increase the computation time correspondently . In the same time , we need to set a high variance ( if we use Gaussian policy ) to explore unseen regions . Thinking a situation that you were in desert and you were looking for water , but you did not have GPS/Map over the whole scene . What would you do to find water ? You might need to search each direction ( east , south , north , and west ) , that is random searching and time-consuming . In this paper , we want to provide a global view to localize where to search . For example , the process of human perception is to scan the whole scene , and then may be attracted by specific regions of interest . In a high level view , it can help us to ignore non-important parts and focus information on where to look , and further to guide us to different fixations for better understanding the internal representation of the scene . To solve ‘ where to look ’ using reinforcement learning , the desired algorithm needs to satisfy the following criterions : ( 1 ) it is stable and effective ; ( 2 ) it is computationally efficient ; ( 3 ) it must be more goal-oriented with global information ; ( 4 ) instead of random searching , it attends regions of interest even over large image ; ( 5 ) it has better exploration strategy on ‘ where to look ’ . For a large image , the weakness of RAM is exposed throughly , because it relies on local fixations to search the next location , which increasing the policy uncertainty as well as the learning time to optimizing the best in the interaction sequence . Inspired by the hierarchical image pyramids Adelson et al . ( 1984 ) ; Lin et al . ( 2017 ) and reinforcement learning Watkins & Dayan ( 1992 ) ; Kulkarni et al . ( 2016 ) , we propose to unify the top-down and bottom-up mechanism to overcome the weakness of recurrent visual attention model . In the high-level , we take a top-down mechanism to extract information at multiple scales and levels of abstraction , and learn to where to attend regions of interests via reinforcement learning . While in the low-level , we use the similar recurrent visual attention model to localize objects . In particular , we add another two constraints over the bottom-up recurrent neural networks for better exploration . Specifically , we add the entropy of image patch in the trajectory to enhance the policy search . In addition , we constrain the region that attend to should be more related to target objects . By combining the sequential information together , we can understand the big picture and make better decision while interacting with the environment . We train our model in an end-to-end reinforcement learning framework , and evaluate our method on visual classification tasks including MNIST , CIFAR 10 classes dataset and street view house number ( SVHN ) dataset . The experimental results outperform convolutional neural networks ( CNNs ) baseline and the bottom-up recurrent attention models ( RAM ) . 2 RELATED WORK . Since the recurrent models with visual attention ( RAM ) was proposed , many works have been inspired to either use RNNs or reinforcement learning to improve performance on computer vision problems , such as object recognition , location and question answering problems . One direction is the top-down attention mechanism . For example , Caicedo and Lazebnik extends the recurrent attention model and designs an agent with actions to deform a bounding box to determine the most specific location of target objects Caicedo & Lazebnik ( 2015 ) . Xu et al . ( 2015 ) introduces an attention based model to generate words over an given image . Basically , it partitions image into regions , and then models the importance of the attention locations as latent variables , that can be automatically learned to attend to the salient regions of image for corresponding words . Wang et al . Wang et al . ( 2018 ) leverages the hierarchical reinforcement learning to capture multiple fine-grained actions in sub-goals and shows promising results on video captioning . Another trend is to build the interpretation of the scene from bottom-up . Butko & Movellan ( 2009 ) builds a Bayesian model to decide the location to attend while interacting with local patches in a sequential process . Its idea is based on reinforcement learning , such as partially observed Markov decision processes ( POMDP ) to integrate patches into high-level understanding of the whole image . The design of RAM Mnih et al . ( 2014 ) takes a similar approach as Butko & Movellan ( 2008 ; 2009 ) , where reinforcement learning is used to learn ‘ where to attend ’ . Specifically , at each point , the agent only senses locally with limited bandwidth , not observe the full environment . Compared Butko & Movellan ( 2009 ) , RAM leverages RNNs with hidden states to summarize observations for better understanding the environment . Similarly , drl-RPN Pirinen & Sminchisescu ( 2018 ) proposes a sequential attention model for object detection , which generates object proposal using deep reinforcement learning , as well as automatically determines when to stop the search process . Other related work combine local and global information to improve performance on computer vision problems . Attention to Context Convolution Neural Network ( AC-CNN ) exploits both global and local contextual information and incorporates them effectively into the region-based CNN to improve object detection performance Li et al . ( 2017 ) . Anderson et al . Anderson et al . ( 2018 ) proposes to combine bottom-up and top-down attention mechanism for image captioning and visual question answering . They uses the recurrent neural networks ( specifically , LSTM ) to predict an attention distribution over image regions from bottom-up . However , no reinforcement learning technique used to adaptively focus objects and other salient image regions of interests . RAM learns to attend local regions while interacting with environment in a sequential decision process with the purpose to maximum the cumulative return . However , if the scope of local glimpse is small , then we lose the high level context information in the image when making decision in an sequential manner . In contrast , if we increase the patch size to the scale as large as the original image , then we lose the attention mechanism , as well as its efficiency of this model . Moreover , when we increase the scale with the fixed patch size , we will deprive distinguished features after resizing back to original patch size . Also it is a challenge to balance the Gaussian policy variance and patch size . Since the initial location is randomly generated , we need to set a large variance in Gaussian policy to increase its policy exploration . However , it will result in high instability in the learning process . In this paper , we propose to unify bottom-up and top-down mechanism to address these issues mentioned above . In addition , we introduce entropy into policy gradient in order to attend regions with high information content . 3 BACKGROUND . 3.1 REINFORCEMENT LEARNING . The objective of reinforcement learning is to maximize a cumulative return with sequential interactions between an agent and its environment Sutton & Barto ( 1998 ) . At every time step t , the agent selects an action at in the state st according its policy and receives a scalar reward rt ( st , at ) , and then transit to the next state st+1 with probability p ( st+1|st , at ) . We model the agent ’ s behavior with πθ ( a|s ) , which is a parametric distribution from a neural network . Suppose we have the finite trajectory length while the agent interacting with the environment . The return under the policy π for a trajectory τ = ( st , at ) T t=0 J ( θ ) = Eτ∼πθ ( τ ) [ T∑ t=0 γtr ( st , at ) ] = Eτ∼πθ ( τ ) [ R T 0 ] ( 1 ) where γ is the return discount factor , which is necessary to decay the future rewards ensuring bounded returns . πθ ( τ ) denotes the distribution of trajectories below ρ ( τ ) = π ( s0 , a0 , s1 , ... , sT , aT ) = p ( s0 ) T∏ t=0 πθ ( at|st ) p ( st+1|st , at ) ( 2 ) The goal of reinforcement learning is to learn a policy π which can maximize the expected returns . θ = arg max J ( θ ) = arg maxEτ∼πθ ( τ ) [ R T 0 ] ( 3 ) Policy gradient : Take the derivative w.r.t . θ ∇θJ ( θ ) = ∇θEτ∼πθ ( τ ) [ R T 0 ] = ∇θ ∫ ρ ( τ ) RT0 dτ = ∫ ∇θρ ( τ ) RT0 dτ = ∫ ρ ( τ ) ∇θρ ( τ ) ρ ( τ ) RT0 dτ = Eτ∼πθ ( τ ) [ ∇θlogπθ ( τ ) R T 0 ] = Eτ∼πθ ( τ ) [ ∇θlogπθ ( τ ) ( R T 0 − b ) ] ( 4 ) Q-learning : The action-value function describes what the expected return of the agent is in state s and action a under the policy π . The advantage of action value function is to make actions explicit , so we can select actions even in the model-free environment . After taking an action at in state st and thereafter following policy π , the action value function is formatted as : Qπ ( st , at ) = Eτ∼πθ ( τ ) [ Rt|st , at ] = Eτ∼πθ ( τ ) [ T∑ i=t γ ( i−t ) r ( si , ai ) |st , at ] ( 5 ) To get the optimal value function , we can use the maximum over actions , denoted as Q∗ ( st , at ) = maxπ Q π ( st , at ) , and the corresponding optimal policy π can be easily derived by π∗ ( s ) ∈ arg maxat Q ∗ ( st , at ) .

Summary: This submission proposes to offer a better initialization by using image pyramids and Q-learning (top-down manner) for the original recurrent visual attention (RAM) model (bottom-up manner). Two new constraints are also proposed for better exploration for RAM. The proposed method has been tested on several image classification datasets, including MNIST, cluttered translated MNIST, SNHN (sequential multi-digit recognition). They also test the robustness to adversarial attack (PGD attack) on CIFAR10.