Hard Attention Control By Mutual Information Maximization
Authors: Himanshu Sahni
Charles Lee Isbell
Venue:  (2021)
Keywords: reinforcement learning
attention
partial observability
mutual information
information theory

Abstract: Biological agents have adopted the principle of attention to limit the rate of incoming information from the environment. One question that arises is if an artificial agent has access to only a limited view of its surroundings, how can it control its attention to effectively solve tasks? We propose an approach for learning how to control a hard attention window by maximizing the mutual information between the environment state and the attention location at each step. The agent employs an internal world model to make predictions about its state and focuses attention towards where the predictions may be wrong. Attention is trained jointly with a dynamic memory architecture that stores partial observations and keeps track of the unobserved state. We demonstrate that our approach is effective in predicting the full state from a sequence of partial observations. We also show that the agent's internal representation of the surroundings, a live mental map, can be used for control in two partially observable reinforcement learning tasks. Videos of the trained agent can be found at ~\url{https://sites.google.com/view/hard-attention-control}.

Under review as a conference paper at ICLR 2021
HARD ATTENTION CONTROL BY MUTUAL INFORMA- TION MAXIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Biological agents have adopted the principle of attention to limit the rate of in- coming information from the environment. One question that arises is an artificial agent has access only limited view of its surroundings, how can it control its attention to effectively solve tasks? We propose an approach for learning how control hard attention window by maximizing the mutual in- formation between the environment state and the attention location at each step. The agent employs an internal world model make predictions about its state and focuses attention towards where the predictions may be wrong. Attention is trained jointly with dynamic memory architecture that stores partial obser- vations and keeps track of the unobserved state. We demonstrate that ap- proach is effective in predicting the full state from sequence of partial ob- servations. We also show that the agent's internal representation of the roundings, a live mental map, can be used for control in two partially observ- able reinforcement learning tasks. Videos of the trained agent can be found nttps::/sitess. http::///itess.coglee.c///// www.wiww./c/c.eeec.. http://aaaddaattttteeae attent ittentionnccotroll
INTRODUCTION
Reinforcement learning (RL) algorithms have successfully employed neural networks over the past few years, surpassing human level performance in many tasks (Mnih al. 2015 Silver al. 2017 Berner et al. 2019 Schulman et al., 2017). But key difference in the way tasks are performed by humans versus RL algorithms that humans have the ability focus on parts of the state time, using attention to limit the amount of information gathered at every step. We actively control our attention to build an internal representation of our surroundings over multiple fixations (Fourtassi et al. 2017) Barrouillet et al. 2004 Yarbus 2013) Itii 2005). We also use memory and internal world models to predict motions of dynamic objects in the scene when they are not under direct observation (Bosco et al. 2012). By limiting the amount of input information in these two ways, ie. directing attention only where needed and internally modeling the rest of the environment, we able to be more efficient in terms of data that needs be collected from the environment and processed at each time step.
Figure Illustration of the PhysEnv domain (Du Narasimhan, 2019). Here, the agent (dark blue) needs to navigate to the goal (red) while avoiding enemies (light blue). Typically the full environment states (top) are provided to the RI. RI. algorithm learn. We consider the case where only observations from under hard attention window (bottom) (yellow square) at each time step controllable by the agent. Using only partial observations, the agent must learn to represent its surroundings and complete its task.
available. The attention's location

---

Under review as a conference paper at ICLR 2021
By contrast, modern reinforcement learning methods often operate the entire state. Observing the entire state simultaneously may be difficult in realistic environments. Consider an embodied agent that must actuate its camera to gather visual information about its surroundings learning how to cross a busy street. At every moment, there are different objects in the environment competing for its attention. The agent needs to learn to look left and right to store locations, heading, and speed of nearby vehicles, and perhaps other pedestrians. It must learn to create a live map of its surroundings, frequently checking back on moving objects update their dynamics. In other words, the agent must leam to selectively move its attention so as to maximize the amount of information it collects from the environment at a time, while internally modeling motions of the other, more predictable parts of the state. Its internal representation, built using successive glimpses, must be sufficient to learn how to complete tasks in this partially observable environment.
We consider the problem of acting in an environment that is only partially observable through a controllable, fixed-size, hard attention window (see figure Only the part of the state that is under the attention window is available to the agent as its observation. The rest must be inferred from previous observations and experience, We assume that the location of the attention window at every time step is under control of the agent and its size compared to the full environment state is known to the agent. We distinguish this task from that of learning soft attention (Vaswani et al. 2017), where the full state is attended to, weighted by a vector, and then fed into subsequent layers. Our system must learn to 1) decide where to place the attention in order to gather more information about its surroundings, 2) record the observation made into an internal memory and model the motion within parts of the state that were unobserved, and 3) use this internal representation to learn how to solve its task within the environment.
Our approach for controlling attention uses RL to maximize an information theoretic objective closely related to the notion of surprise or novelty (Schmidhuber 199I]. is unsupervised in terms of environment rewards, i.e. it can be trained on offline data (states and actions) without knowing the task or related rewards. We discuss this in more detail in section [5] Memory also plays a crucial role in allowing agents to solve tasks in partially observable environments. We pair our attention control mechanism with a memory architecture inspired largely by Du & Narasimhan (211)'ss 's SpatialNet, but modified to work in partially observable domains. This is described in section we show in section J.ha that our system is able to reconstruct the full state image inc including dynamic objects all time steps given only partial observations. Further, we show in section 6.2 that the in- ternal representation built by our attention control mechanism and memory architecture is sufficient for the agent to learn solve tasks in this challenging partially observable environment.
Empirically,
2 RELATED WORKS
Using hard attention for image classification or object recognition is well studied in computer vision
(Alexe et al. 2012 Butko& & Movellan 2009 Larochelle & Hinton| 2010; Paletta et al.) 2005: Zoran
eet al.) 2020 Welleck et al., 2017).
Attention allows for processing only the salient interesting
parts of the image (Itti et al. 1998).
Similarly, attention control has been applied to tracking objects
within video (Denil al. 2012 Kamkar al. 2020 Yu al. 2020). Surprisingly, not lot of recent work exists on the topic of hard attention control in reinforcement learning domains, where sequential decision making task has to be solved by using partial observations from under the attention.
Mnih al. 22044) proposed framework for hard attention control in the classification setting and a simple reinforcement learning task. Their approach consists of using environment rewards to train an attention control RL agent. Our approach differs mainly in that we train the attention control using our novel information theoretic objective as reward. Mnih et al. (2014)ss 's approach leads to task specific policy for attention control, whereas our approach is unsupervised in terms of the task and can be applied generally to downstream tasks in the environment. Our approach also differs in that we use a memory architecture that is more suited to partially observable tasks with 2D images as input, compared to RNN used by Mnih et al. (2014).
There has been much prior work on memory and world models for reinforcement learning (a & Schmidhuber 2018 Graves et al. 2016 Hausknecht & Stone 2015; Khan al.) 2017). The work closest to our own is Du & Narasimhan (019))ss 's SpatialNet, which attempts to learn a task-agnostic world model for multi-task settings. Our memory architecture is largely inspired by SpatialNet
2

---

Under review as a conference paper at ICLR 2021
and adapted to work in the partially observable setting. We also their PhysEnv environment to evaluate our approach. Closely related work is Neural Map (Parisotto Salakhutdinov 2017). which uses structured 2D memory map to store information about the environment. Their approach also applies to partially observable RI. tasks, but the attention is fixed to the agent. contrast, we consider the problem of learning control attention window that can move independently of the agent location. Recently, Freeman et al. (2019) showed that world models can be learnt by simply limiting the agent's ability observe the environment. They apply observational dropout, where the output of the agent's 's world model, rather than the environment state, is occasionally provided to the policy. We consider the related scenario where only part of the true environment state is provided to the agent at each time step and the rest must be modeled using previous observations.
Finally, mutual information has been used to train self-supervised RI. agents. This line of work originates in curiosity driven and intrinsically motivated RI (Schmidhuber 1991, Pathak et al.) 2017 Bellemare et al., 2016). Typically, some notion of predictive error or novelty about an aspect of the environment is optimized in lieu of environment rewards. Multiple papers have successfully used different formulations of mutual information to learn how to efficiently explore the environment without extrinsic rewards (Mohamed & Rezende [0013] Houthooft et al. [0166] AAchamm al. [018 Gregor et al.] [0016] Eysenbach et al., 2018;) Shharma et al., 2019). We apply the idea of using mutual information the problem of curiosity driven attention control.
3 PRELIMINARIES
In this work, we are concerned with tasks solved using reinforcement learning (RL). RL is the study of agents acting an environment maximize some notion of long term utility. It is almost al- ways formalized using the language of Markov decision problems (MDPs). States, actions, rewards and transitions form the components of an MDP, often represented as the tuple (S, A, R, T). Max- imizing the expected sum of (discounted) rewards over rollouts in the environment is usually set as the objective of learning. Once a problem formulated as an MDP, such that the components (S, A, R,, T") are well defined, one can apply a range of model-free RL algorithms to attempt to solve (Schulman et al. 2017; Mnih et al. 2016 Wu et al. 2017). A solution to an MDP is typically sought optimal policy, n²' S A. which is function that maps every state to an action that maximizes the long term expected rewards. In this section, we will attempt to formalize the components of the partially observable reinforcement learning problem under study.
3.1 INTERNAL REPRESENTATION
fomnbaerwwc)
0g
H4+1
Figure 2: The agent starts with representation (n₂) from which it attempts reconstruction (Tt) of the likely current full (unobserved) state st. Then, it picks an attention location (1.) and receives an observation of the state from under the attention (ot). We assume that the size of the attention window is known to the agent. The observation written into μr to form n²" The entire map is then stepped through an internal world model to update the dynamical objects, forming 1441, the representation for the next step.
First, we give a brief description of how the agent stores observations in an internal representation of its surroundings (figure 2). An map (μ)) tracks the full environment state (s), which is never directly observed. The map is empty at the start of the episode and gets sequentially written into as observations made. The agent also creates reconstruction of the full state (7) at every time step
3

---

Under review as a conference paper at ICLR 2021
based on its map. This will become useful later for training. For now, it suffices that μ4 is the map before an observation is made and jur is the one after the observation is made. j4+1 is the map after the dynamics of the system for the next time step taken into account.
3.2 THE TWo AGENTS
We formulate the solutions to controlling the attention at every time step and completing the envi- ronment task as two separate reinforcement learning agents. The attention location is controlled by the glimpse agent (the eye in figure 2. and actions within the environment are taken by the regular agent. These two agents have their own separate MDPs with their (S, A, R) tuples defined below.
The glimpse agent's state every timestep is (4e- Its set of actions is all possible attention locations within the full state (Width x Height actions). Its reward is based on the information theoretic objective discussed in section Thus, the glimpse agent provided the map before an observation is made and it must decide where the observation should be made from in order to optimize its reward.
The environment agent acting in the environment receives as input μ²",, i.e. the internal representa- tion after the observation has been recorded. Its actions are the normal set of actions in the environ- ment and its reward is the normal environment reward. We emphasize that neither agent has access to the full state any time. They must both act based on the internal representation alone. They also cannot make multiple observations from the same environment state. Once an observation is made, an action must be selected that will change the environment state. In the next section, we will describe in detail how 14,", Pe++1 and are formed and how the internal map is trained through sequence of partial observations.
DYNAMIC MEMORY MAP
Memory plays critical role in enabling agents to act in partially observable environments. It allows the agent to remember parts of the state that are not under direct observation. Along with memory, the agent must be able to model the dynamics of objects in its environment, observation of them for long periods. Finally, the effect of its own actions must be reflected in its belief about the environment. For this purpose, visual environments called the Dynamic Memory Map (DMM). DMM is inspired largely by Du & Narasimhan (2099'ss 's work SpatialNett, but modified handle partial observations and work in landem with the glimpse agent.
it may not receive
design special recurrent memory unit for
4.1 MEMORY MODULES
The DMM consists three major modules: write, step and reconstruct.
Write. This module encodes incoming observation, O₁, into the current memory representa- tion, f4-- The observation is first passed through a series of convolutional operations, W, possibly downsampling it for more efficient representation. Then it is blended with μ using series of con- volutions, B. Finally it is written into the memory but only in the locations where the observation was made, leaving the rest of the memory intact. The write operation can be written as:
T0 fr = C. ((W pe) + (1-C,) * fe
where C is a mask which under the attention window and otherwise.
Step. This module is responsible for modeling the dynamics of the environment and updating the memory to track the full state from one time step to the next. Objects in the environment can be static dynamic and may be affected by the agent's own actions. This module is implemented as a residual network, S, similar to SpatialNet (Du Narasimhan] 2019). We additionally condition S on the agent's action de, The entire map is updated at each time step to ensure that objects observed previously are also updated to new likely locations.
Me+1 14 S(p .a)
4

---

Under review as a conference paper at ICLR 2021
Reconstruct. This module converts the memory representation to reconstruction of the full state using a series of deconvolutions (R).
72 = R(
reconstruction can also be made immediately after the observation, i.e. Tt R(p))) Recon- struction is essential for training the other two modules of DMM reconstruction error, or the discrepancy between the observation and the reconstruction under the attention window can be back-propagated through the write and step layers.
explain in section 4.2 The
4.2 TRAINING LOSS
The write loss, L is incurred under the current attention window immediately after the observation made into 14" This ensures that the immediate observation is correctly encoded into the memory and trains the write and reconstruction modules.
L C2 || 0al12-
The step loss, Lg is incurred after stepping the map and under the attention window in the next step.
Lg C++1 * ||Te+1 --+1|12-
This trains the step module, S, to accurately model the motion of objects in the entire state, so that a faithful representation of the state can be guessed before the next observation taken. Once the next observation is made, we can calculate the difference between what the agent's model expected to be at the glimpse location and what actually was there. In Sauure@this is the difference between the images under the yellow rectangle in т and or- This quantity is the amount of surprise in the dynamics model and we will use this again when training the glimpse agent.
In addition the reconstruction loss, the absolute value of the output by the write and step modules also penalized in order to regularize the contents of the DMM. The total loss for single step within the rollout (where a 0.01 here):
L L. L, + C. |n
|
+ 3 |S(( a:)l.
We sum this loss the entire rollout and jointly minimize it over all steps. The reconstruction emor at every location can back- pao-pproe to the step an observation was recorded there.
5 MAXIMIZING MUTUAL INFORMATION TO CONTROL ATTENTION
The intuition behind our approach is that attention should be used to gather information from the environment when and where is required. The DMM is constantly modeling its environment and attention should be directed to parts of the world where the model is uncertain. This directly informs the agent about difficult to model parts the environment, thus reducing uncertainty in the state as whole. Secondly, by focusing areas with hard to predict dynamics, more data be collected from those areas, updating the model for future predictions.
5.1 MUTUAL INFORMATION OBJECTIVE
The idea of redueing uncertainty about the environment can be captured the language of infor- mation theory by using mutual information. Specifically, we propose selecting the location of the attention such that its mutual information with the state at the following step is maximized,
max (8++1144) 4
Where 4 is the location of the attention window at time Eq, [ can be expanded as max H(s++1) H(s
4,
where H denotes the entropy. This expansion brings out very intuitive explanation of the objective. We would like to pick an attention location that maximizes the reduction entropy (uncertainty) of the environment state after an observation is made.
5

---

Under review as a conference paper at ICLR 2021
5.2 CURIOSITY DRIVEN ATTENTION
In this section, we show how maximizing the amount of surprise can lead to maximizing the mutual information. Let us begin by further expanding eq.
[(s+1:4)) = (((++1) H($++114)
Σ p(s(+1) logp(s(+1) + E p(1)) Σ
#6+1
61+1
E **41~p(#(+1)
[ogp(s(+1)1 +
Σ Σ
8e+1
E (4410((941)
logp(s(+1))
E 4.11/eppess1d)
llo p(s)p(((1))))))))
So, maximizing mutual information between the attention location and the environment state is equivalent to minimizing log p($++1) and maximizing p(g 144)+ under expectation. The first term is the log-likelihood of the next state prior to selecting 4. Computing it requires marginalization over all possible 4, which is prohibited by our environment as only single partial observation from the full state can be provided to the agent. The second term, log p(s++1 [4), is computable at single attention location from single state and hence we will focus on maximizing this term.
Now, assume the agent's belief over the true environment state at the next step, Se+1, is represented by a Gaussian with unit variance around the reconstruction, T++1- So, 8141 ~ -N(T++1, and
pog m++1) log carpeplpsss -T41)" (8t+1 T+11
= = 1iih1-T--p 12/
(3)
So, minimizing the squared difference between 8 and T leads to maximizing log p(5 (7+4))
Maximizing [ in turn means maximizing log (og(($#+114 142), since T++11 is constructed using the deterministic functions W, S,, and R once 4 has been picked (see section 4).
In order to minimize the squared difference between 81+1 and T the agent needs to pick a location that observes the maximum error between s, and reconstruction Tę. This is because at step t, if the agent observes the location with the highest reconstruction error, i.c. the least likelihood p((elTe) (T), will ensure that at the next time step + 1 the agent has the most recent information on that region and can make good reconstruction of it. Hence, maximizing log p(s 144) is equivalent to picking an le that will lead to the highest reconstruction error at step f.
max 4
max 4 KET.
Σ (st -
(4)
where I, forms the set of indices under the window at location 4. Directly optimizing this quantity with respect to le is not possible as it can only be computed once an observation has been made and only at a single location. Hence, the glimpse agent must learn to predict the location that is most likely to result in high reconstruction error before an observation is made. Our approach is to formulate the glimpse agent reinforcement learner with the reconstruction error its reward.
An interpretation of this objective is that the glimpse agent is surprise sceking, or curiosity driven. It is attending to parts of the state that are novel in the sense that they difficult for the agent's current model to predict. Another interpretation is that the glimpse agent is acting adversarially to the DMM's model of the environment. At each step the glimpse agent tries focus parts of the state that are the most difficult for DMM to reconstruct. By doing so, it is indirectly creating a curriculum of increasingly difficult model aspects of the environment.
5.3 FULL TRAINING OBJECTIVE
Let us look at different expansion of the objective in eq.
I(8 4)) = max H((1.) (((1
(5)
4

---

Under review as a conference paper at ICLR 2021
The first term is 1(44), the entropy over the attention location, which is controlled by the policy of the glimpse agent. This term can easily be maximized by standard RL algorithms, such as A2C
(Mnih al. 2016: Wu et al. 2017), using weighted entropy maximization.
Combining the first term from equation S and second term from equation 2 the final objective that train the glimpse agent is
max H(4) H(*++1114 (4) = O'L(T gliinpse) + max
Σ (s)
le
iEL.
where a is the entropy weighting set to 0.001 in our experiments.
Note that this objective is the mutual information from equation It is possible to design variational algorithm that maximizes the approximate mutual information based on the expansion in eq. Slalone (instead of mixing terms from both the expansions). Empirically, however, we found that our objective performs better than the variational approximation to mutual information. See section for more detail.
RESULTS
We evaluate our approach in two environments: gridworld environment (figure (5) and PhysEnv (figure []. In both environments, the task is to navigate to goal while avoiding obstacles and enemies. The agent's actions are movement in the four discrete cardinal directions. In the gridworld, the state is one-hot image of the objects in the environment (agent, walls, enemies and goal). The agent's 's and enemies' movements are restricted to the grid squares and the dynamics of the enemies simple: moving in straight lines until an obstacle is hit in which case it reflects. In PhysEnv, observations are provided as RGB images and the motions of the objects are more varied.
The glimpse agent DMM are first trained offline data of state and action trajectories (no reward) collected from each environment. More details on data collection are provided in section A. We evaluate this part of the training by how accurately the full state is reconstructed by DMM each step. Once the glimpse agent + DMM have converged, i.e. the reconstruction loss has plateaued, we freeze their parameters and initialize an agent within the environment. The glimpse agent DMM serve as fixed components within the agent while it is learns to solve task. The agent is trained using an off-the-shelf RL algorithm PPO (Schulman et al., 2017 and is evaluated by the total environment reward it collects during testing episodes.
We compare our method against baselines inspired from related work. In the follow baseline the attention is always focused on the agent itself. This is formulation of the partial observability as seen in Parisotto & Salakhutdinov (2017). Next, the environment baseline mimics the approach by Mnih al. (2014), where the environment rewards are used to train the glimpse agent. Finally, the random bascline moves the attention randomly at each time step. For the environment task, we have an additional baseline, full, where the full state is provided to the agent to solve the task. This is upper limit for how well any method that only receives partial observations can perform.
6.1 STATE RECONSTRECTION
12 reward (ours)
random
follow
environment
Gridworld 0.00555
0.007666
0.01941
0.00827
PhysEnn 0.0521
0.0614
0.1186
0.0826
Table 1: Per-pixel L2 loss between agent's reconstruction 7 and the ground truth full state s. All methods only see partial observations, but must attempt to reconstruct the full state from internal representation. In both environments, our method has the lowest reconstruction error.
We measure the performance of the glimpse agent DMM by the average reconstruction error the true full (unobserved) state in unseen testing episodes. 25 consecutive observations from 6 unseen test episodes passed through each model and error is averaged over reconstructions at each time step. This gauges the ability of our attention mechanism and memory to reconstruct the full state using only partial observations. Table shows that using surprise driven reward (our approach) for
7

---

Under review as a conference paper at ICLR 2021
learning attention control leads to lower reconstruction than the other baselines. See figures throuuh 9 in the appendix for visualizations of the reconstructions and training error. In particular, hgurr|| 7 and show qualitative comparisons between our method with the baselines. Broadly speak- ing, our method learns to focus on dynamic objects within the environment and preserve them over many steps in the memory, whereas the baselines lose track of or blur objects seen in their state reconstructions.
goel
0.04
0oal
chany well
Averace Occurance frame
agont
wal
0.3
0.2
0.2-
10000
20030
50000
40000
50000
10000
20000
50000
40000
50000
Itars
Bors
(left) Object-wise breakdown of reconstruction error in gridworld. (right) % of frames in Figure 3:
which the object occurs, i.c. the frequency with which the attention is focuses on a particular object.
Figure shows analysis of what objects under focus during training in the gridworld envi- ronment and their corresponding reconstruction error. The attention forms an interesting curriculum for training the DMM, where it learns focus on different objects in the environment over time. starts with walls, then moves to the agent itself, then goal and finally the enemies. Thus, it moves gradually from static to dynamic, harder to model parts of the state, allowing the DMM learn how to accurately model each object. This very interesting curriculum-lik behavior emerges naturally and not pre-programmed into DMM or the attention agent. It is a consequence of maximizing surprise at every time step. See section A.2 for more detailed discussion.
6.2
REINFORCEMENT LEARNING
full (upper)
1.75
full (upper)
17.5
(ours)
1.50
(ours)
15.0
12 online (ours)
environment
environment
1.25
follow
12.5
follow
1.00
Reward
Roward
10.0
0.75
7.5
0.30
0.25
0.00
NuM
-0.25
0.0
0.5
1.0
1.5
2.0 Iters
2.5
3.0
3.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
1e7
Iters
le7
Figure 4: Average testing reward for goal seeking task in PhysEnv (left) and gridworld (right). Comparison between agent with full observations (upper limit) and partial observations trained with surprise (ours), environment rewards and simply following the agent's 's location.
Figure(4shows the average episodic reward over five testing episodes during the training of the agent. Our method approaches the performance of the upper bound agent that receives full environment states in the gridworld environment. The environment baseline performs similarly to our approach in gridworld, but does poorly in PhysEnv, where our approach performs the best. This may be because the attention policy of the environment baseline has high entropy and ends up exploring large parts of the state within episode. But it does not learn focus the most unpredictable parts of the state as our method does. The follow baseline performs poorly in both environments, likely because only focuses the agent and does not explore the rest of the state.
8

---

Under review as a conference paper at ICLR 2021
We also show the performance of our agent (12 online) in the condition where the DMM and glimpse agent is trained online along with the policy, all components starting from random initialization. This agent did not have pretraining step and was trained entirely on random exploratory data, therefore obviating the need of a pretraining dataset. The performance slightly worse than the pretrained agent. This is encouraging result in particularly challenging setting of non-stationary state space as the memory representation and glimpse policy are changing while the agent learns.
7 CONCLUSIONS
We have presented an approach of using mutual information to control a hard attention window in environments with dynamic objects. From partial observations under the attention window, our system is able to reconstruct the full state at every time step with the least error. The representa- tion learned using our method enables RL agents to solve tasks within the environment, while the baseline methods are unable to learn useful representations on the same memory architecture. This demonstrates that attention control plays a large role in enabling task completion.
Note that our attention control objective is independent of the task. is solely concerned with gath- ering information from the environment where most unpredictable. This is similar to curiosity driven RL that seeks novel states surprising transitions. Hence, attention control learned in this manner is a generic solution that can be applied to many downstream tasks. It may be able to be com- bined with task-specific attention control by fine-tuning model may also be used instead of the fixed size hard attention used here. We made an assumption that only one glimpse allowed per environment state. But the rate of collecting glimpses may be variable with respect to environment speed.
task reward. A better foveal glimpse
Curiosity based approaches tend to suffer from the "noisy TV problem (Savinov et al. 2018) where the agent tends to get distracted by and focus on random, unpredictable events in the environment. Orthogonal approaches have been developed taking into account the agent's own actions (Pathak et al. 2017). These approaches can be combined with our method as well. We leave the incorporation of this in attention control for future work.
counter this common weakness by, for example,
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv:1807.10299 2018.
Bogdan Alexe, Nicolas Heess, Yee W Teh, and Vittorio Ferrari. Searching for objects driven by context. In Advances in Neural Information Processing Systems, pp. 881-889, 2012.
Pierre Barrouillet, Sophie Bernardin, and Valérie Camos. Time constraints and resource sharing in adults' working memory spans. Journal of Experimental Psychology: General, 133(1):83, 2004.
Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances neural information processing systems, pp. 1471 14799, 2016.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint 1912: 066880, 2019.
Gianfranco Bosco, Sergio Delle Monache, and Francesco Lacquaniti. Catching what we can't see: manual interception of occluded fly-ball trajectories. PLoS One, 7(1 7(11):49381, 2012.
Nicholas J Butko and Javier R Movellan. Optimal scanning for faster object detection. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2751-2758. IEEE, 2009.
Misha Denil, Loris Bazzani, Hugo Larochelle, and Nando de Freitas. Learning where to attend with deep architectures for image tracking. Neural computation, 24(8):2111-2184, 2012.
Yilun Du and Karthik Narasimhan. agnostii dynamics priors for deep reinforcement learning.
arXiv preprint arXiv: arXiv:1905.045 1905094855099... 2019.

---

Under review as a conference paper at ICLR 2021
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06770, 2018.
Maryam Fourtassi, Gilles Rode, and Laure Pisella. Using eye movements to explore mental repre- sentations of space. Annals of Physical and Rehabilitation Medicine, 6063):160-163, 2017.
Daniel Freeman, David Ha, and Luke Metz. Learning to predict without looking ahead: World models without forward prediction. In Advances in Neural Information Processing Systems, pp- 5379-5390, 2019.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwiñiska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538 (7626)4471-777 2016.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv: /tt.://6110050/ 1.07507, 2016.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint 2018.
2018.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. arXiv preprint 1XX.::5000065555 2015.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1109-11911111111 1117,1111111 2016.
Laurent Itti. Quantifying the contribution of low-level saliency to human eye movements in dynamic scenes. Visual Cognition, 12(6):1093-112 19110093311,,,0 2005.
Laurent Itti, Christof Koch, and Ernst Niebur. A model saliency-based visual attention for rapid scene analysis. IEEE Transactions on pattern analysis and machine intelligence, 20(1 (I(11))224- 1259, 1998.
Shiva Kamkar, Fatemeh Ghezloo, Hamid Abrishami Moghaddam, Ali Borji, and Reza Lashgari. Multiple-target tracking in human and machine vision. PLOS Computational Biology, 16(4): c1007698, 2020.
Arbaaz Khan, Clark Zhang, Nikolay Atanasov, Konstantinos Karydis, Vijay Kumar, and Daniel D Lee. Memory augmented control networks. arXiv preprint arXiv:1709.057000 2017.
Hugo Larochelle and Geoffrey E Hinton. Learning combine foveal glimpses with a third-order boltzmann machine. In J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor R. S. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems 23, pp. 2251-125. Curran Associates, Inc., 2010.
Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In Advances in neural information processing systems, pp. 2204-2212, 2014.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 (7775))))529555555 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937,22 2016.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsi- cally motivated reinforcement learning. In Advances in neural information processing systems, pp. 1122-2133, 2015.
Lucas Paletta, Gerald Fritz, and Christin Seifert. Q-learning of sequential attention for visual object recognition from informative local descriptors. In Proceedings of the 22nd international confer- on Machine learning, pp. 649-656, 2005.
10

---

Under review as a conference paper at ICLR 2021
Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning. arXiv preprint arXiv:1002.08360, 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Mare Pollefeys, Timo- thy Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. arXiv preprint arXiv:1810.02274, 2018.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building ral controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pp. 222-22) 1991.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint 1977:0990.006655 2019.
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676))354-359,, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Sean Welleck, Jialin Mao, Kyunghyun Cho, and Zheng Zhang. Saliency-based sequential image attention with multiset prediction. In Advances in neural information processing systems, pp. 5173-5183 2017.
Yuhuai Wu, Elman Mansimov, Roger В Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using tronecker-factoree approximation. In Advances in neural information processing systems, pp. 5279-5288, 2017.
Alfred L Yarbus. Eye movements and vision. Springer, 2013.
Yuechen Yu, Yilei Xiong, Weilin Huang, and Matthew Scott. Deformable siamese attention networks for visual object tracking. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6728-6737, 2020.
Daniel Zoran, Mike Chrzanowski, Po-Sen Huang, Sven Gowal, Alex Mott, and Pushmeet Kohli. Towards robust image classification using sequential attention models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9483-9492, 2020.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 DETAILS OF DATA COLLECTION
For gridworld we used 20000 random steps in the environment to create this training set. In PhysEnv used previously trained agent to generate 500,000 steps of demonstrations. We experienced that using a random policy in PhysEnv led to quick deaths of the agent with insufficient episode lengths for learning. In both cases, the data episodes. In PhysEnv, the map size was 21 x 21 compared to the full state size of 84 x 84 and in gridworld both the map and state were 10 x 10. The size of the attention window was fixed to be 3 x 3 in the gridworld (9% of the full state visible in an observation) and 21 x 21 in PhysEnv (6.25% of the state visible).
sequences of full states and agent actions from multiple
A.2 VISUALIZATIONS OF GLIMPSES DURING TRAINING
We will discuss where the attention focuses and how that effects the reconstruction (and underlying belief) as the training progresses for the gridworld environment. Please refer to to for this discussion.
Initially, the attention learns to focus on the wall objects and the corresponding reconstruction error for the walls decreases rapidly b)). This because the walls are long objects the environment,
present at multiple pixels and side-by-side, providing the largest and most reliable source of error under the attention window initially. The error is provided as reward to the glimpse agent, hence seeks more walls in the state.
Soon, the error it receives from walls decreases as the agent's 's DMM learns to persistently record the location of stationary walls over multiple time steps. Then around 10000 iters, the attention switches to focusing on the agent's 's location itself. This is because the agent's apparent erratic movements provide a larger reward due to their unpredictability than the stationary walls can anymore. This has dual consequences. First, since now the DMM is seeing a lot of examples of the agent's movement, it learns to accurately model the agent based error corresponding to the agent falls. At its peak, over 70% of glimpses contain the agent within them (figure b). Interestingly, the error corresponding the wall and goal objects rises as some forgetting of the representation and dynamics of these objects occurs. A little before 20000 iters though, the agent has learnt to represent simultaneously all three objects and it switches attention to the enemies, the only remaining un-modeled objects in the environment. The enemies move predictably, bouncing off walls and other objects, but otherwise in straight lines. The agent does not have direct access to their actions and must infer their motion over multiple time steps using only glimpses.
initial state and its actions and the reconstruction
Figures 5 and 6 visualize the behavior of the glimpse agent during the initial stages of training. Initially, (100 iters), the attention mostly moves around randomly as can be seen by the heatmap being active throughout the state. The reconstruction does not correspond to anything in the real state since the agent has only just begun learning. The glimpse agent here is receiving high reward for pretty much every location in the state. By 2000 iters, the glimpse agent has learnt mainly focus on the wall once it locates it. This can be seen in the high probabilities of the glimpse agent policy near the wall locations and the attention repeatedly fixating on the wall. The agent is simultaneously striving to learn to represent the walls as they are showing up lot in its training data, and steadily reducing the reconstruction error (the glimpse agent's reward) coming from walls.
By iter 1000, the focus has shifted to tracking the agent as can be seen in the heatmap. The heatmap shows that the glimpse agent is singularly focused on the agent's movements as the error reward is getting from the walls has dropped off and the agent's 's movements are not fully predicatable yet. In the figure for iter 19000, it can seen that the agent can now track its own location very reliably with an initial position and the actions it is taking, without having to focus the attention on itself. In fact, after step 8 when the agent is discovered, the attention almost never moved back to it and yet the location in the reconstruction perfectly matches that of the unobserved full state. Hence the agent's 's location based reward source has largely dried up. The heatmap of attention policy shows that the glimpse agent has become interested in the goals and walls again briefly as it learns to represent all three objects together. can also be seen from the reconstructions here that the agent is unable to accurately track and model the enemies locations at this point.
12

---

Under review as a conference paper at ICLR 2021
By iter 21000, the attention has shifted focus to the remaining object in the state that is producing a high reconstruction error, i.c. the enemy. Once the enemy is discovered on step 14, the glimpse agent's 's policy puts all probability near its location so it can keep observing and model its behavior.
The glimpse agent creates a natural curriculum for training the models within DMM, starting at large stationary objects that do not move, then moving to the agent's followed by the objects with the most complex dynamics. We conjecture this automatically discovered curriculum is the reason system is able learn better reconstructions than the baselines.
13

---

Under review as a conference paper at ICLR 2021
(a)
(b) 2000 (c) 11000 iters
(d) 19000
(e) 21000
Figure 5: Visualizations of glimpse behavior in gridworld environment. Each sub-figure is the same testing episode at different stages of training, with the rows corresponding to the time within the episode. In each subfigure, the first column shows the full (unobserved) state. The second column a heatmap showing the glimpse policy, i.c. the likelihood the glimpse agent will select location for placing attention. The location that was sampled is indicated in the first column by the yellow square. The final, third, column is the reconstruction based on the agent's 's current current belief about the environment.
14

---

Under review as a conference paper at ICLR 2021
(a) 1000 iters
(b) 7000 iters
(c) 10000 iters
(d) 200000 iters
Figure 6: Visualizations of glimpse behavior in PhysEnv environment.
15

---

Under review as a conference paper at ICLR 2021
Next show qualitative comparisons of the glimpse policy and state reconstruction between method and the baselines. Each subfigure in figures 7 and shows the same episode of length 20 played out with different glimpse agents. The first column form random policy for the glimpse agent, the third shows a glimpse agent trained on environment (task) rewards, and lastly the fourth shows the glimpse following the agent. In both environments, it can be seen that our method learns to focus on dynamic objects within the environment and the resulting state reconstruction shows that these objects are preserved in the memory.
method, the second shows uni-
Note that the glimpse policy learned by the environment rewards is significantly more entropic than the one learned by our method, despite having the same entropy weighting hyperparameter. We conjecture that this is because the task reward does not send a clear and consistent learning signal to control the attention window create better reconstructions, or as seen in our RI. results, to facilitate downstream tasks. As such, the reconstructions appear qualitatively similar to the random baseline.
The random baseline quantitatively performs the closest to approach, but it fails to focus on dynamic objects consistently like over time, and hence fails to track these objects over long periods in the episode. The reconstructions therefore have streaks and blurs where the objects were last spotted and the physics model has not learnt how to update their positions properly.
16

---

Under review as a conference paper at ICLR 2021
(a) 12 (ours)
(b) random
(c) environment
(d) follow
Figure 7: Glimpse behavior for fully trained models our method and the baselines in gridworld. Each subfigure is different baseline, with columns within the subfigure being the full unobserved state, heatmap of attention policy, and reconstruction of full state respectively.
17

---

Under review as a conference paper at ICLR 2021
(a) 12 (ours)
(b) random
(c) environment
(d) follow
Figure 8: Glimpse behavior for fully trained models of our method and the baselines in PhysEnv.
18

---

Under review as a conference paper at ICLR 2021
A.3
ERROR CURVES FOR GLIMPSE AGENT DMM
0.20
random
0.10
random
Reconstruction Error pixe!
0.18
follow
(ours)
Reconstruction Error pixel
follow
(ours)
0.16
environment
environment
0.14
0.12
0.10
0.08
0.02
0.06
0.04
50000
100000 Iters
150000
200000
100000 Iters
150000
200000
Figure 9: (left) PhysEnv environment. (right) Gridworld environment. Reconstruction error (L2) from the full state during training of the DMM glimpse agent. All approaches have converged by 200k iters, with our approach having the lowest error, i.e. the most faithful reconstruction of the full unobserved state.
These training curves correspond to the results in table | The reconstruction error for each method has stabilized. Our method has the lowest reconstruction error for both environments.
14
random
random
Reconstruction Error pixee
12
follow
(ours)
Reconstruction Error pixe!
follow
I2 (ours)
environment
environment
10
50000
100000
150000
200000
50000
100000 Iters
150000
200000
iters
Figure 10: (left) PhysEnv environment. (right) Gridworld environment. Episodic reward for glimpse agent during training (higher is better).
Figure 10 shows the episodic reward for the glimpse agent six testing episodes as the training progresses. This corresponds to the mutual information objective in equation|4] or the reconstruction eror under the attention window in the next step. The modules within DMM are constantly improving to reduce prediction error and hence the reward goes down as the training progresses. But the glimpse agent simultaneously trying maximize the objective by searching for areas within the state that still provide high error. Our method maximizes this objective both environments. This leads lower overall reconstruction error as the glimpse agent learning predict where the highest source of error is going to be at the next step. The environment baseline performs close to random and the follow baseline gathers the least reward in both environments.
19

---

Under review as a conference paper at ICLR 2021
A.4 VARIATIONAL APPROXIMATION TO MUTUAL INFORMATION MAXIMIZATION
The mutual information between attention and environment state as expanded in eq. is
[ss (6) H(4) H((4(()).
(6)
In our algorithm, we combine the first term in this expansion with another term in eq. This works well empirically. Theoretically, however, it is computationally possible to maximize both terms in eq. [ hence maximizing the actual mutual information. However, as we shall show here, this does not work well practically.
The first term can be maximized by using maximum entropy RL algorithms for the attention policy as discussed in the main text. Let consider the second term.
-H(1(())) = Σ ((())))((11s)))
ΣE((s p(()) Σ p(1elst) log
[log(([sss)
While logs((e(ss)) is not directly known, construct variational function, ((4))s)) to approx- imate
Since KL(p(4, (4)p(44))))]l((()s)))))))) 2 O, we have,
p((lsss)logg((e 2
Ep(leley
log ((e)(8))
E
[ogp((e|se)]
log ((4))1)) [t)]
add sedemplsede)
Plugging this into eq. 6
1(s(4) 2 H(I)) +
E suderp(ss)d)
[log (l((s))
Therefore, we can maximize the mutual information by maximizing this lower bound w.r.t. the policy parameters of the glimpse agent. This can be achieved by providing log ((,,),,) as a reward to the glimpse agent and using RL algorithms to maximize its long term value along with the policy entropy. As such, maximizing the mutual information.
We can additionally condition g on the internal state and action of the agent at previous time step. Since we do not have access to the full environment state, st, substitute it with the agent's 's internal state. The final variational function looks like g(el|ee, (x-1,4;-11 must be trained to match the true posterior p(()ss)) to provide accurate rewards the glimpse agent.
The question now is how to learn q? this work, is a neural network trained on samples. This is done by minimizing the KL divergence,
min dederplledd))
H4 ((-1))))))
on samples from the environment and the glimpse agent's policy. This will ensure q is likely to assign high likelihood to actual locations conditioned its inputs.
20

---

Under review as a conference paper at ICLR 2021
Intuitively, 9 will be high at locations of attention that are easy identify by looking at the previous state, the resulting state, and the agent action. Otherwise, its output will be flat and hard to prediet. Since q is provided to the glimpse agent reward, it will be rewarded for selecting easily identifiable locations and prefer such a policy. These can be locations that produce a high amount of localized surprising information in 14 as compared
14-1, unexpected appearance of goal somewhere will signal that the attention was just moved there, hence producing a high likelihood in q. Conversely, low reward locations will be ones that are hard identify by q as they do not impact any specific part of μt, leaving it largely the same as 14-1. or observe a location that was already well-modeled by the agent and hence do not leave an identifiable mark on μt that is a tell-tale sign for q-
enemies
0.0200
12 (ours)
I2 (ours)
Reconai Recarniruution Etror pixel
0.0175
variational
0.0175
variational
0.0150
pixel
0.0125
0.0125
0.0100
0.0100
0.0075
0.0050
0,0050
0.0025
0.0025
0.0000
50000
100000 Iters
150000
200000
50000
100000
150000
200000
(a) Overall state reconstruction error per pixel.
(b) Write error of observation under attention window.
Figure 11: State reconstruction and observation reconstruction errors in gridworld environment. Observation reconstruction error falls to almost zero for both methods while full state reconstruction remains high for variational approach.
The above method does not work well in practice. Figure 11 shows the reconstruction error of the unobserved state with our proposed method and the variational approach. Figure shows the observation reconstruction error, which is the error in reproducing just the observation under the attention immediately after it is made. It can be seen that the latter quantity goes to near zero for both methods, meaning that immediate observations are being faithfully recorded into memory and reconstructed. Yet, the overall state reconstruction eтrоr remains high for the variational approach, which means that the agent not able to retain observations over time steps as the attention moves around the state and the environment model is not learning to predict the evolution of the state over time without direct observation.
A reason why the variational approach fails in practice could be that do not have direct access to p((()e) in order to train q. Instead we must rely on the agent's 's internal estimate of the environment state, M4, which itself is being trained alongside the glimpse agent. This results in noisy rewards to the glimpse agent as the g function is not able to accurately predict where the attention is. As we have seen in section attention control creates curriculum for training the agent's 's state estimate. Hence this forms a loop, where attention control is dependent on agent's 's state estimate which is in return dependent the currieulum generated by the attention.
Extensive hyperparameter tuning may also be required for this approach to work as the glimpse agent can easily default to chosing the same attention location at each time step it ensures the highest g(4els)).
21


Reviews:
[{"review_id": "j55KhoCf7As", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work restricts agent observations to a small viewport within the full image representing the environment’s state, and trains one agent to control the location of this viewport so as to reduce the reconstruction error of an internal model of the full image state. The proposed method produces lower reconstruction error than baselines tested, and the reconstructed image is used to train another agent to obtain reward from the environment.\n\nPros\n- The work is carefully motivated and focused on the question of how to successfully control hard attention.\n- The solution (observing the location with the highest predicted reconstruction error) is intuitively appealing.\n\nCons\n- The two environments considered are small and simple.\n- The memory is structured to match the environment’s actual image state, and the memory write operation is automatically masked to the viewport’s corresponding location within the memory. This contrived arrangement injects a strong inductive bias that seems unlikely to generalize to environments that don’t follow these strict assumptions.\n- The automatic masking reduces the challenge of state reconstruction. \n- The reasoning related to mutual information seems circular. It begins with the intuition of using attention to gather information where it is needed, and ends with the minimization of reconstruction error, which is essentially the starting point. The mathematical excursion provides little insight beyond the original intuition.\n\nSuggestions\n- An intrinsic reward based on image reconstruction could drive the agent’s exploration during an initial pretraining phase, where no external reward is provided. If the agent learns enough about its environment during this first (model-building) phase, it may learn more quickly than the full-observation oracle during a second phase of training, where the external reward is provided. Such a demonstration of benefit from restricted observations would be an important contribution, if it held for more complex environments, and if the results included systematic hyperparameter tuning and multiple runs using different random seeds.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZpxzMP1Tts", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n1. This paper proposes a new way to solve tasks under the limited view of the surroundings.\n2. To solve the tasks, three components, 1) decide where to place the attention, 2) record the observation and make internal memory, 3) solve the task with the memory, are necessary\n3. This paper utilizes hard attention with mutual information technique to solve the problems, and mutual information objective reduces the uncertainty about the environment.\n4. The proposed system of this paper is able to reconstruct the full state at every time step.\n5. As a result, the l2 reward shows the lowest state reconstruction and the highest reward except full (upper).\n\nStrengths\n1. The problem definitions and the proposed models are interesting and clear.\n2. This paper investigates the unexplored research area, hard attention in reinforcement learning domains\n3. This paper provides the demo videos and implementation codes.\n\nWeaknesses\n1. This paper has empirical contributions, but the technical contributions are not clarified. \n2. The results are not enough to support the author's claims.\nThe interesting qualitative results are provided in the appendix and videos.\nHowever, there is no comparison between baselines and this paper qualitatively.\n\nQuestions and Additional Feedback\n1. The intuition of total loss in section 4.2 is not well described. It is hard to follow that the necessity of regularization term and alpha (in total loss and entropy weighting), beta.\n2. How much alpha (in total loss and entropy weighting) and beta influence the performance? (sensitive analysis)\n3. L_t in 4.2 is a total loss, but L_t in 5.2 denotes the indices. I suggest that this paper should be modified carefully.\n4. Table 1 denotes the reconstruction loss, but the difference between random and l2 reward seems doesn't large.\nIs there another experiment or settings that l2 reward solves well, but random fails to solve?\n5. Figure 4 shows the test performance, and the gap between full and l2 is very large.\nWhat is the reason that l2 performs worse than full on the PhysEnv, while full and l2 shows similar performance on the gridworld? \n6. Could you summarize your technical contributions instead of empirical contributions?\n\nTypos\nEmperically -> Empirically\n\n\nAfter Rebuttal\nThank you for your detailed response, and I will keep my positive score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "R4R_ejrejGZ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper proposes a new architecture and training method to learn tasks that require hard attention control. Specifically, the paper proposes to learn the “glimpse agent” (which controls the hard attention window) by task-agnostic loss that seeks to maximize information gain by the glimpse to the learned world model. The authors also proposed a specific architecture that incorporates consecutive glimpses to learn the world model. \n\n==============\n\nPositives\n\nThe paper is written clearly and cleanly, with helpful diagrams. \n\nThe mutual information objective the authors proposed to learn the glimpse agent is novel and intuitive.\n\n==============\n\nNegatives \n\nExperiments are not thorough at all. Two similar tasks are chosen, with relatively low level of complexity. As a comparison, the experiments in SpatialNet, a work that inspired the authors, include a more diverse range of tasks including Atari tasks and 3D tasks (PhysShooter3D) and are generally more difficult.\n\nThe existing experiment results are not convincing. While results from PhysEnv show clear advantage of the proposed approach, the gridworld task does not. The authors can strength the result by having more complex tasks where the previous approach (labeled “environment”) would not work well. \n\nIt’s useful to see more analysis of the experiment. For example, it’d be interesting to track the mutual informative objective, which the glimpse agent optimizes, over the course of training. It’s also helpful to see if the learned glimpse agent + DMM can indeed be used for learning different tasks (with same physics environment), as the authors claimed to be task-agnostic. \n\n==============\n\nRecommendation\n\nOverall I find the experiment results to be unconvincing and therefore recommend a reject.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0K9KEhtM4p7", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents a method for learning a hard attention controller using an information maximization approach. As the authors point out, such a method could be very useful for reasoning in terms of high-dimensional observations, like vision. In brief, the method learning to choose the next attention position to be the most informative by maximizing the uncertainty of the next observation. Uncertainty is quantified using a spatial memory model that is trained to reconstruct and predict the scene. The authors validate this approach by showing that the resulting attention mechanism can be used for two simple downstream tasks. The resulting agent outperforms others trained using baseline attention mechanisms: a hard attention mechanism that is trained on task reward (\"environment\"; similar to Mnih et al 2014), as well as models that attend to random positions or to the agent's location.\n\nThis work is well motivated, easy to read, and appears technically correct. The information maximization objective is a sensible way to learn an attention mechanism, and I found the exposition very easy to follow.\n\nThe main appeal of the method is that it's trained independently of the task, and for this reason might be useful on many tasks. And indeed, the authors highlight that this is the promise of this line of research (\" ...our approach is unsupervised in terms of the task\nand can be applied generally to downstream tasks in the environment\"). Accordingly, my main concern is that the paper presents little evidence that the attention mechanism presented here will work in a task-agnostic manner. The paper only shows results on two simple 2D environments, and one of these evaluations has caveats that I feel significantly weaken the paper's case. \n\nIn particular, I don't find the comparison to baselines on the PhysEnv environment to be fair. This is for two reasons:\n(1) while the strongest baseline method is trained on data from the learning agent, the proposed method is trained on data from an expert policy (as described in appendix section A) instead of from random transitions. In the typical RL setup, we can't generally assume that expert demonstrations will be available when training an agent from scratch, so this restricts the applicability of the attention mechanism for RL. The resulting attention policy is likely to indirectly leak task information to the agent, which makes it hard to compare to models trained without expert data on task reward alone. \n(2) the authors report that the strongest baseline (called \"environment\"; Mnih et al 2014) doesn't perform well because the resulting policy is entropic. It's difficult to evaluate whether this behavior is due to problems with the baseline method or with the hyperparameter settings of the RL algorithm used to optimize it. For example, PPO uses a policy entropy bonus, and the authors don't report tuning this hyperparameter, but it will presumably play a large role in the entropy level of the learned attention mechanism. Because of this, it's hard to know whether the proposed methods outperforms the baseline because the baseline is poorly tuned or because the proposed method is generally better. I'm generally surprised by how poor the results of the \"environment\" method shown here are, given the it performs comparably on the other task and given that it's trained on task reward. More analysis or discussion would be very useful.\n\nAt a more fundamental level, I'm not convinced that the task-agnostic strategy for information maximization proposed here is the correct one for all tasks. This method will suffer from the \"noisy TV\" problem faced by many curiosity-based methods (as described e.g. in the introduction to Savinov et al 2019: https://arxiv.org/abs/1810.02274) and will attend to regions of high variability whether or not they're task relevant. In settings where the task of interest involves direct interaction with a relatively small number of pixels, but other things in the scene are also changing, there is no guarantee that an information maximization strategy will attend to the most task-relevant pixels. Without evaluation on more challenging tasks, it's hard to know how good a strategy information maximization is. The paper should address these issues explicitly. These results would be even stronger if shown on a perceptually harder task, such as one involving natural scenes, 3D content, or more realistic dynamics.\n\nOther questions and comments (less central to my evaluation):\n- What happens if the attention policy and the agent are trained simultaneously? If this approach works, it might allow the attention model to be trained on PhysEnv without requiring expert demonstrations.\n- How does the \"random\" attention baseline perform on the two tasks? On PhysEnv, the random baseline gives the second best reconstruction results, so it would be very interesting to see how well it works for control.\n- The claims made about human cognition in the introduction need to be better justified with references to the literature. The one reference provided (Barrouillet et al 2004) is primarily about working memory spans and cognitive load (i.e. internal bottlenecks) rather than about perceptual bottlenecks and the need to build world models or use hard attention mechanisms, as the surrounding text implies.\n- The reconstruction results in section 6.1 are a good sanity check of the proposed model, but the baselines used here are very weak. This is because the proposed method uses an attention mechanism that's trained for reconstruction (via the infomax objective), while the other methods are trained either for an RL task (which may be only loosely correlated with reconstruction) or are heuristic. These results would be much more compelling with stronger baselines (and model ablations).\n- Generally, the paper would benefit from more analysis of the contribution of model components. E.g. how important is the architecture of the dynamic memory module to the reconstruction and RL results? \n\nMinor:\n- Section 4.1: \"This quantity is the amount of surprise in the dynamics model and we will use of this again when training the glimpse agent.\" -> \"...we will use this again...\"\n\n- Section 4.2: \"In addition to reconstruction loss,\" -> \"In addition to the reconstruction loss,\"\n\n- Section 4.2: \"The total loss for as single step\" -> \"The total loss for a single step\"\n\nIn summary: the authors present an interesting application of information maximization-based curiosity to hard attention control. A hard attention mechanism trained in a purely unsupervised fashion (as proposed here) that performs well on many downstream tasks would be very useful. As it stands, I am in favor of rejecting this paper because of the limitations of the evaluation and analysis. My concerns would be addressed by evaluating on more challenging, benchmark tasks and ideally on tasks with more challenging visual structure, and with a thorough analysis of how the model performs in settings where information maximization is uncorrelated with the task (as in the \"noisy TV\" problem). ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Inspired by biological agents that have developed mechanisms like attention as an information bottleneck to help function more effectively under various constraints of life, this paper looks at an approach of learning a hard attention scheme by leveraging off the prediction errors of an internal world model. They demonstrate their approach via a simple but easy to understand 2D pixel multi-agent game, a gridworld env, and also PhysEnv, to show the effectiveness of the learned hard attention, and go on to discuss interesting aspects such as curiosity attention.

Overall, I thought more highly of the paper than the reviewers, and might have proposed a score of 6 if I were a reviewer, but I also read each review and respect the points given by all four reviewers, and also agree with much of their feedback in the end. I think if this work was submitted to ALife (Journal or Conference), it might have been accepted. Not that those venues are easier, if anything they can often be more selective, but I think what ICLR (and similar venues like ICML) tends to expect is a bit different than what this paper offers.

To improve this work, I recommend following some of the reviewers' advice (especially R4), particularly on experimental design. Reviewers suggest that the current experiments are small and simple, but while true to some extent, I think more importantly missing are clear baseline methods to compare your approach against. What can your approach do that existing popular approaches in RL will totally fail at doing? It can be worthwhile to try your approach on a larger task domain, such as Atari (but perhaps modified) or ProcGen [1] to show the benefits of hard attention compared to existing approaches. For instance, some recent work [2] demonstrated that hard attention can help agents generalize to out-of-training domain tasks the agent has not seen before during training - something that traditional approaches without attention tend to fail at doing.

In the current state, the work will be a great workshop paper. But I recommend the authors to continue improving the work in the direction that can help the idea gain acceptance by the broader community.

[1] https://openai.com/blog/procgen-benchmark/
[2] https://arxiv.org/abs/2003.08165

Decision: Reject