R-LAtte: Attention Module for Visual Control via Reinforcement Learning
Authors: Mandi Zhao
Qiyang Li
Aravind Srinivas
Ignasi Clavera
Kimin Lee
Pieter Abbeel
Venue:  (2021)

Abstract: Attention mechanisms are generic inductive biases that have played a critical role in improving the state-of-the-art in supervised learning, unsupervised pre-training and generative modeling for multiple domains including vision, language and speech. However, they remain relatively under-explored for neural network architectures typically used in reinforcement learning (RL) from high dimensional inputs such as pixels. In this paper, we propose and study the effectiveness of augmenting a simple attention module in the convolutional encoder of an RL agent. Through experiments on the widely benchmarked DeepMind Control Suite environments, we demonstrate that our proposed module can (i) extract interpretable task-relevant information such as agent locations and movements without the need for data augmentations or contrastive losses; (ii) significantly improve the sample-efficiency and final performance of the agents. We hope our simple and effective approach will serve as a strong baseline for future research incorporating attention mechanisms in reinforcement learning and control.

Under review as a conference paper at ICLR 2021
R-LATTE: VISUAL CONTROL VIA DEEP REINFORCE- MENT LEARNING WITH ATTENTION NETWORK
Anonymous authors Paper under double-blind review
ABSTRACT
Attention mechanisms are generic inductive biases that have played a critical role in improving the state-of-the-arr in supervised learning, unsupervised pre- training and generative modeling for multiple domains including vision, language and speech. However, they remain relatively under-explored for neural network ar- chitectures typically used in reinforcement learning (RL) from high dimensional inputs such pixels. In this paper, we propose and study the effectiveness of aug- menting a simple attention module the convolutional encoder of an RL agent, Through experiments the widely benchmarked DeepMind Control Suite envi- ronments, we demonstrate that our proposed module can (i) extract interpretable task-relevant information such as agent locations and movements without the need for data augmentations or contrastive losses; (ii) significantly improve the sample- efficiency and final performance of the agents. We hope our simple and effective approach will serve as a strong baseline for future research incorporating attention mechanisms in reinforcement learning and control.
INTRODUCTION
Attention plays a crucial rule in human cognitive processing: a commonly-accepted hypothesis is that humans rely on combination of top-down selection (e.g.. zooming into certain parts in our perception field) and bottom-up disruption (e.g., getting distracted by novel stimuli) to prioritize information that is most likely to be important for survival or task completion (Corbetta & Shul- man L., 2002; Kastner & Ungerleider G., 2000). In machine learning, attention mechanisms for neural networks have been studied for various purposes in multiple domains, such as computer vi- sion (Xu et al., 2015; Zhang et al., 2019a), natural language processing ( (Vaswani et al., 2017; Brown al., 2020), and speech recognition (Chorowski et al., 2015; Bahdanau et al., 2016). For example, Zagoruyko & Komodakis (2017) proposed transferring knowledge between two neural networks by aligning the activation (attention) maps of one network with the other. Springenberg et al. (2014) and Selvaraju al. (2017) proposed gradient-based method to extract attention maps from neural net- works for interpretabiliity. Attention also stands behind the success of Transformers (Vaswani et al., 2017), which uses self-attention mechanism model dependencies in long-sequence language data.
However, attention mechanisms have received relatively little attention in deep reinforcement learn- ing (RL), even though this generic inductive bias shows the potential to improve the sample- efficiency of agents, especially from pixel-based environments. More frequently explored directions learning (Oord et al., 2018: 8; Srinivas et al., 2020; Lee al., 2020; Stooke al., 2020; Kipf et al., 2020): Jaderberg et al. (2017) introduced unsu- pervised auxiliary tasks, such as the Pixel Control task. Srinivas et al. (2020) applied contrastive learning for data-efficient RL and Stooke et (2020) further improved the gains using temporal contrast. Another promising direction has focused on latent variable modeling (Watter et al., 2015; Zhang et al., 2019b; Hafner et al., 2020; Sekar et al., 2020; Watters et al., 2019); Hafner et al. (2019) proposed to leverage world-modeling in latent-space for planning, and Hafner et al. (2020) uti- lized the latent dynamics model to generate synthetic roll-outs. However, using such representation learning methods may incurs expensive Pack-and-fome costs (c.g.. hyperparameter tuning). This motivates our search in a more effective neural network architecture in the convolutional encoder of an RI. agent.
in RL from pixels are mussupprviised/sll--spsrrrssssss

---

Under review as a conference paper at ICLR 2021
In this paper, propose R-LAtte: Reinforcement Learning with Attention module, simple yet effective architecture for encoding image pixels in vision-basee RL. In particular, the major compo- nents of R-LAtte are
Two-stream encoding: We use two streams of encoders to extract non-attentional and atten- tional features from the images. We define our attention masks by applying spatial Softmax to unnormalized saliencies computed by element-wise product between non-attentional and attentional features. We show that the attentional features contain more task-relevant infor- mation (e.g.. agent movements) while the non-attentional features contain more task-invariant information (e.g., agent shape).
Adaptive scaling: Once the attention masks are obtained, they are combined with original non-attentional features, and then provided to RL agents. Here, to balance the trade-off be- tween original and attentional features, we introduce a learn-able scaling parameter that is being optimized jointly with other parameters in the encoder.
We test our architecture on the widely used benchmarks, Deep.Mnnd Control Suite environ- ments (Tassa et al., 2018), to demonstrate that our proposed module can (i) extract interpretable task-relevant information such as agent locations and movements; (ii) significantly improve the sample-efficiency and final performance of the agents without the need for data augmentations or contrastive losses. We also provide results from detailed ablation study, which shows the contri- bution of each component to overall performance.
To summarize, the main contributions of this paper are as follows:
We present R-LAtte, a simple yet effective architecture that can be used in conjunction for encoding image pixels vision-based RL.
We show that significantly improve the sample-efficiency and final performance of the agents on continuous control tasks from DeepMind Control Suite (Tassa et al., 2018).
RELATED WORK
Reinforcement learning from pixels RL on image-inputs has shown be benefited from rep- resentation learning methods using contrastive losses (Oord et al., 2018; Srinivas et al., 2020; Lee al.. 2020; Stooke et al., 2020; Kipf et al., 2020), sulf-super auxiliary task learning (Jader- berg et al., 2017; Goel et al., 2018; Sekar et al., 2020), and latent variable modeling (Watter et al., 2015; Zhang et al., 2019b; Hafner et al., 2020; Sekar et al., 2020; Watters al., 2019). Along with those successes, Laskin et al. (2020) and Kostrikov et al. (2020) recently showed that proper data augmentations alone could achieve competitive performance against previous representation learn- ing methods. Different from existing representation learning and data augmentation methods, focus on architecture improvements specific to RL from pixels, which has been largely uncharted by previous works.
Attention in machine learning Human understand scenes by attending local region of the view and aggregating information over time to form an internal scene representation (Ungerleider & G, 2000; Rensink, 2000). Inspired by this mechanism, researchers developed attention modules in neural networks, which directly contributed to lots of recent advances in deep learning especially on natural language processing and computer vision applications (Vaswani et al., 2017; Bahdanau et al., 2015; Mnih et al., 2014; Xu et al., 2015). Using attention mechanism in RL from pixels has also been explored by previous works for various purposes. Zambaldi et al. (2018) proposed self- attention module for tasks that require strong relational reasoning. Different from their work, we study a more computationally-effie
attention module on robot control tasks that may or may not require strong relational reasoning. Choi et al. (2019) is the closest study in terms of the architectural choice. One main difference is in the objective of the study. Choi al. (2019) investigate the effect of attention on Atari games and focus the exploration aspect, while we focus improving the learning of robot controls using the DeepMind Control Suite environments (Tassa et al., 2018). attention-like module for robot control tasks. There several architectural design differences (e.g., the addition of a residual connection and a Hadamard product), which we will show crucial for achieving good performance on control tasks (See Section 5.4).
Levine et al. (2016) also
2

---

Under review as a conference paper at ICLR 2021
0
Feature Encoder
Fully Connected
π
C M
0t-1
f
Ot
Stack
P
*
O
1x1
1x1
3x3x H x X
Spatial Softmax
O My Wy
Attention Block
Figure 1: Illustration of our attention -augtion-- encoder. First, stack of three image frames are passed as input to a sequence of convolutional layers inside feature encoder (blue box). The feature encoder produces two sets of feature maps: (i) non-attentional features z with shape C'x Hy x W (ii) attentional features Tg with shape C' x H Wy. Attention block (orange box) takes in the two sets of feature maps as input, and outputs an attended copy of zy. Non-attentional and attentional features re-weighted and added together via residual connection.
BACKGROUND
Reinforcement learning We formulate visual control task partially observable Markov de- cision process (POMDP; Sutton & Barto 2018; Kaclbling et al. 1998). Formally, at each time step t, the agent receives high-dimensional observation ot, which is an indirect representation of the state 8t. and chooses an action a based on its policy т. The environment returns a reward r2 and the agent transitions to the next observation 0++1- The return R, = - r*r*+ is the total cumulated rewards from time step t with a discount factor y E [0, 1). The goal of RL is to learn policy that maximizes the expected return over trajectories. Following the common practice in DQN (Mnih et al., 2015), the state information from partially observable environments is approx- imated using stacked input observations, i.e. 8t equation 1 is approximated by l consecutive frames (typically k- 3): = (0t, 9t-1, 1, 0t-k+1)
Soft actor-critic SAC (Haamoja et al., 2018) is an off-policy actor-critic method based on the maximum entropy RL framework (Ziebart, 2010), which enhances robustness to noise and encour- ages exploration by maximizing weighted objective of the reward and the policy entropy. To update the parameters, SAC alternates between soft policy evaluation and a soft policy improvement. At the policy evaluation step, soft Q-function, which is modeled as a neural network with parameters 0, is updated by minimizing the following soft Bellman residual:
(f) = ET (Qo(se, az) -EE [Qa($141, 4441) - a log To (01-1110411)))
(1)
where = (st, at, Tt, St+ is a transition, B is replay buffer, Ø are the delayed parameters, and a temperature parameter. At the soft policy improvement step, the policy π with its parameter Ø updated by minimizing the following objective:
(Ø) = EB,--B.,-mR, [ox log (a)ls)) - Qo(sr, az)]
Here, the policy modeled a Gaussian with mean and diagonal covariance given by neural networks to handle continuous action spaces. In this paper, unless mentioned otherwise, all policies are modeled in this way.
R-LATTE
In this section, present R-LAtte: Reinforcement Learning with Attention module, simple, yet effective architecture for encoding image pixels in vision-based RL, which typically faces the challenge of partial observability and high-dimensional inputs. By using attention mechanism, our encoder enforces the agent to select and focus on a subset of its perception field. There are two key components in architecture (see Figure 1): two-stream encoding, and 2) adaptive scaling.
3

---

Under review as a conference paper at ICLR 2021
4.1 TWO-STREAM ENCODING FOR ATTENTIONAL FEATURES AND NON-ATTENLIONAL FEATURES
Our two-stream encoding starts by processing each frame in the observation 8t [o; 0t-1"'. Ot-k+1| independently using a shared encoder p. This results in a stacked, encoded observation & = p(o:) p(or-1)... p(o==x+]]]- We choose to deviate from the common prac- tice of stacking the frames together before encoding them using convolutional encoder together (e.g., (Mnih et al., 2013; 2015; Kostrikov et al., 2020; Laskin et al., 2020)) since using a shared encoder for each frame yields better performance in experiments (See Figure 5 for supporting
experimental results).
With the stacked encoded observations, the two streams start branch with convolutional encoders, f and which encode the stacked observations into non con-attentional feature maps 2 = fs() and attentional feature maps Zg = g(5)). Both feature maps Zg E RC have C channels and H x W spatial dimensions.
We then use dot product between these two feature maps to produce the soft attention mask as follows:
A = Softmax (q(x)) @ xg).
where q is a convolution and denotes the wise product. Here, the element-wise product (xx)) @ By accentuates where strong signals come from both feature maps, and suppresses those otherwise. This accentuation effect is further normalized using spatial Softmax function to obtain the soft attention mask.
4.2 ADAPTIVE SCALING
Finally, we use the attention maps A to obtain attentional feature maps Ty E RCXH, and provide the agent with mix of features by adding y back its original with a residual connection:
x= z + where 3 h(A o v(z,)),
(4)
where * denotes multiplying one scalar value to each channel of features in T h, v are two 1 convolutions, and a RC are scalars that define the importance of attentional feature maps. modeled as learnable network parameters, and get optimized together with other parts of the encoder. In our experiments, they are initialized be all zeros, but quickly adapted to different values as the training proceeds: intuitively, a controls the amount of annaattetiioal versus attentional features for the agent to use, and making a learnable allows it to adapt to different features as training proceeds. Ablation set 8(b) provides more experimental results using alternatives to o-weights that under-perform using as, and show this need for dynamically control the two-stream features.
EXPERIMENTS
5.1 SETUP
We demonstrate the effectiveness of R-LAtte on a set of challenging visual control tasks from Deep- Mind Control Suite (Tassa al., 2018), which are commonly used as benchmarks in previous work (Srinivas et al. 2020, Laskin et al. 2020, Kostrikov et al. 2020). This benchmark present a vari- ety of complex tasks including bipedal balance, locomotion, contact forces. and goal-reaching with both sparse and dense reward signals. For evaluation, we consider two state-of-the
RL methods: Dreamer (Kostrikov et al., 2020), state-of-the-aa model-based RL method that utilizes the latent dynamics model to generate synthetic roll-outs; and DrQ (Kostrikov et al., 2020), state-of-the-art model-free RL method that applies data augmentations to SAC (Haarnoja et al., 2018). For our method, we train an agent using SAC with R-LAtte encoder, which applies the proposed attention module to the image encoder architecture from SAC-AE (Yarats et al., 2019). We report the learning curves across three runs various task domains including Hopper, Finger, Walker, HalfCheetah, Ball-in-C Cup and record the learning curve. The details of architecture and experimental setups are provided in Appendix B.
4

---

Under review as a conference paper at ICLR 2021
Czeeko Bun
Froox Sen
-000X Saad
1200
800
Cara SACPoal
SAG
2.5
Figure 2: Performances of DeepMind control suite tasks trained with SAC using different encoder architectures for pixel inputs. Our attention network (blue curve) significantly outperforms the base- line encoder, and achieves similar final performance with SAC trained from state inputs.
Ball Cup Catch
Cartpole Swingup Sparse
Cheetah Run
Finger Spmn
1003
1000
1000
Ep1000 820m
Retum
000
E00
600
4000
200
200
200
200
DXQ
DM
DnG
Donmmer
Creamnr
Dreamre
Doeamar
-200g
0.1
02
024
05
200
0.
0.2
84
B5
08
1.0
-20g
0.5
1.0
20
-200
00
332
04
0.8
Environmenn (x-10))
Envrooment Stepe (-:0
Enviroment Steps (x107)
Endironment Stepe (x-IP)
1200
Hogper Stand
1200
Pendulur Swingup
1200
Walker Stand
1200
Walker Walk
1000
1000
11000
500
Spoo0 Rot.m
600
1330
Sploods Itatam
800
100
400
600
600
400
220
400
Olurs
Cur
Dara
Cara
DVQ
2200
Creamer
DKQ
200
Dreamme
DrQ
Dreamer
DVG
0.5
0.7
1.0
8.00
21.
08
87
0.7
0.4
1.0
85
23
Ervironment (x-1F)
Environment (x10")
Ensironment
Figure 3: Learning curves DeepMind control suite. The solid line and shaded regions represent the mean and standard deviation, respectively, across three runs. Our attention network alone, with- any representation learning, world model fitting, data augmentation techniques, performs par with the state-of-the-art methods.
5.2 MAIN RESULTS
Comparison with other encoder architectures for pixel input. We first compare our attention- augmented encoder with a 4-layer (each contains 32 filters) convolution encoder from SAC- AE (Yarats et al., 2019), which is standardized in recent works (Srinivas et al. 2020, Laskin et al. 2020, Kostrikov al. 2020). For both ours and baseline (denoted by SAC:Pixel), we train agents using SAC without representation learning and data augmentation in order to verify the gains from architecture. Figure/ shows that our method significantly outperforms SAC:Pixel in terms of both sample-efficiency and final performance. In particular, our method matches the performance of agents trained with state inputs (denoted by State:SAC), which clearly shows the benefits of atten- tion module in RL from pixels.
Comparative evaluation. In Figure 3, we compare the performance our method with the state-of- the-art DrQ and Dreamer. One note that our attention network can perform par with the state- of-the-art methods without any representation learning, world model fitting, or data augmentation techniques. We remark that method has lower variance compared to other baselines in almost all environments.
5.3 ANALYSIS
Supervised setting. To further verify the effectiveness of the attention module, test our architec- ture on an offline supervised learning task. Specifically, we collect replay buffer from a well-train agent that achieves around 1000 average reward and optimize the policy with the attention module
5

---

Under review as a conference paper at ICLR 2021
0.
0.6
Validation Behaniou
Ctring Loss
0.3
02
wt Attention Module (Fitter Size 32)
Attention Module (Fitter
weo Attertion Module (Fliter Siza 64)
10030
20000
30000
40003
50000
6100C 70000
Training Steps
Figure 4: Behaviour cloning performance comparison of our architecture (w/ Attention Module) and the architecture used in RAD (Laskin et al., 2020) (w/o Attention Module). Additionally, we use random cropping data augmentation when training the network with the RAD architecture. Since our architecture have additional encoders that add more parameters, we also include wider version of the architecture used in RAD (Filter Size 64) that has more parameters than architecture (Filter Size = 32) for comparison. Even with random cropping, the architecture used RAD under- performs our architecture. See more details of the offline experimental setup in Appendix A.
TJ
Fosture encoder
xpes
T g
F.f
Figure 5: Visualization of outputs from the proposed attention module. Our agent demonstrates selective feature-cxtraction scheme that pays more attention to the most relevant locations for each chosen action.
using the behaviour cloning objective (see equation in Appendix A). We compare the validation performance of our architecture against the architecture used in RAD (Laskin et al., 2020) (see Fig- ure 4). Our architecture outperforms the baseline consistently on walker walk task with a faster convergence rate and better final performance.
Visualize attention. To understand the effects of our attention module, we visualize its outputs in Figure 5. First, we remark that two streams of feature outputs, æy (blue box) and Ty (red box) extract very different information from sample input sources, and the attended output & (green box) highlights the activated locations in both features. Also, shown in the final combined features Ty + ł (yellow box), we find that agent is able to consistently locate itself, captures body movements, and ignores lask-irrelevann information in the perception field such as background or floor patterns. Over a course of actions taken, the agent also dynamically changes its attention focus (highlighted by the attention maps) in interpretable manner, e.g., paying extra attention its knee joint when taking stride.
5.4 ABLATION STUDIES
The performance gain presented in the above section comes solely from attention- augmented neural network architecture for encoding pixel images. Here, to provide further insights and jus- tifications for design, provide set of ablation studies the proposed encoder's major

---

Under review as a conference paper at ICLR 2021
Chacaa Pus
Figger 5p.1 Crink
Sland
Wulter Yiat
Festura
Dedomen Crown
Op- 1
(a) Illustration stack-then- (b) Performance comparison between the ablation and our original encoder encode architecture
Figure 6: We evaluate the ablation encoder p' on a subset of visual control tasks, and demonstrate that, although the original shallow encoder p uses a smaller capacity, it outperforms the ablation encoder consistently across the tested domains.
components. For each ablation, we report the average and standard deviation of learning curves from three ramdom seeds.
6 ABLATION ON THE ARCHITECTURE FOR SHARED ENCODER MODULE 9
Shared-encoder p. As described in section 4.1, inside our convolutional feature encoder module, we use both a shared encoder p and two parallel encoders f and g. In our implementation, p is shallow, one-layer convolution that, in contrast to common practices of stacking and then encoding consecutive image frames, first encodes each time-step's ss image observation separately, then stack the features together afterwards. In our implementation, we use one-layer convolution that, takes in 3 channels (for RGB input) and outputs 10 channels, such that stacking three encoded frames will give 30 total feature channels. For ablations, we swap this convolution layer with a bigger filter size: for stack of 3 frames, takes in 9 input channels and output 30. With other parts of the architecture kept the same, we test this stack-then-encode layer on Walker-walk, as shown in Figure 6. With increased filter size, this ablation encoder effectively has bigger capacity, but the performance gap suggests, this does not work as well as using a smaller layer, which preserves the temporal changes between consecutive frames.
Chostah
Weler
Pho-Sefinee-O Oity
Seftren- Only
Only
300
Epoods Raamm
H00
200
-000
Eeviommat
Envinomer
(a) Performance comparison between the ablation attention block and our main architecture
1x1
Speii Sattioo
Spotia Sotwax
Pro Softrax Dot Product Only
Post-Softmax Dot Product Only
(b) Illustration for the ablation attention blocks that use only one Hadamard product
Figure 7: Ablation illustrations and learning curves on Walker-walk, Cheetah-run, Finger-spin and Hopper-stand.

---

Under review as a conference paper at ICLR 2021
Cherah Pace
Walber
190
Diret adoite
Cuncal
additon
Corcaleracion
325
080
031
105 330 31
(a) Illustration of two alternative ways of combining y and y
(b) Learning curves
Figure 8: (a) Illustration of direction addition and concatenation, as alternatives to the adaptive scal- ing method used in our main architecture. (b) Performance comparison between using the ablation combination methods and adaptive scaling using a.
Hadamard products inside the attention block. Given two inputs 2 and Zg described in section 4.1, the attention module performs Hadamard product twice, and applies spatial Softmax function in between. The first Hadamard product feature array with another, and the second product is an "attending" step that utilizes the attention maps produced by the spatial Softmax, which we can verify through visualized outputs from the encoder. As ablations, we test two variants of this operation sequence, both does Hadamard product only once, but differs in location, i.e. before or after the Softmax function, as illustrated in Figure 7.
be seen
element-wise masking of one
Adaptive scaling. Here, with other parts of the encoder architecture fixed, we perform two ablations on alternative ways combine Ty and f namely the non-attentional and attentional features. (A) We experiment direct residual combination If Ty. i.e. without re-weighting. addition, we concatenate the two features, and correspondingly increase the input dimension for the subsequent fully connected (FC) layers (output feature dimension for FC layer are kept the same). As the learning curve shows, the two ablations clearly underperforms the original version, which suggests the need for more flexible, cearningbbase combination of the features.
(B) Instead of
7 CONCLUSION
We present Reinforcement Learning with Attention module (R-LAtte), a simple yet effective ar- chitecture for RL from pixels. On DeepMind Control benchmarks, our simple architecture is able to achieve similar performance compared against representation learning and data augmentation methods, which are currently the best-performing ones in the literature. We also illustrate that our proposed module extracts interpretable rask-relevant information without the need for data augmen- tations or contrastive losses. We hope that our architecture will serve as a simple and strong baseline for future research in RI. from pixels, especially for investigations on neural network architectures for encoding high-dimensional inputs. Potential future directions from this work include combining representation learning or data augmentation methods, both with the attention module, as well as exploring better attention architecture that is more suitable for visual-based RL.
REFERENCES
D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio. End-to-end attention large vocabulary speech recognition. In 2016 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 4945-4949 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations, 2015.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin,
8

---

Under review as a conference paper at ICLR 2021
Scott Gray, Benjamin Chess, Jack Clark, Christopher Bernr,, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Jongwook Choi, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, and Honglak Lee. Contingency-aware exploration in reinforcement learning. In International Con- ference Learning Representations, 2019.
Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition, 2015.
Maurizio Corbetta and Gordon Shulman L. Control of goal-directed and stimulus-driven attention in the brain. 2002.
Vikash Goel, Jameson Weng, and Pascal Poupart. Unsupervised video object segmentation for deep reinforcement learning. Advances in Neural Information Processing Systems, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- on Machine Learning, 2018
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lec, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference Machine Learning, 2019.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference on Learning Representations, 2017.
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134 1998.
Sabine Kastner and Leslie Ungerleider G. Mechanisms of visual attention in the human cortex. 2000.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Thomas Kipf, Elise der Pol, and Max Welling. Contrastive learning of structured world models. In International Conference on Learning Representations, 2020.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 10044,13349,,.. 2020.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Re- inforcement learning with augmented data. arXiv preprint 200e:0900419990 2020.
Kuang-Huee Huei Lee, Ian Fischer, Anthony Liu, Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarraaa. Predictive information accelerates learning in rl. arXiv preprint arXiv:2007.1240) 2020.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):1334-333,, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. 2013.
Volodymyr Mnih, Nicolas Heess, Alex Graves, and koray kavukcuoglu. Recurrent models of visual attention. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, 2014.

---

Under review as a conference paper at ICLR 2021
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning, Nature, 518(7540):529, 2015.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03745. 2018.
Ronald A Rensink. The dynamic representation of scenes. Visual cognition, 7(1-3)117-42, 2000.
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International conference machine learning, 2020.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In IEEE international conference on computer vision, 2017.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint 14Xi 6806, 2014.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. arXiv preprint arXiv:2004.044166, 2020.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. arXiv preprint arXiv:2004.14990, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint arXiv:1801.00690 2018.
Sabine Kastner Ungerleider and Leslie G. Mechanisms of visual attention in the human cortex. Annual review of neuroxcience, 231)33155344, 2000.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, 2017.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in Neural Information Processing Systems, 2015.
Nicholas Watters, Loie Matthey, Matko Bosnjak, Christopher Burgess, and Alexander Lerchner. Cobra: Data-efficient model-based rl through unsupervised object discovery and curiosity-driven exploration. arXiv preprint arXiv:1905.0922. 09955,9 2019.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, 2015.
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Im- proving sample efficiency in model-free reinforeement learning from images. arXiv preprint 190.0.1910 2019.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the per- formance of convolutional neural networks via attention transfer. In International Conference Learning Representations, 2017.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillierap, Edward Lockhart, et al. Relational deep reinforcement learning. arXiv preprint /000.v/800.00 830, 2018.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks, 2019a.
10

---

Under review as a conference paper at ICLR 2021
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine. Solar: Deep structured representations for model-based reinforcement learning. In International Conference Machine Learning, 2019b.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010.

---

Under review as a conference paper at ICLR 2021
Appendix
A OFFLINE BEHAVIOUR CLONING EXPERIMENTAL SETUP
We first train a RAD (Laskin et al., 2020) agent to convergence where it achieved around 1000 average reward (in DeepMind Control Suite, the reward is normalized be between 0 to 1000). Then, we collect 100,000 transitions and store in a replay buffer by rolling out the agent in the environment. After the replay buffer is collected, our baseline) and our architecture using stochastic gradient descent with the following behaviour cloning loss:
train both the architecture used in RAD (as
(o) -, [log (T(aa))s))
We use a batch size of 128 and Adam optimizer (Kingma & Ba, 2014) with (31, (2) = (0.9, 0.999), é 10-5 To determine the best learning rate for each method, we performed learning rate search over 0.001, 0.0003 and 0.003. We found that learning rate of 0.0003 worked the best for both the baseline and our architecture. We repeat the same experiment for three different seeds and report the mean and the standard deviation.
B EXPERIMENTAL SETUPS
We use the network architecture in https://githi:: com com/MishaLaskin/Sa for our im- plementation. We show a full list of hyperparameters in Table We will release our code in the camera-ready version.
12

---

Under review as a conference paper at ICLR 2021
Table 1: Hyperparameters used for DMControl experiments. Most hyperparameter values are un- changed across environments with the exception of initial replay buffer size, action repeat, and learning rate.
Hyperparameter
Value
Augmentation
Crop
Observation rendering
(100, 100)
Observation down/upsampling
(84, 84)
2000 reacher, easy; walker, walk
Number of updates per training step
1
Initial steps
1000
Stacked frames
3
Action repeat
2 finger, spin; walker, walk reacher, easy
8 cartpole, swingup
Hidden units (MLP) Evaluation episodes
1024
10
Evaluation frequency
2500 cartpole, swingup
1000 finger. spin; reacher, easy; walker, walk
Optimizer
Adam
(81, 82) (fa, To, Qo) (81, (2) (a)
(.9.9999)
(.5,9999)
Learning rate (fur No Qo)
le
Learning rate (a)
le
Batch Size
128
Q function EMA
0.01
Critic target update freq Convolutional layers
2
4
Number of filters
32
Non-lincarity
ReIUU
Encoder EMA
0.05
Latent dimension
50
Discount
.99
Initial lemperature
0.1
13


Reviews:
[{"review_id": "bXEi01yVAd", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an alternative encoder architecture for an image-based deep RL agent that leverages attention. The authors suggest to compute attentional and non-attentional convolutional features, which later are combined together via a weighted (learned) residual connection. The designed architecture seems to improve performance of the agent on standard continuous control tasks from the DeepMind suite.\n\nSignificance:\nThe overall novelty and significance of the paper is low. While I appreciate the authors’ drive to research alternative network architectures in deep RL, I’m unfortunately unable to find any significant insight either from a theoretical (not studied) or empirical (the experimental setup is questionable) perspectives.  More details:\n\nPros:\n + Interesting investigation into an architecture choice for a deep RL agent. I think this is an important direction to tackle and unfortunately there has been very little attention given to this problem.\n + Interesting ablation that demonstrates the learned attention map and what different features are responsible for.\n\nCons:\n- The novelty, insight, and contribution are limited. Moreover, the empirical evidence provides inconclusive support for the proposed method.\n- Weak baselines. In Fig 2 performance of SAC:Pixels is weaker than in the literature (See Fig 6a in https://arxiv.org/pdf/1910.01741.pdf). In Fig 3 performance of DrQ is significantly weaker than reported in the original paper (https://arxiv.org/pdf/2004.13649.pdf) and given that DrQ’s code is publicly available and more importantly fully reproducible (personal experience), this casts doubt on the rigor and validity of conducted experiments. \n- This work goes against a common practice in RL and reports performance of their algorithm using only 3 random seeds.\n- There is very little information provided about the exact architecture of the attention block, I would appreciate more details here.\n- Absence of the source code makes me skeptical that the reported results are reproducible, given that the authors build on a publicly available code base (SAC-AE: https://github.com/denisyarats/pytorch_sac_ae).\n- The supervised setting experiment (Sect 5.3) is also inconclusive since the authors compare train loss performance for each of the agents. Obviously agents that use data augmentation would overfit less and thus report higher loss on training dataset. Besides, it is not clear if the dataset for this experiment came from the agent with attention module or not. If so, this will also make learning harder for RAD as it has weaker affinity with the generated data.\n- The authors suggest that they don’t use data augmentation, but in Appendix B Table 1 they also suggest that random crops are being used. This needs to be further clarified.\n\nQuality:\nThe paper’s quality is mediocre. A fully empirical paper requires greater experimental rigor and clarity. \n\nClarity:\nThe paper would benefit from providing more transparency over the experimental setup and clear reporting of the used hyper parameters, especially given the simplicity of the proposed method. Several places in the paper (that I detailed above) were either conflicting or ambiguous. \n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Bg60I4S2Cbm", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\n**Summary**:\n\nThis paper proposes to apply the soft attention mechanism in a CNN network to boost the learning speed of an RL agent in environments such as DeepMind Control Suite. The key idea is that the attention mechanism is a better network architecture and can extract interpretable task-relevant information to improve the learning. However, attention has been used in several past RL works. And paper misses some important comparisons. Overall, the contributions of the paper are limited. \n\n##########################################################################\n\n**Strengths**:\n\nThe proposed method shows comparable performance against the baselines such as Dreamer and DrQ, and outperforms pure vision-based SAC.\n\n##########################################################################\n\n**Weaknesses**:\n\nThe motivation in the introduction section does not seem to be strong. In particular, \nthe second paragraph seems to be disconnected from the paper. Why do authors spend many words explaining self-supervised/unsupervised learning, model-based learning/planning, etc.?\n\nUsing attention in RL is not novel and has been explored in many works [1-5]. For example, [2] also proposed a CNN architecture with a self-attention mechanism. How does the proposed architecture in the paper compare to the one in [2]?\n\nIt's also not clear how good the proposed attention block is compared to other attention architectures that are proposed in previous works (self-attention, attention on flattened feature vectors [1], etc.).\n\nThe proposed method uses a stack of 3 observations, which can provide the policy more temporal information, while the baselines do not seem to use a stack of historical observations. Hence, it's not clear whether the claim that the proposed method achieves similar performance with SOTA still hold if SOTA algorithms are also given the same amount of information as input.\n\nThe keypoint-based method [6] has shown that the network can capture task-relevant information such as keypoint locations. This paper also shows some visualization about the attended regions on the observations in Figure 5. From what both papers show in the visualization, it seems the keypoint-based method can capture better interpretable task-relevant information. It would provide more a more comprehensive view of the series of works if the authors can comment on [6].\n\n\nThe paper compares to SAC with images as input. Such learning is typically slow. [7] shows that using an asymmetric actor and critic can significantly speed up the learning while requiring no state information at test time. How does the attention-based policy compare to this line of works?\n\n**Missing Ablations**:\n\n(1) For the ablation on shared-encoder, the paper provides a comparison between two settings: \n* 3 RGB images as input (3 channels as input), each of them are encoded using the same encoder (10 channels for each image output), and the outputs are then stacked (30 channels as output in the end); \n* 3 RGB images are stacked first (9 channels as input) and processed by an encoder (30 channels as output). These two settings have a different number of network parameters. It could be the latter one has more parameters and hence takes a bit longer time to train.\n\nTo provide a more thorough analysis, one should also compare to the following setting: \n* Convert each RGB image into grayscale, and stack the three grayscale images (3 channels as input), and then process them by an encoder that produces an output with 30 channels. While converting RGB images to grayscale lose some color information, such information should not affect the learning or final performance in the environments that the paper experimented with. \n\nAlso, I wonder how the curves look after 0.2M or 0.3M steps as shown in Figure (6, 7, 8). Could you provide the training curves for at least 1M training steps for the ablation study figure (Figure 6, 7, 8)?\n\n(2) How does the number of times steps (in the paper, it is 3) in the observation input affect the learning? An ablation on different stacking length will better demonstrate the importance and effect of having a history of observations.\n\n##########################################################################\n\n**Minor points**:\n\nIn Section 4.1 (Page 4), it should be Figure 6 rather than Figure 5 to support the claim that using a shared encoder gives better performance.\n\nThe structure of the writing is a bit off. Shouldn't section 6 be a subsection for section 5.4?\n\n\n##########################################################################\n\n**Reference**:\n\n[1] Mishra, Nikhil, et al. \"A simple neural attentive meta-learner.\" arXiv preprint arXiv:1707.03141 (2017).\n\n[2] Manchin, Anthony, Ehsan Abbasnejad, and Anton van den Hengel. \"Reinforcement learning with attention that works: A self-supervised approach.\" International Conference on Neural Information Processing. Springer, Cham, 2019.\n\n[3] Fang, Kuan, et al. \"Scene memory transformer for embodied agents in long-horizon tasks.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.\n\n[4] Sorokin, Ivan, et al. \"Deep attention recurrent Q-network.\" arXiv preprint arXiv:1512.01693 (2015).\n\n[5] Mott, Alexander, et al. \"Towards interpretable reinforcement learning using attention augmented agents.\" Advances in Neural Information Processing Systems. 2019.\n\n[6] Kulkarni, Tejas D., et al. \"Unsupervised learning of object keypoints for perception and control.\" Advances in neural information processing systems. 2019.\n\n[7] Pinto, Lerrel, et al. \"Asymmetric actor critic for image-based robot learning.\" arXiv preprint arXiv:1710.06542 (2017).\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5DmwuZcKrNh", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n###################################\n\nSummary: This paper aims to combine visual attention mechanisms to deep RL by proposing a simple attention module in the convolutional encoder of the deep RL agent. Two-stream encodings and adaptive scaling of balance between non-attentional/attentional masks are major components of this module. The authors present empirical results of their algorithm and compares with SOTA baselines in deep mind control suite. \n\n###################################\n\nPros\n\n1. This paper is fairly well written and easy to follow. One of the main contributions of the attention module is two-stream encoding, where there are separate non-attentional features and attentional features extractor. Attentional features have more task-specific information, while non-attentional features have task-agnostic information. This attention module can be combined with any RL baseline algorithm, like SAC. The structure of attention module is simple and easy to understand. \n\n2. R-Latte shows competitive performance in deepmind control suite. R-Latte performs much better than SAC+pixel, and almost achieves as good performance with SAC+state. When compared with SOTA algorithms (DrQ and Dreamer), R-Latte shows comparable performance in many domains. \n\n3. Visualization of attention module is helpful. As shown in Figure 5, the attended output \\hat{x_f} (green box) captures relevant activated locations, and the agent can dynamically change the attention focus depending on each action.\n\n###################################\n\nCons & Questions\n\nMajor\n\n1. Adaptive scaling of \\sigma: the \\sigma is an alternative parameter that should be tuned to balance between attentional and non-attentional features. I think the role of this parameter is consequential, but the paper lacks detailed description about this parameter. Could you plot how \\sigma changes over the course of training? (\\sigma is C-dimensional vector and initialized as zero, so you could plot the average of \\sigma over the course of training, and how it evolves over time.) Considering that adaptive scaling of balance between two encoder is one of the two major contributions of this work, the paper lacks sufficient explanations on this part. \n\n2. Need more justification for shared encoder for consecutive frames:  one of the difference of this encoder from other conventional encoders is the use of shared-encoder p. The original algorithms stack the consecutive frames first, and then put them altogether into the encoder; whereas in this work, each frame is encoded by p, and then stacked after the encoding. Although the difference between stack-first-baselines and your algorithm is shown in the ablation studies (Figure 6), I don’t see reasonable explanations on why this would benefit the performance. Do you have any intuition of why separate-encoding-then-stacking performs better? To my knowledge, there is a reason in stacking consecutive frames first (e.g. in DQN), because stacking consecutive frames contains its own information (Mnih et al. 2015) and separating them might break those information. \n\nMinor\n\n3. The performance of R-Latte is comparable with SOTA (Dreamer, DrQ), but doesn’t outperform them. In most case, R-Latte performs slightly worse than DrQ or Dreamer; it’s totally fine, but I think you should change the phrase like “significantly improve sample-efficiency and final performance of the agents”. \n\n4. Does your method show lower variances than SOTA baselines as mentioned in page 5? Only looking from the graph plots, I don't see that. Could you support this claim with statistical/numerical values? \n\n\n####################\n\nTypos & citation suggestions\n\n- In section 5.1. setup, I think you cited a wrong paper for Dreamer; you cited Kostrikov et al. 2020 for both Dreamer and DrQ. But you should fix it to Hafner et al. 2018 for Dreamer. \n- Consider citing these works at the related work section and adding some discussions about them too:\n\n[1] Mott, A.; Zoran, D.; Chrzanowski, M.; Wierstra, D, Rezende, D. J. 2019. Towards interpretable reinforcement learning using attention augmented agents. In Advances in Neural Information Processing Systems, 12350–12359.\n\n[2] Greydanus, S.; Koul, A.; Dodge, J.; Fern, A. 2018. Visualizing and understanding atari agents. In International Conference on Machine Learning, 1792–1801.\n\n[3] Yuezhang, L.; Zhang, R.; and Ballard, D. H. 2018. An initial attempt of combining visual selective attention with deep reinforcement learning. arXiv preprint arXiv:1811.04407\n\n\n##########################################\n\nOverall, I think this is a nice work, but given the concerns I have above and marginal/incremental advances in empirical results, I think this paper doesn't yet meet the threshold of ICLR. However, if these concerns are addressed, I am happy to raise my score after the rebuttal period. Also, related work section can be improved by covering more past works: there are a lot of works on visual attention/saliency, and there have been some recent works attempting to use attention for deep RL agents (not just the ones that I mentioned above). ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an attention-endowed architecture for deep image-based RL. While some positive points were raised by the reviewers, most comments were on the negative side.
The reviewers noted marginal/incremental advances in terms of empirical results and low novelty and significance. Moreover, the provided baselines seem weak.
Because of this, the present submission unfortunately does not meet the publication bar.
I recommend the authors take into account the constructive feedback from reviews and discussion and submit an improved version to another venue.
Decision: Reject