Attention Privileged Reinforcement Learning for Domain Transfer
Authors: Sasha Salter
Dushyant Rao
Markus Wulfmeier
Raia Hadsell
Ingmar Posner
Venue:  (2020)
Keywords: sim-to-real
domain randomisation
attention
transfer learning
reinforcement learning

Abstract: Applying reinforcement learning (RL) to physical systems presents notable challenges, given requirements regarding sample efficiency, safety, and physical constraints compared to simulated environments. To enable transfer of policies trained in simulation, randomising simulation parameters leads to more robust policies, but also in significantly extended training time. In this paper, we exploit access to privileged information (such as environment states) often available in simulation, in order to improve and accelerate learning over randomised environments. We introduce Attention Privileged Reinforcement Learning (APRiL), which equips the agent with an attention mechanism and makes use of state information in simulation, learning to align attention between state- and image-based policies while additionally sharing generated data. During deployment we can apply the image-based policy to remove the requirement of access to additional information. We experimentally demonstrate accelerated and more robust learning on a number of diverse domains, leading to improved final performance for environments both within and outside the training distribution.

Under review as a conference paper at ICLR 2020
ATTENTION PRIVILEGED REINFORCEMENT LEARNING FOR DOMAIN TRANSFER
Anonymous authors Paper under double-blind review
ABSTRACT
Applying reinforcement learning (RL) to physical systems presents notable chal- lenges, given requirements regarding sample efficiency, safety, and physical con- straints compared to simulated environments. To enable transfer of policies trained in simulation, randomising simulation paramcters leads to more robust policies, but also significantly extends training time. In this paper, we exploit access to privileged information (such as environment states) often available in simulation, in order to improve and accelerate learning over randomised envi- ronments. We introduce Attention Privileged Reinforcement Learning (APRiL), which equips the agent with an attention mechanism and makes use of state infor- mation in simulation, learning to align attention between state- and image-base based policies while additionally sharing generated data. During deployment we can apply the image-based policy to remove the requirement of access to additional information. We experimentally demonstrate accelerated and more robust learn- ing number of diverse domains, leading to improved final performance for environments both within and outside the training distribution.
INTRODUCTION
Deep Reinforcement Learning (RL) has recently provided significant successes in a range of areas,
including video games (Mnih al. 2015), board games (Silver et al. 2017). simulated continu- ous control tasks (Lillicrap et al. (2155). and robotic manipulation (Haamoja et al. [2018; Haarnoja 2018 Riedmiller et al.] al.]] 2018 OpenAl et al. 2018 Schwab et al.) Andrychowwc al.) [017)) However, application to physical systems has proven to be challenging in general, due to expen- sive and slow data generation as well as safety challenges when running untrained policies. A common approach to circumvent these issues is to transfer models trained in simulation to the real world (Tobin et al. 2017 Rusu et al.] 2016 Held et al. 2017). However, simulators only repre- sent approximations of a physical system. Due to physical, visual, and behavioural discrepancies, naively transferring RL agents trained in simulation onto the real world can be challenging.
To bridge the gap between simulation and the real world, we can either aim to align both do- mains (Ganin et al., 2016; [Bousmalis et al.) 2016; Wulfmeier et al.] (017) or ensure that the real
system covered by the distribution of simulated training data (OpenAl al. 2018 Tobin et al. 2017 Pinto et al. 2018 [Sadehhi & Levine (016)) Viereck et al., 2017). However, training under a distribution of randomised visual attributes of the simulator, such textures and lighting (Sadeghi
& Levine 2016) Viereck et al.] (207)), as well as physics (OpenAl et al., al. 2018), can be substantially more difficult and slower due to the increased variability of the learning domain (OpenAl et al. 2018 Tobin et al., (2017).
The more structured and informative the input representation is with respect the task, the quicker the agent can be trained. A clear example of this effect can be found when an agent is trained with image inputs, versus training with access to the exact simulator states (Tassa al.) 20188] |Pinto et al.,
[Pinoo
2018). However, visual perception is more general and access to more compressed representations can often be limited. When exact states are available during training but not deployment, we can make use of information asymmetric actor-critic methods (Pinto et al.) 2018 Schwab et al. 2019) to train the critic faster via access to the state while providing only images for the actor.
Videos comparing the policy hehaviours of APRIL the asymmetric DDPG haseline he found here

---

Under review as a conference paper at ICLR 2020
By introducing Attention Privileged Reinforcement Learning (APRiL), aim to further leverage access to exact states. APRiL leverages states not only to train the critic, but indirectly also for an image-based actor. Extending asymmetric actor-critic methods, APRiL concurrently trains two actor-critic systems (one symmetric, state-based agent, and another asymmetric agent with image- dependent actor). Both actors utilise an attention mechanism filter input data and by having access to the simulation rendering system, we can optimise image and state based attention masks to align.
By additionally sharing the replay buffer between both agents, we can accelerate the learning process of the image-based actor by training on better performing states that are more quickly discovered by the state-based actor due to its lower dimensional input that is invariant to visual randomisation.
The key benefits of APRiL lie in its application to domain transfer. When training with domain randomisation for transfer, bootstrapping via asymmetric information has displayed crucial bene- fits (Pinto et al., 2018). Visual randomisation substantially increases the complexity of the image- based actor's task. Under this setting, the attention network can support invariance with respect to the irrelevant, but highly varying, parts of the image. Furthermore, the convergence of the state- space actor remains unaffected by visual randomisation.
We experimentally demonstrate considerable improvements regarding learning convergence and more robust transfer on set of continuous action domains including: 21) navigation, 2D locomotion and 3D robotic manipulation.
2 PROBLEM SETUP
Before introducing Attention Privileged Reinforcement Learning (APRiL), this section provides a background for the RL algorithms used. For more in-depth introduction please refer to Lillierap et al. (015) and Pinto et al. 2018)
2.1 REINFOR LEARNING
We describe an agent's environment as Partially Observable Markov Decision Process which is represented as the tuple (S, 0, P, 7, where S denotes a set continuous states, denotes set of either discrete continuous actions, P S x A x S + {a R|0 S } is the transition probability function, r S x A + I is the reward function, y is the discount factor, and 8o is the initial state distribution. O is set of continuous observations corresponding to continuous states in S. At every time-step t, the agent takes action a = (- $t) according to its policy π : + A. The policy is optimised to maximize the expected return R. Exg |Ee (rssss) The agent's Q-function defined as Q (St, az)) = FRReese at)-
2.2 ASYMMETRIC DEEP DETERMINISTIC POLICY GRADIENTS
Asymmetric Deep Deterministic Policy Gradients (asymmetric DDPG) PPito et al.) (018)) repre- sents type of actor-critic algorithm designed specifically for efficient learning of a deterministic, observation-baut policy in simulation for sim-to-real transfer. This is achieved by leveraging ac- cess to more compressed, informative environment states, available in simulation, to speed up and stabilise training of the critic.
The algorithm maintains two neural networks: an observation-based actor or policy म. : : + A (with parameters 8) used during training and test time, and state-based Q-function (also known as critic) 0 S x A + R (with parameters Φ) which is only used during training.
2014)) relies on a noisy version of the policy (called behavioural policy), e.g. Tb(o) T(o) + where ~ () (see Appendix C for our particular instantiation). The transition tuples (st, or, St:1, oc+1) encountered during training are stored in replay buffer (Mnih et al. 2015). Training exam- ples sampled from the replay buffer are used to optimize the critic and actor. By minimizing the Bellman error loss Lorttic (Q(st, a.) - 22 where y/ rı - 2Q( T(0(41)), the critic is optimized to approximate the true Q values. The actor is optimized by minimizing the loss fucter F. Fs.o [Q(s, (o))).
To enable exploration, the method (like its symmetric version (iilerr et al.
St+1,
2

---

Under review as a conference paper at ICLR 2020
A
State Attention
State Critic
State
Q2
State- Mepping
At
Object Attention
Target Obs Attention
weighted
Aa
Obs Attention
Obs Actor
Critic
Obs
Tie
Bc
Qu
Environment
Figure 1: Attention Privileged Reinforcement Learning model structure. Dashed lines indicate at- tention alignment process. The operator signifies that experiences evenly sampled from both agents. The @ operator represents element-wise multiplication.
ATTENTION PRIVILEGED REINFORCEMENT LEARNING (APRIL)
APRiL proposes to improve the performance and sample efficiency of an observation-bant agent by using a quicker learning actor that has access to exact environment states, sharing replay buffers, and aligning attention mechanisms between both actors. While we focus following sections on extending asymmetric DDPG (Pinto al. 20188), these ideas generally applicable off off-policy actor-critic methods (Konda Tsitsiklis 2000).
APRiL is comprised of three modules displayed in Figure The first two modules, A, and Aas are actor-critic algorithms with an attention network incorporated over the input to each actor. For the state-based module A, A, builds on asymmetric DDPG. Finally, the third part Ar represents the alignment process between attention mechanisms of both actor-critic agents to effectively transfer knowledge between the quicker and slower learners, A, and Ag, respectively.
standard symmetric DDPG, while the intervation--aeee module
A, consists three networks: Q7 n.. h2 (respectively critic, actor, and attention) with parameters {o., 0s, v.J. Given input state the attention network outputs soft gating mask he of same dimensionality the input, with values ranging between [0, 1]. The input to the actor an attention- filtered version of the state, sft = h2 (st) To encourage sparse masking function, we found that training this attention module on both the traditional DDPG loss as well as an entropy loss helped:
Cr. = E. (2,(s, Ta (s")) - 3H (ho(s)))
(1)
where & is hyperparameter to weight the additional entropy objective, and my is the behaviour pol- icy used to obtain experience (in this case from shared replay buffer). The actor and critic networks t, and Q2 are trained with the symmetric DDPG actor and Bellman error losses respectively.
Within Ar, the -tate-attention obtained in A, converted to corresponding abservation-attentie T to act as a self-supervised target for the observation- based agent Ag. This achieved in a two- step process. First, state-attention h, (s) converted into object-attention which specifies how task-relevant each object in the scene is. Second, object-attention converted to observation-space attention by performing weighted over object-specific segmentation maps:
N T
= M (s))
(2)
=
C
24
(3)
0
Here, M E (0, 10,1)NxM, (where n. is the dimensionality of s) an environment-specfff, predefined adjacency matrix that maps the dimensions of each corresponding object, and E 1]M is
3

---

Under review as a conference paper at ICLR 2020
then attention vector the N objects in the environment. C corresponds to the ith object attention value. 2 E {0, 1| x is the binary segmentation mag of of the jt object segmenting the object with the rest of the scene, and has the same dimensions as the image observation. 2, assigns values of 1 for pixels in the image occupied by the jt object, and O elsewhere T E [o. 1j x is the converted state-attention observation-span -pacinnn--pceeeeeee attention act target to train the observation- attention network tt on.
The observation-based module A. also consists of three networks: Q5, Tos ho (respectively critic, actor, and attention) with parameters { 1o)- The structure of this module is the same as A, except the actor and critic now have asymmetric inputs. The imput the actor is the attention- filtered version of the observation, of = (ho(or) @ of The actor and critic networks and Qo are trained with the standard asymmetric DDPG actor and Bellman error losses respectively defined in Section 2.2 The main difference between A, and A, is that the observation attention network ho is trained on both the actor loss and object-weighted mean squared error loss:
2 Σ wis (ho(o) Tÿ =2008,
Ch. Bosswas
(4)
where weights Way correspond to the fraction of the partial observation that the object present in of..1:3 occupies, and represents the relative weighting of both loss components. The weight terms, w, ensure that the attention network becomes invariant to the size of objects during training and does not simply fit to the most predominant object in the scene. Combining the self-supervised attention loss and the RL loss leverages efficient state-space learning unaffected by visual randomisation.
During training, experiences are collected evenly from both state and observation based agents and stored in a shared replay buffer (similar to (Scwwab et al. (2019). This to ensure that: 1. Both state- based critic Q, and observation-bast based critic Q. observe states that would be visited by either of their respective policies. 2. The attention modules ht and ha are trained on the same data distribution to better facilitate alignment. Efficient discovery of highly performing states from Re are used to speed up learning of n..
Algorithm shows pseudocode for a single actor implementation of APRiL. In practice, in order to speed up data collection and gradient computation, we parallelise the agents and environments and ensure data collection from state- and image- based agents even.
EXPERIMENTS
To demonstrate the performance and generality of our method, we apply APRiL to a range of envi- ronments, and compare with competitive asymmetric DDPG baseline and various ablations. We evaluate APRiL. over different metrics to investigate how attention helps with robustness and gener- alisation unseen environments and transfer scenarios. Further experimental details can be found in Appendix C
4.1 EVALUATION PROTOCOL
In order to investigate APRiL under varying conditions, we evaluate in scenarios of increasing com- plexity covering simple 2D navigation, 3D reaching and 2D dynamic locomotion.
We use the following continuous action-space environments (see Appendix A for further details):
1. NavWorld: In this 2D environment, the goal is for the circular agent to reach the triangular target in the presence of distractors. The agent is sparsely rewarded if the target is reached.
2. JacoReach: In this 3D environment the goal of the Kinova arm (Campeau-Lecous 2017) agent is to reach the diamond ShapeStacks object (Groth et 20118) 118 in the presence of distractors. The agent rewarded for approaching and reaching its goal.
al.)
Many simulators, like Todorov et al. 2012). natively provide functionality to access these segmentations In practice, the output of (o:) tiled match the numher of channels that the image contains
4

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Attention Privileged Reinforcement Learning
Initialize (a)symmetric actor-critic modules Ag, Ag, attention alignment module Ar, replay buffer for episode= to M do
Initial state while DONE do Render image observation o and segmentation maps 24: 2ς f- renderer(s:)
if episode mod 2 0 then
Obtain action using ohs-hehavioral policy and obs-attention network: d <- (ox) (.o) o.)
else
Obtain action using state- beate-bbhhvioall policy and state-attention network:
- (8.( ② st)
end if
Execute action as, receive reward rt, DONE flag, and transition to 8++1 Store (8t, Ot, Ze, de, Te, St+1, 0++1) in R
end while
for = 1 N do
Sample minibatch (s, 0, r
from R
Optimise state- critic, actor, and attention using [s, a,ar,, 3 with A.
Convert state-attention to target observation-attenicc {T) 0 using (s.0, 2 with Ar
Optimise observation- critic, actor, and attention using {s, T, a,, 8, jo with Ao
end for end
3. Walker2D: In this slightly modified 2D Deepmind Control Suite environment TTsss et al., 2018 the goal of the agent to walk forward far possible within time-limit, The agent receives reward for moving forward as well as reward for keeping its torso upright.
For these domains randomise visuals during training as enable generalisation to these variable aspects of the environment. We randomise combination of: camera position and orientation, textures, materials, colours, object locations, background. Refer Appendix B for more details.
4.2 KEY RESEARCH QUESTIONS
We investigate the following questions to evaluate how well APRiL accommodates for the transfer- ring of policies across visually distinct environments: Does APRiL Increase sample-efficiency during training? 2. Affect interpolation performance on unseen environments from the training dis- tribution? 3. Affect extrapolation performance on environments outside the training distribution?
We qualitatively analyse the learnt attention maps (both on interpolated and extrapolated domains). Finally, we perform an ablation study to investigate which parts of the APRiL contribute perfor- mance gains. This ablation consists of the following models:
APRiL no self-supervision (APRiL no sup): APRiL except without the self-supervision provided by the state agent to train the observation-baut attention. Both agents are still equipped with an attention module, but the observation attention must now learn without guidance from the state agent. Without bootstrapping from the state agent in this way expect learning of informative observation-based attention to be hindered.
2. APRIL no shared buffer (APRiL. no share): APRiL except each agent has its own replay buffer, instead of one shared replay buffer, and hence does not share experiences during training. Under this setting, the observation agent will not be able to benefit from earlier visitation of lucrative states by the state agent. Both agents have attention module and attention alignment still occurs.
3. APRiL background (APRiL back): APRiL except the state agent's attention is no longer used to calculate object-space attention values c. Instead, all objects are given equal attention and we hence learn background suppressor. This most competitive ablation in- vestigates how important object suppression is for learning, robustness, and generalisation. Both agents still maintain attention have shared replay buffer.
5

---

Under review as a conference paper at ICLR 2020
NevWarld
Walker20
baseline
APRIL
APRIL
share
APRIL no bask
Rilloo Steps
Steps
Steps
Figure 2: Learning curves during training of APRiL, its ablations, and the asymmetric DDPG base- line. Solid line: mean performance. Shaded region: covers minimum and maximum performances across 5 seeds.
4.3 PERFORMANCE ON THE TRAINING DISTRIBUTION
We evaluate the performance on all domains during training and observe APRiL 's benefits. As seen in Figure 2 APRiL provides performance gains across all continuous action domains. APRiL not only helps learn useful representations quicker (improving learning rate) but also improves final policy performance (within the allotted training time).
The ablations demonstrate that superviin and shared replay both independently provide per- formance gains for JacoReach and Walker2111 For Walker21), shared replay is crucial as stabilises learning (observe APRIL. APRIL back, APRiL sup), due to constant visitation for highly performing states. Suppression of task-irrelevant, yet highly varying, information also speed up learning as simplifies the observation space. For this reason, APRiL back proves to be a com- petitive ablation, approaching the performance of APRiL for JacoReach and Walker2D. For these domains, the background occupies the majority of the observation space and ignoring it already suppresses most of the irrelevant information. Minimal improvement can be achieved by suppress- ing additional irrelevant objects. None of the ablations, however, are able to outperform the full APRiL. framework, demonstrating that the combination of shared replay buffer and state-space- informed image-attention module cooperate constructively toward more efficient feature learning and effective policy and critic updates.
4.4 INTERPOLATION: TRANSFER TO DOMAINS FROM THE TRAINING DISTRIBUTION
We evaluate the performance of all actor-critic algorithms hold out of simulation parameters, unseen during training, from the training distribution. For a detailed description of the training dis- tribution for each domain please refer to Appendix B For both NavWorld and JacoReach, the inter- polated environments have the same number of distractors, sampled from the same object catalogue, as the training distribution. Table displays final policy performance on these domains. For APRiL, we observe no degradation in policy performance between training and interpolated domains. We see very similar trend for the asymmetric DDPG baseline. However, as APRiL. performs better on the training distribution, its final performance on the interpolated domains is significantly better. We therefore demonstrate that on these domains APRiL's attention mechanism does not hurt with respect to overlitting.
4.5 EXTRAPOLATION: TRANSFER TO DOMAINS OUTSIDE THE TRAINING DISTRIBUTION
We investigate performances simulation parameters outside the training distribution. In partic- ular, we investigate how well APRiL, its ablations, and asymmetric DDPG, generalise to environ- ments with more distractor objects than seen during training. For NavWorld and JacoReach, run two sets of increasingly extrapolated experiments with an additional 4 or 8 distractors (refered ext-4 and ext-8 in Table M. The textures and colours of these objects sampled from held-old out set of simulation parameters not seen during training. For Nav NavWorld, the locations and orientations of the additional distractors are randomly sampled. For JacoReach, the locations are
'We suspect that this is due to the simplicity of NavWorld, both visually and due to the small confined state-space, that of the ablation by themselves outperform the baseline.

---

Under review as a conference paper at ICLR 2020
sampled from ares of two concentric circles of different radii (extrapolated ares and radii to those seen during training), in such a way that each object remains visible. The shapes of the additional distractor object are sampled from the training catalogue of distractor objects. Please refer to Figure 3 for examples of the extrapolated domains.
Table compares performances the extrapolated sets (except Walker2D) varying in difficulty (ext-4 and ext-8). APRIL yields performance gains over the asymmetric DDPG baseline on every extrapolated domain. For JacoReach, APRiL's generalisation is effective that, for the hardest domain with additional 8 distractors, its performance degrades by only 9 opposed to 41 % (base- line).
APRiL generalises favorably due to the attention module. Figure 3 shows that attention generalises and suppresses the additional distractors, thereby effectively converting the hold-out observations to those seen during training, which the image-pon amage-policy can handle. The ablations in Table confirm that in this setting, distractor suppression is crucial. This is seen when comparing the maximum degra- dation in policy performance of APRiL, APRiL no share, APRiL no back and APRiL no sup (9%, 16%, 27% and 47% respectively). APRIL and APRIL no share both align attention between image and state agents during training, and therefore effectively suppress distractors (yielding favourable decrease in policy performance of only 9% and 16%). APRiL no back learns a background suppres- sor, but does not suppress the distractors (leading to larger degradation of 27%). APRiL sup has an attention module trained only on the asymmetric actor-critic loss and yields the worst extrapolated performance (47% policy degradation). For these extrapolated domains, the successful suppression of the background and additional distractors (achieved only by the full APRIL framework), creates policy invariance with respect to them and helps generalise.
Table I: Ablation comparing average return over training, interpolated and extrapolated environ- ments (100 each). Results reflect mean and standard deviation of average return over 5 seeds.
Domain
Baseline
APRiL. no sup
APRIL. no share
APRiL. no back
APRiL.
Nav (train)
0.72 = 0.02
0.60 0.02
0.61 0.02
0.67 1 0.02
0.75 + 0.02
WavWorld (inter)
0.70 = 0.03
0.60 0.60
0.62 0.02
0.69 + 0.02 0.80 0.02
NavWorld (cxt-4)
0.45 + 0.02
0.42
0.43 0.02
0.45 + 0.02
0.53 0.02
Nav (ext-8)
0.46 + 0.02
0.42
0.38 + 0.02
0.47 + 0.02
0.50 + 0.02
JacoReach (train)
-2.38 1 0.23
-3.03 0.30
-0.10
1.10 + 0.16
0.03 0.19
JacoReach (inter)
+
-3.38 + 0.31 0.31
0.81 + 0.18
0.13 0.20
JacoReach (cxt-4)
0.225
-6.81 + 0.33
11.26 0.22
-0.79 + 0.23
-0.70 - 0.23
JacoReach (ext-8)
-8.02 0.24
-9.26 = 0.28
-2.69 0.24
-3.53 = 0.26
-1.47 0.25
Walker2D (train)
3.15
107 3.31
63.0
126 4.33
155 + 3.25
Walker2D (inter)
68.9 100:33.. 3.25
107 3.30
62.6 02161
126 1 4.31
156 - 3.20
4.6 ATTENTION MODULE ANALYSIS
To better comprehend the role of the attention, we visualise APRiL's attention maps (Figure 5) on both interpolated and extrapolated domains. For NavWorld, attention is correctly paid to relevant aspects (agent and target; circle and triangle respectively). Attention generalises reasonably well to the extrapolated environments. For JacoReach, attention looks at the target, diamond -saped, object well every other link (alternating links) of the Kinova arm. Interestingly, APRiL learnt that as the arm is a constrained system, the state of every other link can be indirectly inferred without explicit attention. The state of the unobserved link can be inferred by observing the links either side of it. The entropy loss over the state-attention module encourages this form of attention over minimal of objects. Attention here generalises very well to the extrapolated domains. For Walker2D, we observe attention that is dynamic in object space. The attention module attends different subsets of links depending on the state of the system (see Figure 5).. When the walker is upright, walking, and collapsing, APRiL pays attention to the lower limbs, every other link, and foot and upper body, respectively. We suspect that these scenarios, the magnitude of the optimal action depends on the state of and as is largest for the lower links (due to stability), every link (coordination), and foot and upper body (large torque required), respectively.
Percenttgge decrease is taken with respect initial and final policy performance on training distribution

---

Under review as a conference paper at ICLR 2020
Ittepelatin
Eatrapolaion
Jicthaec
Auchhadi
Faind
Saass
Desd
Figure 3: Example held-out domains (top) and APRiL attention maps (bottom). White and black signify high and low attention values. Attention correctly suppresses background and distractors.
RELATED WORK
Domain Randomisation has been applied for reinforcement learning facilitate transfer between
domains (Tobin et al. 2017, Pinno et al., 2 & Levine 2016 Viereck et 2017 OpenA
2018 Held et al.) (017) and increase robustness of the learned policies (Rajeswaran et al.,
2016 However, while domain randomisation enables to generate more robust and transferable
policies, it leads a significant increase required training time (OpenAl et al. 2018
Existing comparisons in the literature demonstrate that, even without domain randomisation, the in- creased dimensionality and potential partial observability complicates learning for RL agents (Tasaa al. 2018 Schwab et al., 2019 Watter et al. 2015 Lesort et al.) 2018). In this context, accelerated training has been achieved by using access to privileged information such as environment states to
asymmetrically train the critic in actor-critic RL (Schwab al. 2019 Pinto et al. 2018). In addition to using additional information to train the critic, [Schwab et al. (2019) use shared replay buffer for data generated by image- and state-based actors further accelerate training for the image- based agent. Our method extends these approaches by sharing information about relevant objects by aligning agent-integrated attention mechanisms between an image- and state-based actors.
Recent experiments have demonstrated the strong dependency and bidirectional interaction between attention and learning in human subjects (Leong et al.) 2017). In the context of machine learning. attention mechanisms have been integrated into RL agents to increase robustness and enable in- terpretability of an agent's ' behaviour (Sorokin al., 2015 Choi et al. 2017 Mott al. 2019). In comparison to these works, we focus on utilising the attention mechanism as an interface to transfer information between two agents to enable faster training.
CONCLUSION
We introduce Attention Privileged Reinforcement Learning (APRiL). an extension to asymmetric actor-critic algorithms that leverages access to privileged information like exact simulator states. The method benefits two ways, via sharing replay buffer well as aligning attention masks between image- and state-space agents. By leveraging simulator ground-truth information about system states, we are able to learn efficiently in the image domain especially during domain ran- domisation where feature learning becomes increasingly difficult. Our evaluation a diverse set of environments demonstrates significant improvements over the competitive asymmetric DDPG baseline and reveals that APRiL learns to generalise favourably to environments not seen during training (both within and outside the training distribution) comparison to the strong baseline; emphasising the importance of attention and shared experience for robustness of the learnt policies.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large- scale machine learning. In OSDI, volume 16, pp. 265-283, 2016.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay, 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.064500 2016.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In Advances in Neural Information Processing Systems, pp. 343- 351, 2016.
Alexandre Campeau-LLcouus Hugo Lamontagne, Simon Latour, Philippe Fauteux, Véronique Ma- heu, François Boucher, Charles Deguire, and Louis-Joseph Caron L'Ecuyer. Kinova modular robot arms for service robotics applications. Int. J. Robot. Appl. Technol., 5(2):49-71, July 2017.
ISSN 2166-7195. doi: 10.401 11411///IRAAT221070000000 URL hettp:://cci.orgg///
IJRAT 201 7070104
Jinyoung Choi, Beom-Jin Lee, and Byoung- Tak Zhang. Multi-focus attention network for efficient deep reinforcement learning. In Workshops the Thirty-Fitt First AAAI Conference on Artificial Intelligence, 2017.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversario training of neural net- works. The Journal of Machine Learning Research, 17(1):2096-200 2016.
Oliver Groth, Fabian B. Fuchs, Ingmar Posner, and Andrea Vedaldi. Shapestacks: Learning vision- based physical intuition for generalised object stacking. In ECCV (I), volume 11205 of Lecture Notes in Computer Science, pp. 739. Springer, 2018
Haarnoja, S. Ha, Zhou, Tan, G. Tucker, and S. Levine. Learning Walk via Deep Rein- forcement Learning. arXiv e-prints, December 2018.
Tuomas Haarojja. Acquiring Diverse Robot Skills via Maximum Entropy Deep Reinforcement Learning. PhD thesis, UC Berkeley, 2018.
David Held, Zoe McCarthy, Michael Zhang, Fred Shentu, and Pieter Abbeel. Probabilistically safe policy transfer. In Robotics and Automation (ICRA), 2017 IEEE International Conference pp- IEEE, 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 2014.
Vijay Konda and John N Tsitsiklis. Actor-critic algorithms. In Advances in neural information processing systems, pp. 1008-1014, 2000.
Yuan Chang Leong, Angela Radulescu, Reka Daniel, Vivian DeWoskin, and Yael Niv. Dy- namic interaction between reinforcement learning and attention in multidimensional environ- ments. Neuron, 93(2):451 463, 2017. ISSN 0896-6273. doi: https://doi.org/10..00 20uron.2016.12.040 URL http: hwwp//wwwwk wttp://wwwwwciieeetccommm com/ wtt.s/ciece//rrttc....ooo p11/808962731651033
Timothée Lesort, Natalia Díaz-Rodrígued Jean-Franois Goudou, and David Filliat. State represen- tation learning for control: An overview. Neural Networks, 2018.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.

---

Under review as a conference paper at ICLR 2020
Alex Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo J. Rezende. Towards interpretable reinforcement learning using attention augmented agents. ArXiv, abs/ Abss/190.00250, 2019.
OpenAI, :, Marein Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray. Jonas Schneider, Szy- mon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. Learning dexterous in-hand manipulation, 2018.
Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. Asym- metric actor critic for image-based robot learning. Robotics: Science and Systems, 2018.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint arXiv: 70900, 2017.
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv:1610.0128; 2016.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing- solving sparse reward tasks from scratch, 2018.
A Romero, N Ballas, SE Kahou, A Chassang, C Gatta, and Y Bengio. Imagenet classification with deep convolutional neural networks. In International Conference on Learning Representations, 2015.
Andrei Rusu, Mel Vecerik, Thomas Rothörl, Nicolas Heess, Razvan Pascanu, and Raia Hadsell. Sim-to-real robot learning from pixels with progressive nets. arXiv preprint arXiv:1610. 04286, 2016.
Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without a single real image. arXiv preprint 1rXhv:1611.04201, 2016.
Devin Schwab, Tobias Springenberg, Murilo F Martins, Thomas Lampe, Michael Neunert, Abbas Abdolmaleki, Tim Herkweck, Roland Hafner, Francesco Nori, and Martin Riedmiller. Simul- taneously learning vision and feature-based control policies for real-world ball-in-a-cup. arXiv preprint
04706, 2019.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In ICMI., 2014.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, :30(767))3544, 2017.
Ivan Sorokin, Alexey Seleznev, Mikhail Pavlov, Aleksandr Fedorov, and Anastasiia Ignateva. Deep attention recurrent nettwork. arXiv preprint arXiv: 2015.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint
2018
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ International Conference on, pp. 23-30. IEEE, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 JEEE/RSJ International Conference on, pp. 5026- 5033. IEFF, 2012.
10

---

Under review as a conference paper at ICLR 2020
Ulrich Viereck, Andreas ten Pas, Kate Saenko, and Robert Platt. Learning visuomotor controller for real world robotic grasping using simulated depth images. arXiv preprint arXiv:1706.04652. 2017.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in neural information processing systems, pp. 2746-2754, 2015.
Markus Wulfmeier, Ingmar Posner, and Pieter Abbeel. Mutual alignment transfer learning. arXiv preprint arXiv:1707.079907 2017.

---

Under review as a conference paper at ICLR 2020
Appendix
A ENVIRONMENTS
1. NavWorld: In this sparse reward, 2D environment, the goal is for the circular agent to reach the triangular target in the presence of distractor objects. Distractor objects have 4 or more sides and apart from changing the visual appearance of the environment cannot affect the agent. The state space consists of the [r. v] locations of all objects. The observation space comprises RGB images of dimension (60 x 60 x 3). The action space corresponds to the velocity of the agent. The agent only obtains a sparse reward of +1 if the particle is within of the target, after which the episode is terminated prematurely. The maximum episodic length is 20 steps, and all object locations are randomised between episodes.
2. JacoReach: In this 3D environment the goal of the agent is move the Kinova arm (Campeau-Lecours et al.] (017)) such that the distance between its hand and the di- amond ShapeStacks object (Groth el al. 2018) is minimised. The state space consists of the quaternion position and velocity of each joint as well as the Cartesian positions of each ShapeStacks object. The observation space comprises RGB images and is of dimension (100 x 100 x 3). The action space consists of the desired relative quaternion positions of each joint (excluding the digits) with respect to their current positions. Mujoco uses a PD controller to execute 20 steps that minimises the error between each joint's actual and target positions. The agent's reward is the negative squared Euclidean distance between the Kinova hand and diamond object plus additional discrete reward of +55 if it is within of the target. The episode is terminated early the target is reached. All objects are out of reach of the arm and equally far from its base. Between episodes the locations of the objects are randomised along an are of fixed radius with respect to the base of the Kinova arm. The maximum episodic length is 20 agent steps.
3. Walker2D: In this 2D modified Deepmind Control Suite environment Tassa et al., 2018) with continuous action-space the goal of the agent is to walk forward as far as possible within 300 steps. We introduce limit to episodic length as we found that in practice this helped stabilise learning across all tested algorithms. The observation space comprises of stacked RGB images and is of dimension (40 x 40 x 6). Images are stacked that velocity of the walker can be inferred. The state space consists of quaternion position and velocities of all joints. The absolute positions of the walker along the x-axis is omitted such that the walker learns to become invariant to this. The action space is setup in the same way as for the JacoReach environment. The reward is the same defined in (Tassa et al.] 2018) and consists of two multiplicative terms: one encouraging moving forward beyond a given speed, the other encouraging the torso of the walker remain as upright as possible. The episode is terminated early if the walker's 's torso falls beyond either [-1, 1] radians with the vertex or [0.8, 2.0Jm along the axis.
B RANDOMISATION PROCEDURE
In this section we outline the randomisation procedure taken for each environment during training.
1. NavWorld: Randomisation occurs at the start of every episode. We randomise the location, orientation and colour of every object as well the colour of the background. We therefore hope that our agent can become invariant to these aspects of the environment.
2. JacoReach: Randomisation occurs the start of every episode. We randomise the textures and materials of every ShapeStacks object, Kinova arm and background. We randomise the locations of each object along an are of fixed radius with respect to the base of the Kinova arm. Materials vary in reflectance, specularity, shininess and repeated textures. Textures vary between the following: noisy (where RGB noise of a given colour is superimposed on top of another base colour), gradient (where the colour varies linearly between two predefined colours), uniform (only one colour). Camera location and orientation are also randomised. The camera is randomised along spherical sector of sphere of varying radius whilst always facing the Kinova arm. We hope that our agent can become invariant to these randomised aspects of the environment.
12

---

Under review as a conference paper at ICLR 2020
3. Walker2D: Randomisation occurs the start of every episode well after every 50 agent steps. We introduce additional randomisation between episodes due to their increased duration. Due to the MDP setup, intra-episodic randomisation is not issue. Materials, textures, camera location and orientation, are randomised in the same procedure as for JacoReach. The camera is setup always face the upper torso of the walker.
C IMPLEMENTATION DETAILS
Table 2: Model architecture. FCQ represents a (multi-layered) fully connected network with the number of nodes per layer stated argument. Conv( represents (multi-layered) convolutional network whose arguments take the form [channels, square kernel size, stride] for each hidden layer.
Domain
Nav World and JacoReach
Walker2D
State Actor
FC(256]) Actor Conv([[18,
FC(256))
[32, 1], [32,3, 1J)
Conv( [18, 2), [32, 5, 1], [16, [4.3,
State Critic
FC 64|)
1000 300]
Ohs Critic
FC 64)
FC( [000, 300)
State Attention
FC( (266)
FC((56)
Obs Attention
Conv( 8, 1), 32,55, 1), 64. 1))
Conv( 32, 8, 1]. [32, 5, 1), 361,
Replay Buffer Size
10
2 x 10"
In this section we provide more details on our training setup. Refer table for the model ar- chitecture for each component of APRiL and the asymmetric DDPG baseline. Obs Actor and Obs Critic setup are the same for both APRIL and the baseline. Obs Actor model structure comprises of the convolutional layers (without padding) defined in ableefollloeer by one fully connected layer with 256 hidden units (FC((266)) All layers use ReLU (Romero et al., 2015) activations and layer normalisation (Ba et al., 2016) unless otherwise stated. Each actor network is followed by tanh activation and rescaled to match the limits of the environment's 's action space.
The State Attention module includes the fully connected layer defined in table followed by Softmax operation. The Obs Attention module has the convolutional layers (with padding to en- sure constant dimensionality) outlined in table 2 followed by a fully connected convolutional layer (Conv([1, 1, I)) with a Sigmoid activation to ensure the outputs vary between 0 and 1. The output of this module is tiled in order match the dimensionality of the observation space.
During each iteration of APRiL (for both A, and A₂) we perform 50 optimization steps on mini- batches size 64 from the shared replay buffer. The target actor and critic networks are updated every iteration with a Polyak averaging of 0.999. We use Adam (Kingma & Ba 2014) optimization with a learning rate of 10-3 10 and 10 4 for critic, actor and attention networks respectively. We use default TensorFlow (Abadi et al.) 2016) values for the other hyperparameters. The discount factor, entropy weighting and self-supervised learning hyperparameters are y = 0.99, B = 0.0008 and y = 1 respectively. To stabilize learning, all input states are normalized by running averages of the means and standard deviations of encountered states.
Both actors employ adaptive parameter noise (Plappert et al. 2017 exploration strategy with initial std 0.1, desired action std of 0.1 and adoption coefficient of 1.01. The settings for the baseline kept the same as for APRiL where appropriate.
13

---

Under review as a conference paper at ICLR 2020
D ATTENTION VISUALISATION
NavWorld
Inter
Inter
Extra
Extra
Jaco Reach
Inter
Inter
Extra
Extra
Figure 4: APRIL attention maps for policy rollouts NavWorld and Jaco domains. White and black signify high and low attention values respectively. For Nav World and JacoReach, attention correctly paid only the relevant objects (and Jaco links), even for the extrapolated domains. Refer to section for more details.
14

---

Under review as a conference paper at ICLR 2020
Walker2D
Inter
Inter
Inter
Inter
rollout
Figure 5: APRiL attention maps for policy rollouts Walker domain. White and black signify high and low attention values respectively. Attention varies based on the state of the walker. When the walker is upright, high attention is paid to lower limbs. When walking, even attention is paid to every other limb. When about to collapse, high attention is paid to the foot and upper torso. Refer section for more details.
15


Reviews:
[{"review_id": "ryew8BLsFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The topic addressed by the paper is domain adaptation and transfer learning in the text context of deep reinforcement learning, in particular the “sim2real” problem, where a policy is learned in simulation and should be transferred to a physical agent in a real-world scenario. The work builds on the existing “asymmetric DDPG” formulation (Pinto et al., 2017), which exploits the fact that full states are sometimes available in simulated environments but not during deployment. In Pinto et al., this is addressed by learning an actor taking as input observations, and a critic which has access to the state.\n\nThe contribution of the paper is an extension of Pinto et al. by adding additional communication between the state head (which learns a critic) and the observation head (which learns the policy). This is done through an attention mechanism, now very classical in deep learning, which weights the different elements of the (state) input. The particularity here is that the attention mechanism is trained on the critic, which takes the full state as input, and then transferred to the observed input of the policy. The problem is that the state is very different from the observations, which are images. An alignment module expands the distribution over state variables to a distribution over observed objects.\n\nOne of my main concerns here is lack of justification and lack of clarity. While the abstract of the paper and introduction section are well written, and wants us to learn more about this interesting idea, the paper kind of falls apart in the subsequent chapters. The authors mention alignment of attention, but the exact motivation for the algorithm, i.e. the motivation for its formulation, are never provided. The description itself is also far from clear, as key design choices are not introduced and we discover them, or guess them, from equations. We don’t know, for instance, what the observations are, and think that they are images. However, apparently an object mask detector is run over the input images, as segmentation maps are input to the state-object alignment module, which needs to distribute the attention over states to attention over objects.\n\nThis also means that the formulation is quite specific to the task at hand, since it seems to be object-centric.\n\nFigure 1 is another example of the same problem: There are many arrows, but their signification is unclear. Dashed lines are indicated to mean alignment, but alignment is not a standard term, for instance comparable to a computation connection, or a loss signal.\n\nA similar confusion is found in the equations themselves, as h(s) is mapped to a vector c, but this vector decomposes into different objects of the environment. What is T? It being upper case could mean a matrix or a scalar value, but we are not sure. From the loss function we see that this seems to be vector (a distribution) of the same size as the attention vector h_o … which is over what?\n\nBasically, as I understand it, the alignment loss minimizes some structured mapping between some attention vector over objects on the observation side and some attention vector over states on the critics side. This seems to be a weak loss signal, as the attention transformation needs to be learned together with the attention mechanism itself (and of course the actor and the critic).\n\nAt the beginning of section 3, the method is introduced of having an asymmetric part and a symmetric part, but I don’t see this, as this would require learning 4 predictors and not 2 (an actor and a critic). An asymmetric model, as given in Pinto et al., uses two predictors, an actor and a critic, with different loss functions and, more importantly, different inputs. Here, the authors claim that the observation module itself is asymmetric … but how can something by asymmetric if it contains only an actor? Same, the critic is supposed to be symmetric … but without an actor?\n\nAs I see it, w.r.t to the question whether the model is symmetric or asymmetric, the formulation is identical to Pinto et al., thus asymmetric. The difference lies in the attention mechanism and the alignment module.\n\nAt some point in the paper, this method is mentioned to be self-supervised … I am not sure which aspect of this work could be called self-supervised, but I agree that the definition of this relatively new term is sometimes ambiguous.\n\nOn the other hand, I think the method should be compared to “classical” self-supervision in RL, which corresponds to predicting information which is available during training but not available as input (depth prediction etc.). Here, a natural baseline seems to be a symmetric model (same input for actor and critic: observations) and to predict the state of the model and self-supervise it during training but not deployment.\n\nThe evaluation is unfortunately not convincing.\n\nTwo important baselines are missing:\n-\tSelf-supervision, as mentioned above\n-\tPinto et al., on which this method is based.\n\nThere are some hics in the results … for instance the curves in Figure 2 show learning in progress, they are not yet converged. Also, the standard deviations are quite big.\n\nAnother downside is that, although sim2real is used as a motivation for this paper (since this is the most typical scenario where states are available during training but not during testing), the experiments have not been performed using physical agents. Testing is performed on degraded simulated agents. I do understand that physical environments are harder to manage than simulated ones, but tiny physical environments can be obtained for reasonable prices, and this would have made the paper much stronger and more aligned with its core motivation. Simulated noise cannot replace a physical environment.\n\nNo details have been given on the additional distractor objects unseen during training.\n\nThe Walker 2D environment is described as “modified”, but how exactly, and why?\n\nSection 2.1 (definition of POMDPs) seems to be unnecessary and can be deleted.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "rJxee2j0tB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary - Building on top of the domain randomization principle (used to train policies robust to domain-variations) to learn policies which transfer well to new domains, the paper proposes an approach to improve and speed-up learning / training over randomized environments. The paper operates in a settings where the policy to be transferred only has access to observations -- images, etc -- and not the complete underlying state of a (simulated) environment. The underlying idea is to -- (1) maintain two sets of actor-critic networks - a symmetric pair where the actor has access to the underlying state and an asymmetric pair where the actor has access only to image observations; (2) evenly gather experiences from behavioral policies of both actors and store them in a shared replay buffer and (3) learn to align the attention placed by the policies over objects in the environment for the state and observation based actors. The idea is to leverage privileged information about the state (which is strictly more informative compared to observations) to learn robust observation based policies. Experimental results indicate the proposed approach improves generalization performance compared to several ablations of the same on both in-distribution and out-of-distribution environments.\n\nStrengths\n\n- The paper is generally well-written and easy to follow. The authors generally do a good job of motivating the proposed approach by leveraging access to privileged information in the environment / simulation / renderer during training to get more robust observation policies to transfer to novel settings. The proposed approach is presented after appropriately grounding the problem setting and preliminaries and the authors clearly state and evaluate on the axes of research questions they care about.\n\n- The proposed approach is somewhat novel and extends prior work on using shared replay buffers and asymmetric actor critic methods to accelerate training. Although the specific focus is on aligning object-level attention from a state-based actor and an observation based actor, the authors adopt design choices that help in preventing degenerate solutions -- for instance, the object-weighted squared error loss component to learn the observation attention module.\n\n- The experimental results more-or-less support the claims of the paper (however, only in comparison with ablations and 1 baseline) in the sense we see improvements in terms of average returns and sample-efficiency plots. Furthermore, the authors conduct ablations to understand which components of the proposed approach contribute significantly in different environments -- for both extrapolated and interpolated environments.\n\n- Sec 4.6 presents interesting analysis of the learned attention (observation) attention mechanism for both interpolated and extrapolated environments. It seems that attention is generally placed over relevant aspects (objects / links) of the environment and the associated takeaways seem feasible.\n\nWeaknesses\n\n- Having said that, there some weaknesses / questions which if addressed would make the paper stronger and help in increasing the rating of the paper.\n\n- Access to the object-specific attention maps seems to be an assumption that might not scale well across simulators. More realistic / richer simulations (say, 3D reconstructions of indoor rooms) may not always offer this much privileged information -- one might have access to fine-grained reconstructions from multiple viewpoints, but not object level maps. This, combined with the fact that major gains have only been demonstrated over the specified continuous control domains (ignoring the Atari results), makes me slightly concerned about the scalability of the proposed approach in terms of more real-world + applicable domain-transfer scenarios. Maybe a more general approach that learns to match some intermediate representations of the state and observation based actors is a more general approach. Can the authors comment on this?\n\n- Including the entropy loss while training the attention mechanism for the state based actor is justified only through feasible interpretations of the attention visualizations for the JacoReach experiments. I’m curious how important is it for the state-based attention to be sparse? Does it actually affect performance if the entropy loss is not included while learning the state-based attention module?\n\n- Although the paper mentions experience is gathered evenly over the behavioral policies of both the actors it’s slightly unclear how that is being performed without actually referring to the algorithm pseudocode in the appendix. I would encourage the authors to include / move the same to the main paper. It makes the entire pipeline much easier to grasp.\n\n- Transfer to novel environments (not necessarily with domain-shifts) has also been studied in context of providing exploration incentives (see InfoBot - https://arxiv.org/abs/1901.10902) in addition to a (sparse / dense) episodic reward. I would be curious to see how well does APRiL compare to such approaches in a setting where both are applicable - say the MultiRoomNXSY set of experiments in InfoBot (pointed above). This is to understand if the gains obtained from APRiL are very specific to domain-shifts or are largely applicable to any form of novel environment transfer. Can the authors comment on this?\n\nReasons for rating\n\nBeyond the above points of discussion, I don’t have major weaknesses to point \tout. I generally like the paper. The authors do a good job of identifying the sliver in which they make their contribution and motivate the same appropriately. My major point of concern is centered is around the fact that APRiL (probably) assumes access to much privileged information which may not be generally available across all kinds of environments, etc. My rating of the paper is based on the strengths and weaknesses highlighted above. Addressing / Responding to those appropriately would definitely help in improving the rating of the paper.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1gwk6-RFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall the method is interesting but I am not overly convinced the method provides a significant improvement over simpler methods that are not compared to the work here. Also, the generalization and extrapolation experiments need more description. As well, the work talks about how this method can be used to accelerate learning for transfer to real-robotic systems but does not have an example of this.\n\nMore detailed comments:\n- When you are referencing a paper that is very similar to your method and you build off of it is good to link to the published version ( the reference for Asymmetric DDPG was published at RSS2018.)\n- While the method is very interesting I feel that one of the obvious baselines that would be good to try is to train a model that maps o -> s (image to state). This is a very simple method and appears to accomplish the goals for the authors. A discussion on why this would not be good enough and some results to show that this performs poor would be good addition.\n- Related to the last point why are object-specific segmentation maps needed? It seems like the method alone was not capable enough to learn from pixels alone. Also, Can these segmentation maps be visually described? It is not very clear. Where do the segmentation maps come from?\n- Can you describe the shared replay buffer in more detail? What does no shared replay buffer mean?\n- The caption in Figure 2 is rather sparse and comes before the explication of the different methods in the figure. This figure needs more explanation. What are the other versions of April? What is the baseline? There does not appear to be a significant improvement? Can you show more of a qualitative improvement via videos of the policy performance?\n- It appears the method does not work as well on the Atari games can the authors provide some insights into why the method does not offer as much of an improvement? Is there no Attentive DQN for Pong, natural?\n- I do not understand what experiment is being performed in section 4.3.2. Maybe it would help if you explained what a canonical image is? Where do you get the compressed state information?\n- In 4.4 you show that APRiL does not do better than the baseline? What is the baseline?\n- In section 4.5 you describe how the method generalizes to the task with additional distractors. Can you describe specifically how this experiment is designed? How do the distractors compare to the other objects in the scene? are their colors changed? How many are there? This section needs a lot more detail to understand how much the reader can surmise about the methods ability to generalize.\n- In the same section, the authors say the image segmentation is crucial. Does this imply that maybe we should just put an image segmentation network in between the observation and the policy inputs? What is using this additional attention mechanism better than that simpler method?\n- For Table 1: It is interesting that the baseline does better than the APRiL no share or no sup. It might be important to perform an exploration as to why having those features reduces performance and the baseline appears more robust.\n- It would be nice to see this work applied to more tasks. For now the reacher and walking are the most interesting but those tasks are also somewhat basic. How would this method work on the walker 3d where there is even more partial observation?\n- Can the authors explain more how they have deduced from the attention map that the attention has figured out how to indirectly discover where the Jaco links are without explicit attention? I do not see this.\n- For figure 4, how the attention is visualized could be explained better. Are the white regions receiving all the attention? Also, What makes the difference between a Interpolation example and a Extrapolation example? Is it just more objects in the scene?\n- The authors should also reference \"Sim-to-Real Transfer of Robotic Control with Dynamics Randomization\".", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles the problem of transferring an RL policy learned in simulation to the real world (sim2real). More specifically, the authors address the situation where the agent can access privileged information available during simulation, for example access to exact states instead of compressed representations. They perform experiments in various simulated domains where different aspects of the environment are modified to evaluate generalization.

Major concerns remain following the rebuttal. First, it is not clear how realistic it is to assume access to such privileged information in practice. Second, the experiments are not convincing since the algorithms do not appear to have reached convergence in the presented results. Finally, a sim2real work would highly benefit from real-world experiments.

In light of the above issues, I recommend to reject this paper.
Decision: Reject