Title: ATTENTION PRIVILEGED REINFORCEMENT LEARNING FOR DOMAIN TRANSFER

Abstract: Applying reinforcement learning (RL) to physical systems presents notable challenges, given requirements regarding sample efficiency, safety, and physical constraints compared to simulated environments. To enable transfer of policies trained in simulation, randomising simulation parameters leads to more robust policies, but also significantly extends training time. In this paper, we exploit access to privileged information (such as environment states) often available in simulation, in order to improve and accelerate learning over randomised environments. We introduce Attention Privileged Reinforcement Learning (APRiL), which equips the agent with an attention mechanism and makes use of state information in simulation, learning to align attention between state-and image-based policies while additionally sharing generated data. During deployment we can apply the image-based policy to remove the requirement of access to additional information. We experimentally demonstrate accelerated and more robust learning on a number of diverse domains, leading to improved final performance for environments both within and outside the training distribution. 1 .

1. INTRODUCTION:

Deep Reinforcement Learning (RL) has recently provided significant successes in a range of areas, including video games (Mnih et al., 2015) , board games (Silver et al., 2017) , simulated continuous control tasks (Lillicrap et al., 2015) , and robotic manipulation (Haarnoja et al., 2018; Haarnoja, 2018; Riedmiller et al., 2018; OpenAI et al., 2018; Schwab et al., 2019; Andrychowicz et al., 2017) . However, application to physical systems has proven to be challenging in general, due to expensive and slow data generation as well as safety challenges when running untrained policies. A common approach to circumvent these issues is to transfer models trained in simulation to the real world (Tobin et al., 2017; Rusu et al., 2016; Held et al., 2017) . However, simulators only represent approximations of a physical system. Due to physical, visual, and behavioural discrepancies, naively transferring RL agents trained in simulation onto the real world can be challenging.

To bridge the gap between simulation and the real world, we can either aim to align both domains (Ganin et al., 2016; Bousmalis et al., 2016; Wulfmeier et al., 2017) or ensure that the real system is covered by the distribution of simulated training data (OpenAI et al., 2018; Tobin et al., 2017; Pinto et al., 2018; Sadeghi & Levine, 2016; Viereck et al., 2017) . However, training under a distribution of randomised visual attributes of the simulator, such as textures and lighting (Sadeghi & Levine, 2016; Viereck et al., 2017) , as well as physics (OpenAI et al., 2018) , can be substantially more difficult and slower due to the increased variability of the learning domain (OpenAI et al., 2018; Tobin et al., 2017) .

The more structured and informative the input representation is with respect to the task, the quicker the agent can be trained. A clear example of this effect can be found when an agent is trained with image inputs, versus training with access to the exact simulator states (Tassa et al., 2018; Pinto et al., 2018) . However, visual perception is more general and access to more compressed representations can often be limited. When exact states are available during training but not deployment, we can make use of information asymmetric actor-critic methods (Pinto et al., 2018; Schwab et al., 2019) to train the critic faster via access to the state while providing only images for the actor.

By introducing Attention Privileged Reinforcement Learning (APRiL), we aim to further leverage access to exact states. APRiL leverages states not only to train the critic, but indirectly also for an image-based actor. Extending asymmetric actor-critic methods, APRiL concurrently trains two actor-critic systems (one symmetric, state-based agent, and another asymmetric agent with imagedependent actor). Both actors utilise an attention mechanism to filter input data and by having access to the simulation rendering system, we can optimise image and state based attention masks to align.

By additionally sharing the replay buffer between both agents, we can accelerate the learning process of the image-based actor by training on better performing states that are more quickly discovered by the state-based actor due to its lower dimensional input that is invariant to visual randomisation.

The key benefits of APRiL lie in its application to domain transfer. When training with domain randomisation for transfer, bootstrapping via asymmetric information has displayed crucial benefits (Pinto et al., 2018) . Visual randomisation substantially increases the complexity of the imagebased actor's task. Under this setting, the attention network can support invariance with respect to the irrelevant, but highly varying, parts of the image. Furthermore, the convergence of the statespace actor remains unaffected by visual randomisation.

We experimentally demonstrate considerable improvements regarding learning convergence and more robust transfer on a set of continuous action domains including: 2D navigation, 2D locomotion and 3D robotic manipulation.

2. PROBLEM SETUP:

Before introducing Attention Privileged Reinforcement Learning (APRiL), this section provides a background for the RL algorithms used. For a more in-depth introduction please refer to Lillicrap et al. (2015) and Pinto et al. (2018) .

2.1. REINFORCEMENT LEARNING:

We describe an agent's environment as a Partially Observable Markov Decision Process which is represented as the tuple (S, O, A, P, r, γ, s 0 ), where S denotes a set of continuous states, A denotes a set of either discrete or continuous actions, P : S × A × S → {x ∈ R|0 ≤ x ≤ 1} is the transition probability function, r : S × A → R is the reward function, γ is the discount factor, and s 0 is the initial state distribution. O is a set of continuous observations corresponding to continuous states in S. At every time-step t, the agent takes action a t = π(•|s t ) according to its policy π : S → A. The policy is optimised as to maximize the expected return

R t = E s0 [ ∞ i=t γ i-t r i |s 0 ]. The agent's Q-function is defined as Q π (s t , a t ) = E[R t |s t , a t ].

2.2. ASYMMETRIC DEEP DETERMINISTIC POLICY GRADIENTS:

Asymmetric Deep Deterministic Policy Gradients (asymmetric DDPG) (Pinto et al., 2018) represents a type of actor-critic algorithm designed specifically for efficient learning of a deterministic, observation-based policy in simulation for sim-to-real transfer. This is achieved by leveraging access to more compressed, informative environment states, available in simulation, to speed up and stabilise training of the critic.

The algorithm maintains two neural networks: an observation-based actor or policy π θ : O → A (with parameters θ) used during training and test time, and a state-based Q-function (also known as critic) Q π φ : S × A → R (with parameters φ) which is only used during training. To enable exploration, the method (like its symmetric version (Silver et al., 2014) ) relies on a noisy version of the policy (called behavioural policy), e.g. π b (o) = π(o) + z where z ∼ N (0, 1) (see Appendix C for our particular instantiation). The transition tuples (s t , o t , a t , r t , s t+1 , o t+1 ) encountered during training are stored in a replay buffer (Mnih et al., 2015) . Training examples sampled from the replay buffer are used to optimize the critic and actor. By minimizing the Bellman error loss L critic = (Q(s t , a t ) -y t ) 2 , where y t = r t + γQ(s t+1 , π(o t+1 )), the critic is optimized to approximate the true Q values. The actor is optimized by minimizing the loss 

L actor = -E s,o∼π b (o) [Q(s, π(o))].

3. ATTENTION PRIVILEGED REINFORCEMENT LEARNING (APRIL):

APRiL proposes to improve the performance and sample efficiency of an observation-based agent by using a quicker learning actor that has access to exact environment states, sharing replay buffers, and aligning attention mechanisms between both actors. While we focus in the following sections on extending asymmetric DDPG (Pinto et al., 2018) , these ideas are generally applicable to off-policy actor-critic methods (Konda & Tsitsiklis, 2000) .

APRiL is comprised of three modules as displayed in Figure 1 . The first two modules, A s and A o , are actor-critic algorithms with an attention network incorporated over the input to each actor. For the state-based module A s we use standard symmetric DDPG, while the observation-based module A o builds on asymmetric DDPG. Finally, the third part A T represents the alignment process between attention mechanisms of both actor-critic agents to more effectively transfer knowledge between the quicker and slower learners, A s and A o , respectively.

A s consists of three networks: Q π s , π s , h s (respectively critic, actor, and attention) with parameters {φ s , θ s , ψ s }. Given input state s t , the attention network outputs a soft gating mask h t of same dimensionality as the input, with values ranging between [0, 1]. The input to the actor is an attentionfiltered version of the state, s a t = h s (s t ) s t . To encourage a sparse masking function, we found that training this attention module on both the traditional DDPG loss as well as an entropy loss helped:

EQUATION

where β is a hyperparameter to weight the additional entropy objective, and π b is the behaviour policy used to obtain experience (in this case from a shared replay buffer). The actor and critic networks π s and Q s are trained with the symmetric DDPG actor and Bellman error losses respectively.

Within A T , the state-attention obtained in A s is converted to corresponding observation-attention T to act as a self-supervised target for the observation-based agent in A o . This is achieved in a twostep process. First, state-attention h s (s) is converted into object-attention c, which specifies how task-relevant each object in the scene is. Second, object-attention is converted to observation-space attention by performing a weighted sum over object-specific segmentation maps:

c = M • h s (s) (2) T = N -1 i=0 c i • z i (3)

Here, M ∈ {0, 1} N ×ns (where n s is the dimensionality of s) is an environment-specific, predefined adjacency matrix that maps the dimensions of s to each corresponding object, and c ∈ [0, 1] N is then an attention vector over the N objects in the environment. c i corresponds to the i th object attention value. z i ∈ {0, 1} W ×H is the binary segmentation map2 of the i th object segmenting the object with the rest of the scene, and has the same dimensions as the image observation. z i assigns values of 1 for pixels in the image occupied by the i th object, and 0 elsewhere. T ∈ [0, 1] W ×H is the converted state-attention to observation-space attention to act as a target to train the observationattention network h o on.

The observation-based module A o also consists of three networks: Q π o , π o , h o (respectively critic, actor, and attention) with parameters {φ o , θ o , ψ o }. The structure of this module is the same as A s except the actor and critic now have asymmetric inputs. The input to the actor is the attentionfiltered version of the observation,

o a t = h o (o t ) o t 3 .

The actor and critic networks π o and Q o are trained with the standard asymmetric DDPG actor and Bellman error losses respectively defined in Section 2.2. The main difference between A o and A s is that the observation attention network h o is trained on both the actor loss and an object-weighted mean squared error loss:

EQUATION

where weights w ij correspond to the fraction of the partial observation o that the object present in o i,j,1:3 occupies, and ν represents the relative weighting of both loss components. The weight terms, w, ensure that the attention network becomes invariant to the size of objects during training and does not simply fit to the most predominant object in the scene. Combining the self-supervised attention loss and the RL loss leverages efficient state-space learning unaffected by visual randomisation.

During training, experiences are collected evenly from both state and observation based agents and stored in a shared replay buffer (similar to Schwab et al. (2019) ). This is to ensure that: 1. Both statebased critic Q s and observation-based critic Q o observe states that would be visited by either of their respective policies. 2. The attention modules h s and h o are trained on the same data distribution to better facilitate alignment. 3. Efficient discovery of highly performing states from π s are used to speed up learning of π o .

Algorithm 1 shows pseudocode for a single actor implementation of APRiL. In practice, in order to speed up data collection and gradient computation, we parallelise the agents and environments and ensure data collection from state-and image-based agents is even.

4. EXPERIMENTS:

To demonstrate the performance and generality of our method, we apply APRiL to a range of environments, and compare with a competitive asymmetric DDPG baseline and various ablations. We evaluate APRiL over different metrics to investigate how attention helps with robustness and generalisation to unseen environments and transfer scenarios. Further experimental details can be found in Appendix C.

4.1. EVALUATION PROTOCOL:

In order to investigate APRiL under varying conditions, we evaluate in scenarios of increasing complexity covering simple 2D navigation, 3D reaching and 2D dynamic locomotion.

We use the following continuous action-space environments (see Appendix A for further details):

1. NavWorld: In this 2D environment, the goal is for the circular agent to reach the triangular target in the presence of distractors. The agent is sparsely rewarded if the target is reached.

2. JacoReach: In this 3D environment the goal of the Kinova arm (Campeau-Lecours et al., 2017) agent is to reach the diamond ShapeStacks object (Groth et al., 2018) in the presence of distractors. The agent is rewarded for approaching and reaching its goal. 3. Walker2D: In this slightly modified 2D Deepmind Control Suite environment (Tassa et al., 2018) the goal of the agent is to walk forward as far as possible within a time-limit. The agent receives a reward for moving forward as well as a reward for keeping its torso upright.

For these domains we randomise visuals during training as to enable generalisation to these variable aspects of the environment. We randomise a combination of: camera position and orientation, textures, materials, colours, object locations, background. Refer to Appendix B for more details.

4.2. KEY RESEARCH QUESTIONS:

We investigate the following questions to evaluate how well APRiL accommodates for the transferring of policies across visually distinct environments: Does APRiL 1. Increase sample-efficiency during training? 2. Affect interpolation performance on unseen environments from the training distribution? 3. Affect extrapolation performance on environments outside the training distribution?

We qualitatively analyse the learnt attention maps (both on interpolated and extrapolated domains). Finally, we perform an ablation study to investigate which parts of the APRiL contribute to performance gains. This ablation consists of the following models:

1. APRiL no self-supervision (APRiL no sup): APRiL except without the self-supervision provided by the state agent to train the observation-based attention. Both agents are still equipped with an attention module, but the observation attention must now learn without guidance from the state agent. Without bootstrapping from the state agent in this way we expect learning of informative observation-based attention to be hindered. 2. APRiL no shared buffer (APRiL no share): APRiL except each agent has its own replay buffer, instead of one shared replay buffer, and hence does not share experiences during training. Under this setting, the observation agent will not be able to benefit from earlier visitation of lucrative states by the state agent. Both agents have an attention module and attention alignment still occurs. 3. APRiL no background (APRiL no back): APRiL except the state agent's attention is no longer used to calculate object-space attention values c. Instead, all objects are given equal attention and we hence learn a background suppressor. This most competitive ablation investigates how important object suppression is for learning, robustness, and generalisation. Both agents still maintain attention have a shared replay buffer. 

4.3. PERFORMANCE ON THE TRAINING DISTRIBUTION:

We evaluate the performance on all domains during training and observe APRiL 's benefits. As seen in Figure 2 , APRiL provides performance gains across all continuous action domains. APRiL not only helps learn useful representations quicker (improving learning rate) but also improves final policy performance (within the allotted training time).

The ablations demonstrate that self-supervision and shared replay both independently provide performance gains for JacoReach and Walker2D4 . For Walker2D, shared replay is crucial as stabilises learning (observe APRiL , APRiL no back, APRiL no sup), due to constant visitation for highly performing states. Suppression of task-irrelevant, yet highly varying, information also speed up learning as simplifies the observation space. For this reason, APRiL no back proves to be a competitive ablation, approaching the performance of APRiL for JacoReach and Walker2D. For these domains, the background occupies the majority of the observation space and ignoring it already suppresses most of the irrelevant information. Minimal improvement can be achieved by suppressing additional irrelevant objects. None of the ablations, however, are able to outperform the full APRiL framework, demonstrating that the combination of a shared replay buffer and state-spaceinformed image-attention module cooperate constructively toward more efficient feature learning and effective policy and critic updates.

4.4. INTERPOLATION: TRANSFER TO DOMAINS FROM THE TRAINING DISTRIBUTION:

We evaluate the performance of all actor-critic algorithms on a hold out set of simulation parameters, unseen during training, from the training distribution. For a detailed description of the training distribution for each domain please refer to Appendix B. For both NavWorld and JacoReach, the interpolated environments have the same number of distractors, sampled from the same object catalogue, as the training distribution. Table 1 displays final policy performance on these domains. For APRiL , we observe no degradation in policy performance between training and interpolated domains. We see a very similar trend for the asymmetric DDPG baseline. However, as APRiL performs better on the training distribution, its final performance on the interpolated domains is significantly better.

We therefore demonstrate that on these domains APRiL's attention mechanism does not hurt with respect to overfitting.

4.5. EXTRAPOLATION: TRANSFER TO DOMAINS OUTSIDE THE TRAINING DISTRIBUTION:

We investigate performances on simulation parameters outside the training distribution. In particular, we investigate how well APRiL , its ablations, and asymmetric DDPG, generalise to environments with more distractor objects than seen during training. For NavWorld and JacoReach, we run two sets of increasingly extrapolated experiments with an additional 4 or 8 distractors (refered to as ext-4 and ext-8 in Table 1 ). The textures and colours of these objects are sampled from a held-old out set of simulation parameters not seen during training. For NavWorld, the locations and orientations of the additional distractors are randomly sampled. For JacoReach, the locations are sampled from arcs of two concentric circles of different radii (extrapolated arcs and radii to those seen during training), in such a way that each object remains visible. The shapes of the additional distractor object are sampled from the training catalogue of distractor objects. Please refer to Figure 3 for examples of the extrapolated domains.

Table 1 compares performances on the extrapolated sets (except Walker2D) varying in difficulty (ext-4 and ext-8). APRiL yields performance gains over the asymmetric DDPG baseline on every extrapolated domain. For JacoReach, APRiL's generalisation is so effective that, for the hardest domain with additional 8 distractors, its performance degrades by only 9%5 opposed to 41% (baseline).

APRiL generalises favorably due to the attention module. Figure 3 shows that attention generalises and suppresses the additional distractors, thereby effectively converting the hold-out observations to those seen during training, which the image-policy can handle. The ablations in Table 1 confirm that in this setting, distractor suppression is crucial. This is seen when comparing the maximum degradation in policy performance of APRiL, APRiL no share, APRiL no back and APRiL no sup (9%, 16%, 27% and 47% respectively). APRiL and APRiL no share both align attention between image and state agents during training, and therefore effectively suppress distractors (yielding a favourable decrease in policy performance of only 9% and 16%). APRiL no back learns a background suppressor, but does not suppress the distractors (leading to a larger degradation of 27%). APRiL no sup has an attention module trained only on the asymmetric actor-critic loss and yields the worst extrapolated performance (47% policy degradation). For these extrapolated domains, the successful suppression of the background and additional distractors (achieved only by the full APRiL framework), creates policy invariance with respect to them and helps generalise. 

4.6. ATTENTION MODULE ANALYSIS:

To better comprehend the role of the attention, we visualise APRiL's attention maps (Figure 3 , 4, 5) on both interpolated and extrapolated domains. For NavWorld, attention is correctly paid to all relevant aspects (agent and target; circle and triangle respectively). Attention generalises reasonably well to the extrapolated environments. For JacoReach, attention looks at the target, diamond-shaped, object as well as every other link (alternating links) of the Kinova arm. Interestingly, APRiL learnt that as the arm is a constrained system, the state of every other link can be indirectly inferred without explicit attention. The state of the unobserved link can be inferred by observing the links either side of it. The entropy loss over the state-attention module encourages this form of attention over minimal set of objects. Attention here generalises very well to the extrapolated domains. For Walker2D, we observe attention that is dynamic in object space. The attention module attends different subsets of links depending on the state of the system (see Figure 5 ). When the walker is upright, walking, and collapsing, APRiL pays attention to the lower limbs, every other link, and foot and upper body, respectively. We suspect that in these scenarios, the magnitude of the optimal action depends on the state of and as is largest for the lower links (due to stability), every link (coordination), and foot and upper body (large torque required), respectively. 

5. RELATED WORK:

Domain Randomisation has been applied for reinforcement learning to facilitate transfer between domains (Tobin et al., 2017; Pinto et al., 2018; Sadeghi & Levine, 2016; Viereck et al., 2017; OpenAI et al., 2018; Held et al., 2017) and increase robustness of the learned policies (Rajeswaran et al., 2016) . However, while domain randomisation enables us to generate more robust and transferable policies, it leads to a significant increase in required training time (OpenAI et al., 2018) .

Existing comparisons in the literature demonstrate that, even without domain randomisation, the increased dimensionality and potential partial observability complicates learning for RL agents (Tassa et al., 2018; Schwab et al., 2019; Watter et al., 2015; Lesort et al., 2018) . In this context, accelerated training has been achieved by using access to privileged information such as environment states to asymmetrically train the critic in actor-critic RL (Schwab et al., 2019; Pinto et al., 2018) . In addition to using additional information to train the critic, Schwab et al. (2019) Recent experiments have demonstrated the strong dependency and bidirectional interaction between attention and learning in human subjects (Leong et al., 2017) . In the context of machine learning, attention mechanisms have been integrated into RL agents to increase robustness and enable interpretability of an agent's behaviour (Sorokin et al., 2015; Choi et al., 2017; Mott et al., 2019) . In comparison to these works, we focus on utilising the attention mechanism as an interface to transfer information between two agents to enable faster training.

6. CONCLUSION:

We introduce Attention Privileged Reinforcement Learning (APRiL), an extension to asymmetric actor-critic algorithms that leverages access to privileged information like exact simulator states. The method benefits in two ways, via sharing a replay buffer as well as aligning attention masks between image-and state-space agents. By leveraging simulator ground-truth information about system states, we are able to learn efficiently in the image domain especially during domain randomisation where feature learning becomes increasingly difficult. Our evaluation on a diverse set of environments demonstrates significant improvements over the competitive asymmetric DDPG baseline and reveals that APRiL learns to generalise favourably to environments not seen during training (both within and outside of the training distribution) in comparison to the strong baseline; emphasising the importance of attention and shared experience for robustness of the learnt policies.

3. Walker2D: Randomisation occurs at the start of every episode as well as after every 50 agent steps. We introduce additional randomisation between episodes due to their increased duration. Due to the MDP setup, intra-episodic randomisation is not an issue. Materials, textures, camera location and orientation, are randomised in the same procedure as for JacoReach. The camera is setup to always face the upper torso of the walker. )). All layers use ReLU (Romero et al., 2015) activations and layer normalisation (Ba et al., 2016) unless otherwise stated. Each actor network is followed by a tanh activation and rescaled to match the limits of the environment's action space.

C IMPLEMENTATION DETAILS:

The State Attention module includes the fully connected layer defined in table 2 followed by a Softmax operation. The Obs Attention module has the convolutional layers (with padding to ensure constant dimensionality) outlined in table 2 followed by a fully connected convolutional layer (Conv([1, 1, 1])) with a Sigmoid activation to ensure the outputs vary between 0 and 1. The output of this module is tiled in order to match the dimensionality of the observation space.

During each iteration of APRiL (for both A o and A s ) we perform 50 optimization steps on minibatches of size 64 from the shared replay buffer. The target actor and critic networks are updated every iteration with a Polyak averaging of 0.999. We use Adam (Kingma & Ba, 2014) optimization with a learning rate of 10 -3 , 10 -4 and 10 -4 for critic, actor and attention networks respectively. We use default TensorFlow (Abadi et al., 2016) values for the other hyperparameters. The discount factor, entropy weighting and self-supervised learning hyperparameters are γ = 0.99, β = 0.0008 and ν = 1 respectively. To stabilize learning, all input states are normalized by running averages of the means and standard deviations of encountered states.

Both actors employ adaptive parameter noise (Plappert et al., 2017) exploration strategy with initial std of 0.1, desired action std of 0.1 and adoption coefficient of 1.01. The settings for the baseline are kept the same as for APRiL where appropriate.

D ATTENTION VISUALISATION:

Figure 4 : APRiL attention maps for policy rollouts on NavWorld and Jaco domains. White and black signify high and low attention values respectively. For NavWorld and JacoReach, attention is correctly paid only to the relevant objects (and Jaco links), even for the extrapolated domains. Refer to section 4.6 for more details. 

:

Videos comparing the policy behaviours of APRiL to the asymmetric DDPG baseline can be found here

Many simulators, like(Todorov et al., 2012), natively provide functionality to access these segmentations

In practice, the output of ho(ot) is tiled to match the number of channels that the image contains

We suspect that this is due to the simplicity of NavWorld, both visually and due to the small confined state-space, that none of the ablation by themselves outperform the baseline.

Percentage decrease is taken with respect to initial and final policy performance on training distribution

Figures and Tables:

Figure 1: Attention Privileged Reinforcement Learning model structure. Dashed lines indicate attention alignment process. The ∼ operator signifies that experiences are evenly sampled from both agents. The ⊗ operator represents element-wise multiplication.

Figure 2: Learning curves during training of APRiL , its ablations, and the asymmetric DDPG baseline. Solid line: mean performance. Shaded region: covers minimum and maximum performances across 5 seeds.

Figure 3: Example held-out domains (top) and APRiL attention maps (bottom). White and black signify high and low attention values. Attention correctly suppresses background and distractors.

Figure5: APRiL attention maps for policy rollouts on Walker domain. White and black signify high and low attention values respectively. Attention varies based on the state of the walker. When the walker is upright, high attention is paid to lower limbs. When walking, even attention is paid to every other limb. When about to collapse, high attention is paid to the foot and upper torso. Refer to section 4.6 for more details.

Algorithm 1 Attention Privileged Reinforcement Learning Initialize (a)symmetric actor-critic modules As, Ao, attention alignment module AT , replay buffer R for episode= 1 to M do Initial state s0 while ¬ DONE do Render image observation ot and segmentation maps zt: ot, zt ← renderer(st) if episode mod 2 = 0 then Obtain action at using obs-behavioral policy and obs-attention network: at ← πo(ho(ot) ot) else Obtain action at using state-behavioral policy and state-attention network: at ← πs(hs(st) st) end if Execute action at, receive reward rt, DONE flag, and transition to st+1 Store (st, ot, zt, at, rt, st+1, ot+1) in R end while for n = 1 to N do Sample minibatch {s, o, z, a, r, s , o } B 0 from R Optimise state-critic, actor, and attention using {s, a, r, s } B 0 with As Convert state-attention to target observation-attention {T } B 0 using {s, o, z} B 0 with AT Optimise observation-critic, actor, and attention using {s, o, T, a, r, s , o } B

Table content: <table><tr><td>0 with Ao</td></tr><tr><td>end for</td></tr><tr><td>end for</td></tr></table>

Ablation comparing average return over training, interpolated and extrapolated environments (100 each). Results reflect mean and standard deviation of average return over 5 seeds.

Table content: <table><tr><td>Domain</td><td>Baseline</td><td colspan="3">APRiL no sup APRiL no share APRiL no back</td><td>APRiL</td></tr><tr><td>NavWorld (train)</td><td>0.72 ± 0.02</td><td>0.60 ± 0.02</td><td>0.61 ± 0.02</td><td>0.67 ± 0.02</td><td>0.75 ± 0.02</td></tr><tr><td>NavWorld (inter)</td><td>0.70 ± 0.03</td><td>0.60 ± 0.02</td><td>0.62 ± 0.02</td><td>0.69 ± 0.02</td><td>0.80 ± 0.02</td></tr><tr><td>NavWorld (ext-4)</td><td>0.45 ± 0.02</td><td>0.42 ± 0.02</td><td>0.43 ± 0.02</td><td>0.45 ± 0.02</td><td>0.53 ± 0.02</td></tr><tr><td>NavWorld (ext-8)</td><td>0.46 ± 0.02</td><td>0.42 ± 0.02</td><td>0.38 ± 0.02</td><td>0.47 ± 0.02</td><td>0.50 ± 0.02</td></tr><tr><td colspan="4">JacoReach (train) -2.38 ± 0.23 -3.03 ± 0.30 -0.10 ± 0.19</td><td>1.10 ± 0.16</td><td>0.03 ± 0.19</td></tr><tr><td colspan="4">JacoReach (inter) -2.19 ± 0.24 -3.38 ± 0.31 -0.29 ± 0.20</td><td>0.81 ± 0.18</td><td>0.13 ± 0.20</td></tr><tr><td colspan="4">JacoReach (ext-4) -6.07 ± 0.25 -6.81 ± 0.33 -1.26 ± 0.22</td><td colspan="2">-0.79 ± 0.23 -0.70 ± 0.23</td></tr><tr><td colspan="4">JacoReach (ext-8) -8.02 ± 0.24 -9.26 ± 0.28 -2.69 ± 0.24</td><td colspan="2">-3.53 ± 0.26 -1.47 ± 0.25</td></tr><tr><td>Walker2D (train)</td><td>67.4 ± 3.15</td><td>107 ± 3.31</td><td>63.0 ± 2.13</td><td>126 ± 4.33</td><td>155 ± 3.25</td></tr><tr><td>Walker2D (inter)</td><td>68.9 ± 3.25</td><td>107 ± 3.30</td><td>62.6 ± 2.11</td><td>126 ± 4.31</td><td>156 ± 3.20</td></tr></table>

use a shared replay buffer for data generated by image-and state-based actors to further accelerate training for the imagebased agent. Our method extends these approaches by sharing information about relevant objects by aligning agent-integrated attention mechanisms between an image-and state-based actors.

Table content: <table/>

Model architecture. FC() represents a (multi-layered) fully connected network with the number of nodes per layer stated as argument. Conv() represents a (multi-layered) convolutional network whose arguments take the form [channels, square kernel size, stride] for each hidden layer.In this section we provide more details on our training setup. Refer to table 2 for the model architecture for each component of APRiL and the asymmetric DDPG baseline. Obs Actor and Obs Critic setup are the same for both APRiL and the baseline. Obs Actor model structure comprises of the convolutional layers (without padding) defined in table 2 followed by one fully connected layer with 256 hidden units (FC([256]

Table content: <table><tr><td>Domain</td><td>NavWorld and JacoReach</td><td>Walker2D</td></tr><tr><td>State Actor</td><td>FC([256])</td><td>FC([256])</td></tr><tr><td>Obs Actor</td><td colspan="2">Conv([[18, 7, 1], [32, 5, 1], [32, 3, 1]]) Conv([[18, 8, 2], [32, 5, 1], [16, 3, 1], [4, 3, 1]])</td></tr><tr><td>State Critic</td><td>FC([64, 64])</td><td>FC([400, 300])</td></tr><tr><td>Obs Critic</td><td>FC([64, 64])</td><td>FC([400, 300])</td></tr><tr><td>State Attention</td><td>FC([256])</td><td>FC([256])</td></tr><tr><td>Obs Attention</td><td>Conv([[32, 8, 1], [32, 5, 1], [64, 3, 1]])</td><td>Conv([[32, 8, 1], [32, 5, 1], [64, 3, 1]])</td></tr><tr><td>Replay Buffer Size</td><td>10 4</td><td>2 × 10 5</td></tr></table>

Peer review (iclr2020):
The topic addressed by the paper is domain adaptation and transfer learning in the text context of deep reinforcement learning, in particular the “sim2real” problem, where a policy is learned in simulation and should be transferred to a physical agent in a real-world scenario. The work builds on the existing “asymmetric DDPG” formulation (Pinto et al., 2017), which exploits the fact that full states are sometimes available in simulated environments but not during deployment. In Pinto et al., this is addressed by learning an actor taking as input observations, and a critic which has access to the state. The contribution of the paper is an extension of Pinto et al. by adding additional communication between the state head (which learns a critic) and the observation head (which learns the policy). This is done through an attention mechanism, now very classical in deep learning, which weights the different elements of the (state) input. The particularity here is that the attention mechanism is trained on the critic, which takes the full state as input, and then transferred to the observed input of the policy. The problem is that the state is very different from the observations, which are images. An alignment module expands the distribution over state variables to a distribution over observed objects. One of my main concerns here is lack of justification and lack of clarity. While the abstract of the paper and introduction section are well written, and wants us to learn more about this interesting idea, the paper kind of falls apart in the subsequent chapters. The authors mention alignment of attention, but the exact motivation for the algorithm, i.e. the motivation for its formulation, are never provided. The description itself is also far from clear, as key design choices are not introduced and we discover them, or guess them, from equations. We don’t know, for instance, what the observations are, and think that they are images. However, apparently an object mask detector is run over the input images, as segmentation maps are input to the state-object alignment module, which needs to distribute the attention over states to attention over objects. This also means that the formulation is quite specific to the task at hand, since it seems to be object-centric. Figure 1 is another example of the same problem: There are many arrows, but their signification is unclear. Dashed lines are indicated to mean alignment, but alignment is not a standard term, for instance comparable to a computation connection, or a loss signal. A similar confusion is found in the equations themselves, as h(s) is mapped to a vector c, but this vector decomposes into different objects of the environment. What is T? It being upper case could mean a matrix or a scalar value, but we are not sure. From the loss function we see that this seems to be vector (a distribution) of the same size as the attention vector h_o … which is over what? Basically, as I understand it, the alignment loss minimizes some structured mapping between some attention vector over objects on the observation side and some attention vector over states on the critics side. This seems to be a weak loss signal, as the attention transformation needs to be learned together with the attention mechanism itself (and of course the actor and the critic). At the beginning of section 3, the method is introduced of having an asymmetric part and a symmetric part, but I don’t see this, as this would require learning 4 predictors and not 2 (an actor and a critic). An asymmetric model, as given in Pinto et al., uses two predictors, an actor and a critic, with different loss functions and, more importantly, different inputs. Here, the authors claim that the observation module itself is asymmetric … but how can something by asymmetric if it contains only an actor? Same, the critic is supposed to be symmetric … but without an actor? As I see it, w.r.t to the question whether the model is symmetric or asymmetric, the formulation is identical to Pinto et al., thus asymmetric. The difference lies in the attention mechanism and the alignment module. At some point in the paper, this method is mentioned to be self-supervised … I am not sure which aspect of this work could be called self-supervised, but I agree that the definition of this relatively new term is sometimes ambiguous. On the other hand, I think the method should be compared to “classical” self-supervision in RL, which corresponds to predicting information which is available during training but not available as input (depth prediction etc.). Here, a natural baseline seems to be a symmetric model (same input for actor and critic: observations) and to predict the state of the model and self-supervise it during training but not deployment. The evaluation is unfortunately not convincing. Two important baselines are missing: - Self-supervision, as mentioned above - Pinto et al., on which this method is based. There are some hics in the results … for instance the curves in Figure 2 show learning in progress, they are not yet converged. Also, the standard deviations are quite big. Another downside is that, although sim2real is used as a motivation for this paper (since this is the most typical scenario where states are available during training but not during testing), the experiments have not been performed using physical agents. Testing is performed on degraded simulated agents. I do understand that physical environments are harder to manage than simulated ones, but tiny physical environments can be obtained for reasonable prices, and this would have made the paper much stronger and more aligned with its core motivation. Simulated noise cannot replace a physical environment. No details have been given on the additional distractor objects unseen during training. The Walker 2D environment is described as “modified”, but how exactly, and why? Section 2.1 (definition of POMDPs) seems to be unnecessary and can be deleted.