Zero-Shot Policy Transfer with Disentangled Attention
Authors: Josh Roy
George Konidaris
Venue:  (2020)
Keywords: Transfer Learning
Reinforcement Learning
Attention
Domain Adaptation
Representation Learning
Feature Extraction

Abstract: Domain adaptation is an open problem in deep reinforcement learning (RL). Often, agents are asked to perform in environments where data is difficult to obtain. In such settings, agents are trained in similar environments, such as simulators, and are then transferred to the original environment. The gap between visual observations of the source and target environments often causes the agent to fail in the target environment. We present a new RL agent, SADALA (Soft Attention DisentAngled representation Learning Agent). SADALA first learns a compressed state representation. It then jointly learns to ignore distracting features and solve the task presented. SADALA's separation of important and unimportant visual features leads to robust domain transfer. SADALA outperforms both prior disentangled-representation based RL and domain randomization approaches across RL environments (Visual Cartpole and DeepMind Lab).

Under review as a conference paper at ICLR 2020
ZERO-SHOT POLICY TRANSFER WITH DISENTANGLED ATTENTION
Anonymous authors Paper under double-blind review
ABSTRACT
Domain adaptation is an open problem in deep reinforcement learning (RL). Of- ten, agents are asked to perform environments where data difficult to ob- tain. In such settings, agents are trained in similar environments, such sim- ulators, and are then transferred to the original environment. The gap between visual observations of the source and target environments often causes the agent to fail in the target environment. We present a new RL agent, SADALA (Soft Attention DisentAngled representation Learning Agent). SADALA first learns compressed state representation. It then jointly learns to ignore distracting fea- tures and solve the task presented. SADALA's separation of important and unim- portant visual features leads to robust domain transfer. SADALA outperforms both prior
based RL and domain randomization ap- proaches across RL environments (Visual Cartpole and DeepMind Lab).
INTRODUCTION
RL agents learn maximize rewards within task by taking actions based on observations. The advent of deep learning has enabled RL agents to learn high dimensional feature spaces and complex action spaces (Krizhevsky et al.) 2012; Mnih et al., 2016). Deep RL methods have beat human performance in a variety of tasks, such as Go (Silver & Hassabis 2016). However, deep RL has two crippling drawbacks: high sample complexity and specificity training task.
There are many domains where data collection is expensive and time consuming, such as healthcare, autonomous vehicles, and roboties (Gottesman al. 2019; Barrett et al., (010). Thus, agents often trained in simulation and must transfer the resulting knowledge to reality. While this solves the issue of sample complexity, reality and simulated domains sufficiently different that it is infeasible to naively train a deep RL agent in a simulation and transfer. This is known as the reality gap (Sadeghi Levine 2016). Jumping the reality gap is difficult for two orthogonal reasons. The first is that dynamics of simulation are an approximation to the dynamics of the real world. Prior work has shown success in transfer between domains with different dynamics (Killian et al. 2017] Yao et al. 2018 Doshi-Veler. & Konidaris (016). In this paper, we address the second diffiulty: the difference in visual observations of states. Due to limitations in current photorealistic rendering, simulation and the real world are effectively two different visual domains (Sadeghi & Levine 2016).
We present method of robust transfer between visual RL domains, using attention and / varia- tional autoencoder to automatically learn state representation sufficient to solve both source and target domains. By learning disetangled and relevant state representation, our approach does not re- quire target domain samples when training. The state representation enables the RL agent to attend to only the relevant state information and ignore all other, potentially distracting information.
2 RELATED WORK
Domain randomization is currently the most popular transfer method between different visual do- mains RL (Tobin et al.) 2017 OpenAl et al. 2018, Sadeghi & Levine (016). By training on many source domains, an RL agent implicitly learns ignore factors of variation present. Sample complexity scales with the number of source domains and randomized factors. Modes varia- tion maually selected. When faced with different variation, domain randomization will fail.

---

Under review as a conference paper at ICLR 2020
Furthermore, there is evidence that domain randomization can destablize the training of some RL algorithms, such as A3C and DDPG (Matas et al. 2018 Zhang et al. 2019). Though domain ran- domization has shown success in transfer, has three downsides: high sample complexity, manual
selection of irrelevant features, and ustable training (Tobin et al. 2017 OpenAl et al. 2018).
Some visual domain adaptation work is based on image to image translation (Ajakan et al. 2014
Tzeng et af.] 2017, Hoffman et al. 2017 Zhu et al. 2017; Pan et al., [017]] Isola et al.
2017
Liu et al.] (2017). These methods learn a mapping from source domain inputs target domain inputs, utilizing adversarial methods, such as Generative adversarial Networks trained on data from both the source and target domain. During inference, the target domain imput is translated to the source domain before being processed. While these methods have shown promising results, they have two downsides. First, translation incurs additional overhead at inference time. This overhead is especially large since the inputs are high-dimensional images. Second, these methods require samples from the target domaín. In applications such as robotics, sampling the target domain is expensive and difficult to obtain. Further, the target domain may not be known at training time.
Some recent work focuses on modifying the inputs from the source domain so that they are similar to the target domain, effectively matching their distributions (Shrivastava et al. 2017). This approach uses a GAN to refine images from a (simulated) source domain to match images from the target domain: the real world. While this directly addresses domain shift and does not add any additional overhead at inference time in the target domain, it still assumes that samples from the target domain present during training. It enables transfer between two differing visual domains, but it does not enable adaptation to previously unseen domains.
There is body of work that takes this idea further. Rather than translating from the target domain to the source domain, it learns to map all domains to a canonical domain in which the RL agent operates (James al., 2019). By learning to translate from randomized domains to a canonical domain, effectively training translation network using domain randomization. While this approach has shown simulation to reality transfer, it icurrs overhead through the use of the mapping network; Rather than learning to map to a canoical domain, it would be more efficient to map to a disentagled representation of the canoical environment.
Other work learns to map image inputs to a latent space for use in RL. Specifically, it utilizes B Variational Autoencoder (B-VAE) to learn a compressed, disentagled state representation (Higgins) et al. 2017). While this work focuses on domain adaptation in the RL setting, it utilizes a general- purpose generative model. Thus, the state representation is not tailored to the task posed; it preserves information needed to reconstruct the image that may not be needed to solve the RL task given. Additionally, unlike image translation, this method does not attempt to match the distribution of inputs from source and target domains. Since the B- V-V is trained to reconstruct input states, the distributions of compressed and input states will match. Accordingly, the distribution shift between source and target domains will be preserved, even when using compressed states. To circumvent this issue, the RL agent is trained on multiple source domains, as in domain randomization.
BACKGROUND
We formalize the concept of transfer between related MDPs We denote the source domain Ds and the target domain as Dr. Both domains are MDPs, defined as D = (S, T, R),, where S are states, A are actions, T is the transition function, and R is the reward function. We assume that both Ds and Dr share discount factor 7. In general transfer, states and actions may be quite different, while transition and reward functions are similar. We denote a family of related MDPs as M. Within M, transitions, rewards, and actions are the same across MDPs, and states are different. These states are generated from a set of latent state factors Sa, each passed through a different MDP- specific renderer Gy for some MDP U E M. In the case of visual cartpole. S contains the position and velocity of the cart, the angle and position of the pole, and the colors of the cart, pole, axle, and track. The observed states are then rendered by the renderer. Specifically, Su Gu and Sy = GGvSS*, where Gy and Gy are any two MDPs within M.
In this paper, we target transfers where T3 Tr, Rs Rr, As Ar, but Ss (S() + ST (SF). Namely, when target dynamics are similar to source dynamics, rewards, actions, and true states identical, but observed states S. and Sr differ due differing renderers Gs and Gr.
2

---

Under review as a conference paper at ICLR 2020
4 SADALA
4.1 FEATURE EXTRACTION WITH A- VVRRIATTONNLL AUTOENCODERS
Deep RL agents learn a mapping from states S actions a E A. either by directly maximizing reward or through a Q function, using a deep neural network. The deep network implicitly learns mapping function F Sy + $², for some MDP U E M and policy function π S= + A that maps from latent state factors actions. These functions are learned as a result of the feed-forward nature of deep neural networks. The policy и is a function of the last layer of the network. The output of the last layer of the neural network is highly processed feature representation of the input that is optimized for use in generating π. While this implicit mapping is sufficient for deep RL agent to solve a single MDP from M, it is not general enough to transfer to other MDPs within M. By optimizing for F overfits to the specific Sy from the source domain U.
To circumvent this overfitting, DARLA proposed learning a feature extractor F" that maps from any observed state Sx to the latent state factors S (Higgins et al. 2017). The RL agent is the trained on the (approximate) extracted latent state factors S*, separating the "vision" and "action" of the RL. agent. DARLA uses a / variational autoencoder to learn the mapping F" (Higgins et al. 2017).
The extracted latent state factors Sx are (mostly) disentangled in practice, leading to better transfer results than a baseline RL agent. The factors each correlate with an independent factor in the true state S of the task. However, it preserves all latent state variables z E S", incorrectly assuming that they are all relevant to solving the MDP. When attempting to locate an object such as an enemy in a video game, an object for a robot, or a road for an autonomous vehicle, the position and orientation of the object are the most important factors. Visual distractions such as the color of the enemy, opacity of the object, or texture of the road are entirely irrelevant to its localization. Dependence on these distracting factors causes the RL agent to fail to robustly transfer to target visual domains.
4.2 ADDING AN ATTENTION MECHANISM
SADALA is similar to DARLA, but has one key difference it explicitly learns to ignore irrelevant extracted latent state variables.
SADALA learns the same mapping function F' -Sy - S* DARLA, also utilizing a 3 variational autoencoder. It then learns a policy $= + A. but utilizes soft attention mechanism to ignore irrelevant features. Using this mechanism, implicitly learns two functions: A S= - Sa that maps from extracted latent state factors to latent state factors relevant to solving the MDP, and A : S + A that maps from relevant latent state factors to actions.
Soft attention mechanisms learn an importance weight E [0, 1] for each feature in part of neural network. These weights are then multiplied element-wise with the corresponding feature. Thus, the attention layer can be defined as follows:
((Z)
W の Z,
where W is the vector of importance weights wy and ( is element-wiss multiplication. W is defined
W = (h((Z))
where h is differentiable function parameterized by dense layers of neural network. Thus, the values w, E W are constrained between 0 and 1, inclusive. This allows the RL agent to ignore a feature z; by setting w; 0. It also allows the agent to down-weight the importance of feature by setting to a fractional value or pay full attention by setting w, to 1.
4.3 SADALA FRAMEWORK
The SADALA framework is shown in Figure ! At inference time, an input state s E is fed into 3 variational autoencoder. The bottleneck representation of the 3-VAE is then used extracted latent state factors $= Those features are then fed into the attention mechanism. which outputs the weighted features S", according to the formula f above. Finally, the weighted features are used as the inputs to the deep reinforcement learning algorithm.
3

---

Under review as a conference paper at ICLR 2020
State Representation
Feature
Selection
Weighted
(High Dimensional)
Input State
Beta VAE
Features (Low Dimensional)
Attention Mechanism
(Low Dimensional)
Features
Policy
Learning
Algoritm
Policy
Figure 1: SADALA Framework
Input State Reconstruction
Input State
Encoder
Latent State Features
Decoder
Figure 2: B Variational Autoencoder.
Though the feature selection and policy learning stages are represented separately in figure | they are contained within the same neural network and optimized jointly.
4.4 SADALA TRAINING
Training SADALA requires a few steps. First, the 8 variational autoencoder, shown in figure must be trained. To do so, we use the loss term of variational autoencoder, modified fit the 3-VAE (Higgins et al. 2017). The standard VAE loss is
CVAe =
The 3-VAE loss is
La-VAE =By(=|a>) (log 3D kL (2g((()))))l))))
where 3 is weighting term that pressures the encoder q to output latent space similar to an isotropic unit gaussian. Increasing the value of 3 encourages disentanglement of the latent space while sacrificing minor reconstruction details.
We sample states from the environment, following a random policy. We then train the 3-VAE reconstruct these images and freeze the network weights, As shown by DARLA, freezing the network weights of the 3-VAE is necessary (Higgins et al. (017). If the weights are allowed to train jointly with the policy network on the source domain, they will optimize for and overfit to the source domain. When using one source domain, this will cause SADALA to be equivalent to single-task learner. When using multiple source domains, SADALA would be equivalent domain randomization agent.
The feature selection and policy learning stages are then jointly optimized. The SADALA frame- work can utilize any deep RL algorithm (A3C, REINFORCE, DQN, etc), with minor modification. Specifically, we add a loss term to the loss function LRI. of the original algorithm. Our new loss
CARL = CRL + [
where W is the learned attention weights This added term is an LI regularization on the attention weights, which encourages sparsity in feature selection (Kolter & Ng 2009 Ng 2004).
4

---

Under review as a conference paper at ICLR 2020
We additionally test other modification to the loss term. We make the assumption that the ordering of the extracted latent state variables S does not change between subsequent frames within an episode. This, the weight W; for some latent state factor 2; should not change based on the frame. To enforce this loss on the feature selection stage, use the following loss
CPA-RL = CRL + I +
Σ Var[W*],
JElen(=)
where W is vector of weights corresponding to the jth extracted latent state factor vector across the set of training data and Var is the variance. This loss is added to enforce corresponding to the feature 2; to be the same regardless of input. This is similar to learning pseudo-attention weight vector, rather than an attention weight vector.
See the appendix for additional training details.
RESULTS AND DISCUSSION
We test the SADALA framework on two transfer learning tasks, using A3C as the deep RL algo- rithm. The first task is Visual Cartpole. This domain is the same Cartpole-vl in OpenAl Gym with two key differences (Brockman et al. (016). The observed state is now the pixel rendering of the cartpole as well as the velocities of the cart and pole. Thus, the agent must learn predict the position of the cart and pole from the rendering. Additionally, we modify this domain to include a transfer task. The agent must learn to transfer its knowledge of the game across different color configurations for the cart, pole, and track. Thus, Visual Cartpole is defined as family of MDPs M where the true state S is the positions and velocities the cart and pole and the observed state Sy = Gr for MDP U E M is the pixel observations and velocity values. Optimally, the agent should learn to ignore the factors in extracted latent state factors Sx that correspond to color, as they do not aid the agent in balancing the pole. This task tests the agent's ability to ignore irrelevant latent state factors.
Figure 3: 3-VAE reconstructions of samples from the Visual Cartpole environment. Top row con- tains original images and the bottom row contains their reconstructions.
The second task the "Collect Good Objects" task from Deepmind Lab (Beattie al. 2016). The agent must learn to navigate in first person and pick up "good" objects while avoiding "bad" objects. This task defined as family of MDPs M where the true state S² contains the position of the agent, the position of good and bad objects, the type of good and bad objects, and the color of the walls and floor. In a single MDP U M, all good objects are hats or all good objects are balloons. Similarly, all bad objects are either cans or cakes. The walls and floor can either take a green and orange colorscheme or a red and blue colorscheme. The agent is trained on hats/cans with the green/orange colorscheme and balloons/cakes with both colorschemes. It is then tested on hats/cans with the red/blue colorscheme. Optimally, the agent should learn to ignore the color of the floor and walls. Additionally, it should use the type of object to determine if it is good or bad. This task tests the agent's 's ability to ignore distracting latent state factors (the color the walls and floor) while attending relevant factors (the positions and types of objects and its own position).
5

---

Under review as a conference paper at ICLR 2020
To test the results of the SADALA algorithm, first test the reconstruction and disentanglement properties the 3-VAE used in the state representation stage. Note that this stage is identical to that of DARLA (Higgins et al. 2017). As such, expect the disentanglement properties to be similar.
900
Figure 4: B-VAE reconstructions samples from the Deepmind Lab environment. Top row contains original images and bottom row contains their reconstructions.
See figure for reconstructions of the cartpole state. Based on the reconstructions, it is apparent that the 8- VAE has learned to represent cart position and pole angle. Though the angle of the poles is slightly incorrect in the first set of images, the pole is tilted in the correct direction, yielding sufficiently correct extracted latent state factors.
Additionally, the color of the cart, pole, and background is incorrect in the third pair of images. While this demonstrates that the identification and reconstructions of colors is not infallible, the position of the cart and pole remains correct, yielding set of extracted latent state parameters that is sufficient to solve the MDP.
See figure for visualization of reconstruction with attention. In the original image, the pole is standing straight and the cart is centered. In the reconstruction, the cart is centered, and the pole is almost upright. However, the reconstruction does not include the colors of the cart or pole. Instead it fills the cart and pole with the mean color of the dataset. This shows that the attention weights are properly learning to ignore color and instead pay attention the position of the cart and pole.
Figures [6 and 7 gives comparison of the performance of the algorithms across environments. Note that all of the algorithms are sufficient solving the source task, with the single-task learner performing slightly better. This is due to the fact that the single-task learner can optimize its con-
Figure 5: 3- VAE reconstructions from Visual Cartpole The image on the left is the original image from the simulation and the image on the left is the reconstruction after multiplying with the learned attention weights.

---

Under review as a conference paper at ICLR 2020
Rewards on Visual Cartpole
Visual Caripole (Source)
Visual Cartpale (Target)
200
150
Resend
100
50
Single-Task Leamer
Domain Rand.
DARLA
SADALA
SADALA (Feduced Var.)
Figure 6: Source and target rewards on Visual Cartpole.
Rewards on Deepmind Lab
Deepmind Lab (Source)
Deepmind Lab (Target)
35
30
25
20
Reward
15
10
Single
SADALA
SADALA (Reduced
Figure 7: Source and target rewards Deepmind Lab.
volutional filters to the source domain the expense of failing to transfer to the target domain. For numerical reward values, see table T in the appendix.
The single-task learner achieves better rewards all source tasks than any of the transfer-specific agents. Domain randomization performs less well because of the complexity the domain random- ization task. Rather than optimizing reward for single domain, the agent must optimize reward across a large set domains.
DARLA, SADALA, and SADALA with reduced variance also perform less well on the source task than the baseline agent. This is due to imperfections the 3- B-V. Though the 3- 3-VAE attempts to reconstruct the input image, it does not do perfectly, shown in figure This shows that its

---

Under review as a conference paper at ICLR 2020
extraction of latent state features is not perfect, leading potentially confusing states given to the RL agent. Additionally, while the 3-VA''s goal is to learn a disentangled representation, it does not do so perfectly. As such, (partially) entangled latent state factors may further confuse the RL agent.
The single-task learner fails to transfer its policy to the target domain. DARLA transfers some of its knowledge the target domain. Domain randomization also transfers some knowledge. Finally, SADALA transfers more of its knowledge. The single-task agent has no incentive to learn factored state representation that enables transfer. Its convolutional filters will directly optimize maximize reward in the source policy. Thus, if the filters are searching for a hat on a blue floor but tested on a hat on a red floor the convolutional filters will fail transfer.
DARLA learns state representation that is mostl disentangled. This allows the RL agent to learn to ignore unimportant features such cart color and utilize important features such cart posi- tion. The factored state representation forces the RL agent to learn more robust policy. However, the neural network parameterization of the RL policy must implicitly learn to ignore unimportant factors. Therefore, when presented with unseen information, the RL agent may not properly ignore unimportant factors.
Domain randomization forces the neural network to implicitly learn state representation sufficient to transfer between tasks. This requires large amounts of training data and less robust than explicit modeling of latent state factors.
SADALA builds on DARLA by adding an explicit attention mechanism, allowing to more ef- fectively ignore unimportant features. Due to the use of the sigmoid activation in the attention mechanism, the attention weights W are bounded between 0 and 1. In addition to providing a direct weight the importance of feature, this bound prevents high variance of attention weights across different inputs.
SADALA with the variance reduction term performs worse than both DARLA and SADALA with- out variance reduction on the Deepmind lab task but better on the other two. In the scenario where the extracted latent state factors from the 3-VAE are perfectly disentangled, static attention weights should be sufficient to solve the source task and should transfer better to the target task, as in the Visual Cartpole task. However, the B- BAE does not output perfectly disentangled factors, especially in more complex visual domains such as the Deepmind lab. Thus, the amount of attention paid to each feature from the 3-VAE may differ across tasks, violating the assumption that attention weights should be zero variance.
CONCLUSION
In this paper we propose SADALA, a three stage method for zero-shot domain transfer. First, SADALA learns feature extractor that represents input states (images) as disentangled factors. It then filters these latent factors using an attention mechanism to select those most important to solving source task. Jointly, it learns a policy for the source task that is robust to changes in states, and is able to transfer to related target tasks. We validate the performance of SADALA on both high-dimensional continuous-come simulated environments (Deepmind Lab). We show that the attention mechanism introduced is able to differentiate between important and unimportant latent features, enabling robust transfer.
problem (Visual Cartpole) and 3D naturalistic first-person
REFERENCES
Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, and Mario Marchand. Domain-adversarial neural networks. arXiv preprint arXiv: 1412,444, 2014.
Samuel Barrett, Matthew E Taylor, and Peter Stone. Transfer learning for reinforcement learning on a physical robot. In Ninth International Conference on Autonomous Agents and Multiagent Systems-Adaptive Learning Agents Workshop (AAMAS-ALA), volume 1, 2010.
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Victor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint 16Xiv:1612.0333 2016.
8

---

Under review as a conference paper at ICLR 2020
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
Finale Doshi- and George Konidaris. Hidden parameter markov decision processes: A semi- parametric regression approach for discovering latent task parametrizations. In IJCAI: proceed- ings of the conference, volume 2016, pp. 1432. NIH Public Access, 2016.
Omer Gottesman, Fredrik Johansson, Matthieu Komorowski, Aldo Faisal, David Sontag, Finale Doshi-Velez, and Leo Anthony Celi. Guidelines for reinforcement learning in healthcare. Nar Med, 18 2019.
Irina Higgins, Arka Pal, Andrei A. Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. DARLA: improving zero-shot transfer reinforcement learning. CoRR, abs/1 0084750 2017. URL AKKLattpp//axxiv.crr
abs/1107. 08475)
Judy Hoffman, Eric Tzeng, Taesung Park, Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. arXiv preprint arXiv:1711.03215 2017.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data- efficient robotic grasping via mondomized-to-canonic adaptation networks. In Proceedings of the IEEE Conference on Computer Vision and Putlern Recognition, pp. 12627-12637, 2019.
Taylor W Killian, Samuel Daulton, George Konidaris, and Finale Doshi-Velez. Robust and effi- cient transfer learning with hidden parameter markov decision processes. In Advances in Neural Information Processing Systems, pp. 2250-6261, 2017.
J Zico Kolter and Andrew Y Ng. Regularization and feature selection in least-squares temporal dif- ference learning. In Proceedings of the 26th annual international conference on machine learning, pp. 521-528. ACM, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097- 1105. Curran Associates, Inc., 2012. URL http::/ nips. cc/paper/
4824 imagemettttt.......m............ classi Fictatiiinn coation-wee con-with deep- coneoputooollll.ona......o neurall networks. pdf
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Advances neural information processing systems, pp. 700-708, 2017.
Jan Matas, Stephen James, and Andrew J Davison. Sim-to-real reinforcement learning for de- formable object manipulation. arXiv preprint ar arXiv:1806.0785/, 2018
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. CoRR, abs/1602.01 788, 2016. URL httt::/t///aaat
org/ abs 602 01783
Andrew Ng. Feature selection, 1 vs. regularization, and rotational invariance. In Proceedings of the twenty-first international conference on Machine learning, pp. 78. ACM, 2004.
OpenAl, :, M. Andrychowicz. B. Baker, M. Chociej. R. Jozefowicz, B. McGrew, Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray. J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba. Learning Dexterous In-Hand Manipulation. ArXiv e-prints, August 2018
Xinlei Pan, Yurong You, Ziyan Wang, and Cewu Lu. Virtual to real reinforcement learning for autonomous driving. arXiv preprint arXiv: 1704.0997 2017.
2017.

---

Under review as a conference paper at ICLR 2020
Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without single real image. 2016.
arXiv preprint
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 2107-2116, 2017.
David Silver and Demis Hassabis. Alphago: Mastering the ancient game of go with machine learn- ing. Research Blog, 2016.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RS) International Conference on Intelligent Robots and Systems (IROS), pp. 23-30. IEEE, 2017.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7167-7176, 2017.
Jiayu Yao, Taylor Killian, George Konidaris, and Finale Roshi-Veles Direct policy transfer via hidden parameter markov decision processes. 2018.
Jingwei Zhang, Lei Tai, Peng Yun, Yufeng Xiong, Ming Liu, Joschka Boedecker, and Wolfram Burgard. Vr-goggles for robots: Real-to-sim domain adaptation for visual control. IEEE Robotics and Automation Letters, 4(2): 1155, 2019.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
A APPENDIX
A.1 TASKS
We test the SADALA framework on two transfer learning tasks, using A3C as the deep RL algo- rithm. These tasks are descrihed in detail below.
A.1.1 VISUAL CARTPOLE
As in the original cartpole domain, agent must learn to keep pole balanced cart upright for as long as possible. The states in the original domain are the position and velocity of the cart and pole, and the states in the modified domain are image renderings of the cart and pole. The agent acts in both domains by applying a force of +1 or -1 to the cart, and the environment transitions based on deterministic physics simulation of the cart and pole. The reward is +1 every timestep that the pole has not fallen over, and the episode terminates when the pole falls over or the cart moves too far to the left or right. Similar to OpenAI Gym's 's Cartpole V1, Visual Cartpole ends after a maximum of 500 steps, yielding a maximum reward of 500 per episode (Brockman et al. 2016).
Note that visual cartpole described above is Partially Observable Markov Decision Process (POMDP), in which there is a hidden state that is not directly observable. Specifically, it is not possible for an agent to determine the velocity of the cart and pole solely based on one image. In order to simply the visual cartpole task to an MDP, velocity information is included in the state. This enforces the Markov property on the environment, making better suited to test visual transfer.
When framing this MDP as transfer learning task, the agent is trained on some number of source domains, where the colors of the cart, pole, axle, and track are varied. Specifically, the color each element (cart, pole, etc) is sampled randomly from normal distribution with mean 0.5 and standard deviation 0.5 and clipped to the interval [0, 1]. Note that each color is represented as a vector of three numbers, each within the interval [0, 1]. The colors randomly sampled the end of each episode. The agent is then tested on a target domain with held-out colors, which are sufficiently different from those of the source environments. Specifically the colors of the source environments
10

---

Under review as a conference paper at ICLR 2020
constrained to be outside of hypersphere of radius 0.1 the test environment. When evaluating source domain performance, the agents trained on multiple source domains are all evaluated on the same domain, randomly sampled from the set of source domains. The single task learner is evaluated on the source domain it trained on.
A.1.2 DEEPMIND LAB
The second task is the Collect Good Objects task from Deepmind Lab (Beattie et al., 2016). This environment be defined as family of MDPs that share transition function, reward function, and action space. The goal of the agent to pick up as many "good" objects (+1 reward) and avoid picking up "bad" objects (-1 reward) as it can within minute.
These objects can be hats, cans, cakes, and balloons. Additionally the color combinations of the walls and floors can vary between green walls and orange floors or red walls and blue floors. These variations in state space correspond to the different MDPs within the family N of Deepmind Lab MDPs. The 3-VAE is trained set of MDPs covering all latent state factor combinations, the RL network and attention mechanism is trained on subset of N and tested on held-out MDP.
A.2 ADDITIONAL RESULTS
Table compares the numerical reward values displayed in figures 6 and The source domains VC (Source) and DM (Source) are listed for completion. It is better to sacrifice some reward in the source domain for additional reward in target domains.
SADALA
Task
Single-Tass Learner
Domain Rand.
DARLA
SADALA
(Var. Reduction)
VC (Source)
219.53
194.2
193.48
187.0
190.3
VC (Target)
31.2
147.2
150.136
170.16
185.87
DM (Source)
30.8
29.09
28.45
28.06
DM (Target) 6.2
19.7
20.7
15.3
Table 1: Mean reward of A3C, Domain Randomization, DARLA, and SADALA agents Visual Cartpole and Deepmind Lab. Higher is better. Each mean reward is taken over series of 5000 test episodes according to 5 trials.
B NEURAL NETWORK DETAILS
The neural network for the Visual Cartpole task consists of two parts: the 3-VAE network and the policy network. First, the encoder for the 3-VAE network is composed three convolutional layers, each with kernel size 3 and stride In order, they have 32, 32, and 64 filters. The output of the last convolutional layer is then frozen and passed through dense layer with 256 neurons. This is then passed through another dense layer with 64 neurons, which outputs the latent space. The latent space is composed of 32 gaussians, each parameterized by mean and variance. The decoder is an inverse of the encoder, utilizing deconvolutional layers. The encoder from the 3-VAE is then frozen and used as feature extractor for the policy network. The policy network takes the encoders outputs and passes them through one dense layer with 64 neurons, which outputs the attention weights. These weights are multiplied elementwise with the encoder's outputs and passed through two dense layers, each with 128 neurons. Finally, the output is passed through dense layer with two neurons, outputting the policy given the input state. The output is also passed through dense layer with one neuron, which outputs the value of the input state.
The networks used for the Domain Randomization agent and Single-Task learner have similar structure. They first pass the input through three convolutional filters, identical in architecture to the encoder. They then pass the outputs of the comvolutional filters through two dense layers, each with 128 neurons. Finally, this output is passed through the dense layer with two neurons which outputs the policy, and the dense layer with one neuron, which outputs the value.
11

---

Under review as a conference paper at ICLR 2020
The neural networks used for the Deepmind Lab task are slight modifications to the neural networks described above. They have four convolutional layers instead three, with 32, 32, 64, and 64 filters, respectively. Additionally, the hidden dense layers in the policy network have 256 rather than 128 neurons. As in DARLA, the reconstruction loss of the 3- V-V for Deepmind Lab takes place in the latent space of denoising autoencoder with 100 latent factors Higggnn et al. (2017).
Relu activations are used throughout. All Visual Cartpole networks are trained with Adam with a learning rale of 5e 4 and all Deepmind Lab networks are trained Adam with learning rate of le - 4. The V-VAE networks are trained on a set of 1, 000, 000 images sampled from the observed state space of source policies. The RL networks are trained on 000, 000 and 16, 000, 000 states, actions, and rewards for the Visual Cartpole and Deepmind Lab tasks, respectively. The domain randomization agent trained on 3, 000, 000 Visual Cartpole states, actions, and rewards.
C DOMAIN RANDOMIZATION DETAILS
Domain randomization was implemented by varying parameters me
12


Reviews:
[{"review_id": "S1xiYtYZ9S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summarize what the paper claims to do/contribute.\n- The paper proposes a new method for zero-shot visual transfer for RL, SADALA. The method first learns a feature extractor with attention (to focus on realted features only) and then learns a policy in the source task and is able to transfer zero-shot int he target domain. The method is evaluated on two tasks: Cartpole-v1 (Gym) and \"Collect Good Objects\" (Deepmind Lab). It is compared against DARLA for both tasks and against Domain Randomization only for Cartpole. \n\nClearly state your decision (accept or reject) with one or two key reasons for this choice.\nReject.\n- The experiments of the paper were particularly weak. \n--More standard visual adaptation techniques like DANNs,ADDA, PixelDA/SimGAN, CycleGAN were not considered. \n--The results on domain randomization were not convincing: more details are necessary to determine what the experimental protocol was. One major question: what is the source domain in the case of domain randomization (for Fig. 6) In any case, I find it very hard to believe that simple domain randomization considered here can not fully solve this task for all visual pertrubations considered. \n-- In Fig. 5 the reconstruction is not correct.\n-- Domain randomization was not tried on the DeepMind Lab example because of compute. However, I'd encourage the authors to try this. Converging will surely not be  linear to the number of perturbations considered as it seems to be implied. Also the OpenAI paper cited as an example where domain randomization took 100 years of simulation required for transfer is a problem of rather different scale: the domain gap there is between simulation and reality for an anthropomorfic robotic hand, and not a simple visual gap where the color of an identical environment are changed. \n\n-Related work discussion was insufficient\n-- Related work section is missing and work is not adequately placed in the context of existing literature in the Introduction where some related work is indeed discussed.\n-- Related work at the last sentence of the introduction is not discussed correctly. It is implied that all these works are on domain randomization which is not true. Also one work (Chebotar et al) is not relevant as from what I recall there was no visual gap. Finally most of these works deal with much more complex visual gaps so sample complexity is hard to be compared.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HyxtTsDjKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\nThis paper proposed a new method for zero-shot transfer learning under the reinforcement learning setting. The use of attention weights to regularize the latent states was fairly interesting.\n\nCons:\nLimited applicability of the proposed methods\n- The paper was restricted in a setting where rewards, actions, and true states were identical between source and target environments, and only the observed states differed due to differing renderers. Working under such a restricted setting was interesting in its own right, but it might also lead to limited applicability of the proposed method in the real-world setting.\n- The proposed method focused on solving a very specific problem: learning a dis-entangled latent representation for images. As a result, the potential impact of the proposed methods could be minimal.\n\nLimited technical novelty\n- The proposed method, SADALA, was built on top of Higgins et al., 2017 (DARLA). The only difference was an added attention layer to the learning of latent states. As a result, the novelty of the proposed method was very incremental and limited from a technology perspective.\n- Even with additional attention layer, the paper could have performed a more thorough study to help the readers understand and appreciate the idea. For example, this paper didn’t discuss the tradeoff between training SADALA over separate stages, versus training it from end to end. For example, why the weights of the pre-trained beta-VAE had to be frozen and used as weights in the state representation stage.\n\nInsufficient experiments\n-More thorough discussion of the qualitative results should be helpful to understand whether the attention weights helped the model to focus on the right thing. For example, this paper did study the quality of reconstruction in Figure 3-5 of the proposed method. When comparing Figure 3 and Figure 5, it appeared to me that the reconstructed the angle of the pole was different from the original one. And it seemed like attention weights did successfully ignored the color of the cart and pole, but it ignored the angle of the pole, which should be important to the learning task. Unfortunately, the paper didn't further explain the implication of such misrepresentation.\n\n-Quantitative results \n* It would be interesting to all compare the proposed methods against model-agonistic methods like MAML\n* It would be useful to include confidence intervals over different tasks.\n* It would be useful to compare different methods with different parameter settings\n* The authors mentioned “Visual Pendulum tasks” but didn’t include them in the paper\n\n\nReproducibility\n- It's unclear to me how reproducible the research conducted in this paper was, and it would be useful to open source the code used to conduct the experiments.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1gu5-h6Yr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes adding an attention mechanism to the DARLA beta-VAE approach to transfer learning. The beta-VAE, soft attention and policy are trained on appropriate source tasks and evaluated zero-shot on target tasks, using two more difficult continuous control domains with RGB observations. Results indicate some improvements to compared to the immediate relevant baseline which may be statistically significant, but it is not clear whether over 10% in practice. \n\nI cannot at this stage recommend acceptance for the following reasons:\n1) The paper augments an existing method with a well understood attention mechanism, so the novelty of the approach is relatively low.\n2) The experimental results are interesting, but I don't find them compelling enough to recommend acceptance based on the results alone.  The paper does not solve a major problem with the approach it is based on. In fact, the improvement seems to be smaller when the environment is more complex.\n3) Several baselines which are cited in the paper are actually missing in the experiments, so it is hard to determine how important is that roughly 10% improvement compared to SOTA.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method for zero-shot policy transfer in RL. The authors propose learning the policy over a disentangled representation that is augmented with attention. Hence, the paper is a simple modification of an existing approach (DARLA). The reviewers agreed that the novelty of the proposed approach and the experimental evaluation are limited. For this reason I recommend rejection.
Decision: Reject