Cross-Modal Domain Adaptation for Reinforcement Learning

To overcome the unbearable reinforcement training of agents in the real-world , the sim-to-real approach , i.e. , training in simulators and adapting to target environments , is a promising direction . However , crafting a delicately simulator can also be difficult and costly . For example , to simulate vision-based robotics , simulators have to render high-fidelity images , which can cost tremendous effort . This work aims at learning a cross-modal mapping between intrinsic states of the simulator and high-dimensional observations of the target environments . This cross-modal mapping allows agents trained on the source domain of state input to adapt well to the target domain of image input . However , learning the cross-modal mapping can be ill-posed for previous same-modal domain adaptation methods , since the structural constraints no longer exist . We propose to leveraging the sequential information in the trajectories and incorporating the policy to guide the training process . Experiments on MuJoCo environments show that the proposed crossmodal domain adaptation approach enables the agents to be deployed directly in the target domain with only a small performance gap , while previous methods designed for same-modal domain adaptation fail on this task . 1 INTRODUCTION . Deep Reinforcement Learning ( DRL ) for vision-based robotic-control tasks has achieved remarkable success in recent years ( Francis et al. , 2020 ; Zhang et al. , 2019 ; Zeng et al. , 2018 ; Riedmiller et al. , 2018 ; Levine et al. , 2018 ) . However , current RL algorithms necessitate a substantial number of interactions with the environment , which is costly both in time and money on real robots . An appealing alternative is to train policies in simulators , then transfer these policies onto real-world systems ( Rao et al. , 2020 ; James et al. , 2019 ; Yan et al. , 2017 ) . Due to inevitable differences between simulators and the real world , which is also known as the “ reality gap ” ( Jakobi et al. , 1995 ) , applying policies trained in one domain directly to another almost surely fail , especially in visual-input tasks , due to the poor generalization of RL polices ( Cobbe et al. , 2019 ) . Domain adaptation is a common way to improve transferability by mapping inputs from two domains to an aligned distribution . Although distribution alignment is difficult with limited data , many recent works have adopted unsupervised visual domain adaptation ( Hoffman et al. , 2018 ; Ganin et al. , 2017 ; Yi et al. , 2017 ; Kim et al. , 2017 ) to learn the mapping function without a groundtruth pairing . These adaptation methods exploit structural constraints ( Fu et al. , 2019 ) in two samemodal domains ( i.e. , learned on simulated images and deployed on real images ) to overcome the intrinsic ill-posedness of distribution matching as shown in Fig . 1 ( a ) — mapping an instance in the target domain to anything of a similar probability in the source domain is “ reasonable ” if we only consider distribution matching . However , training on simulated images introduces unwanted costs and difficulties , which are ignored in current works . First , a rendering engine needs more human engineering and runs much slower ( can be up to 20× slower according to Xia et al . ( 2018 ) ) than a pure rigid body simulator , which adds considerable cost to the overall process . Second , using RL methods to train a policy with image inputs is usually harder than training with state inputs ( Kaiser et al. , 2020 ; Tenenbaum , 2018 ) , resulting in a sub-optimal simulation policy . An ideal solution to avoid such problems is to train policies with simulated states and adapt the learned polices to real-world images . However , all the structural constraints based on the modality consistency can not be used and the distribution alignment task by learning a mapping function becomes hard to solve . In this paper , we propose Cross-mOdal Domain Adaptation with Sequential structure ( CODAS ) that learns a mapping function from images in the target domain to states in the source domain . With the help of the learned mapping function , policies trained on states in the source domain can be deployed in the target domain of images directly . Specifically , based on the sequential nature of reinforcement learning problems , we formulate the cross-domain adaptation problem as a sequential variational inference problem and derive a series of solvable optimization objectives in CODAS . It is worth noting that our work is different from recent works that learn state embeddings from image inputs , which map images to an arbitrary subspace in a low-dimensional vector space . In CODAS , we embed the image space into a vector space with clear meanings ( defined in the statebased simulator ) , which improves the interpretability of the policy when deployed in the real world . We evaluate our method on 6 MuJoCo ( Todorov et al. , 2012 ) environments provided in OpenAI Gym ( Brockman et al. , 2016 ) , where we treat states as the source domain , and rendered images as the target domain , respectively . Experiments are conducted in the scenario where only offline real data are available . Experiment results show that the mapping function learned by our method can help transfer the policy to target domain images with a small performance degradation . Previous methods that use unaligned Generative Adversarial Networks ( GANs ) suffer from a severe performance degradation on this cross-modal transfer problem . The experiments provide an optimistic result which indicates cross-modal domain adaptation can serve as a low-cost Sim2Real approach . 2 RELATED WORK . To our best knowledge , this work is the first to address cross-modal domain adaptation in RL setting . We will discuss two research areas closely related to this topic , which are , ( 1 ) unsupervised visual domain adaptation in RL and ( 2 ) image-input representation learning in RL . 2.1 VISUAL DOMAIN ADAPTATION IN RL . Unsupervised visual domain adaptation aims to map the source domain and the target domain to an aligned distribution without pairing the data . Prior methods fall into two major categories : featurelevel adaptation , where domain-invariant features are learnt ( Gopalan et al. , 2011 ; Caseiro et al. , 2015 ; Long et al. , 2015 ; Ganin et al. , 2017 ) , and pixel-level adaptation , where pixels from a source image used to generate an image that looks like one from the target domain ( Bousmalis et al. , 2017 ; Yoo et al. , 2016 ; Taigman et al. , 2017 ; Hoffman et al. , 2018 ) . Pixel-level adaptation is challenging when data from two domains are unpaired . Prior works tackle this problem by using GANs ( Goodfellow et al. , 2014 ) conditioned on simulated images to generate realistic images . Gamrian & Goldberg ( 2019 ) transfers policies from Atari Games ( Bellemare et al. , 2015 ) to modified variants by training a GAN to map images from the target domain to the source domain . GraspGAN ( Bousmalis et al. , 2018 ) addresses domain adaptation in robotic grasping by having the GAN reproduce the segmentation mask for the simulated image as an auxiliary task , including the robot arm , objects , and the bin . RCAN ( James et al. , 2019 ) adopts ideas from domain randomization by learning a mapping of images from randomized simulations to a canonical simulation and treating the real world just as one of the random simulations . RL-CycleGAN ( Rao et al. , 2020 ) unifies the learning of a CycleGAN ( Zhu et al. , 2017 ) and an RL policy , claiming better performance by learning features that are most crucial to the Q-function in RL . Image-to-image domain adaptation can somewhat bypass the ill-posedness for distribution matching ( See Fig . 1 ( a ) ) since it often enjoys an implicit advantage that images differ locally , in color , textile , lighting , but resembles globally between two domains , while images and states differ essentially . Some works impose extra structural constraints ( e.g. , segmentation , geometry ) ( Fu et al. , 2019 ; Bousmalis et al. , 2018 ) , but such tricks fail in image-to-state domain adaptation either . In this work , we force the mapped states to follow transition consistency by using a recurrent structure ( See Fig . 1 ( b ) ) and to be able to recover the pre-learned policy . We also employ a stochastic mapping function with the help of a variational encoder that is more robust to target domain data noise . 2.2 REPRESENTATION LEARNING IN RL . Representation learning aims to transform high-dimensional data into lower-dimensional vector representations , which suit RL better . It is widely accepted that learning policies from states ( embeddings ) is significantly more sample-efficient than learning from pixels , both empirically ( Kaiser et al. , 2020 ; Tenenbaum , 2018 ; Tassa et al. , 2018 ) and theoretically ( Jin et al. , 2020 ) . Sequential auto-encoder is a common network structure to learn state representations by minimizing reconstruction loss . Early works on DRL from images ( Ha & Schmidhuber , 2018 ; Lange et al. , 2012 ; Lange & Riedmiller , 2010 ) use a two-step learning process where an auto-encoder is first trained to learn a low-dimensional representation , and subsequently a policy or model is learned based on this representation . Later works on model-based RL improve representation learning by jointly training the encoder and the dynamics model end-to-end ( Watter et al. , 2015 ) – this has been proved effective in learning useful task-oriented representations . PlaNet ( Hafner et al. , 2019 ) learns a hybrid of stochastic and deterministic latent state models using a reconstruction loss . SOLAR ( Zhang et al. , 2019 ) combines probabilistic graphic models with a simple network structure to fit local linear transitions . Some recent works adopt advancements in unsupervised representation learning . CURL ( Laskin et al. , 2020b ) utilizes contrastive learning methods to capture essential information in an image that distinguishes from others , though later works ( Laskin et al. , 2020a ; Kostrikov et al. , 2020 ) point out that data augmentation may play the major role here . Our work utilizes a sequential variational encoder structure to capture sequential information from trajectories . The main difference between our work and representation learning is whether the state space is predefined . We add extra supervised information to guide the training of the mapping by minimizing the distance between the distributions of the mapped states and the original states , and by enforcing the policy to recover the actions from the mapped states . As a result , we successfully learn states that match the ground-truth simulator states well . The mapped states can be directly fed into the pre-trained policy network . 3 CROSS-MODAL DOMAIN ADAPTATION WITH SEQUENTIAL STRUCTURE . Our work follows the problem setting similar to previous methods that tackle visual domain adaptation problems in RL . We have a policy π pre-trained in the source domain ( state ) and a dataset pre-collected in the target domain ( image ) . The task is to learn a mapping qφ from images to states . In the deployment , agents interact with a new policy π̃ ( o ) = ( π ◦ qφ ) ( o ) , where ◦ denotes function composition . During the training of the mapping function , only the source domain is accessible . This section is organized as follows . Sec . 3.1 formulates the cross-modal domain adaptation as a variational inference problem . Sec . 3.2 decomposes the variational inference problem into several feasible optimization objectives . Sec . 3.3 proposes a residual network structure to handle the complex long-horizon training of the sequential structure .

Summary: The paper proposes a new approach for performing cross-modal domain adaptation, i.e. adapting a policy trained with inputs from modality A (eg low-dimensional environment state) to work with inputs from domain B (eg images). The main use case demonstrated in the paper is the adaptation of policies trained on states in a simulator to work on image inputs, which can be useful for eg real world deployment where states might not be available. While it is a very classic approach to separately train a perception module images --> state (eg in robotics), the main novelty of the presented method is, that this mapping can be learned without the need for paired [image, state] data.