Cross-modal Domain Adaptation for Cost-Efficient Visual Reinforcement Learning

1 Introduction . Reinforcement learning ( RL ) for vision-based robotic control tasks has achieved remarkable success in recent years [ 1 , 2 ] . However , current RL algorithms necessitate a substantial number of interactions with the environment , which are costly both in time and money on real robots . An appealing alternative is to train policies in simulators , then transfer these policies to real-world systems [ 3 ] . Due to inevitable differences of representation between simulators and the real world , which is also known as the “ reality gap ” [ 4 ] , applying policies trained in one domain directly to another almost surely fail , especially in visual-input tasks , which is due to the poor generalization of RL policies [ 5 ] . Domain adaptation is a promising direction to handle the gap by mapping representation from two domains to an aligned representation and then training and deploying policies in the aligned representation . Many recent works , which learn a mapping function to align the data distributions of the two domains , have adopted unsupervised visual domain adaptation [ 6 , 7 , 8 ] . We point out that , as illustrated in Fig . 1 ( a ) , the objective is ill-posed for learning a correct mapping function . These adaptation methods exploit structural constraints [ 9 ] in two domains of the same modality ( e.g. , learned on simulated images and deployed on real images ) . These methods implicitly alleviate the intrinsic ill-posedness of distribution matching . ∗Equal Contribution †Corresponding Author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , such a kind of same-modal domain adaptation requires the simulator to render images when training the model , which introduces unwanted costs and difficulties that are ignored in previous works . First , building a rendering engine is a laborious task . Second , using RL methods to train a policy with an image-based simulator is usually harder [ 10 ] and slower ( can be up to 20× slower [ 11 ] ) than with a state-based simulator . An ideal solution to these problems is to train policies with states in the simulator and adapt the learned policies to real-world images . Current domain adaptation methods generally fail in this setting since the structural constraints based on the modality consistency are no longer available , which makes the representation alignment task harder . To learn such a cross-modal mapping , we propose Cross-mOdal Domain Adaptation with Sequential structure ( CODAS ) that learns a mapping function from images in the target domain to states in the source domain , as illustrated in Fig . 1 ( b ) . With the help of the learned mapping function , policies trained on states in the source domain can be deployed in the target domain of images directly . Specifically , based on the sequential nature of RL problems , we formulate the cross-domain adaptation problem as a variational inference problem and decompose it into a series of solvable optimization objectives . We also design a special residual model structure in the recurrent neural network ( RNN ) to enforce additional inductive bias and stabilize the training process . We evaluate our method on six MuJoCo [ 12 ] tasks and four Robot Hand Manipulation tasks [ 13 ] , where we treat states as the source domain , and rendered images as the target domain . Results show that the learned mapping function can help transfer the policy to the target domain with only a small performance degradation in most of the tasks while existing methods [ 14 , 15 ] suffer from a larger performance gap . 2 Related Work . Unsupervised visual domain adaptation ( UDA ) aims to map the source domain and the target domain to an aligned distribution without pairing the data . UDA is originally designed for image translation in computer vision [ 14 , 6 , 16 , 17 ] . In RL , UDA transfers policies from simulators to the real world , to overcome the reality gap between rendered images and real-world images . Previous methods fall into two major categories : feature-level adaptation , where domain-invariant representations are learned [ 7 ] , and image-to-image adaptation , where pixels from source images are used to generate images that look like those from the target domain [ 18 ] . Feature-level adaptation usually adopts domain randomization techniques for domain-invariant representation learning [ 19 , 20 , 21 , 22 , 23 ] . In domain randomization , a meta-simulator is required to generate a mass of variants on rendered images . The learner aims to extract an invariant representation from the variants of representations . Therefore , the training process is generally costly . Besides , these methods implicitly assume that the variants can cover the target domain so that the target domain is only an instance of the variants . Image-to-image adaptation is challenging when data from two domains are unpaired . Most previous works attempt to solve this problem by using generative adversarial networks ( GANs ) [ 15 , 24 , 3 , 17 , 25 , 26 ] . [ 24 ] transfers policies from Atari games to their modified variants by training a GAN to map images from the target domain to the source domain . RL-CycleGAN [ 3 ] unifies the learning of a CycleGAN [ 14 ] and an RL policy , claiming better performance by learning features that are most crucial to the Q-function in RL . Despite the success of the image-to-image domain adaptation methods in previous settings , we point out that these methods can somewhat bypass the ill-posedness for distribution matching , which is illustrated in the example in Fig . 1 . Since images generally differ only locally in color , textile , and lighting , but resemble globally between two domains , the effect of ill-posedness can be implicitly handled by leveraging the consistency of modality [ 17 , 25 , 26 ] . However , since images and states differ essentially , we can no longer utilize the consistency of modality . Thus , as shown in our experiments , the effect of ill-posedness is revealed in RL tasks . Some works impose extra structural constraints [ 9 , 27 ] but fail in image-to-state domain adaptation either . In our work , we force the mapped states to follow transition consistency by considering the inner relationship between the sampled data from the decision-making process in RL tasks . 3 Cross-modal Domain Adaptation with Sequential Structure . Without loss of generality , the visual domain adaptation problem in RL can be formulated as learning a mapping function to align the representation of the source and target domains , then training and deploying policies in the aligned representation . To learn the mapping function and policies , a dataset is pre-collected by some policies ( e.g. , human-expert policies or random policies ) in the target domain , and an agent can only interact with the environment in the source domain . Since our major concern is the reality gap between the representation spaces , in this formulation , we follow the setting in previous methods which omits the gap between the dynamics models in the two domains . In our setting of cross-modal domain adaptation , the source domain is a low-dimensional state domain and the target domain is a high-dimensional image domain . In this work , we regard the state space as the aligned representation space . We first pre-train a policy π in the source domain . By learning a mapping qφ , parameterized by φ , from image space O to state-space S , the agent can be deployed with a new policy π̃ ( o ) = ( π ◦qφ ) ( o ) , where ◦ denotes function composition , i.e. , image observations are mapped to states , then the pre-trained policy directly acts on the states . 3.1 Domain Adaptation in RL as Variational Inference . In this section , we first define our objective of domain adaptation in RL based on the framework of variational inference . Then we give a comparison between our objective and the objectives used in previous methods . The modeling of the generation and inference process of the decision process in the source and target domains used in our method is illustrated in Fig . 2 . In the source domain , an initial state follows the distribution p ( s1 ) . A transition function st ∼ p ( s | st−1 , at−1 ) outputs the current state st from the previous state st−1 and the previous action at−1 . In the target domain , the generation process also follows the transition of states , but the states are not available to the agent , and the agent only observes the images corresponding to the states . Ideally , an image observation ot is generated by the corresponding state st . In other words , there exists a decoder p ( ot | st ) that can construct observations from states . It is because that the state contains full of information for a transition function to compute the next state , while image observations are rendered based on the states . However , there would have some irrelevant patterns in images in real-world applications . In practice , our decoder is defined as p ( ot | st , ot−1 ) . We allow pθ ( ot | · ) to be dependent on ot−1 so that some irrelevant patterns in images can be decoded easily in an auto-regressive manner . The inference process can be regarded as the mapping process in domain adaptation in RL . The inference function is defined as qφ ( st | st−1 , at−1 , ot ) , which is similar to the transition function but with additional ot as an input . A simple insight is that the inputs st−1 and at−1 give information to predict the distribution of st , as the transition function p ( s | st−1 , at−1 ) does , and ot is used to determine the state sample st from the distribution . The training process aims at learning an inference function ( i.e. , the mapping function ) that approximates the ground-truth posterior distribution . The raw optimization objective of variational inference in this setting can be formulated as : min φ Eτo [ DKL [ qφ ( τs | τo ) || p ( τs | τo ) ] ] , ( 1 ) where τ denotes a trajectory , the superscripts s and o of τ indicate the trajectories from the source and target domains respectively , p ( · ) is the ground-truth distribution , qφ ( · ) is the mapping function that we want to learn , and DKL computes the Kullback-Leibler divergence . τo contains a trajectory of observation-action pairs { ( o1 , a1 ) , ( o2 , a2 ) , ... , ( oT , aT ) } , and τs contains a trajectory of state-action pairs { ( s1 , a1 ) , ( s2 , a2 ) , ... , ( sT , aT ) } . Here we assume the trajectories of source and target domains are collected by the same policy π , which is a mild assumption and has been implicitly or explicitly introduced in previous works [ 24 ] . Since the policy is fixed , for the simplification of notations , we omit the dependence of π and just use qφ and p to indicate the distribution . The Evidence Lower BOund ( ELBO ) of this variational problem can be formulated as : max φ , θ Eτo [ Eτ̂s∼qφ ( τs|τo ) [ log pθ ( τ o | τ̂s ) ] −DKL [ qφ ( τs | τo ) || p ( τs ) ] ] , ( 2 ) where τ̂s denotes the inference trajectories of qφ , and pθ ( τo | τ̂s ) is an approximation of the generation process p ( τo | τs ) , with parameter θ . The derivation of ELBO can be found in Appendix A . The first term maximizes the reconstruction probability , in order to enforce that the mapped state ŝ can recover observation o . The second term enforces the alignment of the distributions of the mapped trajectories qφ ( τs | τo ) and the ground-truth trajectories distribution p ( τs ) in the source domain . Compared with previous works which learn the mapping function by minimizing the divergence between the distributions of qφ ( s | o ) and p ( s ) , we model the optimization objective as a matching of trajectory distributions via the Bayesian graphical model . As illustrated in Fig . 1 , the trajectory-level modeling is important since the data points are not i.i.d. , since they are collected in the RL scenario . Without considering the inner structure of the dataset , mismatching of mapping might occur easily .

Summary: This paper studies the problem setting in which one has a "source" simulator with low-level state information in which one can efficiently train an agent to perform a task, and a "target" simulator that shares the initial state distribution and transition dynamics as the source simulator but only offers high-dimensional observations. The goal is to leverage trajectories sampled from a policy in the target domain to learn a mapping $q_\phi$ from the stream of high-dimensional observations in the target domain to states in the source domain. One can then perform the task in the target domain by feeding observations through $q_\phi$ and then through the source policy. The authors propose to use variational inference to train $q_\phi$ and thoroughly demonstrate what adaptation tasks their framework (CODAS) can and cannot perform on MuJoCo and Hand Manipulation. Impressively, using CODAS, the model in the target domain generally achieves 80-90% the performance of the model in the source domain.