Cross-Modal Domain Adaptation for Reinforcement Learning
Authors: Xiong-Hui Chen
Shengyi Jiang
Feng Xu
Yang Yu
Venue:  (2021)
Keywords: Domain Adaptation
Reinforcement Learning

Abstract: Domain adaptation is a promising direction for deploying RL agents in real-world applications, where vision-based robotics tasks constitute an important part. Cur-rent methods that train polices on simulated images not only require a delicately crafted simulator, but also add extra burdens to the training process. In this paper, we propose a method that can learn a mapping from high-dimensional images to low-level simulator states, allowing agents trained on the source domain of state input to transfer well to the target domain of image input. By fully leveraging the sequential information in the trajectories and incorporating the policy to guide the training process, our method overcomes the intrinsic ill-posedness in cross-modal domain adaptation when structural constraints from the same modality are unavailable. Experiments on MuJoCo environments show that the policy, once combined with the mapping function,  can be deployed directly in the target domain with only a small performance gap, while current methods designed for same-modal domain adaptation fail on this problem.

Under review as a conference paper at ICLR 2021
CROSS-MODAL DOMAIN ADAPTATION FOR REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
To overcome the unbearable reinforcement training of agents in the real-world, the sim-to-real approach, i.e., training in simulators and adapting target environ- ments, is a promising direction. However, crafting delicately simulator can also be difficult and costly. For example, simulate vision-based robotics, simulators have to render high-fidelity images, which can cost tremendous effort. This work aims at learning cross- cross-modal modal mapping between intrinsic states of the simulator and high- -igh-dimensionll observations of the target environments. This modal mapping allows agents trained on the source domain of state input to adapt well to the target domain of image input. However, learning the cross-modal mapping can be ill-posed for previous same-modal domain adaptation methods, since the structural constraints no longer exist. We propose to leveraging the sequential information the trajectories and incorporating the policy to guide the training process. Experiments modal domain adaptation approach enables the agents to be deployed directly in the target domain with only a small performance gap, while previous methods designed for same-modal domain adaptation fail on this task.
MuJoCo environments show that the proposed cross-
INTRODUCTION
Deep Reinforcement Learning (DRL) for dision-based robotic-control tasks has achieved remark- able success in recent years (Francis et al., 2020; Zhang et al., 2019; Zeng et al., 2018; Riedmiller al., 2018; Levine al., 2018). However, current RL algorithms necessitate substantial number of interactions with the environment, which is costly both in time and money on real robots. An appealing alternative is to train policies in simulators, then transfer these policies onto real-world systems (Rao et al., 2020; James et al., 2019; Yan et al., 2017).
Due to inevitable differences between simulators and the real world, which also known as the "reality gap" (Jakobi et al., 1995), applying policies trained in one domain directly another almost surely fail, especially in visual-input tasks, due to the poor generalization of RL polices (Cobbe ct al., 2019). Domain adaptation is common way improve transferability by mapping inputs from two domains to an aligned distribution. Although distribution alignment is difficult with limited data, many recent works have adopted unsupervised visual domain adaptation (Hoffman al., 2018; Ganin et al., 2017; Yi et al., 2017; Kim et al., 2017) to learn the mapping function without a ground- truth pairing. These adaptation methods exploit structural constraints (Fu et al., 2019) in same- modal domains (i.e., learned on simulated images and deployed on real images) to overcome the intrinsic ill poseddddsssss of distribution matching as shown Fig. 1(a) mapping an instance in the target domain to anything of similar probability in the source domain is "reasonable" if we only consider distribution matching.
However, training on simulated images introduces unwanted costs and difficulties, which are ignored in current works. First, a rendering engine needs more human engineering and runs much slower (can be up 20x slower according to Xia et al. (2018)) than pure rigid body simulator, which adds considerable cost to the overall process. Second, using RL methods to train a policy with image inputs is usually harder than training with state inputs (Kaiser et al., 2020; Tenenbaum, 2018), resulting in a sub-optimal simulation policy. An ideal solution to avoid such problems is to train policies with simulated states and adapt the learned polices real-world images. However, all

---

Under review as a conference paper at ICLR 2021
(a) Mapping without Sequential Structure
(b) Mapping with Sequential Structure
Figure I: Illustration of mapping functions with and without sequential structure from the target domain (left) to the source domain (right). Shaded regions denote data distributions, where the darker the color, the higher the probability. In Fig. I(a), both S; and s/ are "realisic" source domain instances, but only 8, correct. Since they are of similar probabilities, distribution matching may map o, to any of them. In RI, the policy may output unreliable actions when taking these incorrectly mapped states as inputs. In Fig. 1(b), sequential structure can help rule out the wrong mapping based on trajectory contexts.
the structural constraints based on the modality consistency can not be used and distribution alignment task by learning mapping function becomes hard to solve.
In this paper, we propose Cross-mOdal Domain Adaptation with Sequential structure (CODAS) that learns mapping function from images in the target domain to states in the source domain. With the help the learned mapping function, policies trained on states in the source domain can be deployed in the target domain of images directly. Specifically, based the sequential nature of reinforcement learning problems, we formulate the cross-domain adaptation problem as a sequential variational inference problem and derive series of solvable optimization objectives in CODAS. is worth noting that our work is different from recent works that learn state embeddings from image inputs, which map images CODAS, we embed the image space into vector space with clear meanings (defined in the state- based simulator), which improves the interpretability of the policy when deployed in the real world.
arbitrary subspace in low-dimensional vector space. In
We evaluate our method on 6 MuJoCo (Todorov et al., 2012) environments provided in OpenAI Gym (Brockman al., 2016), where we treat states as the source domain, and rendered images as the target domain, respectively. Experiments are conducted in the scenario where only offline real data are available. Experiment results show that mapping function learned by our method can help transfer the policy to target domain images with small performance degradation. Previous methods that use unaligned Generative Adversarial Networks (GANs) suffer from a severe perfor- mance degradation this cross moosl transfer problem. The experiments provide optimistic result which indicates cross-modal domain adaptation can serve as a low-cost Sim2Real approach.
2 RELATED WORK
To our best knowledge, this work is the first to address cross cross-modal domain adaptation in RL setting. We will discuss two research areas closely related to this topic, which are, (1) unsupervised visual domain adaptation in RL and (2) image-input representation learning in RL.
2.1 VISUAL. DOMAIN ADAPTATION IN RI.
Unsupervised visual domain adaptation aims to map the source domain and the target domain to an aligned distribution without pairing the data. Prior methods fall into two major categories: feature- level adaptation, where domain-ing invariant features are learnt (Gopalan et al., 2011; Caseiro et al,, 2015; Long al., 2015; Ganin et al., 2017), and pixel-level adaptation, where pixels from source image used to generate an image that looks like one from the target domain (Bousmalis et al., 2017; Yoo et al., 2016; Taigman et al., 2017; Hoffman et al., 2018).
Pixel-level adaptation is challenging when data from two domains are unpaired. Prior works tackle this problem by using GANs (Goodfellow et al., 2014) conditioned on simulated images generate
2

---

Under review as a conference paper at ICLR 2021
realistic images. Gamrian & Goldberg (2019) transfers policies from Atari Games (Bellemare al,, 2015) to modified variants by training a GAN to map images from the target domain to the source domain. GraspGAN (Bousmalis et al., 2018) addresses domain adaptation in robotic grasping by having the GAN reproduce the segmentation mask for the simulated image as an auxiliary task, including the rohot arm, objects, and the bin. RCAN (James al., 2019) adopts ideas from do- main randomization by learning a mapping of images from randomized simulations to canonical simulation and treating the real world just as one of the random simulations. RL-CycleGAN (Rao et al., 2020) unifies the learning of a CycleGAN (Zhu et al., 2017) and an RL policy, claiming better performance by learning features that are most crucial to the Q-function in RL.
Image-to-image domain adaptation can somewhat bypass the ill posedness for distribution matching (See Fig. ((a)) since it often enjoys an implicit advantage that images differ locally, in color, textile, lighting, but resembles globally between two domains, while images and states differ essentially. Some works impose extra structural constraints (e.g., segmentation, geometry) (Fu al., 2019; Bousmalis et al., 2018), but such tricks fail in image-to-state domain adaptation either. In this work, we force the mapped states to follow transition consistency by using a recurrent structure (See Fig. 1(b)) and to be able to recover the pre-leerned policy. We also employ stochastic mapping function with the help of a variational encoder that is more robust to target domain data noise.
2.2 REPRESENTATION LEARNING IN RL
Representation learning aims to transform high-dimensional data into lower-dimensional vector rep- resentations, which suit RL better. It is widely accepted that learning policies from states (embed- dings) is significantly more sample-efficien than learning from pixels, both empirically (Kaiser et al., 2020; Tenenbaum, 2018; Tassa et al., 2018) and theoretically (Jin et al., 2020).
Sequential auto-en is a common network structure to learn state representations by minimizing reconstruction loss. Early works on DRL from images (Ha & Schmidhuber, 2018; Lange et al., 2012; Lange & Riedmiller, 2010) use a two-step learning process where an auto-encoder is first trained to learn a low-dimensional representation, and subsequently a policy or model learned based on this representation. Later works on model-based RL improve representation learning by jointly training the encoder and the dynamics model end-to-end (Watter et al., 2015) this has been proved effective in learning useful task-oriented representations. PlaNet (Hafner et al., 2019) learns hybrid of stochastic and deterministic latent state models using reconstruction loss. SOLAR (Zhang et al., 2019) combines probabilistic graphic models with simple network structure to fit local linear transitions. Some recent works adopt advancements in unsupervised representation learning- CURL (Laskin et al., 2020b) utilizes contrastive learning methods to capture essential information in an image that distinguishes from others, though later works (Laskin et al., 2020a; Kostrikov et al., 2020) point out that data augmentation may play the major role here.
Our work utilizes a sequential variational encoder structure to capture sequential information from trajectories. The main difference between work and representation learning is whether the state space is predefined. We add extra supervised information to guide the training of the mapping by minimizing the distance between the distributions of the mapped states and the original states, and by enforcing the policy to recover the actions from the mapped states. As a result, we successfully learn states that match the ground-truth simulator states well. The mapped states can be directly fed into the pre-trained policy network.
3 CROSS-MODAL DOMAIN ADAPTATION WITH SEQUENTIAL STRUCTURE
Our work follows the problem setting similar to previous methods that tackle visual domain adap- tation problems in RI. We have policy æ pre-trained in the source domain (state) and a dataset pre-collected in the target domain (image). The task is to learn mapping 96 from images to states. In the deployment, agents interact with a new policy i(o) = (I o (o). where denotes function composition. During the training of the mapping function, only the source domain is accessible.
This section is organized as follows. Sec. 3.1 formulates the cross- cross-modal domain adaptation as variational inference problem. Sec. 3.2 decomposes the variational inference problem into sev- eral feasible optimization objectives. Sec. 3.3 proposes a residual network structure to handle the complex long-horizon training of the sequential structure.
3

---

Under review as a conference paper at ICLR 2021
1
OT
S1
ST
S1
S2
ST
a1
Ry-1
Q1
Ry-1
(a) generation process of real world
(b) generation process of simulation
Figure 2: Illustration of the generation in the real world and simulation domains respectively. All nodes random variables. Shaded nodes are observable variables. The solid line denotes the gen- eration process and dashed lines denote the inference process. There is an unpaired correspondence between state trajectories surrounded by the rounded rectangle. Note that include policy in both generation process, which corresponds to the edge from state S; to action aç.
3.1 DOMAIN ADAPTATION AS VARIATIONAL INFERENCE
The goal of domain adaptation is to find a mapping from the target domain to the source domain, which is images to states in our case. We first model the generation process of the target domain and source domain respectively and the connection between them, as illustrated in Fig. 2. The initial state follows distribution p(s1). ). The transition function (s St-1, St-1, modeled as a feed-forward neural network with parameters
predicts the current state from the previous state and previous action. The decoder (o St, 01,1, 0t-1), modeled as deconvolution network with parameters 0, reconstructs the current observation from the current state, previous observation and previous action. In practice, we model these distributions as multivariate Gaussian distributions. We allow pe(o; 1) dependent 0r-1 that some irrelevant patterns in images can be reconstructed in an rege-segressive manner. Such a conditional distribution feasible practice since we always have ground-truth 00:-11 at timestep in both training and deployment phases. A sequential generation process enjoys an extra benefit of solving Partially Observable Markov Decision Process (POMDP) since single image not reveal the full state of the environment in general.
The key point that distinguishes our method from conventional representation learning is the ad- ditional constraint that the mapped state trajectories should match the simulation state trajectories. See rounded rectangle parts Fig. 2. Such a constraint may somewhat require that the underlying transition dynamics the same two domains. Introducing distributions the assumption and increase robustness when there is a small mismatch between dynamics in source and target.
some extent relax
$1 ~110(11)
8; ps ((-1,90-1) o ~ (o, | (t-1,
(1)
The overall optimization problem can be formulated as variational inference problem described in Eq. 2, where want learn posterior distribution to approximate the ground-truth distribution.
min E,r [DKL. [go(7" |>") || p(7" |r"]]]
(2)
where 7 is a trajectory, 8 and r indicate whether the trajectory is from the source or target domain, p(-) is the ground- -ruth distribution, (-) the mapping function we want learn, and D computes the Kullback-Leiblee matching can naturally handle the stochasticity of environments and policies, and the possible noise in data collected in the real world, The Evidence Lower Bound (ELBO) of this variational problem can be formulated as:
divergence. Modeling the optimization as trajectory distribution
max Eyr (Ega(r+ [logpo (7" [ - DKL. (go (7" [ 11 (7*)]]
The derivation of ELBO follows common practice using Jensen inequality. A detailed derivation can found in Appendix A. The first term maximizes the reconstruction probability, in order to enforce that the mapped states recover both observations and action The second term

---

Under review as a conference paper at ICLR 2021
Algorithm Training Procedure of CODAS
Input: Simulator with oracle dynamics p; Policy (a s) pre-trained in the simulated dynamics
p(s' 5, s,a); Target-domain Trajectory Dataset of images D {(o dgs..., .., of );); Number of
iteration N..
Output: Mapping function : + S. Initialize the sequential mapping function f(st 9,-1, for 1 to N do
Sample a batch of target domain trajectories T² = (10-- oo: ..., )r}} from D
Initialize RNN with a zero state.
Infer the corresponding state trajectories T' = {( .ii........... 8T); via the mapping function f;
Rollout one step with the oracle simulation dynamies p(s' ss a) for each state-action pair
in T to construct the transition dataset D; = {(S; ag $;+1)]; do
for
Update the discriminator D by maximizing Eq. 13; end for
Update the mapping function (details are in Algorithm 2); end for
enforces the alignment of the distributions of the mapped trajectories and the trajectories collected the simulator. The policy т in source domains included the second term does introduce new assumption that target domain data are collected by known behavioral policy. This assumption mild and is implicitly or explicitly introduced in previous works (Kim et al., 2020; Gamrian & Goldberg, 2019).
3.2 DIFFERENTIABLY OPTIMIZATION OBJECTIVES
The ELBO defined in Eq. contains terms involving distributions over the entire trajectory, and thus impractical to solve directly. Given the generation process defined in the previous section, can decompose the joint probability into the multiplication of one-step probabilities. Eq. shows the result after the decomposition. For brevity, we will use Ô denote s, o outputted by networks and omit the networks themselves.
max E.
Eg Bt pe, (o] 0:-1, Sc, a(-1) log pe, (at- (0t-1/st-1))
(4)
DKi (Ts Tx) (1
A direct computation of the second DKI. term is intractable. Following the idea that the optimiza- tion process of GAN equivalent to minimizing certain distance measure between two distri- use the optimization objective of GAN the surrogate
butions (Nowozin et al., 2016), objective of minimizing the DKL [ga (7" [r") || p (r*)]. For implementation simplicity and train- ing stability, still choose to use the original GAN optimization objective, which is equivalent to minimizing JS divergence. The optimization objective is formulated in Eq. 5.
(0,0,w) w) = E (s, a)] + Epr- [log(1 - Dw a))]
(5)
Here, the discriminator takes state, rather than trajectory, as input for better practicality, where "real" samples are simulation states and "fake" samples are mapped states. The latter term is the objective of the generator To in our case. The sequential structure is preserved in the generator: This part is different from previous works like VAR-GAN (Larsen et al., 2016) and Causal InfoGAN (Kurutach et al., 2018), that discriminate against the reconstructed output. Instead, we discriminate against the states after the variational encoder.
The reconstruction of action can be optimized in an end-to-end manner. However, if differ- entiable æ of simulation states available (which the usual case), we can replace pe, with i, resulting objective in Eq. 6. A fixed here can guide pe, output states that yields similar Previous works (Schrittwieser et al., 2019) also discovered that recovering a is helpful in improving the representation learning in RI.
5

---

Under review as a conference paper at ICLR 2021
Algorithm 2 Detailed Training Procedure of Mapping Function
Input: Simulator with oracle dynamics p: Policy xa 8) pre-trained in the simulation dynamics p(s' s,a);;
Initialization.
Pre-train DM t.p with D {(s), a, 8.+1))-
Update per iteration.
Update the the DM hip p by minimizing Eq. 8 using De and DD; Copy the new parameters of DM lyp to the embed-IM in ga: for m
M do
Update the mapping function g by minimizing Eq. 7;
Update the reconstruction function po by minimizing the first term Eq. 7: end for
ppllicy (r"
Dog x (aa [()]
Combining all the aforementioned losses together, i.c, reconstruction loss, policy loss and generation loss. the complete optimization objective of the mapping function is as follows:
T
mmapping D D
log p(o oz) + Ap log(1 - (a:)) - Am log 8x))
t=1
where An and D hyper-parameters. Both the decoder and the policy are fixed during the training process of the mapping function. The loss function for pe is just the first term in Eq. 7. Similarly, the mapping function qo is fixed during the training process the reconstruction function po.
3.3 STATE INFERENCE MODEL WITH EMBEDDED DYNAMICS
Since trajectory of practical RL problems often lasts hundreds or thousands timesteps, training RNN on such a long-horizon trajectory is difficult. The previous success of ResNet (He et al., 2016) shows that a residual structure can simplify the learning target by predicting small residuals, resulting in remarkable performance increase in learning ultra-deep neural networks.
Adopting similar idea, we incorporate residual structure in the RNN to help stabilize the training process. We have Dynamics Model (DM) trained independently using transition tuples collected the simulator. This structure further forces the mapped states to follow the simulation dynamics. Its param- eters periodically updated to embed- DM, in order to provide an "average" estimation of the next states s, (See Fig. 3). The job of the rest part, instead, is simplified to just output a correction. With the pro- posed structure model, the mapped states follow the transition dynamics in the simulator better.
9--2
pertod update
embed
RNN call
As
0
0b-1
DM first trained using batches of transition tuples collected in the simulator. During the training pro- cess of the mapping function, the dynamics model is trained online using D {(1g, ae S1+1 p(s'|š,, } and is periodically updated to the mapping function. That is, we reset the simulator to mapped states &, and then rollout with a one-step oracle simulation transition to get 8 The optimization objective of the dynamics model is an MSE loss in Eq. 8. Algorithm demonstrates detailed training and updating procedure of the dynamics model and the entire mapping function.
Figure Model structure with embed-DM
laynamics E ((s.a,")=D'D (14p (8.a) a) - 2)
(8)

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTS
We evaluate our method 6 MuloCo environments from OpenAl Gym, namely InvertedPendulum, InvereddDoulleenndtuu,, HalfCheetah, Hopper, Swimmer and Walker2d. We define the rendered images as the target domain, and the original observations as the source domain. The pre-collected dataset in the target domain contains 600 trajectories and collected by a sub-optimal policy. Please refer to Appendix C for detailed experiment setting.
We modify state-of-the-arn art methods in same-modal domain adaptation for comparison, namely GAN and CycleGAN. GAN uses the same model structure of the encoding network in CODAS, and is trained on pre-collected state and image datasets. The state dataset of the same size the image dataset. As described in Sec. D.6, CycleGAN fails in all environments. We also compare with Behavioral Cloning (BC) considering target domain data are collected by a (sub)optimal policy. BC trains policy in supervised manner using (of of) Dr. To mitigate the partial observability issue mentioned in Sec. 3, we stack every consecutive images as the new input to the BC algorithm. All methods trained until convergence. Implementation details for all methods can be found in Sec. B. Due to some numerical instabilities' inside MuJoCo, we cannot get the oracle transition function in HalfCheetah and InvertedDoubleeendduumm CODAS in these two environments implemented without embed-DM.
Experiment results in this section will answer the following three questions:
1) Does CODAS enable polices to transfer from states to images? Does CODAS outperform state-of-the-art methods?
2) Does every component CODAS contribute to the overall performance?
3) Is CODAS robust small mismatches in environment dynamics between the source and target domains?
To focus on the performance of the adaptation process, we use reward ratio as the metric, which is defined Tratio ,r, where r and r* are the cumulative return of the adapted policy and the optimal policy trained on states respectively. The quantitative performance of the optimal policy trained on states and rendered images using PPO is given in Sec. D.2.
4.1 PERFORMANCE
Training curves of all methods are shown in Fig. 5, x-axis being training iterations, y-axis being the reward ratio. Each iteration, every method is updated using batch of 20 trajectories. CO- DAS reaches an average of 70% reward ratio after adaptation, providing an optimistic result on future applications of cross-modal domain adaptation. BC performs well in simple environments (HalfCheetah and Inverted Pendulum), but poorly in rest environments, especially in those with an early termination. GAN performs even worse than BC most environments, which suggests
Figure 4: A visual illustration (a) original images (b) reconstructed images (c) re-rendered images by setting the simulator to mapped states.
MuloCo engine outputs different s' given exact same (s, a) as input due to its inner inaccessible random
states.

---

Under review as a conference paper at ICLR 2021
600
802
1000
200
60X
1830
(a) Inverted Pendulum
(h) Inverted Double Pendulum
(c) HalfCheetah
C224
OROAA
GAN
0.
(d) Hopper
(c) Swimmer
(f) Walker2d
Figure Training curves of different methods on cross- cross-modal domain adaptation
mapping images to predefined interpretable and meaningful state space is more difficult than end- to-end policy learning.
A visual illustration of the mapped states accuracy is shown in Fig. 4. Both reconstructed images and re-dendered images match the original ones well. is worth noting that re-rendered images can match the last falling frames, which are sparse in the dataset, well. Quantitative results mapping states errors of GAN and our methods can be found in Sec. D.3.
4.2
ABLATION STUDIES
mm
100on mbh
2000
2000
5000
R000
82:02
12:00
Einestep
(a) Inverted Pendulum
(b) Inverted Double Pendulum
(c) HalfCheetah
2000
1200
2000
1200
202 2018
1100
1008
2010 1390 2110 18025
(d) Hopper
(c) Swimmer
(1) Walker2d
Figure 6: Ablation studies by removing embed-DM and RNN in CODAS
Since CODAS contains multiple ideas, conduct additional experiments to understand the contri- bution of each component to the overall performance. We conduct ablation studies to demonstrate (1) if sequential structure matters in the cross-modal domain adaptation in RL and (2) if embed-DM
8

---

Under review as a conference paper at ICLR 2021
improves the long-horizon inference. Since the embed-DM is based sequential structure, we only test on two ablated variants, i.., CODAS w/o embed-DM. and w/o both embed-DM and RNN.
A comparison of all ablated variants is shown in Fig. 6. Both axes are of the same meanings as above. In all the environments, mapping functions with sequential structure yield better deployment per- formance. In four enviromments where embed-DM is applicable, embed-DM helps further improve the performance. As we expect, the major purpose of incorporating embed-DM is to simplify the learning process. If RNN itself can learn reasonable mapping, the improvement of embed-DM will be marginal (Walker2d). The learning process is also a little unstable in InvertedDoubleeendulnn where embed-DM. is not available.
4.3 ROBUSTNESS TO DYNAMICS MISMATCH
To analyze the CODAS's robustness dynamics mismatch, we modify the dynamics in the source domain. Specifically, we reconfigure the friction coefficient in Hopper. The result can be found in Figure 11. The results in Hopper both with 110% and 120% friction still demonstrate robust mapping. Besides, the mismatch of dynamics even not slow down the speed of learning. The result suggests that the CODAS technique has the potential to be deployed in realistic applications where the dynamics in two domains can not match exactly. We leave this analysis in future works.
5 CONCLUSION
In this work, we propose Cross-Modal Domain Adaptation with Sequential structure (CODAS). CODAS enables a new paradigm of Sim2Real adapting policies trained on simulated states real- world with image inputs. We believe this setting is valuable in real-world applications of RL by exempting us from the tedious work of building and running rendering engines. Previous methods that use GANs fail on this problem since global structural resemblance between two same modal domains no longer exists, while our method succeeds by fully leveraging the sequential structure and other auxiliary information provided by the policy and dynamics underlying RI. problem. We first model the cross-mes modal domain adaptation problem as a variational inference problem and de- compose it into several feasible optimization objectives. To better solve the complex long-horizon sequential mapping problem, we propose residual network structure. We validate the proposed method by adapting policies from images to states various MuJoCo environments. Our results provide an optimistic results of cross- cross-modal modal domain adaptation as a low-cost Sim2Real approach.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning en- vironment: An evaluation platform for general agents (extended abstract). In Qiang Yang and Michael J. Wooldridge (eds.), Proceedings of the 24th International Joint Conference on Artifi- cial Intelligence, pp. 4148 4152, Buenos Aires, Argentina, 2015.
Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 95-104, Honolulu, Hawaii, 2017.
Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakr- ishnan, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, Sergey Levine, and Vincent Van- houcke. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In Proceedings of the IEEE International Conference on Robotics and Automation, pp. 4243- 4250, Brisbane, Australia, 2018
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Rui Caseiro, João F. Henriques, Pedro Martins, and Jorge Batista. Beyond the shortest path: Un- supervised domain adaptation by sampling subspaces along the spline flow. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3846-38866 3854, Boston, Mas- sachusetts, 2015.
Karl Cobbe, Oleg Klimov, Christopher Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 1282-1289, Long Beach, California, 2019.
Anthony G. Francis, Aleksandra Faust, Hao-Tien Lewis Chiang, Jasmine Hsu, Chase Kew, Marek Fiser, and Tsang- Wei Edward Lee. Long-range indoor navigation with PRM-RL. IEEE Trans. Robotics, 36(4): 1115-11344 2020.
Huan Fu, Mingming Gong, Chaohui Wang. Kayhan Batmanghelich, Kun Zhang, and Dacheng Tao. Geometry-consisten generative adversarial networks for one-sided unsupervised domain map- ping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 2427-2436, Long Beach, California, 2019.
Shani Gamrian and Yoav Goldberg. Transfer learning for related reinforcement learning tasks via image-to-image translation. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 1063-2072, Long Beach, California, 2019.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. Domain-adversarial training of neural networks. In Gabriela Csurka (ed.), Domain Adaptation in Computer Vision Applications, pp- 189-209. 2017.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. CoRR, abs/1 4ods/1406.6611,, 2014.
Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Domain adaptation for object recognition: An unsupervised approach. In Proceedings of the IEEE International Conference on Computer Vision, pp. 999-1006, Barcelona, Spain, 2011.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, pp. 2467, Montréal, Canada, 2018
Danijar Hafner, Timothy P. Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 2555-2565, Long Beach, California, 2019.
10

---

Under review as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, Las Vegas, Nevada, 2016.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei A. Efros, and Trevor Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 1994-2003, Stockholm, Sweden, 2018.
Nick Jakobi, Phil Husbands, and Inman Harvey. Noise and the reality gap: The use of simulation in evolutionary robotics. In Proceedings of the Third European Conference on Artificial Life, volume 929, pp. 704-720, Granada, Spain, 1995.
Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data- efficient robotic grasping via celdoomized-t--caoniieeee adaptation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12627-12637, Long Beach, California, 2019.
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I. Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, volume 125, pp. 2137-2143, Virtual Conference, 2020.
Lukasz Kaiser, Mohammad Babacizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model based reinforcement learning for Atari. In Proceedings of the Sth International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020.
Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Domain adaptive imita- tion learning. In Proceedings of the 37th International Conference Machine Learning, Virtual Conference, 2020.
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. In Proceedings of the 34th Interna- tional Conference on Machine Learning, volume 70, pp. 1857-1865, Sydney, Australia, 2017.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. CoRR, abs/2004.13649, 2020.
Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart J. Russell, and Pieter Abbeel. Learning plannable representations with causal infogan. In Advances in Neural Information Processing Systems, pp. 8747-8758, 2018
Sascha Lange and Martin A. Riedmiller. Deep auto-encoder neural networks in reinforcement learning. In Proceedings of the International Joint Conference on Neural Networks, pp. 1-8, Barcelona, Spain, 2010.
Sascha Lange Martin A. Riedmiller, and Ame Voigtländer. Autonomous reinforcement learning on raw visual input data in a real world application. In Proceedings of the International Joint Conference on Neural Networks, pp. 1-8, Brisbane, Australia, 2012.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using learned similarity metric. In Proceedings of the 33rd International Conference on Machine Learning, volume 48, pp. 1558-1566, New York City. New York, 2016.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Re- inforcement learning with augmented data. CoRR, abs/2004. 14990, 2020a.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised representa- tions for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, Virtual Conference, 2020b.
11

---

Under review as a conference paper at ICLR 2021
Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. J. Robotics Res., 37(4-5):421-436, 2018.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pp. 97-105, Lille, France, 2015.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: Training generative neural sam- plers using variational divergence minimization. In Advances in Neural Information Processing Systems, pp. 271-279, Barcelona, Spain, 2016.
Kanishka Rao, Chris Harris, Alex Irpan, Sergey Levine, Julian Iharz, and Mohi Khansari. RL- CycleGAN: Reinforcement learning aware simulation-to-real. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 11154-11163, Seattle, Washington, 2020.
Martin A. Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 1341-4350, Stockholm, Sweden, 2018.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillicrap, and David Silver. Mastering Atari, go, chess and shogi by planning with a learned model. CoRR, abs/1911. 1.00265, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/ 000000770.37.... 2017.
Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsupervised cross-domain image generation. In Proceedings of the 5th International Conference on Learning Representations, Toulon, France, 2017.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A. Riedmiller. Deepmind control suite. CoRR, abs//801.000 086900 2018
Josh Tenenbaum. Building machines that learn and think like people. In Proceedings of the 17th International Conference on Autonomous Agents and MulfiAgent Systems, pp. 5, Stockholm, Sweden, 2018
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for model- control. In Proceedings of the IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033, Vilamoura, Portugal, 2012.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforce- ment learning. CoRR, 0ass/1907.00057 2019.
Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin A. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in Neural Information Processing Systems, pp. 2746-2754 Montréal, Canada, 2015.
Fei Xia, Amir R. Zamir, Zhi- Zhi-Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: real-world perception for embodied agents. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 9068-96889988 9079, Salt Lake City, Utah, 2018
Mengyuan Yan, Iuri Frosio, Stephen Tyree, and Jan Kautz. Sim-to-real transfer of accurate grasping with eye-in-hand observations and continuous control. CoRR, abs/1712.00300. 2017.
Zili Yi, Hao (Richard) Zhang, Ping Tan, and Minglun Gong. DualGAN: Unsupervised dual learning for sange-to-image translation. Proceedings of the IEEE International Conference on Com- puter Vision, pp. 2876, Venice, Italy, 2017.
12

---

Under review as a conference paper at ICLR 2021
Donggeun Yoo, Namil Kim, Sunggyun Park, Anthony S. Pack, and In-So Kweon. Pixel-level do- main transfer. In Proceedings of the 14th European Conference on Computer Vision, volume 9912, pp. 517-532, Honolulu, Hawaii, 2016.
Andy Zeng, Shuran Song, Stefan Welker, Johnny Lee, Alberto Rodriguez, and Thomas A. Funkhouser. Learning synergies between pushing and grasping with self-supervised deep re- inforcement learning. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4238 4245, Madrid, Spain, 2018.
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew J. Johnson, and Sergey Levine. SOLAR: deep structured representations for model-based reinforcement learning. Proceedings of the 36th International Conference Machine Learning, volume 97, pp. 7444-7453, Long Beach, California, 2019.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2242-2252 Venice, Italy, 2017.
13

---

Under review as a conference paper at ICLR 2021
A DERIVATION OF OPTIMIZATION OBJECTIVES
Assume the distribution of trajectories in the simulator p" (Tx) is calculated from the real world trajectories Ty. We want match %% (Ta(Ty) to the ground-truth distribution p (Ta/TT)) leading the optimization objective:
min E, (DL 10 (Ta (1 (Ta/T,)]]
To maximize the objective, we first transform it into the Evidence Lower Bound (ELBO):
DKL
(Ta/Tr)]
Rog (n.) log 2 (T,(TT))
Pd (Tss Te)
(rg In.) flog 4 (TalTy) log p= (Tr)
= (r. log 18 (TalT,) - log pă (Tr, Tx)] + log pi T,)
The last term p (T.) a constant with regards to the parameter and thus can be ignored in the optimization process. The minimization can be reduced tomin .E,- (T.1(1))))) (n))ooo o (T,IT,) log p (Tr, J.]] Since sampled trajectories can considered as i.i.d., the first term of expec- tation can be further re- written as
(eg(rol (5 - log
(T)TTT
-I (ro Bog (pt (T Io (T) (T.))
(10)
(r, loggi (Tr (T)) 2 (T) (r.)]
Based on the generation process, we can decompose the first term based on trajectories to multipli- cation of terms based on states.
(7 (7) II (st 11, ah-1), Ot, a
t=1
(II)
Tc
pd (r" (7)) = II
Po 00-11,
8t, 4- ~
[=1
E,- BBg (r, (T, [lg p5 (Tr
Er, Bog (v, (m+)
log (ôr a-,a.-1). ax-1)]
(12)
1e po 8e, di-1)]]
A direct computation of the second term DKL intractable. Following the idea that the optimiza- tion process of GAN equivalent to minimizing certain distance measure between two distri- butions (Nowozin al., 2016), can the optimization objectíve of GAN the surrogate objective of minimizing the DKI. [gs (7" (") || px (7*)) by introducing a new discriminator D., parameterized by that maximizes Eq. 13.
(s)
(s) (5)
(p(0,w) w) E.-D, log
+ ErT
(13)
1- Des (s)
14

---

Under review as a conference paper at ICLR 2021
For implementation simplicity and training stability, (i)D.(s)) can become arbitrarily large, we still choose optimize the original GAN optimization objective, which is formulated as Eq. 14.
P_ (6)
(D(0,w) E.ND, (Dw + faloor (
(14)
B IMPLEMENTATION DETAILS
Deployment
Mopt.ml
embed 5,
Sc-1
RNN
A..
mlp encoder
ec-1
visual decoder
Mbehavizoul
discriminnaor
visual encoder
Training
Figure 7: Illustration of full network structure. Blue parts denote mapping function ds: Yellow and green parts denote reconstruction function pa. Green part is always fixed in the entire training process.
Fig. 7 gives an illustration of the overall structure of our method. Both Toptimal and Thehavioral are polices of the source domain, where Toptimal is the optimal policy trained in the source domain and Abehavioral is policy policy in the source domain that mimics the behavior of the dataset collection policy (in terms of the cumulative returns). In rare cases when a differentiable Tbehavioral is not available (e.g. the policy is decision tree), it can be replaced by trainable network here. Detailed network structure and hyperparameters are listed in Sec. B.1, all of which are the same for every environment. To help balance the influence of the residual Ast, introduce new hyperparameter Ares: S.L.
8, = 8t + Avess Ast
(15)
B.1 HYPERPARAMETERS AND NETWORK STRUCTURE
Table 1: Hyperparameters for the Reconstruction Network
Name
Value
General
Ap
50.0
1.0
Ares
1.5
batch size
20 (trajs)
mapping update iter M
5
discriminator update iter
1
DM update iter
10
DM batch size
512
DM sync freq
1
grad clip norm
10
15

---

Under review as a conference paper at ICLR 2021
Table 2: Hyperparameters for the Reconstruction Network (Cont.d)
Name
Value
RNN
RNN type
GRU 128, 128
RNN hidden size
Dynamics Model
Hidden sizes
512, 512, 512 tanh 1 10-1
Activation function
Learning rate
Policy
Hidden sizes
64, tanh
Activation function
Discriminator
Hidden sizes
256, 256, 256, 256, 256 relu true
Activation function
Layer norm
Learning rate
5 x 10
Visual Encoder
Hidden sizes
Conv (4, 4,3),, Conv (4, 64), Conv (4, 128),
Conv (4, 4, 256), 256, 256, 256
Activation function
relu true
Layer norm
Learning rate
5 10-5
Visual Decoder
Hidden sizes
256, 256, 1024, Deconv (4, 4, 128), Deconv (4, 4, 64), Deconv (4, 4, 32), Conv (4, 4),, relu
Activation function
Layer norm
Learning rate
5 x 10
MLP Encoder
Hidden sizes
256, 256 relu
Activation function
Layer norm
Learning rate
5 x 10-5
16

---

Under review as a conference paper at ICLR 2021
C EXPERIMENT SETTING
We evaluate our method 6 MuloCo environments from OpenAl Gym, namely InvertedPendllum, InvereedDoubleeenduumnn HalfCheetah, Hopper, FixedSwimmer and Walker2d. FixedSwimmer is based on modifications proposed in previous works (Wang et al., 2019) to avoid sub-optimal policies by changing one sensor position. We treat the simulation observations as low dimensional simulation states s and rendered images real images.
Images are collected by "track camera" in HalfCheetah, Hopper, FixedSwimmer and Walker2d and "default camera" in InvertedPendulumg and InvereedDoubleePedduuun All images are resized to (64, 64, without any further pre-processing techniques in all environments. Examples of rendered images are shown in Fig. 8.
Two polices are independently trained until convergence using PPO (Schulman al., 2017) for every environment. One of them is regarded as Starget to collect the "real" image dataset. Trarget stochastic policy to mimic the real data collection process. The other is regarded as pre- trained simulator policy source. The image dataset contains 600 episodes, each being truncated to a maximum length of 500. The evaluation of all methods are done based on this truncated dataset.
(a) Pendulum
(b) HalfCheetah
Hopper
Swimmer
(e) Walker2d
Figure 8: Examples of rendered images of MuJoCo environments
17

---

Under review as a conference paper at ICLR 2021
D
EXTRA EXPERIMENT RESULTS
D.1
AVERAGE REWARD RATIO OF ALL ENVIRONMENTS
Method
CODAS(ours)
GAN
BC(max)
BC(finna)
Reward Ratio
70.1%
42.8%
47.83%
34.91%
D.2 PERFORMANCE THE OPTIMAL POLICY
Table. 3 shows the mean value and standard deviation of the un-discounted cumulative return of 100 trajectories collected by the optimal policy trained on states using PPO. The maximum episode length is set 1000. Full training curves of policies trained state space provided in Fig. 9. The performance of policies trained on state space matches the public benchmarking results.?
Fig. 9 also provides training curves of the policy trained images. We modify the network structure of actor and critic to adapt PPO to image input. all environments, the policy perform poorly. The final performance of image input is calculated at 3.0 10 time-step, when the value loss has converged. As far as we know, there is no public performance benchmark of optimal policy trained on MuJoCo images. Some previous results Deepmind Control Suite shows better result than ours, partly due to much clearer robots and background and tuned hyperparams.
Table 3: Performance of Optimal Policy on State Space
Environment
Return
Environment
Return
Hopper
2097 411
Swimmer InvertedPendulum
325 +
Walker2d
3669 + 587
775.5 = 319
HalfCheetah
1580 + 35
InvertedDouble
5201 + 1029
Fale day
Restrned
2.50
(a) Inverted Pendulum
(h) Inverted Double Pendulum
(c) HalfCheetah
14
Inessto
(d) Hopper
(c) Swimmer
(f) Walker2d
Figure Training curves of policy on state space and image space.
hww.:piiinnnnnu.ppppp..p.. openai
18

---

Under review as a conference paper at ICLR 2021
D.3 QUANTITATIVE RESULTS OF STATE MAPPING ERROR
Cood
CoolGreo umbicoo
A Wampyy
(a) Inverted Pendulum
(b) Inverted Double Pendulum
(c) HalfCheetah
cDBdd
GAM
1.0
2040
6000
1003
Emeeraa
(d) Hopper
Swimmer
Walker2d
D.4 TRAINING CURVES OF BEHAVIORAL CLONING
gg) Inverted Pendulum
(h) Inverted Double Pendulum
HalfCheetah
Por
ii) Hopper
(k) Swimmer
Walker2d
Figure 10: Training curves of Behavior Cloning.
19

---

Under review as a conference paper at ICLR 2021
D.5 ROBUSTNESS TO DYNAMICS MISMATCH
To test the robustness of CODAS, we manually change the friction of the target environment to create dynamics mismatches. Fig. 11 shows the reward ratio of CODAS in Hopper environment with different magnitudes friction. The performance of CODAS remains stable even when the amplitude of friction reaches 20%, proving that CODAS robust to mild dynamics mismatch. is worth noting that the policy is trained without any technique such as domain randomization that improves robustness.
1.0x frition (defbult) (defbut)
1.2x fricton
1.3x friciin
0.5
04
0.
0
2000
4900
6000
5000
10000
time-atop
Figure 11: Reward ratio in Hopper with small dynamics mismatches.
D.6 TRAINING CURVES OF CYCLEGAN
We planned to CycleGAN, one of the state-of-the-are methods in domain adaptation, for com- parison. Concretely, we remove the identity loss and use the network structure as ours (See Visual Encoder/Visuall Decoder in Sec. B) the generator network structure. However, the changes in the loss function and network structure may require a complete hyperparameter setting from image-to- image translation. We fail get workable CycleGAN in most environments. The best results so far are shown in Fig. 12. The training of state-to-image GAN in CycleGAN is not stable and may lead to the overall bad performance. It could be of the following reasons:
id loss in the original CycleGAN improve training stability is not applicable;
network structures (e.g. UNet) that are specially designed for images generation applicable:
(a) Inverted Pendulum (b) Inverted Pendulum
Double
(c) HalfCheetah
(d) Hopper
(e) Swimmer
(f) Walker2d
Figure 12: Training curves of CycleGAN.
20

---

Under review as a conference paper at ICLR 2021
D.7 GAN WITH ACTION Loss
Fig. 13 shows the training curve of GAN with additional policy loss defined in Eq.6. Surprisingly, adding fixed policy network does not help the learning process of naive GAN. It may be because the gradient from this fixed policy in the initial training stage useless or even harmful to the training of the mapping function. The performance of GAN with policy loss drops significantly partly due to this loss is large in the modified Swimmer environment (can be found in the BC loss in Fig. 10 as well).
0.8
GAN
codex
CAM
GAN poliey loss
0.6
GMN kn polky
0.5
0.4
2.4
02
0.0
lime-step
timesttpp
Inverted Pendulum
(b) Swimmer
Figure 13: Training curves of GAN with action loss.
E A SIMPLE EXAMPLE OF FAILED DISTRIBUTION MATCHING
Fig. 14 shows example of wrong mapping using GAN. Fig. 14(a) and Fig. 14(b) are two Gaussian Mixture Models of modes with a linear transformation between them. The translucent circles denote the ground-truth correspondence between every mode in two domains and the dots denote the actual correspondence learned by GAN. Although GAN successfully matches the probability densities, it fails map the target instances to their corresponding correct source instances. Under such a wrong mapping function, a neural network trained on samples in the source domain cannot be transferred the target domain directly.
1
-2
2
-1
1
2
(a) Source domain
(b) Target domain
Figure 14: An example of failed distribution matching under linear transformation.
21


Reviews:
[{"review_id": "kcXfeQrgCFb", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThe paper proposes a new approach for performing cross-modal domain adaptation, i.e. adapting a policy trained with inputs from modality A (eg low-dimensional environment state) to work with inputs from domain B (eg images). The main use case demonstrated in the paper is the adaptation of policies trained on states in a simulator to work on image inputs, which can be useful for eg real world deployment where states might not be available. While it is a very classic approach to separately train a perception module images --> state (eg in robotics), the main novelty of the presented method is, that this mapping can be learned without the need for paired [image, state] data.\n\n## Strengths\n- the discussed problem of cross-modal domain adaptation is very relevant, since it can allow for the transfer of policies from training in simulation to deployment in real environments\n- being able to learn the image --> state mapping without paired data can be impactful since it is often tedious / impossible to get exact state annotations for real world data\n- the proposed technical approach is novel to the best of my knowledge\n- the method is evaluated on multiple environments and some of the design decisions are ablated + (preliminary) experiments on the robustness of the method are conducted\n\n## Weaknesses\n- **baselines are not described in detail**: the experimental section only mentions that \"We modify state-of-the-art methods in same-modal domain adaptation for comparison\" -- it is important to describe which method for same-modal adaptation was used and how it was modified for the cross-modal case to properly judge the results. \n- **no investigation into why baseline does not work**: figure 1 provides one possible explanation (because of not taking dynamics into account), but later the paper mentions it could be because the necessary biases present in image-to-image translation are not present in image-to-state translation, finally it could be because the method was not tuned sufficiently. It would be good to show a more detailed investigation of this, particularly since the paper claims to be the first trying to apply domain adaptation techniques to the cross-modal case\n- **only tested on visually clean environments**: the environments used for testing the approach are visually clean -- in particular: the state information is sufficient to fully render / reconstruct the scene. This is likely not true for more realistic scenarios (eg think about autonomous driving where the commonly used state representations are certainly not sufficient to render all details in the environment -- the whole point is to reduce the amount of information in the policy's input). I wonder whether the proposed method would struggle with such environments since it constrains the \"latent\" variable of the prediction model to be equal to the pre-defined state while training it to reconstruct the full scene. If there is lots of detail in the scene that is not covered by the information in the state the model might struggle to reconstruct the scene properly.\n- **requires action trajectories in the image data domain**: at least the version of the model that was experimentally validated requires access to action trajectories in the image domain (for the action reconstruction loss). Such action annotations might be hard to obtain in the real world -- baselines like CycleGAN do not require these since they learn the mapping purely from state and image data.\n\n## Questions\n- does the proposed approach need access to a differentiable behavior policy? I would think that just action samples would be enough and it would only need a differentiable policy on states (which is anyways available) --> however, the formulation in the paragraph before eq6 talks about a \"differentiable behavior policy\" so it would be good if the authors could clarify whether it is indeed needed\n- how stable is the training of the approach? (Cycle)GAN approaches can be notoriously hard to train (see Appendix Sec D5) -- the proposed method also uses a GAN to minimize divergence from the prior. Some discussion on stability of this training or even a quantitative analysis of performance over a range of hyperparameters could help to show that this method might be easier to train than the GAN-based alternatives?\n- when training the dynamics model online the simulator needs to be reset to the predicted state of the model --> how can value ranges be handled? ie what to do if the model predicts an invalid output state?\n- what are the \"numerical instabilities in Mujoco\" mentioned in Section 4? maybe add a little more explanation?\n\n## Suggestions to improve the paper\nAddressing the weaknesses I listed above can help to improve the paper. In particular I would suggest to:\n- clearly describe the baselines used \n- show *why* the baseline is failing to make clear that this is not an issue of tuning\n- include a GAN baseline that operates on short trajectory snippets instead of single states to see whether it is truly an issue of cross-modality or whether merely including dynamics information can help the simpler GAN baselines\n- test on environments with (non-static) visual components that are not captured in the state information (eg add moving visual distractors to the Mujoco scenes which are not part of the state)\n- ablate the action reconstruction component of the mapping loss to show that the method can work without the need for action annotations in the image domain\n- tone down the contribution claims that talk about \"transfer to real-world images\" since the tested scenarios are far from real-world images\n\nSome further, optional improvements:\n- add a baseline that shows RL trained on images from scratch --> this can show how much performance we gain / loose by doing the domain adaptation vs training from scratch (even if we loose some performance it is okay since training from scratch might not be feasible in the real world)\n- it seems that some of the differences to prior work require a better understanding of the technical details of the proposed method (particularly paragraph 4), it might be worth considering to move the related work section after approach before experiments\n- as mentioned in my summary, it is a quite classic approach eg in robotics to separately train a perception module that maps images to states and then use it to work in the real world with policies that operate on state input (eg motion planners etc), ie perform cross-modal domain adaptation. However, they assume access to a paired dataset of [image, state] tuples, which is potentially hard to obtain. I think the related work section could benefit from adding a discussion about this.\n- this sentence is not clear: \"We first formulate the generation process of the real world and simulation\" -- does this talk about how observations are actually generated in the real world or about the generation process of the model used in this paper? maybe reformulate for better clarity?\n- Section 3.3 is a bit confusing, it only later became clear to me that training the separate dynamics model is optional, this could be emphasized a bit more.\n- the dynamics mismatch experiment should be trained to full convergence (which seems to be ~10k steps for hopper), now it is only trained for 5k steps which equals only half-converged performance so it is unclear whether the mismatched runs will actually reach full performance\n\n## Overall Recommendation\nThe proposed method is interesting and the paper certainly has merit. My main concern is that it is currently hard to judge the thoroughness of the experimental evaluation since it remains unclear how the baseline is implemented and why it fails. Therefore I cannot recommend acceptance at this point. If the authors can more convincingly show why prior work fails in the cross-modal case and how their method fixes that I am willing to increase my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "WKCvShjBMYk", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Cross-modal Domain Adaptation for Reinforcement Learning\n### Summary\nThe authors propose CODAS, a domain adaptation algorithm for transferring policies learned in state-space to the image domain. They learn a variational RNN to map image inputs to state space with an adversarial loss to ensure that the distributions of the mapped states and the original states are matched. Their experiment results show improvement over one behavior cloning baseline and one GAN domain adaptation baseline.\n\nOverall, the paper is clearly written, and the topic is new and relevant to the field.\n\n### Strength\n- The first cross-modal adaptation for the RL agent to the best of my knowledge.\n- No paired data is needed.\n- Interesting DM model.\n- Good improvement over the baselines considered in the paper.\n\n\n### Weakness\n- Insufficient experimental results. Specifically:\n   1. What's the performance of the policies trained in the original state-space?\n   2. What's the performance of the policies trained directly in the image space?\n   Knowing these numbers gives us a better idea about how well the adaptation works.\n\n- Unclear description of GAN baselines. Did I miss something? Can't find a description in the appendix as well.\n\n- Need human expert policies to collect image input. It is unclear how this component may affect the final performance of CODAS.\n\n- \"In this paper, we propose Cross-mOdal Domain Adaptation with Sequential structure (CODAS) that learns a mapping function from images in the target domain (real world)\" (page 2): this is misleading. The target images are not really from the real-world. The images are rendered in simulation, with no realistic perturbations.\n\n### Minor Comments\n- Notation: why refer to Eq in the appendix instead of the main paper? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "TQeuj99IQDs", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores learning the mapping from state to sensor observations as part of RL policy learning. This is a relevant problem to enable sim-to-real transfer.  The paper explores cost terms that have been used in GANs to align distributions between the state and observation. The paper shows experiments in simulation, where the observation is images of the robot in the environment. I have some concerns  which are listed below,\n\n1. The paper only shows transfer from the robot's state to images of the robot in the environment. The state of a system might not sufficiently capture high fidelity information from observation. E.g., can this method reconstruct uneven terrain from only observing the robot's pose and other low dimensional representations in state? It's unclear to me how the method can transfer to work with observations on real systems. \n\n2. All the chosen environments do not involve a second object to interact with. E.g., can the method work on the OpenAI cube rotation task where the state could contain the object pose and the observation might not. It's also shown by other researchers that walking doesn't require high fidelity observations~[1] such as vision so the effectiveness of the method to transfer policies trained on state to observations is unclear. Is the method overfitting to recorded trajectories? one good test would be to deploy the policy in a different environment(increased slope of the floor?) that wasn't part of the recorded trajectories but was part of the full state policy.\n\n3. It would be interesting to discuss how this work could leverage or compare against multi-modal representation learning in robotics~[2]. \n\n\nReferences:\n[1] Lee J, Hwangbo J, Wellhausen L, Koltun V, Hutter M. Learning quadrupedal locomotion over challenging terrain. Science Robotics. 2020 Oct 21;5(47).\n[2] Lee MA, Zhu Y, Srinivasan K, Shah P, Savarese S, Fei-Fei L, Garg A, Bohg J. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In2019 International Conference on Robotics and Automation (ICRA) 2019 May 20 (pp. 8943-8950). IEEE.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3M7AqoC6lKu", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors pose a problem of learning a mapping when when a low-dimensional state simulation is given along with target image tragectories. The goal of the paper is to learn a mapping from image to state such that at test time the agent can directly use this mapping with a trained policy from the simulator to perform in the target domain. The contribution of this paper lies in its problem formulation and an algorithm named Cross-mOdal Domain Adaptation with Sequential structure (CODAS).\n\nPros:\n\nThis paper proposes a novel and interesting problem for cross-modal transfer.\n\nI appreciate the paper shows the results when the dynamics of the source is slightly different from the target dynamics. \n\nCons:\n\nAre there any assumption about the target data? Does data collection policy in the target domain resemble the pretrained source policy in the simulator in some way?\nIt seems to me that without this assumption q_\\phi can be different from the groundtruth posterior p(s_t|o_t).\nI am not sure if this is related to this sentence, \"The policy \\pi in source domains included in the second term does introduce a new assumption that real-world data are collected by a known behavioral policy.\" Do we need the policy in the target domain? If so, please clarify where this is used and mention the assumption in the beggining.\n\nThe experimental section needs clarity in baseline details. The GAN baseline is not explained in the paper making it difficult to interpret the comparative performance. Potentially, there should be a sequential version of GAN that generates tupples/sequences rather than just a single image [1].\n\nWhat is the reward ratio? Please elaborate.\nDoes the groundtruth state-based (source) policy achieve the reward ratio of 1?\n\nThe dynamic mismatch should be investigated more. How differ can the dynamics be before things break? This is very important to understand whether this formulation can **actually** be applied to the real world settings.\nI am curious to see what happen the states contain irrelevant information or be represented in a different way?\n\nIn equation 7, where does log p(\\hat{o}_t|o_t) come from? Why do we not use the first term from l_D in equation 5 here?\n\nConclusion:\n\nThis paper proposes an interesting problem of cross-modality domain adaptation. However, there are a few questions to be addressed above to substantiate the potential of the proposed low-cost sim2real.\n\n\nReferences:\n\n[1] Learning Plannable Representations with Causal InfoGAN (http://papers.nips.cc/paper/8090-learning-plannable-representations-with-causal-infogan)\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers generally appreciated the problem statement and topic of the paper, but raised concerns across the board about the empirical evaluation. Since the paper is largely experimental in nature, a compelling experimental evaluation is important. Reviewer concerns generally centered around: (1) the relative simplicity of the evaluated tasks; (2) somewhat questionable baselines. While the authors provided responses to some of these concerns, after discussion the reviewers generally still felt that these issues were rather severe, and preclude publication at this time. I would encourage the authors to take this feedback into account in improving the empirical evaluation in the future.
Decision: Reject