Cross-modal Domain Adaptation for Cost-Efficient Visual Reinforcement Learning
Authors: Xiong-Hui Chen
Shengyi Jiang
Feng Xu
Zongzhang Zhang
Yang Yu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Reinforcement Learning

Abstract: In visual-input sim-to-real scenarios, to overcome the reality gap between images rendered in simulators and those from the real world, domain adaptation, i.e., learning an aligned representation space between simulators and the real world, then training and deploying policies in the aligned representation, is a promising direction. Previous methods focus on same-modal domain adaptation. However, those methods require building and running simulators that render high-quality images, which can be difficult and costly. In this paper, we consider a more cost-efficient setting of visual-input sim-to-real where only low-dimensional states are simulated. We first point out that the objective of learning mapping functions in previous methods that align the representation spaces is ill-posed, prone to yield an incorrect mapping. When the mapping crosses modalities, previous methods are easier to fail. Our algorithm, Cross-mOdal Domain Adaptation with Sequential structure (CODAS), mitigates the ill-posedness by utilizing the sequential nature of the data sampling process in RL tasks. Experiments on MuJoCo and Hand Manipulation Suite tasks show that the agents deployed with our method achieve similar performance as it has in the source domain, while those deployed with previous methods designed for same-modal domain adaptation suffer a larger performance gap.

Cross-Modal Domain Adaptation for Cost-Efficient Visual Reinforcement Learning
Xiong-Hui Chen", Shengyi Jiang', Feng Xu, Zongzhang Zhang', Yang Yu National Key Laboratory of Novel Software Technology
Nanjing University, Nanjing 210023, China
Pazhou Lab, Guangzhou 510330, China {chenxh, jiangsy, xufeng) amda. nju. edu. en, {zzzhang, yuy weweeeeueuuuu.....eeeeeeeeeeeuue edu..edd.ce. cn
Abstract
In visual-input sim-to-real scenarios, to overcome the reality gap between images rendered in simulators and those from the real world, domain adaptation, i.c., learning aligned representation space between simulators and the real world, then training and deploying policies in the aligned representation, a promising direction. Previous methods focus model domain adaptation. However, those methods require building and running simulators that render high-quality images, which can be difficult and costly. In this paper, consider more cost- efficient setting of visual-input sim-to-real where only low-dimensional states are simulated. We first point that the objective of learning mapping functions in previous methods that align the representation spaces is ill-posed, prone to yield an incorrect mapping. When the mapping crosses modalities, previous methods are easier to fail. Our algorithm, Cross-mOdal Domain Adaptation with Sequential structure (CODAS), mitigates the ill pos-eesesssss by utilizing the sequential nature of the data sampling process in RL tasks. Experiments on MuJoCo and Hand Manipulation Suite tasks show that the agents deployed with our method achieve similar performance as it has in the source domain, while those deployed with previous methods designed for same-modal domain adaptation suffer larger performance gap.
1 Introduction
Reinforcement learning (RL) for vision-based robotic control tasks has achieved remarkable success in recent years [1, However, current RL algorithms necessitate substantial number of interactions with the environment, which are costly both in time and money on real robots. An appealing alternative train policies in simulators, then transfer these policies to real-world systems [3]. Due to inevitable differences of representation between simulators and the real world, which is also known the "reality gap" [4], applying policies trained in domain directly another almost surely fail, especially in visual-input tasks, which is due to the poor generalization of RL policies [5]. Domain adaptation a promising direction handle the gap by mapping representation from domains to an aligned representation and then training and deploying policies in the aligned representation.
Many recent works, which learn mapping function to align the data distributions of the two domains, have adopted unsupervised visual domain adaptation [6. 7, 8]. We point out that, as illustrated in Fig. 1(a), the objective is ill-po for learning correct mapping function. These adaptation methods exploit structural constraints [9] in two domains of the same modality (e.g., learned on simulated images and deployed real images). These methods implicitly alleviate the intrinsic ill ill-posedness of distribution matching.
Equual Contribution 'Corresponding Author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

?
O1-1
(a) Mapping only considering state-distribution matching
(b) Mapping with sequential structure
Figure Illustration of training objective of learning mapping function unsupervised domain adaptation. Shaded regions denote data distributions, where the darker the color, the higher the probability. For each figure, the left region is the target domain and the right is the souree domain. In Fig. (a), both and s are "realistic" instances, but only correct. Since they of similar probabilities, mapping instance o, in the target domain somewhere of similar probability in the source domain is "reasonable" if we only consider distribution matching. RL, policy may output unreliable actions when taking these incorrectly mapped states as inputs. In Fig. ((b), sequential structure can help rule out the wrong mapping via trajectory contexts.
However, such kind of same-modal domain adaptation requires the simulator to render images when training the model, which introduces unwanted costs and difficulties that are ignored in previous works. First, building rendering engine is laborious task. Second, using RL methods to train policy with an image-based simulator is usually harder [10] and slower (can be up to 20x slower [11]) than with state-based simulator. An ideal solution to these problems to train policies with states in the simulator and adapt the learned policies to real-world images. Current domain adaptation methods generally fail this setting since the structural constraints based on the modality consistency are no longer available, which makes the representation alignment task harder.
To learn such a cross mross-modal mapping, we propose Cross-mOda Domain Adaptation with Sequential structure (CODAS) that learns mapping function from images in the target domain to states in the source domain, as illustrated in Fig. (bb). With the help of the learned mapping function, policies trained on states in the source domain can be deployed in the target domain of images directly. Specifically, based on the sequential nature of RL problems, we formulate the cross-domain adaptation problem as variational inference problem and decompose it into series of solvable optimization objectives. We also design a special residual model structure the recurrent neural network (RNN) to enforce additional inductive bias and stabilize the training process.
We evaluate our method on six MuJoCo [12] tasks and four Robot Hand Manipulation tasks [13], where we treat states as the source domain, and rendered images as the target domain. Results show that the learned mapping function can help transfer the policy to the target domain with only a small performance degradation in most of the tasks while existing methods [14, 15] suffer from a larger performance gap.
Related Work
Unsupervised visual domain adaptation (UDA) aims to map the source domain and the target domain to an aligned distribution without pairing the data. UDA originally designed for image translation in computer vision [14, 6, 16, 17]. In RL, UDA transfers policies from simulators to the real world, to overcome the reality gap between rendered images and real-world images. Previous methods fall into two major categories: fcature-level adaptation, where domain-invarianl representations learned [7], and amage-to-image adaptation, where pixels from source images are used to generate images that look like those from the target domain [18].
Feature-level adaptation usually adopts domain randomization techniques for domain-invariant representation learning [19, 20, 21, 22, 23]. In domain randomization, meta-simulator is required to generate mass of variants on rendered images. The learner aims to extract an invariant representation from the variants of representations. Therefore, the training process is generally costly. Besides, these methods implicitly assume that the variants can cover the target domain so that the target domain only instance of the variants.

---

Image-to-ine to-g adaptation is challenging when data from two domains are umpaired. Most previous works attempt solve this problem by using generative adversarial networks (GANs) [15, 24, 3, 17, 25, 26]. [24] transfers policies from Atari games to their modified variants by training GAN to map images from the target domain to the source domain. RI-CycleGAN [3] unifies the learning of a CycleGAN [14] and an RL policy, claiming better performance by learning features that are most crucial to the Q-function in RL.
Despite the success of the mmage-to-image domain adaptation methods in previous settings, we point out that these methods can somewhat bypass the ill- -ll-posedsess for distribution matching, which is illustrated in the example in Fig. 1. Since images generally differ only locally in color, textile, and lighting, but resemble globally between two domains, the effect of ill-posedness can be implicitly handled by leveraging the consistency of modality [17, 25, 26]. However, since images and states differ essentially, we can no longer utilize the consistency of modality. Thus, as shown in our experiments, the effect of ill-posedness is revealed in RI. tasks. Some works impose extra structural constraints [9, 27] but fail in image-to-state domain adaptation either. In our work, we force the mapped states to follow transition consistency by considering the inner relationship between the sampled data from the decision-making process in RL tasks.
3 Cross-modal Domain Adaptation with Sequential Structure
Without loss of generality, the visual domain adaptation problem in RL can be formulated as learning a mapping function to align the representation of the source and target domains, then training and deploying policies the aligned representation. To learn the mapping function and policies, a dataset is pre-collected by some policies (c.g.. human-exper policies or random policies) in the target domain, and an agent can only interact with the environment in the source domain. Since our major concern is the reality gap between the representation spaces, in this formulation, we follow the setting in previous methods which omits the gap between the dynamics models in the two domains.
In our setting of cross-modal domain adaptation, the source domain is a low-dimensional state domain and the target domain is high-dimensional image domain. In this work, we regard the state space as the aligned representation space. We first pre-train a policy π in the source domain. By learning a mapping Ts- parameterized by o. from image space O to state-space S, the agent can be deployed with new policy (o)) (x o q6 (o). where denotes function composition, i.c., image observations are mapped states, then the pre-trained policy directly acts on the states.
3.1 Domain Adaptation in RL as Variational Inference
In this section, we first define our objective of domain adaptation in RL based on the framework of variational inference. Then we give a comparison between our objective and the objectives used in previous methods.
The modeling of the generation and inference process of the decision process in the source and target domains used in our method is illustrated in Fig. 2. In the source domain, an initial state follows the distribution p(s1). A transition function $; ~ p(s $t-1, 4-11) outputs the current state 8, from the previous state 8t-1 1 and the previous action a:-1. In the target domain, the generation process also follows the transition of states, but the states are not available to the agent, and the agent only observes the images corresponding generated by the corresponding state 8t- In other words, there exists a decoder p(o, I xx) that can construct observations from states. It because that the state contains full of information for a transition function to compute the next state, while image observations are rendered based on the states. However, there would have some irrelevant patterns in images in real-world applications. In practice, our decoder is defined as p(o: St, 01-)). We allow (o(o: | .) to be dependent on 0 so that some irrelevant patterns in images can be decoded easily in an regreseegee manner. The inference process can be regarded as the mapping process in domain adaptation in RL. The inference function defined as (0; 88-1, at- 0.). which is similar to the transition function but with additional o as an input. A simple insight is that the inputs 8t-1 1 and dt- 1 give information to predict the distribution of 8t, as the transition function p(s 8t-1, a+-1) does, and o is used to determine the state sample s; from the distribution.
the states. Ideally,
image observation o, is
3

---

OT
S2
$2
a1
Q1
(a) Generation process in the source domain
(b) Generation and inference process in the target domain
Figure 2: Illustration of the generation and inference processes based on the decision processes of RL tasks. All nodes random variables. Shaded nodes observable variables. Solid lines denote the generation process and dashed lines denote the inference process. There correspondence between the two trajectories enclosed by the rounded rectangles. Note that we include policy æ in both generation processes, which correspond to the edge from state &, to action
The training process aims at learning an inference function (i.e., the mapping function) that approxi- mates the ground-truth posterior distribution. The raw optimization objective of variational inference in this setting can be formulated
min E,- [DEL (1o(T") _ (7) p(7" [+)]]]
where 7 denotes a trajectory, the superscripts and o of T indicate the trajectories from the source and target domains respectively, p(.) is the ground-truth distribution, (-(-) is the mapping function that we want to learn, and DKL computes the Kullback-Leible divergence. T" contains a trajectory of observation-action pairs {(01, 01), (o2, 02), .... (or, or)). }. and r" contains trajectory of state-action pairs ((11, 01), (82; o2), (8T, ar)). Here we assume the trajectories of source and target domains collected by the same policy æ, which is mild assumption and has been implicitly explicitly introduced in previous works [24]. Since the policy is fixed, for the simplification of notations, we omit the dependence of and just Is and p indicate the distribution. The Evidence Lower BOund (ELBO) of this variational problem can be formulated as:
max Eyy [p (pp [log pa (7" [+*] DKL 190 (T" |TT) (()((l))))) 0.0
(2)
where 4 denotes the inference trajectories of qá. and (7 (*) is an approximation of the generation process p(7" r*), with parameter 0. The derivation of ELBO A. The first term maximizes the reconstruction probability. in order to enforce that the mapped state å recover observation o. The second term enforces the alignment of the distributions of the mapped trajectories Φ6 (T" T") and the ground-trute -tunh trajectories distribution (r") in the source domain.
be found in Appendix
Compared with previous works which learn the mapping function by minimizing the divergence between the distributions of ( o) and p(s), we model the optimization objective as a matching of trajectory distributions via the Bayesian graphical model. As illustrated in Fig. the trajectory- level modeling is important since the data points are not i.i.d., since they are collected in the RL scenario. Without considering the inner structure of the dataset, mismatching of mapping might occur easily.
3.2
Differentiable Optimization Objectives
The ELBO defined in Eq. (2) contains terms involving distributions over the entire trajectory, which impractical to solve. Given the previously defined generation process, we can decompose the joint probability into the multiplication of single-step probabilities. The result of the decomposition is
max E Do (e [log (o Ot-1,, a.-)] DKI. [as(7" 1 p(T)) 0,0
where Do denotes the pre-collected dataset in the target domain, is the timestep and T denotes the horizon. So. ag and on initialized with 0. (s $;-1, ar-1, ox) is RNN that outputs the mean

---

and standard deviation of Gaussian distribution. A direct computation of the second DKL term in Eq. (3) is intractable. Following the idea that the optimization process of GAN equivalent to minimizing a certain distance measurement between two distributions [28], we use the optimization
objective of GAN as the alternative of minimizing DKL. Is (r" T") || p (7*)). For the simplicity of
implementation and stability of training, we use the original optimization objective of GAN, which is equivalent to minimizing the SaasennShannon divergence. The optimization objective is
min max -D [log (Dw(+"))] + Bro [log (1 -t)
(4)
Similarly, we decompose Eq. (4) into single-step formulation:
T
min max Φ) = min max E
log Dw (o,
tm1
(5)
Do
E (0 1 191
log (1 (,,, 1))1))],
ag
where D denotes the dataset collected by the pre-trained policy in the source domain. Here D., is also implemented as an RNN, in which h denotes the hidden state the RNN rollout. particular, the discriminator outputs tuple (yt, h4) (se dr, h h1-1), where y/ is the probability of prediction and h2 is the next hidden state. We omit the output of h for brevity. The latter term the practical objective of the KL-divergence in Eq. (3) of the mapping function 9á-
Combining both the aforementioned reconstruction loss and generation loss together, the optimization objective of the mapping function is
max E, Do 0,0
Es (adl as 1.01) [log 82, 01.01-1
(6)
- Ap log (1 (Be, (s, 14 mt)))
s.t. w = arg max LD(w, o),
where An is the hyper-paramette for the weight controlling. The decoder po and the discriminator D. are fixed during the training process of the mapping function 96. The loss function of the decoder pe is the first term in Eq. (6). Similarly, the mapping function q is fixed during the training process of the decoder pe and discriminator D.,. The detailed derivation can be found Appendix A.
3.3 Embedded Dynamies Model for Stable Training
Trajectories of practical RL problems often last hundreds of time steps. Training RNN such long-horizon trajecto- ries can be difficult. The previous success of ResNet [29] shows that residual structure simplify the learning target to predict small residuals, resulting in a remark- able performance increase in learning ultra-deep neural networks.
Sp-1
DM
perlodicall updisee
Embeddded-DM
RNN cell
As
Adopting similar idea, we incorporate residual structure in the RNN to help stabilize the training process. A deter- ministic Dynamics Model (DM) (s, a) with parameter is trained independently using transition tuples collected in the simulator. Its parameters periodically updated to embedded-DDN DM minimizes the mean-square error to the transition tuples in the source domain. We regard the embedded-DM as part to provide an "average" estimation of the next states 3, (see Fig. 3). The job of the rest of the part, instead, is simplified to just output correction. With this specially designed model structure, the mapped states follow the transition dynamics in the source domain better. In particular, we model the mapping function as:
encoder
Figure 3: Model structure of the inference function with embedded-DM.
St =p(s1-1, a-11 + aAs, As, (o(As) St-1, 9t),
5

---

where a is hyper- paramerrr to control the correction range of As. As takes tanh as the activation function. Thus, the range of aAs is constrained to [-a, a).
DM is first trained using hatches of transition tuples collected in the simulator. However, these data are insufficient for DM training since the DM predictions in unseen state-action pairs are unreliable, due to the extrapolation eπror [30, 31]. Since the aAs is constrained to !-a, a), too larger extrapolation errors of P.p (st-1 a1-1) might obstruct As, to recover the correct $, During the training process of the mapping function, the dynamics model is trained online using D {(,, T(a 8), p(8, π (a 6))]- That is. we reset the simulator to the mapped states & and then rollout with single-step oracle transition to get st+1. The optimization objective of the DM is minimize the following mean- squared-error loss:
min E "' [(p, (s,0) - s')1
When training the mapping function qa, the parameter yp in the embedded-DM is fixed and gradients are back-propagated from in Eq. (7) to & via the reparameterization trick [32]. The detailed training procedure of CODAS is shown in Alg. 1 in the Appendix.
Experiments
We evaluate our method in MuJoCo [12] from OpenAl Gym and Robot Hand Manipulation Tasks. We define the rendered images as the target domain and the original states as the source domain. The pre-collected dataset the target domain contains 600 trajectories and is collected by policies trained by PPO [33] and DAPG [34]. Please refer to Appendix E for the detailed experimental setting. The quantitative performance of the data-collecting policies are given in Appendix F.I.
We compare our method with several state-of of-the-art art methods in same-man model domain adaptation which are also compatible with the cross-modal setting, including:
1. GAN: Conditional GAN trains a mapping function (s o) to align the distributions of D and D A diseriminator tries to discriminate the mapped states from those from D³, and is used to guide the training of the mapping function.
2. CycleGAN: CycleGAN shares the same basic framework with GAN with an extra cycle-consistent loss to constrain the mapping. other words, CycleGAN generates à from ô ~ Par (o ( o))) and trains the mapping function via making the (ō)) output the real labels with high probability.
3. GAN with Stacked Input:
Inspired by common trick that uses stacked images as input in visual RL tasks [35, 36), we modify GAN to take stacked images as input, and name it GAN STACK. The stacked images provide additional local temporal information.
We also compare CODAS with behavior cloning (BC), which trains a policy in a supervised manner using (or, at) DD All domain-adaptation baselines are trained with the same neural network structure and tuned to converge (i.e., the output probabilities of discriminators are close to 0.5). The tasks trained for 10,000 to 40,000 epochs based on their difficulty. For each epoch, every method is updated using batch of 20 trajectories. Implementation details for all these methods can be found in Appendix D and the source code is available at https://github cwwm/c/ooooococe wwwoignnncche//oodaas
To focus on the performance of the adaptation process, we use reward ratio as the metric: Tratio ... where r and rd are the cumulative rewards of the adapted policy deployed images the optimal policy trained on states respectively. Each task is trained with three seeds. In the following plotted curves, the solid lines indicate the mean value and the shaded regions indicate the range of +1 standard deviation. All the details of training and evaluation are given in Appendix E and Appendix F,
4.1 Performance on MuJoCo Tasks
The training curves of all methods in MuJoCo tasks are given in Fig. 4. Firstly, BC performs well in HalfCheetah, but poorly in the other environments. This indicates that the dataset is not large enough to learn a correct mapping from images to actions directly. In Swimmer and HalfCheetah, the deployed policies with mapping functions learned by GAN and CycleGAN achieve good performances. It is because of the stability of the environments. The trajectory will not pre-terminate no matter how badly the agent behaves. Therefore, the agent is allowed some incorrect actions, which come from

---

CODAS CAN STACK
1.0
5.0
regard ratlo
reward retio
6..6
reward ratlo
0.4
STA.K
0.0
COO8S
0.0
05
100
200 epocts
355
0.2
0.4
epochs
epochs
(a) Inverted Pendulum
(b) Inverted Double Pendulum
(c) Half Cheetah
Walkerdd
0.5
1.0
0.7
reward ratio
0.5
0.6
colns CAn
reward rabio
cooaar rablo
0.4
0.4
D0C45
0.0
OyduiAN STAOK
0.2
GAN
STACK
-02
apochs
X1P
epochs
(d) Hopper
Swimmer
(f) Walker2d
Figure 4: Training curves of different methods MuJoCo.
mismatched states. While the rest tasks, the agent might reach unsafe states after performing some undesired actions. Thus the agent is sensitive to the mismatching of mapping states. In these tasks, GAN and CycleGAN perform even worse than BC. Although CycleGAN has been tuned to converge (see Appendix), the performance of CycleGAN is similar to GAN in all of the tasks. We believe although the cycle-consistent loss guarantees the mapping function is a bijective mapping, the mismatching problem shown in Fig. ((a) not be handled, which is more critical in the RL setting. Therefore the performance improvement is not significant. Finally, we found that the GAN STACK algorithm reaches better performances than GAN in most of the tasks. The phenomenon indicates that sequential information important to learn a correct mapping. However, without other constraints that CODAS has, the performance improvement is unstable.
In conclusion, the performance of CODAS is consistently better than the baseline algorithms. The average performance of adapted policies in the target domain is about 85% of their original perfor- mance in the source domain. visual illustration of the mapped states' accuracy shown Fig. Both reconstructed images and re-rendered images match the original ones well. Re-rendered images can even match the original ones well in the last falling frames which are sparse in the dataset.
Figure 5; visual illustration of (a) original images, (b) reconstructed images, and (c) re-rendered images of the mapped states.
4.2 Quantitative Results of State Mapping Error
In our experiments, since each image is rendered based on its corresponding state, we can store the state as the ground-truth state for each image our experiments. For each batch of trajectories in an epoch, compute the root mean squared error (RMSE) between the ground-truth states and
7

---

the mapped states. The curves of the RMSE between the ground-trun truth states and the mapped states are shown Fig.
We regard the RMSE as the quantitative results of the mapped states for each algorithm. The results show that CODAS is consistently better than all of the baseline algorithms among all of the tasks. In HalfCheetah and Swimmer tasks, on which the baseline algorithms can also reach a good reward ratio, the qualities of mapped states of CODAS in the main body that the good reward ratios come from the robustness of environments to some incorrect output actions, which are caused by mismatched mapped states.
significantly better. The results verify our suppose
20
20
CANAS
DAA Cyckerot
3.8
2.8
1.50
GAN SLNCE
3.6
2.6
1.00
2.5
6MN
0.50
1.2
0.25
22
2.5
0.00
2.5
355
2.0
0.0
0.4
2.0
spocts
epochs
spochs
(a) Inverted Pendulum
(b) Inverted Double Pendulum
(c) Half Chectah
2.0
2.0
20
CAN
CAR
3.8
3.8
Pyckenu
Cycknna
3.6
3.6
100t MSE
6.6
02
1.2
4.3
3.0
00
spocts
K20
opocts
spochs
(d) Hopper
(e) Swimmer
(l) Walker2d
Figure 6: Root mean squared error between mapped states and ground-truth states. The solid lines denote the mean value. The shadows denote the standard deviation.
4.3 Ablation Studies
By considering three mapping function modeling methods (i.c., Multilayer Perceptron (MLP), RNN or RNN with embedded-DDL (DM-RNN)) and two discriminator modeling methods (i.e., taking trajectories inputs or not), we construct six CODAS variants do our Ablation studies. We named them with the format of "mapping method- Y/N".. For example, DM-RNN-Y is the algorithm that the mapping function is modeled with RNN and embedded-DMr and the discriminator taking trajectories as inputs, which is the original CODAS. We conduct experiments in Hopper, Inverted Double Pendulum, and Walker2d. We show the results Fig. 7.
0.8
ratle
0.8
reward ralle
reward rabo
1MY SMN MUAY
0.4
ARI (I3300
HIRRY FEODAOO
3.2
0.2
MLFYY MIP-N
NUYY
MPN
0.3
BC
0.53 375
3,25 375 2.02
00
3.0
3.5
3.0
1.0
15 epochs
30
epochs
epochs
(a) Hopper
(b) Inverted Double Pendulum
(c) Walker2d
Figure 7: Ablation studies of CODAS.

---

500
0.40
apocts
edodkt
opodia
(a) Prediction discrim- inator under dynamics mismatch
(b) Prediction discrim- inator under policy mis- match
(c) Performance under (d) Performance under policy mismatch
dynamics mismatch
Figure 8: The training curves of CODAS under two types of mismatches. Y-axis in Fig. 8(a) and Fig. 8(b) is the mean of predicted probabilities of the discriminator given of trajectories with mapped states. The higher probability means the data are more like to real. we use "number-x" to denote the rescaling coefficient to the original friction coefficient. For example, "1.2x" denotes the friction cocfficient in the source domain is 1.2 times to the target domain. In the policy mismatch setting (Fig. 8(b) and Fig. 8(d)), the pereentage is to denote the performance ratio of the policy for data collection compared with the data-collecting policy used in the previous experiments. For example, the legend "77%" means the performance the data-collecting policy this experiment has 23% performance degradation.
dynamics mismatch setting (Fig. 8(a) and Fig. 8(c)),
The comparison of RNN-* methods and MLP- methods in Fig. 7 shows that RNN structure is an important component for mapping function training in Hopper and Inverted Double Pendulum tasks. By taking sequential information into consideration, the actions taken in the target domain are more accurate because the policy acts on the mapped states. Based on the RNN mapping function, feeding the sequential information into the discriminator further improves the performance for InvertedDouule and Walker2d. These results verify our argument that the inner structure of the collected dataset crucial for modeling the universal comain-adaptatioon objective in the RL setting. On the other hand, in these tasks, embedded- -DM helps improve the performance both from RNN N to DM-RNN-Y RNN-Y and from RNN-Y DM- OM-RNN-3 -Y. Both embedded-DDD and the sequential-input discriminator improve the performance of CODAS. Embedded-DD DD standardizes the inference process which simplifies the learning complexity. The sequential input discriminator strengthens the gradient signals back- propagated from the discriminator by taking more information into consideration to discriminate the two datasets.
4.4 Robustness to Mismatches
Current UDA methods omit the gap between the dynamics models and the data-collecting policies of the source and target domains [17, 25). Although we follow these implicit consistency assumptions in CODAS, in some real-world applications, the reality-gap gap of dynamics models is non-negligible [19, 37, 38, 39], and the policies in the two domains might not be the same. Unfortunately, whether considering the reality-gap of dynamics models or the mismatching in the policies, the primitive objective of CODAS and previous UDA methods are biased: Even with an oracle mapping function p(s o), the distribution of the dataset in source domain D' and the mapped dataset D := {
p(s o), E D are not aligned [40]. In this section, we analyze the effect of the violation on the implicit consistency assumptions in the current UDA paradigm.
We first change the friction coefficient in the Hopper environment to analyze the effect of dynamics mismatches CODAS. As shown in Fig. 8(a), with a larger friction difference, the asymptotic probability tends to be smaller. The phenomenon indicates that the gap between dynamics models can lead to the gap between the state-action distributions, which can be distinguished by the discriminator. When the friction is doubled, the gap between state distributions starts to affect the deployment performance of policy (Fig. 8(c)). Then we add Gaussian noise with different variance to the original data-collecting policy to collect datasets and train CODAS to analyze the effect of policy mismatch. The policy mismatch is quantified by the performance degradation. We assume that with large performance degradation, the policy behaviors will be more different. The details of the setting can be seen in Appendix E. As shown in Fig. 8(b), the mismatch of collection policies also induces the gap in the state-action distribution. The gap is larger compared with that of the dynamics mismatch. Such larger gap does cause a more significant performance drop of deployed policy.
However, the experiments with 120% and 160% friction, the mapping function is relatively robust to the mismatch. We think that such tolerance comes from the distributional modeling for each step

---

of inference. The distribution modeling improves the robustness by considering the uncertainty of the prediction. The performance drop in both experiments suggest us pay more attention the implicit assumptions not fully discussed in current UDA methods in RI tasks. In Appendix F.4, we further test the tolerance to the mismatch of initial state distribution, which has similar empirical conclusion.
4.5 Performance on Robot Hand Manipulation Tasks
We further test CODAS in four complex hand- hand-manipul manipuuaunl tasks: door, pen, hammer and relocate?. An example of image observation of the hammer task is given in Fig. 9. In the hammer task, the robot arm should hold the hammer and hit the nail into the wood. The policies trained by DAPG [34] are used as the pre-trained source domain policy. The performance of CODAS is given in Tab. 1. In three out of four tasks, CODAS yields reasonable mapping functions for policy deployment. CODAS fails to output correct mapping the relocate task. We think that this failure can be attributed to the goal-conditional nature of the task, where the robot arm needs to both grasp a ball from and take it to randomly initialized location in every episode. The goal-conditional nature increases the complexity of data distribution. To learn a better mapping function, larger dataset is necessary to capture enough information of the distribution.
Figure 9: An example of im-
age observation of the ham- mer task.
Table 1: The reward ratio in hand manimulannliion tasks.
Tasks
hammer
pen
door
relocate
Reward Ratio
0.820
0.701
0.886
0.090
5 Discussion and Future Work
In this work, we investigate the modal unsupervised domain adaptation problem in RL that more cost-efficient than image-to-iman domain adaption. We first point out the intrinsic ill- posedness of distribution matching in the current formulation of the UDA objective. To handle this issue, we deduce the UDA objective based on the decision processes of RL. tasks and the framework of variational inference, then derive a differentiable objective. We also design special model structure to stabilize the training and enforce better dynamics consistency. These components constitute our method, Cross-Modal Domain Adaptation with Sequential structure (CODAS).
The experiments in MuJoCo and Robot Hand Manipulation Tasks show the ability of CODAS to find a correct mapping function between two completely different domains with unlabeled data. We hope that CODAS and its reasonable performance could attract and enlighten more work in this setting. Such results also corroborate that modeling the process of generation and inference more precisely is a promising way to improve the performance on UDA, in parallel with previous methods that focus on exploiting more efficient structural constraints. In the ablation studies, we analyze the effectiveness of sequential mapping structure, trajectory-inple discriminator, and embedded-DM in CODAS. We also test the robustness of CODAS to two kinds of mismatches, namely dynamics and policy mismatches. The results show that CODAS enjoys a reasonable degree of robustness to both mismatches. However, if the mismatches exceed certain thresholds, the bias of the original objective of CODAS will result in degradation of the deployment performance.
In essence, CODAS tries to formulate and solve the UDA problems in RL without relying on any prior knowledge of the two domains. Image-to-image UDA can be regarded as special case of cross-modal UDA. Therefore, it is possible to extend CODAS to the image-to-image UDA problems, and other proposed techniques focused on mage-ttoimaage UDA to pre-train/constrain a CNN encoder can be adopted into CODAS too, which is valuable to investigate in future work. On the other hand, gaps still exist between our experiments (simulation image to state) and the practical visual sim2real problem (real image to state) in theory. By modeling the mismatching of dynamics models and data-collected policies into the CODAS framework, we can build a more robust UDA algorithm when the assumptions in this paper are not strictly fulfilled. We leave them as our future work.
"The environment are modified to support resetting to an arbitrary state, which are also open-sourced in https//ggihub..om/jjaaggy// envs
10

---

Acknowledgements and Disclosure of Funding
We thank Chenxiao Gao for his feedback on the final presentation of this paper. This work is supported by the National Key Research and Development Program of China (2020AAA0107200) the NSFC (61876077, 61876119) the Fundamental Research Funds for the Central Universities (022114380010), and Huawei Noah's 's Ark Lab (HF2019105005).
References
[1] Anthony G. Francis, Aleksandra Faust, Hao-Tice Lewis Chiang, Jasmine Hsu, J. Chase Kew, Marek Fiser, and Tsang-Wei Edward Lee. Long-range indoor navigation with PRM-RL. IEEE Transactions on Robotics, 36(4): (0(4):1110 1115-1134, 2020.
[2] Andy Zeng, Shuran Song, Stefan Welker, Johnny Lee, Alberto Rodriguez, and Thomas A. Funkhouser. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In IROS, 2018.
[3] Kanishka Rao, Chris Harris, Alex Irpan, Sergey Levine, Julian Ibarz, and Mohi Khansari. RL-CyclGGAN: Reinforcement learning aware simulation-to-real. In CVPR, 2020.
[4] Nick Jakobi, Phil Husbands, and Inman Harvey. Noise and the reality gap: The use of simulation evolutionary robotics. In ECAL, 1995.
[5] Karl Cobbe, Oleg Klimov, Christopher Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In ICML, 2019.
[6] Judy Hoffman, Eric Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei A. Efros, and Trevor Darrell. CyCADA: Cycle-consistent adversarial domain adaptation. In ICML, 2018.
[7] Yaroslav Ganin, Evgeniya Ustinova, Hama Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. networks. Journal of Machine Learning Research, 17(59):1-35, 2016.
adversarial training of neural
[8] Zili Yi, Hao (Richard) Zhang, Ping Tan, and Minglun Gong. DualGAN: Unsupervised dual learning for image-to-image translation. In ICCV, 2017.
[9] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, Kun Zhang, and Dacheng Tao.
generative adversarial networks for one-sided unsupervised domain mapping. In CVPR, 2019.
[10] Lukasz Kaiser, Mohammad Babacizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model-based reinforcement learning for Atari. In ICLR, 2020.
[11] Fei Xia, Amir R. Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: real-world perception for embodied agents. In CVPR, 2018.
[12] Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In IROS, 2012.
[13] Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. ROBEL: robotics benchmarks for learning with low-cost robots. In CoRL, 2019.
[14] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.
[15] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. NIPS, 2014.
[16] Tinghui Zhou, Philipp Krähenbühl, Mathieu Aubry, Qi-Xing Huang, and Alexei A. Efros. Learning dense correspondence via 3d-guided cycle consistency. In CVPR, 2016.
11

---

[17] Gregory J. Stein and Nicholas Roy. GeneSIS-RE Generating synthetic images for training secondary real-world tasks. In ICRA, 2018.
[18] Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In CVPR, 2017.
[19] OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving rubik's cube with robot hand. CoRR, abs/1910.07113,, 2019.
[20] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IROS, 2017.
[21] Fereshtch Sadeghi and Sergey Levine. CAD2RL: real single-image flight without a single real image. In RSS, 2017.
[22] Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via aaddomize-to-ccaooiicl adaptation networks. In CVPR, 2019.
[23] Irina Higgins, Arka Pal, Andrei A. Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. DARLA: improving zero-shot transfer in reinforcement learning. In ICML, 2017.
[24] Shani Gamrian and Yoav Goldberg. Transfer learning for related reinforcement learning tasks via image-to-immae translation. In JCML, 2019.
[25] Jingwei Zhang, Lei Tai, Peng Yun, Yufeng Xiong, Ming Liu, Joschka Boedecker, and Wolfram Burgard. VR-Goggles for robots: Real-to-sim domain adaptation for visual control. IEEE Robotics and Automation Letters, 4(2): 1155, 2019.
[26] Rae Jeong, Yusuf Aytar, David Khosid, Yuxiang Zhou, Jackie Kay, Thomas Lampe, Konstanti- nos Bousmalis, and Francesco Nori. superviedd sim-to-real adaptation for visual robotic manipulation. In ICRA, 2020.
[27] Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakrishman, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, Sergey Levine, and Vincent Vanhoucke. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. ICRA, 2018.
[28] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: Training generative neural samplers using variational divergence minimization. In NIPS, 2016.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[30] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model-based offline policy optimization. In NeurIPS, 2020.
[31] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In ICML, 2019.
[32] Diederik P. Kingma and Max Welling, Auto-encoding variational Bayes. In ICLR, 2014.
[33] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs//707.06347, 2017.
[34] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In RSS, 2018.
12

---

[35] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King. Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 5 518(7540), 2015.
[36] Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Rémi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. In ICLR, 2017.
[37] Xue Bin Peng. Marcin Andrychowicz. Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In ICRA, 2018.
[38] Melissa Mozifian, Juan Camilo Gamboa Higuera, David Meger, and Gregory Dudek. Learning domain randomization distributions for transfer of locomotion policies. CoRR, abs/1906.00410,
2019.
[39] Manuel Kaspar, Juan David Muñoz Osorio, and Jürgen Bock. Sim2real transfer for reinforce- ment learning without dynamics randomization. In IROS, 2020.
[40] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In NIPS, 2016.
13


Reviews:
[{"review_id": "mycXHGvAb4", "reviewer": "Reviewer_6QJy", "summary": "This paper purpose to solve sim to real problem from a new viewpoint. Instead of using image or feature level domain adaptation, the authors use sequential information（trajectories) to adapt the image to state, and define a differential and stabilize object for domain adaptation in reinforcement learning, and results perform better than other baselines in all experiments.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ This paper has a solid theoretical derivation for the objective of domain adaptation in RL using Variational Inference.\n+ Lots of experiment is conducted to show the effect of this method including baseline, an experiment in a complex task, ablation study, mapping state error, etc.\n\nWeakness:\n- Without an image, the agent cannot know the geometry of itself or object which can be hard to train a task need to know the shape (for example grasp for different objects).\n- For a simulation environment that has a simple and unrealistic image, it still needs 600 episodes (minimum 100 steps each episodes) data to train the mapping function, which means need more data for the true sim to real.\n- This method is proposed to solve sim2real, no experiment show this method can map a real image to the corresponding state, even for a demonstration.\n\nDetail comments：\nThe main motivation of this paper is for general domain adaptation in sim2real under the same modal. It has a mismatching problem and is costly, so this paper purpose to solve the problem by cross-modal domain adaptation. \nFor the method, there is some previous work also try to adapt between image and state. For example, in [1], the authors adapt image and state to hidden state and use that to train the policy.  [2] also uses sequential information to unsupervised adapt the policy. It would be better to show the advantage compare with these methods.\nFor the “cost” problem, the main idea of sim2real is to use a simulator to reduce the cost of real data collection, for this method although it reduces the cost in simulation, it seems to require more complex real data collection than general domain adaptation(trajectory need action). Besides, this paper does not show any demonstration in real-word scenario.  Therefore, it is hard to evaluate whether this method still remains low-cost in real-world scenarios. From this point, it would be better to conduct some experiments in real world or evaluate the performance of this method with different amounts of target data episodes to show some potential for low real data requirement.\nFor baselines, there is also another domain adaptation method that can be used for cross-modal domain adaptation, such as[1] and [3], it would be better also use this kind of method as stronger baseline.\n\n[1] Wilson, Matthew, and Tucker Hermans. \"Learning to manipulate object collections using grounded state representations.\" Conference on Robot Learning. PMLR, 2020.\n\n[2] Hansen, Nicklas, et al. \"Self-supervised policy adaptation during deployment.\" arXiv preprint arXiv:2007.04309 (2020).\n\n[3] Tzeng, Eric, et al. \"Adversarial discriminative domain adaptation.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "gRpN4_j5XE1", "reviewer": "Reviewer_kd5t", "summary": "The paper provides a variational formulation to train a observation-to-state mapping, as a potential solution to cross domain transfer. The proposed approach is evaluated in MuJoCo control and hand manipulation tasks, showing its advantage over GAN-based approaches.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n- The method shows empirical improvement over baselines across a range of control tasks. \n- The study of changes in environmental dynamics is very interesting and shows that the proposed method can tolerate some degree of dynamic mismatch.\n- While the baselines are rather naive, the paper addresses an interesting problem and proposes a feasible solution. \n\nWeakness\n- The method relies on the ability to reset the simulator and query the oracle one-step transition, while the GAN baselines do not have this information. It strongly favors the proposed method. Stacking the images could already improve vanilla GAN significantly, and what if transitional information is also taken into accounts? Requiring the ability to reset could also be a problem when scaling to real robots. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "g1H6LrqslQ2", "reviewer": "Reviewer_NDFM", "summary": "This paper studies the problem setting in which one has a \"source\" simulator with low-level state information in which one can efficiently train an agent to perform a task, and a \"target\" simulator that shares the initial state distribution and transition dynamics as the source simulator but only offers high-dimensional observations. The goal is to leverage trajectories sampled from a policy in the target domain to learn a mapping $q_\\phi$ from the stream of high-dimensional observations in the target domain to states in the source domain. One can then perform the task in the target domain by feeding observations through $q_\\phi$ and then through the source policy. The authors propose to use variational inference to train $q_\\phi$ and thoroughly demonstrate what adaptation tasks their framework (CODAS) can and cannot perform on MuJoCo and Hand Manipulation. Impressively, using CODAS, the model in the target domain generally achieves 80-90% the performance of the model in the source domain.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading this paper. The problem domain is largely very practical; the assumption in 125-127 that the source and target policies are identical is somewhat limiting in that in the practical setting, we have an optimal policy in simulation but no optimal policy in the real world target domain. However, the authors do a good job of investigating this limitation and also providing intuition on what makes the alignment problem difficult even when it is assumed that the source policy matches the target policy. The experimental results are impressive and the writing is very clear. The algorithm does seem to be somewhat unstable and possibly difficult to train (based on section 3.3) but the authors provide a comprehensive description of their implementation details in the paper and supplementary.\n\nSome minor comments below:\n\n- Figure 1 - this figure seems to suggest that the \"shape\" of the 3-point trajectory is preserved from one domain to another, and that additional structure is what makes the problem easier. However, trajectory shapes in one domain do not necessarily correspond to the same shape in another domain. Is this interpretation and criticism of the figure correct?\n- Figure 7 - is it also worth considering ablating the importance of the initial state distributions matching?\n- 98 - how well is $q_\\phi$ able to extract the correct state from an observation quantitatively? A training curve would be interesting to see. Figure 5 is just qualitative.\n- 230-231 - what is the intuition for image->action requiring a higher sample complexity to learn than image->state?\n- 245-247 - is there some suggestion for how one could fine-tune either the source policy or $q_\\phi$ to bump the performance from 85% to 100%?\n- 306-307 - is it not easy to collect a larger dataset and test this reasoning?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do a good job of discussing the importance of the assumption in the derivation of CODAS that the dynamics and data collection policies match between source and target. They also offer an example of a task for which CODAS performs poorly (relocate in Table 1) and claim the reason is the \"goal-conditional nature\" of the task.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "_F3wfNSZSo", "reviewer": "Reviewer_cBxr", "summary": "This paper introduces an interesting work that applies UDA technique to connect the visual gap between the real and simulation domain in RL simulation tasks. An inference function is learned with variational inference to transfer the policy learned from the simulator instance to a realistic instance.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ This basic idea of this paper is easy to follow.\n+ This paper shows an easy solution to reduce the visual gap between real and simulate environments when solving the RL tasks. The idea to transfer status and policy from status domain to visual domain with domain adaptation is interesting.\n \n \nThis paper shows some interesting ideas and novelty. But it still contains some uncleared descriptions and concerns. There are a few questions or problems as listed below:\n \n\t1. L135 mentioned that related works learn the mapping function by minimizing the divergence between distributions. What related works exactly here are talking about? Is there any comparison of these works in your experiments? \n\t2. L146 What is the Gaussian distribution here descript? The structure of q() and the output distribution should be clarified. \n\t3. L158 The difference of ELBO loss with VAEGAN should be more specific list here.\n\t4. The definition of D in Eq4. is unclear. \n\t5. L178 What are the structure of the DM and embedded DM model? This is not clear. \n \nWhy name your method CODAS instead of CMDAS ?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper has limited impact on the related community since similar works have been explored.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles the problem of visual-input sim-to-real and tries to overcome the reality gap between images rendered in simulators and those from the real world. It presents an approach that, instead of building and running simulators that render costly high-quality images, generates only low-dimensional states in simulation to learn the policy, and then learns encoders from and decoders to observations that handle sequential observations. This enables to train and transfer state dynamics from a source environment to a target environment with only high-dimensional observations. It then proposes a mathematical derivation for the cross-modal unsupervised domain adaptation problem. The method is then evaluated on MuJoCo tasks (OpenAI Gym and robot hand manipulation) and compared to several GAN, CycleGAN and temporally stacked GAN baselines, where it shows better domain adaptation and transfer performance.

Reviewer NDFM praised the paper and their minor comments were addressed. Reviewer cBxr’s most negative comment was about the limited impact of the work due to similarity with existing work, but the authors contest this review and justify their contribution as a combination of mathematical derivation of the problem as variational inference and the addition of sequential modeling of the dynamics (as opposed to VAE-GAN). Reviewer 6QJy complained about missing real-world data evaluations to make this work relevant (a claim disputed by the authors and reviewer NDFM), as well as missing comparisons to two recent sim2real techniques that “adapt image and state to hidden state and use that to train the policy'' and that modeled sequences of states from observations (the authors provide a clarification).

Review scores are (5, 5, 6, 8, average 6). Reviewers cBxr (score 5) did not respond to the rebuttal and did not update their scores but I believe they could promote their score. While waiting on this reviewer, I am therefore willing to challenge them and promote this paper to an acceptance.

Decision: Accept (Poster)