Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency
Authors: Qiang Zhang
Tete Xiao
Alexei A Efros
Lerrel Pinto
Xiaolong Wang
Venue: ICLR 2021 Oral (2021)
Keywords: self-supervised learning
robotics

Abstract: At the heart of many robotics problems is the challenge of learning correspondences across domains. For instance, imitation learning requires obtaining correspondence between humans and robots; sim-to-real requires correspondence between physics simulators and real hardware; transfer learning requires correspondences between different robot environments. In this paper, we propose to learn correspondence across such domains emphasizing on differing modalities (vision and internal state), physics parameters (mass and friction), and morphologies (number of limbs). Importantly, correspondences are learned using unpaired and randomly collected data from the two domains. We propose dynamics cycles that align dynamic robotic behavior across two domains using a cycle consistency constraint. Once this correspondence is found, we can directly transfer the policy trained on one domain to the other, without needing any additional fine-tuning on the second domain. We perform experiments across a variety of problem domains, both in simulation and on real robots. Our framework is able to align uncalibrated monocular video of a real robot arm to dynamic state-action trajectories of a simulated arm without paired data. Video demonstrations of our results are available at: https://sites.google.com/view/cycledynamics .

Published as a conference paper at ICLR 2021
LEARNING CROSS-DOMAIN CORRESPONDENCE FOR CONTROL WITH DYNAMICS CYCLE-CONSISTENCY
Qiang Zhang
Tete Xiao UC Berkeley tx txiao beccao@eee.bbbekeeeeee..... edu
Shanghai Jiao Tong University zhangqi ang201 fts.ttu.du.......
Alexei A. Efros
Lerrel Pinto New York University lerrelêcs nyu.edu
Xiaolong Wang
UC Berkeley
UC San Diego
efros@eecs berkeley, edu
xiw012@ucsd. edu
ABSTRACT
At the heart of many robotics problems is the challenge of learning correspon- dences across domains. For instance, imitation learning requires obtaining cor- respondence between humans and robots; sim-to-real requires correspondence between physics simulators and the real world; transfer learning requires corre- spondences between different robotics environments. This paper aims to learn correspondence across domains differing in representation (vision vs. internal state). physics parameters (mass and friction), and morphology (number of limbs). Importantly, correspondences are learned using unpaired and randomly collected data from the two domains. We propose dynamics cycles that align dynamic robot behavior across two domains using cycle-consistency constraint. Once this correspondence is found, we can directly transfer the policy trained one domain to the other, without needing any additional fine-tuning on the second domain. We perform experiments across variety of problem domains, both in simulation and on real robot. Our framework is able to align uncalibrated monoc- ular video real robot arm to dynamic state-action trajectories of simulated arm without paired data. Video demonstrations of our results are available at: https::/// http://jjtttug// git..b.tthuuu...g... io/ wwc.cceeeeeeeeooooooooooooooeeee laroool__ddaaaeee dynamics html.
INTRODUCTION
Humans have remarkable ability to learn motor skills by mimicking behaviors of agents that look and act very differently from them. For example, developmental psychologists has shown that 18-month-old children are able to infer the intentions and imitate behaviors of adults (Meltzoff, 1995). Imitation is not easy: children likely need infer correspondences between their observations and their internal representations, which effectively aligns the two domains. Learning such a cross-domain correspondence particularly valuable for robotics and control. For example, in imitation learning, if we want robots to imitate the motor skills of humans (or robots with different morphologies), we need to find the correspondence in both visual observations and morphology dynamics. Similarly, when transferring policy trained in simulation to real robot, we, again, need to align visual inputs and physics parameters across different environments.
To align the skills across different domains, several prior approaches have proposed learning invariant feature representations across the domains (Gupta et al., 2017: Sermanet et al., 2018). Policies or visual representations are trained to be invariant to the changes which are irrelevant to the downstream task, while maintaining useful information for cross-domain alignment. However, these methods require paired and aligned trajectories, usually collected by pre-trained policies or human labeling, which is often too expensive to collect for real-world learning problems. Additionally, invariance is a rather strong constraint, and might not be universally suitable. The reason is that different invariances might be beneficial for different downstream tasks, which has been recently studied in superviee visual representation learning (Tian et al., 2020).

---

Published as a conference paper at ICLR 2021
Render
Ptysics Maddan
observation conespandencc
comegoondecc
HallCheelah cress physics aligmment
Render of Y--1
observation
observation
contespoodeeo
adtion
conteponniiie
(a) X-ammmbtt real swm aligmmett
(c) Swimmeer cross morphology aligmment
Figure I: We propose to learn observation correspondence (blue arrow) and action correspondence (red arrow) across domains using Dynamics Cycle-Cose Codde-Cosiitencc. Our applications include: (a) Aligning real robot images with simulation states: (b) Aligning actions between environments with different physics parameters (We use different rendering indicate that the physics are different); (c) Aligning both actions and observations between agents the same time with different morphology.
Instead of learning invariances, emerging line of research focuses finding correspondences by learning to translate between two different domains with unpaired data (Zhu et al., 2017; Bansal et al., 2018). While this translation technique has shown encouraging results imitation learning (Smith et al., 2019) and sim-to-real transfer (Hoffman et al., 2017; James et al., 2019), it is limited to finding correspondences only in the visual observation space. However, in real-world applications, besides visual observations, the physics parameters and morphology dynamics between two domains are also often misaligned. Hence, solely learning with passive visual correspondences, one unable to reason about the effects of dynamics. We must go beyond the image space and explicitly incorporate dynamics information to truly extend correspondence learning to aligning behaviors.
In this paper, we take the first steps toward learning correspondences which can align behaviors range domains including different modalities (vision vs. agent state), different physical parameters (friction and mass), and different morphologies. Importantly, we use unpaired and unaligned data from the two domains to learn the correspondences. Specifically, we propose to find observation correspondences and action correspondences the same time using dynamics cycle-consistency. Our dynamics cycles chain the observations and actions across time and domains together. The consistency in the dynamics cycle indicates consistent translation and prediction results. The input data to learning algorithm takes the form 3-element tuples from both domains: current state, action and the next state. Figure (a) exemplifies our model, which a 4-cycle chain containing the observations of one domain (X,, 1 (real robot in Figure (a)) at two time steps, and another domain Y++1)) (simulation in Figure (a)). To form a cycle, we learn domain translator G X, + Yt to translate images to states and a predictive forward dynamics model in state space F Y: Xu H Yi+1 where u, represents the action taken at time t. and a, the corresponding action in the real robot domain. The forward model in the real robot domain is not necessary in our framework. The training signal is: given observations in time I, the future prediction in time + should be consistent under the consistent action taken across two domains, namely dynamics cycle-consistencyy
We explore applications both in simulation and with a real robot. In simulation, we adopt multiple tasks in the MuJoCo (Todorov et al., 2012) physics engine, and show that our model can find correspondence and align two domains across different modalities, physical parameters (Figure 1(b),, and morphologies (Figure I(c)). Given the aligmment, policy trained in one domain directly to another domain without further optimizing the RI. objective. the xArm Robot (Figure ((a)). Given only uncalibrated
transfer a reinforcement learning (RL)
For real robot experiments, monocular videos of the xArm performing random actions, our method learns correspondences between the real robot and simulated robot without any paired data. At test time, given video of the robot arm executing a smooth trajectory, we can generate the same trajectory in simulation.
RELATED WORK
Learning invariant representations. To find cross-domain alignment, researchers have proposed to learn representations which are invariant to the changes unrelated to the downstream task (Tobin al., 2017; Peng al., 2018: Gupta et al., 2017; Sermanet al., 2018; Liu et al., 2017b; Pinto et al.,
2

---

Published as a conference paper at ICLR 2021
2017: Sadeghi & Levine, 2016; Yan et al., 2020; Chen al., 2020; Andrychowicz et al., 2018). For example, domain randomization (Tobin et al., 2017: Sadeghi & Levine, 2016; Andrychowicz et al., 2018; Ramos et al., 2019; Zakharov et al., 2019; Wu et al., 2019) aligns the simulated and real world for policy transfer. However, it assumes that differences between two domains can be covered by hand-crafted augmentations, which may not hold when the domain changes happen lie outside of these assumptions. To align two domains where the dynamics are different, Gupta et al. (2017) propose to learn invariant features with pairs of states from two domains. However, paired data is hard to collect, and the method is limited to state space, while real-world observations are often based on images (Taylor& & Stone, 2009).
Learning translation. Instead of learning invariance, our method is related to works which learn the mapping across two domains for alignment (Taylor et al., 2007; Ammar et al., 2015; Tzeng et al., 2015; Joshi & Chowdhary, 2018; Kim et al., 2019; Smith et al., 2019). For example, Tzeng et al. (2015) design an approach to weakly align pairs of images in the source and target domains. Given the paired images, they can perform adaption from simulation to real robot. However, this work has only focused on translation between visual observations. Going beyond visual adaptation, Ammar et al. (2015) utilize unsupervised manifold alignment to find correspondence between states across domains from demonstrations. However, this method uses hand designed features, which restricts its generalization ability. Kim et al. (2019) propose imitation learning with unpaired and unaligned demonstrations. While with less constraint, it requires a trained RL. policy to collect demonstrations in both domains for training, and RL is involved in the correspondence learning process. This leads to learning correspondence only relevant to specific task. In contrast, most of our experiments assume that we do not know the downstream task and we do not have access the rewards for RL. Hence, it is a more general problem setting and can be used for variety of applications. Our method can learn correspondence between simulated and real robot through unpaired and randomly collected trajectories.
In transfer learning, several works have looked at architectural novelties to improve transfer across RL problems (Parisotto et al., 2015; Pinto et al., 2016; Rusu et al., 2016a; Barreto et al., 2017; Omidshafici et al., 2017; Rusu et al., 2016b). Our method of using cycle consistency pursues an orthogonal direction of architecture design and is compatible with these approaches.
Wele-Consistency. Our work inspired by literature on cycle-consistency (Zhou et al., 2016; Zhu et al., 2017; Liu et al., 2017a; Hoffman al., 2017; Bansal et al., 2018; Bousmalis et al., 2018; James et al., 2019). For example. CycleGAN (Zhu et al., 2017) uses cycle-consistency loss with the Generative Adversarial Networks (Goodfellow et al., 2014) for unpaired image-to-image translation, which is subsequently extended for videos (Bansal et al., 2018) and domain adaptation (Hoffman et al., 2017). Similar techniques applied in sim-to-real transfer by training a image translation model between simulation and real-world images (Stein & Roy, 2018) or aligning both the simulation and real images the same canonical space (James et al., 2019). Recently, Rao al. (2020) propose RL-CycleGAN to perform simlreal image translations by adding an extra supervision signal from the Q function. However, all these works are restricted to visual alignments, while ours can align agents cross different dynamics and structures.
LEARN CORRESPONDENCE USING DYNAMICS SYCLE-CONSISTER
Problem setup. We aim to learn correspondence across various domains, i.e., input modalities, physics parameters, and morphology. We formulate the trajectories of domain X and Y as Tx = (Xe, ae, X and Ty (yr, Ur, Yt+1), where x R and y E R"= are observation representations in domain X and Y, a E Rm and u E R are action representations in domain X and Y. and ( is time step. Without loss of generality, we assume to learn correspondence from domain X to domain Y. Suppose that we have observation alignment functions G X + Y. and action alignment function H Ax - U and its inverse counterpart H-1 as a function P : YXU + A. We define two types of correspondence follows.
Observation Correspondence, i.e., what the representation of one observation in domain X should correspond to if it is in domain Y, and vice versa. For example, if X is visual sensing of an agent while Y is the state (e.g.. joint angle) of the same agent, G functions as a state estimator. If X is the
I We will subsequently show that for various applications only action mapping ought be bidirectional, whereas observation mapping can be unidircctional.
3

---

Published as a conference paper at ICLR 2021
domain
domain Y
domain
domain Y
domain X
domain
(Dy)
G (Dy)
x
VE
x
Ye
Y
(Du)
(Du)
a
P (Da)
P (Da)
FDr
F
dyranics tydle luss
detanics spcle luss
dynanics cycle luss
X
Ve13
X
V111
Y
G (Dy)
G (Dy)
(a) Cross-physics only
Cross-modality only
(c) Joint model
Figure 2: Model framework: (a) Model for only cross-piis cross-physics alignment; (b) Model for only cross modality alignment; (c) Joint model for waw--maddlit--nddppyaaae alignment. Red arrows indicate correspondences between actions and blue arrows indicate correspondence between observations.
state of one agent while Y is the state of structurally different agent, such Sawyer arm and URS arm, G' aligns the states at same stage towards a common goal (e.g., robot joint positions). We denote two correspondent observations between X and Y as x + y.
Action Correspondence, i.c., with correspondent initial observations which actions to execute so that the next observations in two domains remain correspondent. For example, if and are two environments with different physics parameters, with the initial observations x,, Yr and X, e yt. after action a, is executed in domain X and leads to the next observation Xt+1, alignment function H should find the action u, which leads to next observation Y++11 in domain Y where X < Ye+1+ and vice versa for H-1 We denote two correspondent actions from and as (x,, ar) < (yx, Ur).
Learning observation correspondence and action correspondence enables estimating states from visual input, adapting to environments with different physics, and being able to function even when the structure of the agent changes.
We begin by simply mapping states across domains by adversarial training. Given unpaired samples {x,} E X, and [y;} E Y, a mapping function G can be learned with a discriminator Dy with the adversarial objective, where G' tries to map x onto the distribution of y, while Dy tries to distinguish translated samples G(x) against real samples y:
Method.
min max Ladv (G. Dr) Eywp(y) [log (y)) Ex-p(x) [log (1 - Dy (C(x))) G Dy
(1)
The adversarial objective reaches global optimal when the mapping function G can perfectly ground the translated samples onto the distribution defined by {y.).
We learn an action mapping function H A A +> U which maps actions from domain X to domain Y. and model its inverse counterpart H-1 as a function P: XU + with separate parameters. Besides using two adversarial losses with discriminators Dy in Y and D in X, i.c., L (H, Dv) and Ladv (P, A) we add cross-domain cycle consistency loss (Zhu et al., 2017) into the objective:
min Ldoo eye (H, P) = En--p(a)) P(y (x, - -ll.] H.P
(2)
which implies that the translated action should be able be translated back: P(y, (x, a))
Nevertheless, the structure of learnt mapping by adversarial training is loosely constrained. Vanilla adversarial training may map all samples X to few samples of Y. which still minimizes the adver- sarial objective. Adding domain cycle consistency loss does not solve the problem fundamentally: for example, given two correspondent but unpaired observations, i.e., Xt, Ye and Xt+1, Yt+1, can map X₁ to Y++1 and G-1 can still map y++1 back to x,, which does not violate domain cycle-consistency.
Beyond only relying domain cycle consistency, exploit the transition dynamics of two domains, termed dynamics cycle-consistency, As illustrated Figure 2(c), we map the observation-action pair at time step t X₁ and a, from domain X to using and H, then exccute the translated observation and action Y; and ú;) in domain Y by its transition dynamics Ty Y + Y to get the next observation, which is expected to be correspondent to the next observation from domain X, i.e. Ty (Yt, ū,) < X According the definition of observation correspondence, Ty (Y,, ú;) should
4

---

Published as a conference paper at ICLR 2021
be the same as G(x++1), as expressed in the objective:
min [(G H) Jaya.eye = (xx G.
[[(((()) - Ty (G(x)), H (x,, a )]]]]
(3)
One obstacle remains. The transition dynamics Ty in Equation 3 is in fact the physical property of a simulator or the real world, hence it is not differentiable for back- propaaaapat In consequence, we train a forward model which takes an iiservation-action pair as input and predicts the next observation to approximate the dynamics of the environment. Since we have access to trajectories from Y. we can directly train the forward model using supervised regression objective:
min foorward (F) vee [|>ve1 - Fly,, m)ll
(4)
Note that forward model F is first pre-trained and it is not optimized together with the dynamics cycle-consistency objective, as otherwise G and can learn to map everything to zero so that Tdyn becomes zero, which leads to a trivial solution. Consequently, our full objective is:
map
Cral Ayy (G, H) + (Ludv (H, Do) + Laav (P, Da) + Cdom sye (H) + Aa (G) (5)
where Ao λ1 and X2 are constants balancing the losses.
Optimization. We collect unpaired tra- jectories Tx and ry by executing random actions from both domains. Directly opti- mizing the full objective end-to-end leads to model collapse, as it involves joint op- timization with multiple neural networks:
Algorithm 1: Alternatingyy Joint Training Algorithm
Input: Domain X: rx {(x,, ar, x(+1)]
Domain Y: ry = {yr, V++1) // Training Forward Model Stage
train Llorward (F) (Eq. 4) to learn transition dynamics
G and H can easily discover a "short- cut" solution, where the translated ob- servations and actions are not valid but they can fool the forward model opti- mize the dynamics sycle-consistency ob- jective. Since the forward model is only optimized on trajectory data ry, thus we first pre-train the forward model and fix its parameters throughout the following training procedure. We initialize the ac- tion mapping function using an algorithm detailed in the Appendix A.4. We pro- pose to employ alternating training procedure for the full objective: When we train the observation mapping function G and its auxiliary discriminator Dy. we fix the action mapping function H and P; then when the action mapping function H and P with Du and DA trained, we fix the observation mapping function G. Since the action mapping functions are reasonably initialized, at the beginning of training procedure the observation mapping function is optimized. It is grounded on good action mappings, as well as the dynamics of environments by dynamics cycle consistency, thus is constrained from learning an arbitrary short cut. Subsequently, action mapping functions can be further fine-tuned once we obtain a good observation mapping function (Algorithm 1).
Ty in domain Y:
/ Alternatingly Training Stage
for: i = to e do
reset \1, set X = 0; fix weight of G;
for j = 1 to e1 do
| using Ctll (Eq. 5) to train model H and P;
reset A2 set W = 0; fix weight of H and P; for to 62 do using Lrull (Eq. 5) to train model G; return State alignment model G Action alignment model H and P
Tasks. Our formation of correspondence learning is broad and general, and it enables many applica- tions which typically require intricately designed frameworks or are hard to solve without paired data. Specifically, we study the following three tasks:
The first task is cross cross-physics alignment, where domain X and domain Y are two environments with different physics parameters but same input modality. As shown Figure 2(a), same input modality indicates that observation correspondence always holds, i.e., x, Yy; different physics parameter indicates that executing a same action at the same initial observation in separate environments results in different next observation. After learning correspondences, assuming we have policy in domain Y. we can transfer it to domain X by mapping the predicted action of the policy u from domain Y to X with action mapping function P. The translated action å can then be executed in domain X.
The second task is cross-modality alignment, where domain X and domain Y are different sensing (observation) modality of the same agent, which implies that action correspondence between two domains always hold (see Figure 2(b)). In other words, H and P are both identity mapping, and a, uy Thus we can set 7 0 in Eq. 5 in training. A predominant choice is X being image while
5

---

Published as a conference paper at ICLR 2021
Y being state, where G essentially leams to perform state estimation. Moreover, we can execute policy which is originally trained on state space in image space, as the imput x; in image space can be translated by G before fed into the policy based on state space, yielding a predicted action u, which can be directly executed in domain X.
Combining the above-diseusses two tasks yields the third task, in which cross-physics and cross- modality alignment are realized simultaneously, thanks to our proposed joint alternative training procedure. We refer to it as cross-modality-and pan--yysissss alignment, as shown in Figure 2(c). This formulation can be further extended to another task, where domain X and Y are two agents with different morphologies, termed as cross- moos-morphhlgy alignment. For example, domain X can be a three-leg cheetah and domain Y can be a two-leg cheetah. In this case, the representations of x x and a / u are fundamentally different, yet intrinsically they share similarities in locomotion.
As the correspondence is established between two domains, it can be applied to different downstream applications. Suppose that our goal is to transfer a policy trained in domain Y to X. Inference includes three steps: (i) Given an observation x, in domain X, use observation mapping function G to translate x, to yr: (ii) Execute the policy in domain Y given yt, and obtain the action output u;; (iii) Translate the action U, from domain Y back to domain X with the action mapping function P.
Implementation Details. The networks D, F, H, P are implemented by MLPs, and network G is a ResNet-18 (He et al., 2016) with 4-layer MLP head. For the inputs of G. instead of using one static image, we concatenate the current frame and two consecutive past frames together to capture any motion information. We first train the forward dynamics model F for 20 epochs using Adam (Kingma & Ba, 2014) with 0.0001 learning rate. We then train the other networks for 50 epochs with the same learning rate. We set e and e2 to 5000 steps in Algorithm 1, See Appendix for more details.
SIMULATION EXPERIMENTS
We first test the efficiency of our framework and conduct ablation studies in simulation environments. We choose MuJoCo physics simulator as test bed. We model domain X and Y two different environments, where input modality, physics parameters, and morphology structures of the agents can vary. We believe that our method can be applied to lot of environments. However, in this paper we focus on the representative ones including four tasks based on OpenAl Gym (Brockman et al., 2016), i.e., "HalfCheetah", "FetchReach", "Walker" and "Hopper", and one task based on DeepMind Control (Tassa et al., 2018), i.e., "FingerSpin". We perform experiments with different settings including: (i) Cross-physics alignment, where only the physical parameters are different in two domains; (ii) Fross-modalle alignment, where only the observation space is different; (iii) fooss-modality-anan aad-iity--nn--hssiicse alignment, joint task of (i) and (ii); (iv) pross-morphology alignment, where agent structures in two domains are different. To sample the training data, we randomly collect 50k unpaired trajectories in both domain and domain Y in most settings. The evaluation dataset size is 10k. Besides evaluating on the alignment errors, we also benchmark how well the pre-trained RL policies in one domain can be transferred to another domain. To pre-train the policy, we use DDPG (Lillicrap et al., 2015) with HER (Andrychowicz et al., 2017) for "FetchReach" and TD3 algorithm (Fujimoto et al., 2018) for other environments. Note that we do not need to further fine- tune domain. We report the task success rate for "FecchReach" and
the policy for transferring task rewards for the other environments. All RL policies are trained with 5 different seeds. More details about our method implementation and the reference policies can be found in the Appendix B.
Cross-physie alignment. In order to create en- vironments with different physics parameters, we modify armature and mass in the environments. We use default armature and torso mass parameters in domain Y. To create do- main X, we increase the ar- mature for tasks including "HalfCheetah", "FetchReach",, "FingerSpin", and modify the torso mass for "Walker" and "Hopper" (see details in Appendix A. Different tasks are sensitive for different physical parameters, e.g.,
Tasks
Oracle, Y Direct, Y + DR, +-X Ours, Y X Oracle,
HalfCheetah
6270+123
3651+666
37632752
3997-1438
67694185
FingerSpin
804
183
492:281
562::124
665
FetchReach*
100%
100%
100%
100%
100%
Walker2d 875=24
5161395
546:1258
667=1744
816:17
Hopper 236-12635
1542=1041
1683:s809
1919-1794
4640++554
Table 1: Cross-physics. Results on transferring a policy trained on domain Y to domain X. DR: domain randomization. Task successful rate is reported.
6

---

Published as a conference paper at ICLR 2021
12
6.200
3000
Narthweihh
143
1600
0.2
1.4
1.2
3400
1200
p %
6350
-000
100
a20
0.6
A39
800
2002
.000
64
samplle dotaset number(k)
10
20
30
40
50
6.000
11 arror
Taisk reward
(a) Data scale ablation
(b) Ablation for discriminator
(c) Ablation for supervised case
Figure Ablation study with HalfChectah. (a) LI error for different dataset scale; (b) Ablation with
discriminators; (c) Combining our method with supervised state estimation (using paired image-state data).
L1 Error Cycle-GAN
RL Score
Tasks
Random
Ours
Oracle, Y Random, X Cycle-GAN Ours, Y-+ Oracle,
HalfCheetah
2.18
2.07
0.57
6270+12a
-89
-199
1504:1256
3689 2247
FingerSpin
1.61
1.92
0.23
804
0=0
Ozo
341+339
765 + 68
FetchReach 0.87
0.94
0.05
100%
0%
0%
92%
100%
Table 2: Evaluation of cross-modality alignment, including I.1 error of state estimation, and RI policy performance on the original domain Y and after transferring domain X. 1; Task successful rate reported.
while changing armature yields noticeable effect on "HalfCheetah", changing mass does not. We tackle the hard cases where physical changes matter. In this setting, we obtain the unpaired training data from two domains with a pre-trained policy in domain Y.
Results are shown in Table The 1st column (Oracle, Y) reports the performance of the policy in the original domain Y. Directly testing this policy in environment X results in significant drop terms of RL scores (2nd column), due the disparity in physies parameters. We also train policy with physics domain randomization (DR, 3rd column) for direct transfer (see Appendix A. for details). Our method (4th column), which maps actions predicted by RL policy from domain Y to X, demonstrates superior performance across all tasks compared to the direct deployment as well as DR baselines. We provide results of training RL in domain X in the last column (Oracle X) as an upperbound. We also implement Cycle-GAN which only learns random projections between actions two domains, thus we do not report the numbers.
Cross-modality alignment. In this setting, we use RGB images as observations in domain X and the internal state of agents as observations in domain Y, while keeping physics parameters the same. G' is then essentially state estimator. We execute random actions without pre-trained policies in both domains to obtain unpaired training trajectories. As even supervised learning for state estimation with the same number of image-state pairs works poorly for "Walker" and "Hopper", report the results on "HalfCheetah", "FetchReach", "FingerSpin" in this setting. We compute L1-distance between the predicted states and the ground-truth states from simulator X (although training) as an evaluation metric. We also use RL performance as another metric: we train an RL policy state space (domain Y). and test it in image space (domain X) by executing the predicted action based on estimated states from images.
do
them for
We compare with baselines: a random projection baseline; image-state Cycle-GAN baseline, performing unpaired image-state translations without using dynamics (see Appendix C for details). As shown Table 2, our approach performs significantly better than the Cycle-GAN Gycle-GAN baseline in both 7.1 error and the RI. scores (the return reward), which shows the importance of incorporating the dynamies into the cycle. Note that even when the ground-truth paired (image, state) samples provided, this is still difficult since images lie in high-dimensional space. By exploiting the dynamics yycle-consistency, our method is able to do state estimation for transferring RL policies. We also provide results directly training policy on the two domains (Oracle, Y and Oracle, X).
We perform ablation studies on different elements in training with the "HalfCheetah" environment: (i) the number of training samples: (ii) the role of the discriminator. We report the results in Figure 3. can be seen that the LI error of state estimation reduces as training data for our dynamic cycles increases (a); training with the discriminators improves both II error and transferring policies by
7

---

Published as a conference paper at ICLR 2021
large margin , comparing to training without the discriminator (b), as adversarial learning with the discriminator can largely reduce the search space for finding correspondence.
We further explore our approach with supervised state estimation: Given paired image-state data, can train a state estimator with supervised learning. We combine our dynamics cycle-consistency objective with the supervised objective train the state estimator. As shown in Figure 3(c), We compare the results on the model using dynamics cycle-consistency ("w") and without the dynamics cycle-consistency ("w/o"). The "Ll error" measured here are on the training states which are randomly collected, the "Task reward" are from running the policy which sees states collected by the policy. We observe improvement on transferring policies with the joint model over the counterpart trained only with the supervised learning objective. In terms of LI error of state estimation, although using paired data (strong supervision) per se will yield the best model on I.I metric (especially the state estimator in this way is trained via L1 loss. However, overfitting to the L.1 loss does not completely transfer to better policy performance on downstream tasks, since the policy can observe novel states that are not in the training set. In our case, adding dynamics objective biases the LI error by a small margin as the model cannot solely optimize for state estimation error, but it must learn dynamics well and in turn increases the task performance. Consequently, incorporating the dynamics cycle-consi can provide extra regularization and improve generalization on test data.
Cross mros--oddllttyande physics alignment. Eval- uation on two domains with different physics parameters and different input modalities. Following the cross- modalityyy setting, we sample our unpaired training data randomly. We report the results of transferring RL policy in Table While this setting is very challenging and the cross-modality Cycle-GAN method fails (3rd column), our method can discover the correspondence from randomly collected unpaired trajectories (last column). We perform ablation by only training to align the observations, without the translator between the actions (4th column). This shows the importance of our joint optimization approach.
Tasks
Oracle, Y Random Cycle-GAN Ours (only M) Ours (Full)
HalfCheetah
6270.123
-448
-226184
856
1251::297
FingerSpin
804+89
Ozo
0=0
213+54
305-1-43
FetchReach 100%
0%
0%
92%
92%
Table 3: Cnos--odaaliy--an--pyyye Results on transferring RI. policies.
morss-morphllogy align- ment. Evaluation two domains with different mor- phology. We experiment with two tasks (see Ap- pendix A.3): (i) domain Y with 2-1eg HalfCheetah and domain X with 3-1eg HalfCheetah; (ii) domain Y with 3-limb Swimmer and domain X with 4-limb Swimmer. In this setting, the unpaired trajectories are also randomly sampled and our model learns to align the observations and actions the same time. Once the correspondence is found, we can transfer the RL policies from domain Y to X. We compare to two baselines: one is the Cycle-GAN to perform both state-state and action-action translations, the other is using our action repetition initialization strategy before training (INIT, this baseline repeats the actions of the shared joints while uses actions for novel joints from its nearest joint, see Appendix A.4). As shown in Table 4, our approach can still perform reasonably well without finetuning the policy while the baselines completely fail. The reason behind this is that both INIT and Cycle-GAN baseline share the critical flaw: they do not take dynamics information into account. Given unpaired agent states, Cycle-GAN cannot find the correspondence between the two domains. In contrast, our method learns dynamics via dynamics cycle-consistency, which outperforms those which don't by a large margin.
Tasks
Oracle, Y Random Cycle-GAN
INIT
Ours, Y-XX
Cheetah Swimmer 366+26
6270=123 2500=59
-43+52
-37
2471
-1#1
14+5
-15+3
204
Table 4: Cross-morphology. Results on transferring RL policies.
5 REAL ROBOT EXPERIMENTS
We use an xArm robot for the cross-modality alignment task. The goal is to estimate the simulation states (domain Y) given the real robot images (domain X), without any paired image-state data. We do not have access to the internal states of the real robot. We use an uncalibrated RGB camera to capture the videos of the robot movements. We collect the real robot videos by randomly executing end-effector positional control. We collect random trajectories in xArm simulator. The training set includes 1 k triplets (image, action, next image) of the real robot. We collect two testing sets from
8

---

Published as a conference paper at ICLR 2021
Method
Random
Smooth
7
Random G
0.30
0.18
Cycle-GAN (B)
0.18
0.21
Oyde-GEN
Ours (E)
0.025
0.033
7
Ours (J)
0.031
0.044
Figure 4: Visualization of learnt correspondence from RGB images to robot joint states with xArm robot. We render the predicted states simulation with green background. While Cycle-GAN struggles to find the correct correspondence, the results of our method highlights the importance of dynamics sele--onistteeyy objective. (Best viewed in Adobe Acrobat to see the GIF of the last column.)
Table 5: Real robot results. We mea- sure the LI enror (smaller better) of end effector position estimation. We experi- ment with either end effector position (E), or (ii) joint positions (J) as the ob- servations in simulator.
the real robot: a) 1,000 samples of random movement (Table 5, 1st col.), and b) 100 samples of smooth movement (Table 5, 2nd col.).
We conduct experiments using either end-effector position or joint poses (7 joint positions) as observations in simulation. Note the action is defined by the delta movement of the end-effector, not the exact position of the end-effector. Thus there is shortcuts for directly estimating the end-effector position and even harder for joint positions. We measure the L1-distance between the method with
predicted and ground-truth end-effector position for evaluation. We compared Cycle-GAN baseline (Appendix C). As shown Table 5, the results from Cycle-GAN is close to random and our method with dyanmics cycle-consistency achieves much lower state estimation error. Besides training with end-effector as observations (Ours (E)), we also use joint poses as observations (Ours (J)) which increase the difficulty on learning the correspondence. The reason why Ours (J) is harder is that end-effector (Ours (E)) estimation does not involve the structural information of the robot, and of lower dimension. The joint poses contain 7 robot joint positions. Therefore both the state mapper and the dynamic model should perform better with end-effector than joint poses, yielding better overall performance. Even so, our results are still much better than Cycle-GAN with end-effector observations. We also visualize the translation results by rendering the states in simulation in Figure 4, and observes that our state estimation results are well aligned with the real robot video.
CONCLUSION
We propose a novel framework find observations and actions correspondence across domains using dynamics cycle-consistency We show the efficacy of our method on multiple downstream applications in both simulation and on a real robot. While previous approaches relies on paired data or RL polices on collecting the data for learning, we provide general framework that can learn correspondence from randomly sampled, unpaired data, independent of the defined RL task. This allows the correspondence be generalized to diverse downstream applications.
Acknowiedgements. We like to thank Jeannette Bohg for helpful discussions this project. This work was supported, in part, by grants from DARPA, NSF 1730158 CI-New: Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CI), NSF Research Platform, research grants from Berkeley DeepDrive and SAP, and gifts from Qualcomm and TuSimple.
CC*DNI Pacific
REFERENCES
Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew E Taylor. Unsupervised cross-domain transfer policy gradient reinforcement learning via manifold alignment. In Twenty-Ninth AAAI Conference Artificial Intelligence, 2015.
Marcin Andrychowiez, Filip Wolski. Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in neural information processing systems, pp. 5048-5058, 2017. 6, 14

---

Published as a conference paper at ICLR 2021
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. arXiv preprint arXhv:1808.00177, 2018.
Aayush Bansal, Shugao Ma, Deva Ramanan, and Yaser Sheikh. Recycle-gan, Unsupervised video retargeting. In Proceedings of the European conference computer vision (ECCV), pp. 119-135, 2018. 2,3
André Barreto, Will Dabney, Rémi Munos. Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In Advances in neural information processing systems, pp. 4055 4065, 2017.
Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakrish- nan, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, et al. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 4243 4250. IEEE, 2018. 3
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint /606vv/606.0/550 2016. 6
Bryan Chen, Alexander Sax, Gene Lewis, Iro Armeni, Silvio Savarese, Amir Zamir, Jitendra Malik, and Lerrel Pinto. Robust policies via mid-level visual representations: An experimental study in manipulation and navigation. arXiv preprint arXiv:201 1.06698, 2020. 3
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1582-1591, 2018. 6, 14
Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Farle-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems. pp. 2672-2680, 2014.
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. arXiv preprint 1rXii:1703.022: 2017. 1, 3,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. arXiv preprint arXiv:1711.03215 2017, 3.
Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-simm Data- efficient robotic grasping via caadomized-to-canoniin adaptation networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 12627-12637, 2019. 2,3
Girish Joshi and Girish Chowdhary. Cross-domain transfer in reinforcement learning using target apprentice. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp- 7525-7532. IEEF, 2018.
Kun Ho Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Cross domain imitation learning. arXiv preprint arXiv: 1910.0000 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv: 1412.6980, 2014. 6, 14
Timothy P Lillicrap Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 1509.0000000000 2015. 6, 14
10

---

Published as a conference paper at ICLR 2021
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to anage-to-image translation networks. In Advances in neural information processing systems, pp. 700-708, 2017a.
YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learn- ing to imitate behaviors from raw video via context translation. arXiv preprint 2017b. 2
Andrew N Meltzoff. Understanding the intentions of others: re-enactment of intended acts by
18-month-old children. Developmental psychology. 1995. 1
Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P How, and John Vian. Deep decentralized multi-task multi-agent reinforcement learning under partial observability, In Pro- ceedings of the 34th International Conference Machine Learning Voolue--e 70, pp. 2681-2690. JMLR. org, 2017.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv:1511.0644 2015. 3
Xue Bin Peng. Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. 2018 IEEE International Conference Roboties and Automation (ICRA), May 2018. 2
Lerrel Pinto, Dhiraj Gandhi, Yuanfeng Han, Yong-Lae Park, and Abhinav Gupta. The curious robot: Learning visual representations via physical interactions. In European Conference on Computer Vision, pp. 3-18. Springer, 2016. 3
Lerrel Pinto, Marcin Andrychowiez, Peter Welinder, Wojeiech Zaremba, and Picter Abbeel. Asym- metric actor critic for image-based robot learning. arXiv preprint arXiv:1710.06544 2017. 2
Fabio Ramos, Rafacl Possas, and Dieter Fox. Bayessim: Adaptive domain randomization via probabilistic inference for robotics simulators. Robotics: Science and Systems XV. Jun 2019. 3
Kanishka Rao, Chris Harris, Alex Irpan, Sergey Levine, Julian Ibarz, and Mohi Khansari. Rl-cyclegan: Reinforcement learning aware simulation-to-real to-rell.. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11 11157-11 1166,, 2020. 3
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 046711 2016a.
Andrei Rusu, Mel Vecerik, Thomas Rothörl, Nicolas Heess, Razvan Pascanu, and Raia Hadsell. Sim-to-real robot learning from pixels with progressive nets. arXiv preprint arXiv:1610.04286 2016b.
Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without single real image.
arXiv preprint arXiv:1611.04200 1.02007, 2016. 3
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1134-1141. IEEE, 2018. 1,2
Laura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel, and Sergey Levine. Avid: Learning multi-stage tasks via pixel-level translation of human videos. arXiv preprint 191.v:1112.0444/ 2019. 2,
Gregory Stein and Nicholas Roy. Genesis-rt: Generating synthetic images for training secondary real-world tasks. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 7151-7158. IEEE, 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li. Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 1801 00690, 2018. 6
11

---

Published as a conference paper at ICLR 2021
Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey.
Journal of Machine Learning Research, 10(Jul):1633-11) 2009. 3
Matthew E Taylor, Peter Stone, and Yaxin Liu. Transfer learning via inter-task mappings for temporal
difference learning. Journal of Machine Learning Research, 8(Sep):2125-2166) 2007. 3
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint arXiv:2005./0244 2020. 1
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Picter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. 2017 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), Sep 2017. 2,
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Eric Tzeng, Coline Devin, Judy Hoffman, Chelsea Finn, Pieter Abbeel, Sergey Levine, Kate Saenko, and Trevor Darrell. Adapting deep visuomotor representations with weak pairwise constraints. arXiv preprint 15 2015. 3
Yilin Wu, Wilson Yan, Thanard Kurutach, Lerrel Pinto, and Pieter Abbeel. Learning to manipulate deformable objects without demonstrations. arXiv preprint arXiv:1910.134333 2019. 3
Wilson Yan, Ashwin Vangipuram, Picter Abbeel, and Lerrel Pinto. Learning predictive representations for deformable objects using contrastive estimation. arXiv preprint wrXiv:2003.054:33 2020. 3
Sergey Zakharov, Wadim Kehl, and Slobodan Ilic. Deceptionnet: Network-driven domain random- ization, In Proceedings of the IEEE International Conference on Computer Vision, pp. 532-541, 2019. 3
Tinghui Zhou, Philipp Krahenbuhl, Mathieu Aubry, Qixing Huang, and Alexei A Efros. Learning dense correspondence via 3d-guided cycle consistency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 117-126, 2016. 3
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017. 3,
12

---

Published as a conference paper at ICLR 2021
A EXPERIMENT SETTING
A.1 CROSS-PHYSICS AGENT SETTING
The physical parameters in cross-physics settings shown in Table 6. The "Parameter" column represents which type of parameter is changed in each task. Recall that we train the RL. policy in domain Y and evaluate in domain X in our evaluation. The numbers the columns of domain X and domain Y are the physics values for each domain. The physics parameter in domain Y is defined by default the OpenAl Gym MuJoCo simulation environment. The physics parameter domain X is selected so that it can cause obvious distortion to the policy trained in domain Y (as shown in Table 1). Note that we did not perform physics parameter search for improving our method. For the domain randomization baseline, the parameter value for each episode is uniformly sampled from the range shown in the last column.
Parameter
Eavs
Domain X
Domain
DR Range
HalfCheetah
0.3
0.1
[0.2, 0.4]
Armature
FingerSpin
2.0
[1.0, 3.0]
FetchReach
3.0
1.0
[2.0, 4.0]
Walker
0.4
[0.0, 0.8]
Torso Mass
Hopper
1.2
1.0
[0.8, 1.6]
Table 6: Cross physies physical hyperparameter settings. The parameters in domain X, domain Y and parameter range for domain randomization baseline for each task.
A.2 CROSS-MODALITY AGENT SETTING
In this setting, the resolution of the image observation in domain X is 256 256 and we concatenate three images (current and previous two frames) as the observation input for estimate state. By concatenating multiple images instead of one, allows the representation capture the velocities and motion of the agent, which a common practice vision-based RI..
A.3 CROSS PROSS-MORPHOLOGY AGENT SETTING
We introduce two tasks for the cross-morphology experiments including the "HalfCheetah" and "Swimmer" environments. As shown Figure 5, for "HalfCheetah", we modify the agent by adding hind leg of the same structure as the original hind leg to obtain three-leg cheetah. For "Swimmer", we add one more limb cloned from the original third limb, leading to a four-limb swimmer.
Two-leg Cheetah
Three-leg Cheetah
Three-limb Swimmer
Four-limb Swimmer
Figure 5: Cross-morphology agent introduction. Left: twa-leg Cheetah and three-leg counterpart. Right:
three-limb swimmer its four-limb counterpart. Please check out our supplementary video for visualization.
A.4 INITIALIZATION FOR ACTION ALIGNMENT MODEL
In the cross-modality-na
alignment, for agents with the same morphology and structure, initialize the action translation between two domains by identity mapping, and only train the observation alignment model (G) in the beginning. Then we unfreeze the action alignment model (H and P) and jointly train all models by our alternative training procedure.
13

---

Published as a conference paper at ICLR 2021
For agents of different morphology (e.g,, different number of limbs and joints), the dimensions of action spaces are naturally different. Thus is impossible to initiate action mapping functions with identity mapping. Note that can still use identity mapping for the original joints and limbs between two domains. For extra joints and limbs, we borrow the mapping function from nearby joints to initialize the novel joints. For example, in our experiments, between the three-leg cheetah and the two-leg cheetah. For the newly-added leg, we use and repeat the nearby original hind leg actions to initialize its actions. There no correspondence established from this initialization, as shown by the experiment results- we provides this initialization baseline for cross- morpholgy policy transfer in Table This baseline perform much worse than our approach.
find correspondence
B IMPLEMENTATION DETAILS
B. NETWORK ARCHITECTURE
The network architectures for each separate settings are as follows:
Cross-physics. The discriminator D is a five-layer MLP (hidden size: 32, 64, 128, 32) which takes states as input and predicts whether a state is true or fake. The forward dynamics model is a four-layer MIP (hidden size: 64, 128, 32) which takes current state and action as input and predicts the next state. The observation alignment function G is an identity mapping. The action alignment functions H and P are MIPs (hidden size: 32, 64, 128, 32) which take current state and action as input and predicts corresponding action in the other domain.
Cross-modality. The discriminator D) and the forward dynamics model F are the same as that in the "cross- "cross-physics" setting. The observation alignment function G is a ResNet-18 18 and followed by an MLP head (hidden size: 256, 64, 32) which outputs corresponding state. The action alignment functions H and P are identity mapping.
Cross-oorpholgg. Newtorks D, F, H and P are the same as that in the "cross-physscs" setting. The observation alignment function the same as that in the "cross-modality" setting.
B.2 TRAINING
Given the dataset of unpaired and unaligned samples from two domains, we first train the forward dynamics model F on domain Y until it converges, we use Adam optimizer (Kingma & Ba, 2014) with initial learning rate 0.001 which is decreased by half every three epochs and train the network for 20 epochs. This is the same for all the settings. Secondly, we train the action alignment functions H, P or observation alignment function G by optimizing Eq. 5. The pipeline and optimization chain for each separate settings are as follows:
Cross-physics. We optimize the alignment functions H, P here. We set A = 200, A = 1, and 0 in Eq. 5, where Ag O means we
A2 =0 that although the forward dynamics model F is involved in the back-propas propagation, we are fixing the weights of F. We train the models by using the Adam optimizer for 50 epochs with batch size of 32. The learning rate is set to 0.001 and decreased by 1/3 for every 10 epochs.
using the observation alignment function G. Note
Cross-modality. We optimize the observation alignment function G. We set Ao = 200. A = 0. and A2- = 3 in Eq. 5, where λ₁ means we are not using the action alignment function H, P. Similarly, the forward dyanmics model F is freezed. We train the model with Adam for 50 epochs with a batch size of 32. The learning rate is set to 0.001 and decreased by 1/3 for every 10 epochs.
mross-morphlloy (Joint training). Networks H, P. and G are optimized by Adam optimizer while the forward dyanmics model F' freezed. We combine the training procedure in cross-physics and cross- cross-modality settings following Algorithm 1. The model is trained for 10 epochs with a batch size of 32. The training alternates every 5000 steps (el and e2). Note each epoch contains more training steps in joint training. The learning rate is set to 0.0001.
B.3 REFERENCE POLICY
We use DDPG (Lillicrap et al., 2015) with HER (Andrychowicz et al., 2017) for "FetchReach" and TD3 (Fujimoto et al., 2018) for other environments, For DDPG, we train the policy for 50 epochs of 400 episodes in each epoch. The policy exploration epsilon ratio is 0.3 and the reward discount factor
14

---

Published as a conference paper at ICLR 2021
is 0.98. For TD3, train the policy for 400k time steps. The initial exploration step is 25k. The reward discount factor is 0.99, the target network update rate 0.005 and exploration noise standard deviation level is 0.1. When evaluating the performance of the policies, calculate 50 episode rollout rewards across 5 different seeds. We report the rewards with variance in all tables.
C DETAILS OF CYCLE-GAN BASELINES
Here we introduce the details of Cycle-GAn baseline for each different settings.
Cross-physies. In this setting, the Cycle-GAN baseline translates actions between domains. The generators and discriminators are MLPs and the cycle loss the L1 loss. The MLP network structures follow our approach. The results show that the policy fails to perform across domains and the mapping is close to random, so we have not included this baseline in the table.
Cross-modality. In this setting, the Cycle-GAR baseline translates between image and its cor- responding state. The image-to-state generator is the same G. The state-to-image generator is composed of MLPs and six upsampling blocks. Each block consists of one transposed convolution layer, one BatchNorm layer and one ReLU activation layer. The cycle loss L1 loss. This setting is also applied to our real robot experiment.
The Cycle-GAN model translates states and actions between two domains. All generators and discriminators are MLPs and the cycle loss is L1 loss.
Cross-morphology
We visualize in Figure the learned correspondence from the cross- mods-ttyyyyty alignment on the HalfCheetah experiment (Table 3). where observations of images are translated states. In each plot, x-axis represents the true-state of the image and the y-axis represents the translation result from the network. Each point in the plot represents random sample. We can see that our method is able to translate the image the correct states as most of the dots are in the diagonal line (y x), while Cycle-GAN yield near random translation similar to the random baseline. The result underlines the significance of incorporating dynamics into cycle-consistency.
Image-State State Cycle-Gan Baseline
Random Shuffle Baseline
Ours
Figure 6: Correspondence visualization for Cycle-GAN baseline and Cycle-GAN model performs nearly the same the random shuffle baseline while our model can correctly find correspondence between the image modality and the state modality.
The qualitative visualizations for Cycle-GAN and ours are shown in Figure for the cross-modality alignment on HalfCheetah, These five sub-figures from left to right sequence are as follows: (a) One random image observation sample from the dataset. (b) The rendered image from the state which predicted by Cycle-GAN model. (c) The rendered image from the state which predicted by our model. (d) The image with new renderer for our model prediction, giving different rendering view of the state. (e) The training curve of LI error for self-supervised state estimation. After training, our model output is almost the same as the ground-trute truth while the Cycle-GAN model completely fails.
(a) Random sample (b) Cycle-GAN
(c) Ours
(d) New render
(e) Training curve
Figure 7: Qualitative visualization for Cycle-GAN baseline and ours. The rendered image (c) from our model prediction looks almost the same like the original input image (a) while Cycle-GAt baseline (b) fails. The last small figure visualizes the LI error between the ground-trute state and our model prediction during the training process and proves the effectiveness of method self-supervised estimation. More visualizations are presented the project page link given abstract.
15


Reviews:
[{"review_id": "iz4Am8vcayz", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "** Paper Summary **\n\nThis paper proposed a novel framework to establish the correspondences across observations and actions, by using dynamic cycle-consistency. The paper focused on robotics applications, but the technical solutions are based on a pioneering method computer vision, generative adversarial networks (GANs) and cycle-consistency constraints, whose robustness has shown in many previous literatures. The proposed method was applied and attained the robustness on various tasks, such as cross-physics alignment, cross-modality alignment, cross-modality-andphysics alignment. \n\n** Paper Strength **\n+ The paper is well organized and written.\n+ Borrowing the concepts of GANs and cycle-consistency to solve the unpaired problems in cross-modal correspondence makes sense and good performance gains are expected.\n+ Solving several simulation-based settings and real robot settings was nice.\n\n** Paper Weakness **\n\nI have no major comments on this paper, but have some minor ones.\n- In Table 5, Ours (E) and Ours (J) have shown the performance gaps. It would be great if the reasons are properly mentioned.\n- In Table 4, the authors compared the quantitative results with Cycle-GAN and INIT. It would be great if the thorough discussion for these experimental settings are provided.\n- Computational complexity analysis is also required.", "extra_scores": {}, "extra_text": {}}, {"review_id": "eW_l5PSdrrB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper provides an interesting (and to my knowledge, novel) approach for learning a mapping of actions and observations from one domain/character to another reasonably similar domain/character. This mainly allows the transfer of skills (i.e. policies) from one domain/character to another. Immediate use cases of this approach are in imitation learning and sim2real transfer. In order to learn these mappings, the authors use the idea of cycle-consistency in generative adversarial networks and adapt it to the imitation learning task. Importantly, this allows them to obviate the need for paired state samples across domains. The authors show that their method not only works across modalities (vision from real robot to states in simulated robot), but it also works to some extent on different character morphologies.\n\nReasons for score:\nOverall, I vote for accepting this paper. To my knowledge, the method is novel and provides a viable and interesting approach for imitation learning and sim2real transfer. The experiments and the final quality are also high, however, the broader applicability to more challenging tasks and its limitations remain to be seen.\n\nCons:\n- The chosen tasks are (understandably) simple, therefore the applicability of the methods to more challenging environments remain to be seen.\n- The limitations of the method are not discussed well. I believe some commentary on the challenges of this method (e.g. the ease of training with GANs) would be useful. Also, more discussion on the use cases of the method and where it excels existing methods is missing.\n- The code is not provided which hampers reproducibility. I strongly suggest providing the code if possible.\n \nQuestion: \n- How does the method handle partial observability? In the robot arm example, a single image of the robot does not contain information about the velocity and angular velocity (of the joints). I'm confused as to how the model can actually infer these or work without knowing them.\n- Can you spend more time explaining Figure 3 (c)? It seems strange that the \"L1 error\" increased when more data was available.\n- Related to the last question: in this case, can you pre-train G the same as F and use cycle-consistency only for H and P? A brief explanation would suffice.\n\n\nFixes or suggestions:\n- I will have to double check the conference style guides, but having tables and algorithms intertwined with the text in a single-column publication is not pleasing to the eyes. I suggest rearranging these elements.\n- The plots in Figure 3 can be improved. At the very least, the distortions caused by resizing the images should be addressed.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Kn4TbMgKF86", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a technique that leverages cycle-consistency to align data across domains given only samples of one-step trajectories (x_t, a_t, x_{t+1}) from each. The cycle-consistency is used over the space of actions across two domains: e.g., given data from X [in the form (x_t, a_t, x_{t+1})] and data from Y [in the form (y_t, u_t, y_{t+1})], the system should be able to discover an \"alignment\" between states across the two domains by enforcing that the action spaces be mappable to one another (either identical, as in the \"cross-modality only\" applications, or via cycle-consistency, as in the \"joint model\" applications). The authors go on to show that their approach, among other things, allows for strong alignment between real-world images and their corresponding underlying state.\n\nOverall, the paper presents some solid results across a handful of interesting domains/applications. The paper is occasionally difficult to follow, particularly on the first read-through, in part because it is unclear precisely what form the data takes until Section 3, a problem not alleviated by a somewhat-unclear Figure 1 (see more detailed comments on this below). The introductory paragraph beginning with \"In this paper\" is quite hard to follow. Providing some more concrete examples in the introduction would be incredibly helpful for clarity. For example, mentioning that the input data to the learning algorithm takes the form of 3-element tuples (and their components), would be helpful.\n\nThe authors might also consider combining and aligning Figures 1 & 2 to help the reader follow along with the early pages of the paper. Perhaps also specifying the domain (i.e. functional domain: the space of the variables) on the figures themselves would help here. This lack of clarity in the beginning of the paper is likely its biggest weakness, as it is otherwise a good theoretical idea supported by strong results.\n\nSmaller comments:\n- Though it is clear why training _all_ of the networks jointly would lead to mode collapse (motivating the need to train F separately), it is not made particularly clear why the remainder of the system needs to be trained using the 'alternating' procedure employed throughout. A sentence or two explaining this decision would be helpful.\n- Particularly in the figures, the authors should make clear that the `y` images are in fact renders corresponding to the predicted state. Without this \"disclaimer\" readers coming from the purely-image-driven cycle-consistency community may mistakingly believe that those images are the direct output of the CycleGAN baselines or the proposed approach. Modifying the green-backgound images in Fig. 1(a) to include \"Render of y_t\" (or similar) would help avoid such a confusion.\n- Figure 1(b) is potentially misleading, since really the images are assumed to be identical between the two domains. Additionally the arrows on (b) and (c) in fig. 1 are also somewhat misleading, since it is not clear what the arrows represent. Consider adding more annotations on this figure describing in more depth what this figure is trying to communicate: namely the types of applications enabled by the approach, rather than the structure of the proposed approach itself.\n- The Related Work section is quite thorough and helpful for providing context. Two other papers that the authors might have missed and consider citing that made progress in solving similar tasks are (1) \"Adapting Deep Visuomotor Representations with Weak Pairwise Constraints\" (Tzeng, Devin, et al, 2016) [which uses a task-specific loss for aligning sim/real robot arm images] and (2) \"GeneSIS-RT: Generating Synthetic Images for training Secondary Real-world Tasks\" (Stein & Roy, 2018) [which uses cycle-consistency for task-agnostic sim-to-real and then uses the translated data for quadcopter flight]. Both papers circumvent the need to collect paired sim and real images for real-world robotics applications.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DWkYYswD1B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##### Summary\n\nThis work is concerned with learning mappings between pairs of domains that may differ in representation (imagery vs. configuration state), physical parameters, morphology, or combinations thereof, relying only on unpaired training data from both domains.  The key idea is to leverage the (existing) concept of cycle consistency, incorporating dynamics to formulate a cycle loss that spans vastly different domains.  The cycle is closed by learning a domain translator $G$ that maps states (the paper calls them observations - this is unimportant as the system makes no such distinction, as either domain might represent a \"state\" or an \"observation\") from domain $x$ to domain $y$, and a forward model $F$ that operates in the $y$ domain.  In addition, functions $H$ and $P$ are learned that map actions $a$ and $u$ between their respective domains $x$ and $y$, in either direction.  The difference between $G(x_{t+1})$ and $F(G(x_t), u_t)$ is then the part of the overall loss function that closes the loop. $G$, $H$ and $P$ are trained adversarially using cycle consistency losses; $F$ can be trained by regression, exploiting training sequences in the $y$ domain.\n\nThe system is evaluated on four different tasks from OpenAI Gym, and is shown to be effective in diverse settings.\n\n##### Strengths\n\n* This work addresses an important, fundamental problem that arises in many applications, including state estimation and sim2real transfer.\n* The paper makes a significant, nontrivial contribution to this problem, allowing hard cross-domain transfer problems to be solved dramatically better than before.\n* The work is well related to prior literature.\n* The claims are supported by the empirical results.\n* The paper is well organized, readable and clear.\n* The appendix provides a lot of additional detail important for reimplementing this method and replicating the results.\n\n##### Weaknesses\n\n* The implementation details, in particular the neural-network architectures, appear to be very specifically tailored to the scenarios of evaluation. Instead it would be helpful to provide insight into how a user of this method should go about designing the networks, given their application domains.\n* For cross-modality alignment, paired training data are typically available.  How can the method take advantage of this? How would this impact the results, e.g. those of Table 2?  Some related results are given in Fig. 3(c) but it is unclear what exactly was done and what exactly the graph shows.\n* An obvious and explicitly-stated application of this work is sim2real transfer, but no such experimental results were provided (I realize that it is not easy to set up such an experiment from which strong conclusions can be drawn).\n\n##### Recommendation\n\nUnless I am missing important similar work, the paper makes a strong contribution in an important area.  It should be accepted.  While I wrote more text under Weaknesses than under Strengths, the weaknesses are minor compared to the strengths.\n\n##### Questions for the Rebuttal\n\n* The results on cross-physics alignment (Table 1) seem to leave a lot of room for further improvement.  What is the limiting factor?\n* Please comment on the listed weaknesses.\n\n##### Details\n\n* \"RL score\" = return?\n* There are many little typos.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new solution for cross-domain correspondence in control, which combines GANs and cycle-consistency, and separates shifts in observation space and in action space. The paper targets unpaired data / simulations, and discovers alignment of state by enforcing that domains are mappable.

The paper was received well by reviewers, who pointed out several strengths: a strong contribution on a fundamental problem, and an interesting formulation; a well written and well positioned paper; This compensates minor weaknesses, in particular the fact that transfer has been tested between two different simulated environments. 

The reviewers unanimously suggested acceptance, the AC concurs.
Decision: Accept (Oral)