Learning a Domain-Agnostic Policy through Adversarial Representation Matching for Cross-Domain Policy Transfer
Authors: Hayato Watahiki
Ryo Iwase
Ryosuke Unno
Yoshimasa Tsuruoka
Venue: Submitted to ICLR 2023 (2023)
Keywords: imitation learning
domain transfer
zero-shot transfer

Abstract: The low transferability of learned policies is one of the most critical problems limiting the applicability of learning-based solutions to decision-making tasks. In this paper, we present a way to align latent representations of states and actions between different domains by optimizing an adversarial objective. We train two models, a policy and a domain discriminator, with unpaired trajectories of proxy tasks through behavioral cloning as well as adversarial training. After the latent representations are aligned between domains, a domain-agnostic part of the policy trained with any method in the source domain can be immediately transferred to the target domain in a zero-shot manner. We empirically show that our simple approach achieves comparable performance to the latest methods in zero-shot cross-domain transfer. We also observe that our method performs better than other approaches in transfer between domains with different complexities, whereas other methods fail catastrophically.

Under review as a conference paper at ICLR 2023
LEARNING A ADVERSARIAL REPRESENTATION MATCHING FOR CROSS-DOMAIH POLICY TRANSFER
POLICY THROUGH
Anonymous authors Paper under double-blind review
ABSTRACT
The low transferability of learned policies is one of the most critical problems limiting the applicability of learning- barning-based solutions to decision-making tasks. In this paper, we present way to align latent representations of states and actions adversarial objective. We train two between different domains by optimizing models, a policy and a domain discriminator, with unpaired trajectories of proxy tasks through behavioral cloning as well as adversarial training. After the latent representations are aligned between domains, a domain-agnostic part of the policy trained with any method in the source domain can be immediately transferred the target domain in a zero-shot manner. We empirically show that our simple ap- proach achieves comparable performance the latest methods in zero-shot cross- domain transfer. We also observe that our method performs better than other ap- proaches in transfer between domains with different complexities, whereas other methods fail catastrophically.
INTRODUCTION
Humans have astonishing ability to learn skills in highly transferable way. Once learn the route from home to the station, for example, we can get to the destination using different vehicles (e.g., walking, cycling, or driving) in different environments (e.g., on map or in the real world) ignoring irrelevant perturbations (e.g., weather, time, or traffic conditions). We find underlying structural similarities between situations, perceive the world, and accumulate knowledge in our way of abstraction. Such abstract knowledge can be readily applicable to various similar situations. This seems easy for humans but not for autonomous agents. Agents trained in reinforcement learning (RL) or imitation learning (IL) often have difficulties in transferring knowledge learned in a specific situation to another. It because the learned policies are strongly tied to the representation acquired in specific configuration of training, which is not generalizable or an environment.
subtle changes in agent
Previous studies have attempted to address this problem with various approaches. Domain random- ization (Tobin et al.] 2017 Peng et al., 2018 AAnryyhowicc et al.) 2020) aims to learn a policy robust environmental changes by having access multiple training domains, but it cannot handle large domain gaps out of the domain distribution assumed in training such as drastically different observations or agent morphologies. To overcome such domain discrepancies, Gupta al. (2017) and Liu et al. (018) proposed methods to learn domain-invariant state representations for imita- tion. but these methods require paired temporally-alige alignnd datasets across domains and. in addition, need expensive RI steps to adapt to the target domain. More recently, Kim et al. (2020) proposed method to find cross-domain correspondence of states and actions from unaligned datasets through adversarial training. This method imposes strong assumption that there is exact correspondence of states and actions between different domains and learns as direct mapping functions. The as- sumption is sometimes problematic when such correspondence is hard to find. For example, if one agent has no leg while another has few legs, we cannot expect all information on how the agent walks to be translated into another domain.
In this work, we propose a method that does not require the existence of exact correspondence between domains. Our approach leamns domain-invariant representations and common abstract

---

Under review as a conference paper at ICLR 2023
Common Latent Space
Expert Domain
Target Domain
Sy
Sx
ax
ay
Figure 1: Illustration of a domain -agaosii-nnosicccccc representation space. Since semantically similar states close together in the latent space regardless of the original domain, transfer knowledge between different domains through the latent space.
policy on them that is shared across different domains (Figure 1 components: a policy and a domain discriminator. The policy has three blocks: a state encoder, a common policy, and action decoder. In the first stage, the model is optimized with imitation objective and an adversarial objective on a learned state representation simultancously using an unaligned dataset of proxy tasks. The adversarial training induces the state encoder to generate latent state representations whose domains are indistinguishable to the domain discriminator. Such representations do contain domain-specifit information and thus can work with the common policy. Next, we freeze the parameters of the state encoder and action decoder and only update the common policy in the source domain on the learned feature space to adapt the model to the target task. In this process, with Kim et al. (2020), we can any learning algorithm for updating the policy and moreover do not require an expensive RL step interacting with the environment. After the update, combined with the fixed encoder and decoder, the learned common policy can be readily used in either domain in a zero-shot manner.
Our model consists of two
challenging maze environment (Fu et al. 2020) with various domain shifts: domain shift in observation, action, dynamics, and morphology of an agent. Our experiments show that approach achieves comparable performance in most settings. We find that our method effective the setting of cross-dynamics or cross-robot transfer, where no exact correspondence between domains exists.
We conduct experiments
In summary, our contributions are as follows:
We provide novel method of cross-domain transfer with an unaligned dataset. In contrast to the latest method that learns mappings between domains, our approach aims to acquire a domain-invariant feature space and a common policy on
Our experiments with various domain shifts show that method achieves comparable performance in transfer within the same agent and better performance than existing meth- ods in cross-dynamics cross-robot transfer by avoiding direct mapping between domains.
RELATED WORK
Transferring learned policy different en- vironment is a long-standing challenge in policy learning. Most previous methods acquire some cross-domain metric to optimize and train policy for target task using standard RL algorithm
Policy Transfer between MDPs
(uppta et al./ 2017: Liu et al.] 2018 2020 Zakka et al. 2022; Fickinger et al.) (022) or a GAIL (Ho & Ermon b016)-based approach (Stadie al. 2017 Franzmeyer
al., 2022). In particular, Franzmeyer et al. (2022) utilize domain confusion as we do in this paper. To calculate the pseudo
corresponding states (Gupta et al.) 2017 Liu et al.) 2018) or the distance from the goal (Zakka et al. 2022) in the latent space is often used. Some recent transfer as our method does, without interacting
reward for RL, the distance between cemporally-corrspondiii
approaches can perform zeros-shot cross-doman with the environment for target task. Kim et al. 2020); Raychaudhuri et al. (2021) learn mappings between domains while Zhang et al. (2020) impose domain confusion on its state representation for domain shift in observation. Our approach predicts actions without learning cross-domain mappings
2

---

Under review as a conference paper at ICLR 2023
x: Source Domain
Rx
4
T2
2: Common Space
R
4y
Ty
Target Domain
Ry
Figure 2:
Common latent MDP between a source MDP and a target MDP. The latent MDP is ex- pected to be expressive enough reproduce the dynamics and the reward of the original MDPs.
and also can handle diverse situations including domain shift in an action space. For cross-robot transfer, Hejna et al. (2020) train a portable high-level policy by using a subgoal position as a cross- robot feature. Gupta al. (022) cover morphology distribution to generalize unseen robots. We intend to perform more direct policy transfer without making domain-specift assumptions.
State Abstraction for Transfer Theoretical aspects of latent state representation have been ana- lyzed in previous studies. There exist several principled methods of state representation learning for transfer such as bisimulation (Castro & Precup 2010) and successor features (Barreto et al. 2017), although we use neither of them. Recently, Gelada et al. (2019) proved that the quality of the value function is guaranteed the representation sufficient to predict the reward and dynamics of the original MDP. In similar context, Zhhng et al. (2020); Sun et al. (2022) provide performance guar- antees in multi-task settings or cross-domain transfer without paired relationships between domains.
Unsupervised Domain Adaptation & Correspondence Learning Domain adaptation with un- aligned datasets has been intensively studied in computer vision. CycleGAN (Zhu et al., (017) finds translation function between domains by generating the corresponding instances in another domain. Similarly to our approach. Tzeng et al. (2017) matches the feature distributions between domains by fooling a domain discriminator and successfully transfers an image classifier across do- mains. Besides, several studies learn domain-invarian temporal relationships of frames in videos (Sermanet et al. 2018 Dwibedi et al., (2019) cycle- consistency in trajectories of an agent (Zhang el al. 2021 Wang et al. 2022)). Such features can be used in the reward shaping for cross-domain imitation through RL (Zakka al. 2022).
features by optimizing an objective related
3 PROBLEM FORMULATION
We consider a Markov decision process (MDP): M (S. 7), where S is state space, A an action space, R x A + R a reward function, T S x A x S + R20 is a transition function, and 7 is a discount factor. The aim of this paper is to transfer knowledge of a source MDP MM = (S.,Az, R2, Tz, 7) to a target MDP My = (Sy. Ry , T 7). Here we assume that these MDPs share a common latent structure which is also MDP: M2 (S., Ts. T). Formally, we assume the existence of state mapping functions dr S2 - Sz, By Sy - S2 and action mapping functions via Ar - Az, thy : Ay A. which translate states Sw, Sy or actions ag ay into shared states 8₂ or actions ag, respectively, satisfying the following relationships:
T2 (0x the (ax)) = dn (Ta(s2, oz)), T. (1,,(ss)) vy(a)) = (T, aw)), (2(1 Vi (ax)) Rn Rx(82, 0z); (x)) 1y (ay)) (y myy)
In short, as depicted in Figure 2 we assume that the common latent MDP is expressive enough to reproduce the dynamics and reward structure of the source and target MDP. Our goal is to learn the state mapping functions &2 θ, and the action mapping function V.., 3 so that any policy learned in the common latent space nz <=(a=(0=)= S. A. -> R20 can immediately used in either MDP combined with the obtained mappings. In this paper, we use a deterministic policy, and thus we sometimes denote the latent policy as (=())) : S. - Ag, although we can easily extend it to a stochastic policy. We learn these mappings using state action trajectories of a set of proxy tasks Da,A = (Ta.e.}} where Td, = {( } } is a successful trajectory of task & in domain d, and use the learned relationships in task K' unseen during training. The alignment of the representation is obtained through behavioral cloning (BC) on the trajectories of proxy tasks and adversarial training on the embeddings, which will be described in the following section.
3

---

Under review as a conference paper at ICLR 2023
Encoder Common Pollcy
Decoder
(s.)
Sdt
Domain: d
az
ad
Task ID: k
Domain Discriminator
Learnable
Frozen
d
(a) Alignment phase
Common Policy
Decoder
Policy
Decoder
(sx)
(sy)
State:
State: Sy
Sz
Sz
Domain: X
ag
ax
Domain:
ag
ay
Task ID: k
Task k'
y
(b) Adaptation phase
Inference
Figure 3: Overview of the training and inference procedure of our method. (a) In the alignment phase, we jointly train the policy and the discriminator using trajectories of proxy tasks match the representation between domains. (b) In the adaptation phase, we only update the common policy to adapt the target task in the source domain. (c) inference, we can use the updated policy trained in the source domain combined with the encoder and decoder already trained in the alignment phase.
4 LEARNING COMMON POLICY THROUGH REPRESENTATION MATCHING
In this work, aim to learn state mapping functions dz. Oy. and action mapping functions thas 1 or equivalents, and use them to transfer the policy learned in one domain to another. Our algorithm of cross-domain transfer consists of steps illustrated Figure (i) Cross-domain representation alignment, (ii) Policy adaptation to target task the source domain. We call them the alignment phase and the adaptation phase. respectively. After the adaptation phase, the learned policy of target task can be immediately used in the target domain without any fine-tuning, additional training interacting with the target domain (Gupta al. 2017 Liu al., 2018), or a policy learning in the mapped target domain (Raychaudhuri et al. 2021).
4.1 GROSS-DOMALL REPRESENTATION ALIGNMENT
In the alignment phase, aim to learn the state and action mappings and acquire domaann-nvariinn feature space that can be used in either the source domain or the target domain. We represent our policy simple feed-forward neural network shown in Figure 3 It consists of three compo- nents: state encoder, common policy, and an action decoder. They correspond to o(s), ), π (Tss), and a (a,),, respectively. (az) can optionally take raw state for the action prediction in domain. Note that we feed domain ID to the encoder and the decoder instead of using two separate networks for the domains to simplify the architecture and the training. Additionally, we feed task ID A the policy to deal with multiple proxy tasks simultancously.
Even when the dataset contains trajectories from both domains performing the same tasks, sim- ply training the policy with such dataset does not necessarily match the representations of both domains. Figure 4a shows the distribution the representation latent state s, acquired by be- havioral cloning using expert trajectories from different domains in experiment (Maze2D-OA) Although we trained a single policy. the learned representations are completely separated for each domain. We need additional objective to obtain better alignment.

---

Under review as a conference paper at ICLR 2023
Source
Target
(a) Distribution without adversarial training
(b) Distribution with adversarial training
Figure 4: Distributions of latent state representation s, with and without adversarial training. These are visualized with t-SNE Van der Maaten & Hinton 2008). Here we sample corresponding states from two domains of the Maze2D environment in our experiment and encode them into the learned feature space. Black lines connect 20 corresponding state pairs in each figure. Without adversarial training, the model can learn the policy for different domains on different feature spaces. The adversarial training encourages the model to learn them the same feature space.
we have access temporally-alignn trajectories (i.e. (dx(s)) vhr(aa)) (0 (s) (a,))) we can directly learn corresponding representations &z (sx), (sy)) ) or & (a(ax), (ay) close together as done in Gupta et al. (017)). However, we do not assume such an alignment in the dataset and hence we instead use adversarial training to match the representations. Our purpose to match the corresponding latent representation of states and actions between domains. In other words, want the latent representations to contain domain-specific information as little as possible. can be expressed mutual information minimization between domains D and latent representation S2 or Ag. Here we can skip this process on latent actions a, since a₂ a function latent states 83. The objective for latent states be formulated follows:
min ((D;s.) = min H(D) - H(DSS,)
@
max HDDSS.)
The second equation holds because we cannot control the domain distribution by the encoder. To optimize this objective, we introduce a domain discriminator q(d[sz) ) as an approximation of p(d) the objective. The training proceeds
similar way to that of generative adversarial networks (GAN) (Goodfellow et al. 2014). The discriminator =2) predicts which domain each given s: came from, while the encoder & tries to fool the discriminator, as indicated in ( Combined with the behavioral cloning objective, we have our final objective Calign for the alignment phase:
min max falign min max LBC + XL 0.50.
2....0
Lac = (((s (v7 (vT (m k)) ad)
(2)
Lad = (sad))D [log ((())(())))
where À is a constant that defines the importance the adversarial term. More detailed derivation and interpretation can be found in Appendix A By optimizing the adversarial objective along with imitation, we can obtain an aligned representation as shown in Figure 4b Intuitively, behavioral cloning shapes the structure of the representation of each domain (Figure 4a). and the adversarial term pulls the entire distributions close together so that they will be identical (Figure 4b). Besides, we also employ task-level a alignment by feeding task ID k to the discriminator. We observe that this improves performance some cases. See Appendix B.3 for the performance comparisons.
4.2 POLICY ADAPTATION
In the adaptation phase, train policy successfully perform the target task which finally would like to solve. We assume that the latent states and the latent actions are already aligned the alignment phase, and also have state and action mapping functions between the common latent
5

---

Under review as a conference paper at ICLR 2023
Next actions (af))
Transformer Encoder
Transformer Decoder
Demonstration {( e()
Observations (xp)
(b) Maze-MR (umaze)
Illustration of cnnnonstration-conitiinnnn policy. This policy cannot be trained only with data source domain.
Figure 5:
(a) Maze2D (medium)
Figure 6: Pictures of Maze2D and Maze-MR. The red points show the goal, which are not observable for agents.
space and each domain. Thus, our task in this phase is to train the common latent policy for the target task. We can train it solely in the source domain using mapping functions. As described in Figure 3 the weights of the encoder and decoder are fixed during the training so that the common policy learned on top of the acquired aligned representations. In this phase, we can use any learning algorithm, including reinforcement learning. In our experiments, we update the common policy by behavioral cloning using expert trajectories in the source domain. Therefore, we minimize the following objective in the adaptation phase.
min Cadape min LBC- 3a
This adaptation phase is one of the advantages of using common policy demonstration- conditioned policy (Figure 5) used in previous studies such as Dasari Gupta (2020). Since a conditional policy cannot be trained with dataset that only contains demonstrations in the source domain, we cannot update it to adapt to the target task. This constraint requires the model to gener- alize demonstrations of an unseen target task in a zero-shot manner, which difficult especially when the target task is much more advanced than proxy tasks used in the training. We demonstrate effectiveness of our method from this perspective our experiment.
5 EXPERIMENTS
In the experiment, our aim to answer these questions: (i) Can our method align the states and ac- tions of source and a target domain? (ii) Can our method achieve zero-shot cross-domain transfer in various settings? (iii) which case is our method more effective compared to previous approaches?
5.1 ENVIRONMENTS
To evaluate the efficacy of our method, choose the Maze environment used in the D4RL Fu cct al., (220)) benchmark. It is a multi-takk robotic locomotion environment suitable for defining proxy tasks, and its simplicity enables to evaluate the discrepancy of the alignment from the ideal one. also offers multiple agents with different morphologies, such as Point and Ant, evaluate the performance of cross-domain transfer between significantly differ- ent observation and action spaces. Concretely, we use four environments for our evaluation (Two of them are shown in Figure 6). In each environment, agents explore the maze toward specific goal. The shape of the maze has Two variations: umaze and medium. A task is defined as a 2D position of a goal, and the dataset has expert trajectories of tasks. Note that the starting point can vary within a task. Maze2D-O A point agent explores the maze. The observation space has four dimensions (position and velocity for each direction), and the action space has two dimensions. The x-axis and the y-axis are swapped in observations of the target domain. Maze2D-OA: In addition to the obser- vation shift of Maze2D-O, actions for both directions are inverted (i.e., multiplied by -1). Maze-D (Dynamics): In the source domain, inertia is removed and action directly determines the velocity. Agents in domains share the same observation, but they have different action spaces, dynamics, and state distributions. Maze-MR tion space and the action space of Ant are 29-dimensional and 8-dimensional, respectively. Transfer between these agents is challenging since they have different morphologies and thus the observation
obtain
with which
An ant agent is used target agent. The observa-

---

Under review as a conference paper at ICLR 2023
Table I:
Alignment scores. The values are mean squared errors between the ground-truth states in the source domain and the ones predicted from the corresponding target states. The results are averaged over nine seeds with three fixed goals.
Task
Ours
Ours-S
GAMA
CCA
IIO
Maze2D-O (umaze)
0.11 + 0.06
1.98 + 0.54
0.09 + 0.04
2.74
3.60 + 1.34
Maze2D-O (medium)
0.21 + 0.26
4.49
2.49
0.08 + 0.02
0.01 0.01
4.71 + 0.76
Maze2D - (umaze)
4.50 + 1.75
1.72 = 0.73
0.11 0.03
2.74
3.60
= 1.34
Maze2D- OA (medium)
2.26 + 1.89
2.88 + 0.57
0.10 0.05
0.01
4.71 + 0.76
space, action space, and dynamics vary significantly. We also measure the performance in a manip- ulation environment and show results in Appendix [B.6] For other experimental settings, including the detail of datasets, architectures, and the training procedures, refer to Appendix C.I
5.2 BASELINES
For our proposed methods, we used two implementations. In Ours, the decoder predicts actions without raw states 8. Ours-S we feed 8 to the decoder to compensate domain-specific infor- mation lost from latent representations. We compare our methods with the following approaches. CCA (Hoteliigg 1992) finds invertible linear transformations to the space where unaligned demon- strations maximally correlated. A target policy is then optimized by reinforcement learning so that the policy can obtain observations similar to the given demonstrations of a target task in the learned space. IIO (Gupta et al. 2017)) uses dynamic time warping (DTW) (Müller (007) to find pairs and get the representations of them close together using a context translation model. A target policy is trained in similar way as CCA using RL. GAMA (Kim al. 2020 is one of the closest approaches to ours. It first learns direct mappings of states and actions between domains solves the target task by translating the states to those of the source domain, applying the source domain policy, and translating the output actions back to the target domain. The critical difference between this approach and ours is that GAMA uses the source domain policy for the target task, while ours uses a common policy. In addition, we do not need a dynamics model and thus the training is simpler. Cond is Transformer (Vaswani et al. 2017)- -based demonstration-conniiioo policy depicted in Figure 5 It digests a state-action demonstration of a task to perform and the ob- servation history of an agent, and outputs the next action take. It only has the alignment step due to its structure as discussed in Section 4.2 See Appendix B.5 for more comparisons and Appendix C.4 for more details of the training and implementation.
an adversarial way. After that, it updates the source domain policy and then
5.3 ALIGNMENT EVALUATION
Quantitative Evaluation We evaluate the quality of the alignment in Maze2D, where we know the ground-truth state correspondence between domains. We expect that the corresponding states are mapped to the same latent representation. We calculate a prediction error of states in the source domain from the corresponding states in the target domain and use it as a metric of the alignment. Here additionally train the state decoder o (s.) to calculate the metric. Note that this metric is advantageous to GAMA since it has a direct mapping function of states. The scores in Table I show that, while Ours achieves similar scores GAMA in Maze2D-O, our methods fail to accurately recover the state in the other cases. It is not only because our methods do not learn direct mappings between domains, but also because the model discards part of the information on the original state when projected into the domain-invarian space. As long as latent states can use the same common policy Tz, it can be valid to map the corresponding states to different positions, and hence a worse alignment score does not necessarily mean worse transfer performance. We examine how it affects the final transfer performance in Section 5.4 Regarding the performance of CCA, it close to the perfect score in some settings since the observation shift is linear in Maze2I), whereas CCA fails in other cases due to the lack of explicit temporal alignment and the variations in the starting positions.
Qualitative Evaluation We visualize the distributions of the learned latent state space using t- SNE (Van der Maaten & Hinton 2008). As shown in F Ours successfully aligns most pairs in successful cases in Maze2D-OA. In the more challenging environment, Maze-MR, the latent states
7

---

Under review as a conference paper at ICLR 2023
Source Target
(a) Maze2D-OA (Ours)
(b) Maze-MR (Ours)
(c) Maze-MR (Ours-S)
Figure 7: Comparison of distributions of latent states 3- in the medium maze visualized by t-SNE Van der Maaten & Hinton 2008). Similar to Figure [ we sample corresponding states from do- mains and encode them into the learned feature space. Black lines connect 20 corresponding state pairs in each figure. In Maze-MR, we do not show the lines since the ideal alignment is not available.
Table 2: Success rates of target tasks. Results are averaged over nine seeds with three fixed goals.
Task
Ours
Ours-S
GAMA
CCA
IrO
Maze2D-O (umaze)
0.76 + 0.33
0.42 + 0.41
0.63 E 0.35
0.45
0.06 + 0.11
Maze2D-O (medium)
0.97
0.54 0.40
0.77 0.24
0.14
0.00 0.01
Maze2D-OA (umaze)
0.40 + 0.34
0 0.36
0.77 +
0.02 0.35
Maze2D-OA --OA (medium)
0.61 + 0.32
0.72
0.61
0.10
0.08
Maze-D (umaze)
0.26 + 0.26
0.50 +
0.03
0.10 1.00
Maze-D (medium)
0.36 0.29
0.72
+.003
0.48
0.00
Maze-MR (umaze)
0.20 + 0.26
0.42 + 0.32
0.00
+.007
0.00
Maze-MR (medium)
0.39 + 0.17
0.48 0.24
0.00
+
0.00 + 0.00
for the target domain (i.e. Ant) have broader distribution than the source domain. If we supply domain-specific state information s to the decoder in Ours-S, the alignment is significantly improved since the encoder can focus more on Domain-agnostee information for the action prediction.
5.4 CROSS-DOMAIN TRANSFER PERFORMANCE
We measure the performance of the cross-domain transfer by the success rate of a target task. We choose a single goal for the target task and use the other goals as proxy tasks for the alignment phase. The values are averaged over three different target tasks. Table summarizee the scores of methods that employ representation alignment various settings. Note that CCA and IfO use RL for adaptation and does not perform zero-shot transfer. In Maze2D, our methods and GAMA achieve better performance than CCA and IfO. Interestingly, our methods show comparable performance to GAMA even though the alignment score is worse than that of GAMA. This shows that aligning the latent representation is sufficient for transferring knowledge, instead of learning a direct mapping between domains. In Maze-D and Maze-Mil GAMA fails to transfer the policy, while methods achieve a non-zero success rate consistently. Since it is difficult to find an exact correspondence between states with different complexities, direct state mapping and action mapping GAMA hard to acquire. Our methods instead reduce each MDP to the common one and thus we do not have learn the direct relationship between the source domain and target domain. However, the performance of Ours is limited since the decoder cannot get sufficient domain-specine information for domain connain-specce specific action prediction. Using raw states s directly fed to the decoder, Ours-S achieves the better performance with the improved alignment shown in Figure 7c Regarding the performance drop of Ours-S Maze2D-O, we give some explanation methods to Cond in Maze2D environments. The results in Figure 8 show that Cond struggles to adapt to the target task since the lack of the adaptation phase requires the model to perform zero- unseen demonstrations of a target task. This supports our claim in Section 4.2 that adaptation through updating a domain-agnostic policy on a shared representation space is effective for cross-domain transfer.
Appendix B.4 We also compare our
shot generalization
8

---

Under review as a conference paper at ICLR 2023
Maze- OA
1.0
1.0
Ours-S Cond
Success Rate
0.8
Success Rate
0.8
0.6
0.4
0.4
Ours
0.2
Ours-S
0.2
0.0
GAMA
5
10 25
0.0
(umaze) O (meeiim) OA (umazze OM (medium)
Num Proxy Tasks
Figure 8: Success rates of our methods and a con- ditional model in Maze2D. Our models achieve better performance consistently because it can be trained only with source domain demonstrations. The error bars show the standard deviations.
Figure 9: success rate Maze2D-OA. The scores are averaged over three runs with a fixed goal. The error bars show the standard deviations. More results are available Appendix B.1
The number of proxy tasks vs
Table 3: Success rates of Ours- in Maze2D (medium) with various coefficients À for the adversarial term. Here we used a fixed single goal. The scores are averaged over three runs.
Task
A =
== 0.1
A= 0.5
X 2.0
A
Maze2D-O
1.00 + 0.00
0.67
0.40
0.67 + 0.40
0.07 + 0.09
0.09
0.53 + 0.33
0.33
Maze2D-OA
0.93 0.09
0.93 0.05
0.87 0.05
0.90 0.08
0.67 + 0.34
Maze-D
0.13
0.09 0.09
0.72 1722:027
0.89
0.55 + 0.35
Maze-MR
0.09 0.03
0.61 + 0.24
0.48 + 0.25
0.47 + 0.10
5.5 ABLATION STUDIES
Alignment Complexity We measure the transfer performance, varying two aspects of the dataset for alignment: the number of trajectories and the number of proxy tasks. We present plots of the number of proxy tasks for Maze2D-OA in Figure 9 and more in Appendix B.I These plots show that a decrease the number of demonstrations or proxy tasks leads to a decrease in performance.
Sensitivity to A Adversarial training is notorious for its difficulty in balancing the training of multiple functions: the encoder and the discriminator in our case. To evaluate how sensitive our method is to the adversarial coefficient A. we measure the success rates of Ours-S with various values of X. The results are shown in Table 3 Surprisingly, in Maze2D-O and OA, the transfer is successful without the adversarial term. In the other environments, on the other hand, adversarial training is necessary to align the representations, and we have to choose the correct value of X.
6 CONCLUSION
In this work, we present novel method to learn a domain-invariant policy in a common feature space for cross-domain policy transfer. Our experimental results show that our method achieves comparable performance to the prior methods in environments with domain shifts on the same agent and dynamics. Our method especially effective for transfer between domains with different com- plexities, where no exact correspondence exists.
The main limitation of our work the instability of the training due adversarial training. Intro- ducing techniques for more stable training of GANs could enhance the performance of our method. Alternatively, some self-supervised objectives can give more signals for the alignment and conse- quently stabilize replace the adversarial training. Besides, relaxing the requirements a promising direction for future work. we do not require expert actions or task ID in a dataset for the alignment, ing. We hope our work provides some suggestions to researchers who will work on the development of domain-free policy, abstract policy that can be applied to any domain in zero-shot manner.
the dataset
utilize prevalent, less-structured datasets including videos to scale the train-

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We provide detailed descriptions our experimental setups and implementations in Section and and Appendix [ These sections contain the key hyperparameters, links to the datasets, modified parts of the existing environments, dataset size, duration of the training, and other necessary information for reproduction. The training time does not exceed about five hours with single GPU. We also release our codebase and created datasets as supplementary materials.
REFERENCES
OpenAl:
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowiez, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement leamiin. Advances in neural information processing systems, 30, 2017.
Pablo Samuel Castro and Doina Precup. Using bisimulation for policy transfer in mdps. In Twenty- Fourth AAAI Conference on Artificial Intelligence, 2010.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. Ad- vances in neural information processing systems, 29, 2016.
Sudeep Dasari and Abhinav Gupta. Transformers for one-shot visual imitation. In Conference Robot Learning, pp. 1439-1456. PMLR, 2020.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Tem- poral sycle-consistency learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1801-1810, 2019.
Arnaud Fickinger, Samuel Cohen, Stuart Russell, and Brandon Amos. Cross-domain imitation learning via optimal transport. In International Conference Learning Representations, 2022.
Tim Franzmeyer, Philip Torr, and Joao F. Henriques. Learn what matters: cross-domain imitation learning with task-relevant embeddings. In Alice Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep
data-driven reinforcement learning, 2020.
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc Bellemare. DeepMDP: Learning continuous latent space models for representation learning. In International Conference on Machine Learning, pp. 2170-2179. PMLR, 2019.
Ian Goodfellow. Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. In International Conference on Learning Representations, 2017.
Agrim Gupta, Linxi Fan, Surya Ganguli, and Li Fei-Fei. MetaMorph: Learning universal controllers with transformers. In International Conference on Learning Representations, 2022.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, 1861-1870. PMLR, 2018.
Donald Hejna, Lerrel Pinto, and Pieter Abbeel. Hierarchically decoupled imitation for morphologi- cal transfer. In International Conference Machine Learning, pp. 4159 PMLR, 2020.
10

---

Under review as a conference paper at ICLR 2023
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Harold Hotelling. Relations between two sets of variates, In Breakthroughs in statistics, pp. 162- 190. Springer, 1992.
Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Domain adaptive imita- tion learning. In International Conference Machine Learning, pp. 2286-529) PMLR, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceedings, 2015.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.022
2015.
Fangchen Liu, Zhan Ling, Tongzhou Mu, and Hao Su. State alignment-based imitation learning, In International Conference on Learning Representations, 2020.
YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learn- ing to imitate behaviors from raw video via context translation. In 2018 IEEE International Con- ference on Robotics and Automation (ICRA), pp. 1118-1 1125. IEEE, 2018.
Meinard Müller. Dynamic time warping. Information retrieval for music and motion, pp. 69-84, 2007.
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. 2018 IEFE international conference on robotics and automation (ICRA), pp. 3803-3810. IEEE, 2018.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Biblle- Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL http://jmml wwt.er/g//pp//.oee wrg/papers/v22) 220
Dripta Raychaudhuri, Sujoy Paul, Jeroen Vanbaar, and Amit K Roy-Chowdhury. Cross-domain imitation from observations. In International Conference on Machine Learning, pp. 8902-8912. PMLR, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 1134-1141. IEEE, 2018.
Bradly C Stadie, Pieter Abbeel, and Ilya Sutskever. Third person imitation learning. In International Conference on Learning Representations, 2017.
Yanchao Sun, Ruijie Zheng, Xiyao Wang, Andrew E Cohen, and Furong Huang. Transfer RL across observation feature spaces via model-based regularization. In International Conference on Learning Representations, 2022.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167-7176. 2017.
11

---

Under review as a conference paper at ICLR 2023
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Zihan Wang, Zhangjie Cao, Yilun Hao, and Dorsa Sadigh. Weakly supervised correspondence learning. In 2022 International Conference Robotics and Automation (ICRA), pp. 469-476, 2022. doi: (009//CRA44639.2222.88888
Kevin Zakka, Andy Zeng. Pete Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi. XIRL: Cross-embodiment inverse reinforcement learning. In Conference on Robot Learning, pp. 537-546. PMLR, 2022.
Amy Zhang, Clare Lyle. Shagun Sodhani, Angelos Filos, Marta Kwiatkowska, Joelle Pineau, Yarin Gal, and Doina Precup. Invariant causal prediction for block mdps. International Conference on Machine Learning, pp. 1 11214-11224. PMLR, 2020.
Qiang Zhang, Tete Xiao, Alexei Efros, Lerrel Pinto, and Xiaolong Wang. Learning cross-domain correspondence for control with dynamics cycle-consistency. In International Conference on Learning Representations, 2021.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings the on Computer Vision (ICCV), Oct 2017.
the IEEE International Conference
Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martin-Martin. robosuite: modular simu- lation framework and benchmark for robot learning. In arXiv preprint urXiv:2009.12293 2020.
A DERIVATION AND INTERPRETATION OF LEARNING OBJECTIVE
Our objective in is to maximize the entropy of domain prediction given a latent state max H(DSS.) = max Eawp(Dis.) [p(d]s=)] &
min (DIS)) p(d[))(sa)))
where su is sampled from the dataset D. Here we cannot directly evaluate p(d[s2), We introduce the variational approximator s2), which can be seen as domain discriminator. Then we have minBa-p(D(s.) (a(d)o(sa)) + p(dlo(sa)) {(d(o(sa))
min Ed-n(Dis.) (g(d (( + DRL (tl.:l/llss))llllddooo
min E (Gndd (g(d)o(sa)) + DxL (p(dló(sa))ll((dddee))
In the second equation, we remove the necessity of the sampling from the posterior following the discussion similar to Lemma 5.1 in Chen et al. (001)).
Since we would like to minimize this objective instead of maximizing as done in the well-known evidence lower-bound (ELBO) objective, we cannot take a lower bound by skipping the second term. In our case, we can consider that the training of the discriminator & toward correct domain classification is minimization of the KL divergence. Given the sufficient quality of the approxima- tion, the KL term should be small, so encoder can focus on optimizing the first term, which final objective in
B ADDITIONAL RESULTS AND DISCUSSIONS
B.1 ADDTIONAL RESULT OF ALIGNMENT COMPLEXITY
Figure and Figure II show the alignment complexity in Maze2D-O and Maze2D-OA with respect to the number of demonstrations and the number of proxy tasks. As mentioned Section [.5 an increase in the number of demonstrations proxy tasks improves the performance.
12

---

Under review as a conference paper at ICLR 2023
Maze-O
Maze-O
1.0
Success Rate
0.8
Success Rate
0.8
0.6
0.4
0.4
0.2
0.2
Ours-S
GAMA
0.0
0.0
400
2000
Num Demonstrations
Num Proxy Tasks
(a) The number of demonstrations vs success rate
(b) The number of proxy tasks vs success rate
Alignment complexity in Maze2D-O. Here we used fixed single goal. The scores are averaged over three runs. The error bars show the standard deviations.
Figure 10:
Maze-Oi -A
Maze- OA
1.0
1.0
Success Rate
0.8
Success Rate
0.8
0.6
0.4
0.4
Ours
Ours-S
GAMA
0.0
0.0
400
2000
10000
10
Num Demonstrations
Num Proxy Tasks
(a) The number of demonstrations vs success rate
The number of proxy tasks vs success rate
Alignment complexity in Maze2D-OA The scores are averaged over three runs with fixed goal. The error bars show standard deviations.
Figure 11:
13

---

Under review as a conference paper at ICLR 2023
Table 4:
Success rates in Maze2D (medium) with and without behavioral cloning. averaged over nine seeds with three fixed goals.
Results are
Task
BC + adversarial
adversarial only
Maze2D-O (Ours)
0.97 0.07
0.03 0.05
Maze2D-OA (Ours)
0.61 + 0.32
0.03 + 0.07
Maze2D-O (Ours-S)
0.54 0.40
0.03 0.04
Maze2D-OA (Ours-S)
0.72 = 0.29
0.00 = 0.00
Table 5: Success rates of target tasks, with and without task conditioning of the discriminator.
Results are averaged over nine seeds with three fixed goals.
Task
Ours-S
Ours-S (w/o task)
GAMA
GAMA (w/o task)
Maze2D-O (umaze)
0.42 = 0.41
0.46 + 0.42
0.63 = 0.35
0.69 = 0.23
Maze2D-O (medium)
0.54 0.40
0.27 + 0.27
0.77 + 0.24
0.78 + 0.16
Maze2D-OA (umaze)
0.36 + 0.36
0.47 0.38
0.77 077::020
0.38 : 0.35
Maze2D-OA (medium)
0.72 + 0.29
0.67 + 0.24
0.61 + 0.42
0.48 + 0.35
B.2 SUCCESS RATES WITHOUT BEHAVIORAL CLONING
Although we align the representation through behavioral cloning and the adversarial objective jointly, one might think that we can first align the representations with the adversarial term only and directly learn the common policy in the learned representation space. To measure how much behavioral cloning helps the alignment, we measure the success rates when the alignment only done with the adversarial term. TableI4showe that behavioral cloning encourages the model to learn the representation space that incorporates the closeness of states in terms the actions to perform in proxy tasks. The adversarial term is insufficient for the alignment because it only matches the distributions of latent states of the source domain and the target domain. implies that we could get a better alignment we impose appropriate additional constraints the representations.
B.3 EFFECT OF TASK CONDITIONING
We employ a task-level alignment signal by feeding task ID & to the discriminator. It encourages the model to align only the entire latent task. We measure the success rates of target tasks with and without task conditioning of the domain discriminator. The results in Table 5 show that the task-level alignment improves the performance of both our method and GAMA in some cases. Based on this result, we use task conditioning by default in our experiments.
distributions but also the in every single
B.4 PERFORMANCE DROP OF OURS-S IN MAZE2D-O
As shown in Table 2 Ours-S, which takes raw states as input of the action decoder (ax), can sometimes show worse performance than Ours. Since the decoder can directly use domain- specific state information for the action prediction, the common policy can ignore and discard the information of the latent state 8; and only pass latent action that is equivalent to a task this case, the transfer will fail because the decoder will encounter an unknown task ID of target task passed from the common policy at inference time. We conjecture that Ours-S fails when it falls into this failure mode. It can possibly be solved by controlling the amount and the content of the information provided to the decoder from raw state s.
In
B.5 ADDITIONAL BASELINE
As an additional baseline, we measure the transfer performance of XIRL (Zakka al. 2022, XIRL first learns a cross-domain state representation with TCC (Dwibedi et al. 2019) that captures task progression, and utilizes it to shape the reward for performing RL in the adaptation. It thus does not perform zero-shot transfer. The results are shown in Table The results show that our methods consistently outperform XIRL without interacting with the environment in the adaptation phase.
14

---

Under review as a conference paper at ICLR 2023
Table 6: Success rates of larget tasks (Ours, Ours-S, and XIRL). The results of the other methods are shown in Table 2
The values are averaged over nine seeds with three fixed goals.
Task
Ours
Ours-S
XTRL
Maze2D-O (umaze)
0.76 0.33
0.42 0.41
0.30 0.33
Maze2D-O (medium)
0.97 + 0.07
0.54 + 0.40
0.03 + 0.09
Maze2D-OA (umaze)
0.40 + 0.34
0.36 0.36
0.30 0.33
Maze2D-on -OA (medium)
0.61 + 0.32
0.72 = 0.29
0.03 = 0.09
Table Success rates target task in the robotic manipulation enviroment (Ours, Ours-S, and GAMA). The values averaged over nine seeds with three fixed target positions.
Ours
Ours-S (A = 0.0) Ours-S (A 0.5)
GAMA
0.00 0.00
0.61 0.33
0.86 0.22
0.18 0.27
Note that Maze2D-O equivalent to Maze2D-OA for XIRL and we only measure the performance in Maze2D-O since XIRL learns the state correspondence only with states.
B.6 PERFORMANCE IN MANIPULATION TASK
We additionally evaluate our methods in a rohotic manipulation task in robosuite (Zhu et al. 2020) framework (Figure T2). We the Block Lifting task, where the robot has to pick up block and lift it to a certain height. A task is defined as the position of the object to lift. We set nine initial locations on the table. We one for the target task and the rest for proxy tasks for aligning representations. To evaluate the transfer between different robots, we use Sawyer for the source domain and UR5e for the target domain. Sawyer is 7-DoF robot and URSe a 6-DoF robot, hence these robots have different observation spaces and dynamics. The results in Table demonstrate that Ours-S outperforms GAMA. As in Maze-D and Maze-MR, GAMA shows suboptimal performance since there is no good correspondence across robots with different DoF' and grippers. Ours does not perform well in this environment either, probably because the domain-specific state input to the decoder is highly necessary for the accurate control in this environment. Ours-S avoids these issues and successfully performs cross-robot zero-shot transfer. We also confirm that the adversarial term boosts the performance of our method.
(a) Sawyer
(b) URSe
Figure 12: The pictures of the robotic manipulation environments.
15

---

Under review as a conference paper at ICLR 2023
C EXPERIMENT DETAILS
C.I ENVIRONMENTS
All Maze environments used in this paper are based on D4RL (Fu et al. 2020). In Maze environ- ments, two types of agents perform actions: Point and Ant. The Point agent has a state space of four dimensions and action space of two dimensions. The states consist of positions and velocities for the x-axis and y-axis, while the actions consist of the force to he applied for each direction. The Ant agent has a state space of 29 dimensions and action space of eight dimensions. The states consist of the position and velocity of the body, and the joint angles and angular velocities of the four legs. The actions consist of the force to be applied to each joint.
In Maze-D and Maze-MR, we use an environment with different dynamics from the original envi- ronment for the Point agent. We removed the inertia from the source domain and thus changes in position depend only on the action at the step. In Maze-D, the task of the Point agent in the target domain is to learn from the trajectories of an agent in the source environment that does not have in- ertia. In Maze-MR, the task of the Ant agent is to learn from the Point agent in the environment with the simpler dynamics. The input positions of x and y coordinates of the Point agent were reversed from those of the Ant agent to prevent leaking information without representation alignment.
For the manipulation task, we use robosuite framework (Zhu et al. 2020 We use Sawyer for the source domain and UR5e for the target to test cross-robot transfer (Figure 12
We choose Block Lifting task, where the robot has to pick up a block and lift it to a certain height. A task is defined as the position of the object to lift. We set nine initial locations on the table. We use one for the target task and the rest for proxy tasks for aligning representations. The initial pose of the robot is randomized. The observation space consists of the position or angular information and the velocity of each joint and the end effector. Sawyer and URSe have state spaces of 32 dimensions and 37 dimensions, respectively. The input positions of state elements of Sawyer were reversed for the same reason as Maze-MR. Both robots are controlled by delta values of the 3D position of the end effector and ID gripper state.
C.2 DATASETS
As described in Section 3 the dataset contains state action sequences of expert demonstrations with various goals (i.e., multiple tasks). For the Maze2D and Maze-D, we provide about 10k trajectories of six tasks for the umaze, and about 10k trajectories of 25 tasks for the medium maze unless plicitly mentioned in the ablation study. We downloaded the expert demonstrations from http:/// / raill.ecs. berkeley edu /at///atasttttt oofsssofffllino r /e/mmaaeee wtt////az/ddda/// (maze2d-umaze--parrc vl,
-paasuuumm-paa----l.... For Maze-MR, we provide about 10k trajectories of six tasks for the umaze and about 5k trajectories of 25 for the medium maze. When we created expert trajecto- ries for the AntMaze and simpler Maze2D environement without inertia in Maze-D, we used PPO (Schulman et al. 2017) from stale-baseeiines (Raffin et al. 202T) to train agents to move only one square up, down, left, or right. We then composed entire demonstrations by solving the maze with BFS and giving the agent the direction of the next square. For the robotic manipulation environment, we provided 600 trajectories for each task ID. We collected the expert demonstrations by scripted policy based on the object position and the gripper pose.
C.3 ARCHITECTURE AND TRANING DETAILS OF OUR METHOD
Our policy is a simple multilayer perceptron. In the experiments of Maze2D, the state encoder, the common policy, and the decoder have three, five, and three hidden layers with 256 units and ReLU activations. Only the last layer of the decoder has Tanh activation. The dimension of latent representations is also 256. We optimized our objective with Adam optimizer (Kingma & Ba 2015). We set the learning rate to le-4 and the batch size to 256, and trained the model for 20 epochs. In Maze-D, we used the same number of layers as in Maze2D, while, in Maze-MR, we used four, six, and four hidden layers for the encoder, the policy, and the decoder, respectively. In Maze-D and Maze-MR, used Tanh activation for the last layer of each component, and Mish activation for the other layers. The number of units and the dimension of the latent representation were set to 512. We set the leamnnng rate to 2 and the batch size to 512, and trained the model for 40 epochs.
16

---

Under review as a conference paper at ICLR 2023
C.4 BASELINES
For GAMA, we re-implemented the algorithm referring to the original paper and an official imple- mentation (https://githe)
When we found a few differences between the paper and the implementation, we followed the description in the paper. We swept the adversarial coefficient from 0.01 to 10, the learning rate from le-4 to 1e-3, and used 0.5 and le-4, respectively, For CCA, we used the trajectories of the proxy tasks to learn linear state correspon- dence mappings. We first padded trajectories to the same length and created single sequence of trajectories for each domain by concatenating them sorted by task ID. The order of trajecories were randomly shuffled within a task ID, In the adaptation phase, a policy is trained by reinforcement learning using an auxiliary reward function r(sg)) defined as follows:
com/ ermongroup/daat
r(s
Σ [1s(o))) ) - [( 112
[Da
TED
where f(x=) is a learned mapping function from the source (expert) domain to the learned latent space, g(sy) is its counterpart for the target (agent) domain, D2 is the expert trajectories of a target task in the source domain, Sy is state of the target agent at time step f, and 82.7 is target state in the source domain at time step in a sampled expert trajectory т. If the state correspondence
(t)
is learned sufficiently, r(sg)) would be reward for tracking features of expert trajectories in the source domain. We used the Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al.) 20155) algorithm for this step and trained MLP policy for 200k environmental steps. For IfO, trained context translation model which translates expert demonstrations in the source domain into the ones in the target domain. We randomly chose pairs of demonstrations performing the same task and trained the model to perform translation between the corresponding states found by dynamic time warping (DTW) (Müller 2007). In the adaptation phase, we used DDPG to train the policy in the target domain for 200k environmental steps. As in CCA, the reward is defined to track the features of the expert demonstrations in the source domain.
XIRL uses the for the adaptation. Given the demonstration the source domain, the reward is calculated as follows:
feature acquired by TCC (Dwibedi et al. 2019) to shape the reward
rss) 1 ((s)
N Σ (
where (ss) is the state embedding, T, is the length of demonstration. i, N is the number of demon- strations. Intuitively, it is a distance of current state from the averaged goal in the feature space. R is a scale parameter and we set it to 10% of the distance of the first state from the goal state. We tried the 1 1%, 10%, and 100% and chose the best one for our setting. The policy was trained with SAC (Haarnoja et al., 2018) as in the original paper. We trained the policy for 800k environmental steps with learning rate 3e-4. TCC was trained with the classifier-based loss, which performed best in our expeirments. For more details, please refer to the original paper.
For the cuamonstraaion-conditoonn based architecture to process sequences of observations and actions (Figure 5) We fed demonstra- tions and observation history as they were without thining-out timesteps. The maximum sequence length was 250 and 400 for the umaze and medium maze, respectively. The model had three encoder layers and three decoder layers with 256 units and eight heads for each layer. The dropout rate was set to 0.2. The activation function was ReLU and it was applied after the normalization. We set the batch size to 32, the learning rate to 1e-3, and trained the model for 100 epochs. We confirmed that the error of behavioral cloning going down to the similar value observed in the training of our method and GAMA.
model (Cond), we used Transformer (Vaswani al. 2017)-
17


Reviews:
[{"review_id": "huKSEmhF1Z", "reviewer": "Reviewer_rLRK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a novel method of cross-domain transfer with an unaligned dataset, and the proposed approach aims to acquire domain-invariant feature space and a common policy on it. And the experiments with various domain shifts show that the proposed method achieves better performance than existing methods in cross-dynamics. ", "summary_of_the_review": "The paper is well written and provided extensive experiments to show the proposed method works better than baseline models.", "strength_and_weaknesses": "Strength \nThe paper proposes to use approach similar to GAN that aims to acquire domain-invariant feature space and a common policy on it. The author provided extensive ablation study, and compared the model with state of the art models, and showed the new model has better performance.\nWeaknesses\nThe training process seems quite complicated. Combined with the instability of adversarial training, the model can be difficult to tune.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Y5_bhUmoB6Q", "reviewer": "Reviewer_737K", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper study the domain adaptation imitation learning. The paper proposes a method to map the state and the action of both domains into a common space. After learning this common space, the target agent can zero-shot perform the new task that only has source demonstrations. Specifically, the paper use domain adversarial learning to learn such common space of the state and the action.", "summary_of_the_review": "Overall the paper clearly states the problem and the framework, which can inspire the readers. The following works can benefit from the view of the paper, which introduces the common state and action space.\n\nHowever, the novelty of the work is limited. The paper follows GAMA and combines more ideas in the domain adversarial learning. Therefore, I will weakly reject this paper and hope the author can improve the idea and the experiments.", "strength_and_weaknesses": "Strength:\n\nThe topic of cross-domain imitation learning is relatively new and has the potential to explore.\n\nThe framework of the paper (the common state and action space) is clearly stated and can inspire the following works.\n\nThe method introduces domain adaptation learning into behavior cloning, which is relatively new in the community.\n\nWeakness:\n\nDomain adversarial learning is a conventional method for domain adaptation when we need to learn domain-invariant representations. The paper directly uses domain adversarial learning to learn the common space of the state. It is not a very novel idea for me.\n\nGAMA uses domain adversarial learning to learn the mapping of the state and the action between domains. The paper is more like a simple version of GAMA, as the domain discriminator of GAMA takes the triples (s, a, s') as input while the paper takes the state s as input. GAMA learns the mapping from the source state and action to the target while the paper learns a common space. Therefore, the contribution of the paper is incremental to GAMA.\n\nIntuitively speaking, the adaptation phase might disturb the aligned space learned by the alignment phase. Because in the adaptation phase, the method will finetune the common policy for the source domain, it will cause inconsistency between the common policy and the decoder of the target domain.\n\nThe alignment scores in Table1 clearly do not favor the proposed method. If the paper would like to claim this alignment score is not a suitable metric, then it needs to propose some new metrics. \n\nThe experimental environments have no overlap with GAMA. Therefore, it is not convincing enough that the method is better than GAMA.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WbCrrdLHvw", "reviewer": "Reviewer_QGQp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The proposed method is as simple as aligning representations in the latent space to learn a domain-invariant policy. Conceptually that makes sense as it can generalise better in the presence of small perturbations and changes in an environment. This allows to do some domain adaptation with zero-shot learning whereby everything learnt in the source domain (environment Χ) can be transferred into the target domain (environment Υ). The results presented are fairly comprehensive and the visualisations alike, but the description of the method is not very convincing, i.e. it seems to work surprisingly well. The authors have provided the code and the corresponding datasets though.", "summary_of_the_review": "The paper provides adequate experiments and visualisations to justify the advantages of the proposed method. I am somewhat hesitant due to the method description seeming too straightforward and 'simple' (not in a bad way, in that I do not mean that complex is needed) to learn representations that are transferable and generalisable. Results prove me wrong though so am erring in caution.", "strength_and_weaknesses": "Strengths:\n\na) Relatively simple method, which can be extended and further improved relatively easily, especially the adversarial training part, and of course the learning part. \nb) Competitive results show promising avenues for expanding on other settings and in other environments\nc) Visualisations of the latent space provide some further understanding of the underlying processes involved.\n\nWeaknesses:\n\na) Background work is rather blunt. The last paragraph on page 2 is not well written. Methods' names are somewhat akwardly provided alongside the in-text citation.\nb) I am not sure that the background covers all recent methods, but because I am not very well aligned with this area I am not very firm on it.\nc) Further to b) more comparisons could have been added to the main text\nd) Methods are not very well described in the main paper - appendix provides further context though so it somewhat compensates. It buffles me though that the method works relatively well, given what the methods employed", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "E564jfVW-d", "reviewer": "Reviewer_X2sb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel method to perform zero-shot cross-domain transfer by learning a domain-agnostic policy in the source domain that can be directly applied to the target domain. To this end, specific training methods (technical contritions) include: \n1. Align representations of states and actions from different domains to a common latent space through adversarial training. \n2. The common policy is trained to adapt to the target task in the source domain.\n3. Directly apply the common policy to the target domain based on the aligned representations. \n\nThe effectiveness of the proposed approach is verified with four Maze environments in the D4RL benchmark, which is shown to outperform other baselines especially when there are sufficient proxy tasks.", "summary_of_the_review": "Overall, I think the paper presents a simple but effective cross-domain policy transfer method. However, I am inclined to reject this paper given its technical novelty compared with GAMA and the insufficient experimental results. ", "strength_and_weaknesses": "Strength:\n1. It is a simple yet reasonable idea for cross-domain policy transfer by aligning and training a policy in a common latent representation space shared by different domains. \n2. The authors provide both qualitative and quantitative results to show the performance of adversarial alignment. \n3. In general, the paper is well-written and easy to follow. \n\nWeaknesses:\n1. My main concern is the novelty of the proposed approach, which seems very similar to GAMA. Both of them use an adversarial objective for state alignment in different domains so that the policy can be directly transferred to the target domain. The major difference is that in this paper, the states are aligned in a common latent space instead of a direct matching.\n2. If I understand correctly, the action $a_z$ from the common policy may only contain compact information. Therefore, if action $a_d$ in the target domain is more complicated than the action in the latent space, only using $a_z$ and the domain ID might not be able to decode $a_d$. For example, $a_z$ may represent a high-level behavior such as “turn left”. However, for a robot to execute this action, it may take multiple steps based on its current domain-specific states (e.g., the pose of its own torso). In other words, the method might only work when the source and target domains have a strong relationship, because the policy in the target domain is completely determined by the common state, and does not depend on any domain-specific skills.\n3. All experiments are conducted in the Maze environments in the D4RL environments, which is inadequate to support the effectiveness of the model in general control scenarios. Therefore, I would suggest validating the model on other benchmarks with a wide variety of dynamics. For a fair comparison, the authors may use the original environments of other baselines, e.g., those used in the GAMA paper.\n4. Except for GAMA, the compared approaches are somewhat \"out-of-date\". Although I understand that this paper is focused on performing policy transfer between offline datasets, without interacting with various environments, it would be nice if the authors could include stronger baseline methods for model comparison, especially the offline RL approaches that show impressive performance on the D4RL dataset.\n5. A typo: “deomnstrations” in Appendix C.2.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject