Domain Adaptive Imitation Learning with Visual Observation
Authors: Sungho Choi
Seungyul Han
Woojun Kim
Jongseong Chae
Whiyoung Jung
Youngchul Sung
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
Deep Reinforcement Learning
Imitation Learning
TL;DR: We propose a new learning architecture and a reward generation procedure for domain-adaptive IL with visual observation to improve behavior feature extraction and learner policy performance.

Abstract: In this paper, we consider domain-adaptive imitation learning with visual observation, where an agent in a target domain learns to perform a task by observing expert demonstrations in a source domain. Domain adaptive imitation learning arises in practical scenarios where a robot, receiving visual sensory data, needs to mimic movements by visually observing other robots from different angles or observing robots of different shapes. To overcome the domain shift in cross-domain imitation learning with visual observation, we propose a novel framework for extracting domain-independent behavioral features from input observations that can be used to train the learner, based on dual feature extraction and image reconstruction. Empirical results demonstrate that our approach outperforms previous algorithms for imitation learning from visual observation with domain shift.

Domain Adaptive Imitation Learning with Visual Observation
Sungho Choi' Jongseong Chac
Seungyul Han
Woojun Kim² Youngchul Sung!
Whiyoung Jung'
KAIST
UNIST
Carnegie Mellon University
LLG Al Research
{sungho. choi, jongseong- chae, c yycsung)kkaaaa @kaist. ac. kr.
syhanCunist.c kr, moojunk@andrew. cmu.edu whiyoung. whiyoung.jungeee jungelgresearchh.e ai
Abstract
In this paper, consider tion, where an agent in a target domain learns to perform a task by observing expert demonstrations in source domain. Domain adaptive imitation learning arises in practical scenarios where a robot, receiving visual sensory data, needs to mimic movements by visually observing other robots from different angles or observing robots of different shapes. To overcome the domain shift in cross-domain imitation learning with visual observation, domain-independent behavioral features from input observations that can be used train the learner, based on dual feature extraction and image reconstruction. Empirical results demonstrate that our approach outperforms previous algorithms for imitation learning from visual observation with domain shift.
imitation learning with visual observa-
propose novel framework for extracting
Introduction
Imitation learning (IL) framework where an agent learns behavior by mimicking an expert's demonstration without access to true rewards. The mainstream IL methods include Behavior Cloning (BC), which trains the learner with supervised learning Learning (IRL), which tries to find the reward function 1
39,, Inverse Reinforcement and Adversarial Imitation Learning (AIL), which trains the learner to match its state-action visitation distribution that of the expert via adversarial learning [10 12
33
29
49
40. 46). IL provides an effective alternative solving reinforcement learning (RL) problems because it circumvents the difficulty in the careful design of reward function for intended behavior [30 47 Despite the effectiveness, conventional IL assumes that the expert and the learner
18
the same domain, but this a limited assumption in the real world. For example, self-driving should learn real-world driving skills from demonstrations in driving simulator. such a case, the expert and the learner do not lie the same domain.
In this paper, we consider the challenge of domain shift in II., i.e., an agent in a target domain learns to perform a task based on an expert's 's demonstration in source domain. In particular, we focus on the case where the expert demonstrations are provided in visual form. In many practical cases, obtaining expert demonstrations as an explicit sequence of states (such as positions and velocities) and actions (such as movement angles) can be challenging. Instead, the expert's 's demonstrations often provided as image sequences. One example can be a robot mimicking a movement by visually observing human behavior. However, learning from visual observations poses major challenge in IL since images are gigh-dimensional and often include only partial information about the expert's 's behavior information unrelated to the expert's 's behavior. Even minor changes between the source and target domains can vastly affect the agent's 's policy and lead to unstable learning. recent years,
"oorresooniigg Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

several works aimed to address the challenge of domain shift in IL. Some methods show impressive performance by training model to learn a mapping between domains 22 32), requiring proxy tasks (i.e., additional tasks to assist in learning the target task) to provide expert demonstrations in both source and target domains. However, this approach assumes expert demonstrations for similar tasks in the target domain, which may not be practical in many realistic domain transfers where expert demonstrations in the target domain are unavailable. Other methods generate metric-based rewards that minimize the distance between the expert and the learner trajectories [9 27 assuming the expert's s state is directly available rather than requiring image observations. Our focus is on solving domain shifts with visual observations without the use of proxy tasks or direct access to expert's 's states. Our work is closely related to the methods that learn a 37. However, removing domain information not satisfactory in situations with large domain gap. In contrast, we propose a novel learning method to train the learner in domain-adaptin II. with visual observation, achieving significantly improved performance in II. tasks with domain shift.
to extract domain-invarian features
2 Related Work
IL with domain shift IL with domain shift is challenging problem where the learner should learn from an expert in a source domain and apply the learned policy in target domain. In recent years, there has been significant progress in addressing this problem in machine learning, RL, and roboties 66] 7 9 11
32 37 45). For example, Fickinger et al. 99] proposed utilizing Gromov- Wasoo--tinnnniinn distance to generate reward minimizing the difference between expert and learner policies. Franzmeyer et al. []] learned a mapping between two domains, where expert embeddings are reduced by selecting task-relevant state elements based on mutual information criterion. These methods assume to have access expert's ' states; in contrast, our method assumes expert demonstrations only available as high -ish-iimensional image sequences with partial information about the expert. Kim et al. [2 and Raychaudhuri et al. 32 trained a model that can map between two domains by using expert demonstrations for proxy tasks on both domains. In comparison, our method does not rely on expert demonstrations of proxy tasks. Sermanet et al. 34] employed a model that learns representations from visual demonstrations of different viewpoints using cont-conrrattive approach. This approach mainly focuses on viewpoint mismatches between domains and relies on time-synchronized demonstrations with simultaneous viewpoints. Chae et al. (7] introduced an IL algorithm designed to train a policy that exhibits robustness against variations in environment dynamics by imitating multiple experts. In contrast, our method encompasses general domain mismatches including the differences in viewpoint, degree of freedom, dynamics, and embodiment.
14 22 27
28
We focus solving domain shift IL with image observation. Liu et al. 22 used time-aligned expert demonstrations in multiple source domains and learned a model to transform the demonstration from one domain into another. Our work is closely related to Third-Person Imitation Learning (TPIL) (37), which trains a model based on domain innreinnnnddpeddenn feature extraction [3] with GAIL [18]. Similarly, Cetin & Celiktutan [6] proposed restricting mutual information between non-expert data and domain labels. Our method improves on the idea of feature extraction from non-time-aligned visual data by applying dual feature extraction and image reconstruction with consistency checks.
IL from observations (IfO) IfO an area of II., where the expert state-action trajectory is not provided explicitly. In IfO, the learner should capture the behavioral information from observation. Demonstrations can be provided either as a set of state vectors without actions [8] 39 or simply as image observations [13 21 28 43). Torabi et al. [39 learned the inverse dynamics to infer the missing actions from observations in the demonstration. Edwards et al. [8] used latent actions to model the transition between two consecutive observations and relabel those latent actions to true actions. We assume that the learner knows its own state and action but expert demonstration is
provided only in the form of visual observations in other works (6) 37
3 System Model
Setup An Markov Decision Process (MDP) is described by tuple AS,A.P,R,Y Pb), where S the state space, A is the action space, P: S x A
+ R+ the state transition probability, S x A + IR is the reward function, 7 E (0, 1) is the discount factor, and pn is the initial state distribution 38 Given an MDP, a stochastic policy æ : S A + R+ a conditional probability
over actions given state E S. The goal of RL is to find an optimal policy n² that maximizes

---

R(s,, at)], '' where su ~ Po. T(-ss)),st $141 P(-Ist,a)) In IL, expert demonstrations are provided instead of the true The goal of IL with single domain is imitate the demonstration generated by an expert policy Tg.
((²) = BxI_i=o
Problem formulation We now define IL with domain adaptation. We have two domains: source domain and target domain. The source domain is where we can obtain expert demonstrations. and the target domain is where MDPs of the source domain (using subscript S) and the target domain (using subscript '") are given Yr. Po.r). respectively.
train the agent. Both domains are modeled MDPs. The
by Ms = (Ss, Ps. Rs. 7s+ Po.s) and Mr (Sr,Ar, Pr. R expert is the one who performs a task optimally. The agent in the target domain learns the task from expert demonstrations in the source domain. We call the agent a learner. The expert policy : ST x Ar - R is
Ss x As + R- is in the source domain and the learner policy in the target domain. We assume that a set of image observations of Tæ are provided, which denoted by OSE, where 'SE" refers to 'source expert'. The set of observations generated by is denoted by OTL, where 'TI' refers to 'target learner'. In addition to expert demonstrations from the source domain,
assume access to the non-optimal trajectories of image observations from both the source and target domains. The visual observations rollouts from a non-expert policy TSN Sş x As + E in the source domain are denoted by OsN, and the visual observations of rollouts from a non-expert policy TrN :S X Ar - R the target domain are denoted by OrN- Here, 'SN' refers to 'source non-expert', and "TN" refers to 'target non- non-expert'. OsN and Orn additional data to assist our feature extraction and image reconstruction process. TSN and TYN can be constructed in several ways, but we choose these policies as ones taking uniformly random actions. Each observation set has size demmmo and each observation o, consists 4 images corresponding to consecutive timesteps. The goal is to train Ho to solve given task in the target domain using OSE. OsN, OTN, and OTL.
4 Motivation
IL with domain adaptation presents a major challenge as the learner cannot directly mimic the expert demonstration, especially when it is given in visual form. Also, we cannot leverage expert demonstrations for proxy tasks. To overcome domain shift, conventional methods 66 37 extracts features from images. We call such feature behavior feature, capturing only expert non-expert behavior information. One example is TPIL (37). TPIL consists of a single behavior encoder BE, a domain discriminator DD, and a behavior discriminator BD, as shown in Fig. | (The label Bx in Fig. 1 means that the input in the X domain with behavior Bx- X can be source S or target T, and Bx can be expert or non-expert N. We use this notation throughout the paper. See Appendix forr more details). The key idea is that BE learns to fool DD by removing domain information from the input while helping BD by preserving behavior information from the input.
X TM
inputs
behavior encoder
behavior feature
DD
predict
SBs
BE
By
By
BD
behavior
TBT
behavioral Info.
domain info.
(h)
Figure I: Basic structure for domain-independent behavior feature extraction: BE- behavior encoder, BD- behavior discriminator, DD domain discrim- inator
Figure 2: T-SNE plots of features from (a)
TPIL and (b) DisentanGAlL. Each point rep- resents a behavior feature: Red circle (SE) (SE), orange square (SN) and blue cross (TN)
However, the behavior feature extraction of the conventional methods does not work to our satisfaction. Fig. shows the t-SNE [42] plots of the behavior features extracted by the conventional methods
6 37 after training for the task of Reacher with two links (source domain) to three links (target domain). As seen in the figure, the behavior feature vectors of SN (orange squares) and those of
TN (blue crosses) are separated in the left figure (note that when domain information is completely
3

---

- dual feature extraction
................................ image generation
cycle-consistency check
09n
domain discriminator
domain encoder
image discriminator
behavior discriminator
fake images
Inputs
DEs
DEy
I
DDg
T
behavior encoder
Gr
G5
B.B
SBg
BEs
TBg
BET
SBs
Bs
contains info. 7 Bs
Bg
behavioral info.
10,
domain info.
8Pp
DET
generator
DEg
[
$
Gs
Gr
80p
TBy
BEy
SBT
BEş
TBy
Br
contains info. By
BT
feature-level cycle-consistency (Feature-CC)
image-level cycle-consisteng, (Image-CC)
Figure 3: The structure of the learning model: Inputs in source and target domains have features S SBg and T BT, respectively. Bs be E (expert) and N (non-expert), and By be N (non-expert) and L (learner). DE is domain encoder (blue), B is behavior encoder (orange) and G is generator (green). DD, BD and ID with gray color domain discriminator, behavior discriminator, and image discriminator, respectively. Small long rectangles represent features. The sub-block in the blue dotted line in the lower left corner corresponds to the basic structure shown in Fig.
removed, the behavior feature vectors of SN and TN should overlap). Furthermore, since the target non-expert performs random actions for each given state, some actions resemble the expert actions in a given state, while other actions do not. So, there should be some overlap between the blue crosses (TN) and the red dots (SE). However, the blue crosses are well separated from the red circles in the right figure. Thus, the domain information not properly removed from the behavior feature. Extracting domain-independent behavior features needs fully exploit available source-domain and target-domain data.
Our approach is to embed the basic feature extraction structure into a bigger model with data translation. Our main idea for this is dual feature extraction and dual ppolecconsistecy.. First, we propose dual feature extraction scheme to extract both the domain feature and behavior feature from the input so that these two features are independent and contain full information about the input. Second, we propose new cycle-consistency scheme that fits domain comain-adaptive II. with visual observation. The proposed model aims to extract better features for reward generation rather than to generate images. Thus, solely applying image-level cycle-consistency is not sufficient to solve the problem of our interest. cycle-consistency top of image-level sycle-consistency. Third, we adopt reward-generating discriminator outside the feature extraction model for proper reward generation by exploiting the generated target-domain expert data from the feature extraction model. Therefore, we name approach as D3IL (Dual feature extraction and Dual sycle-consistency for Domain adaptive IL with visual observation).
circumvent this limitation by additionally applying feature-level
Proposed Methodology
5.1 Dual Feature Extraction
The behavior feature in the target domain the required information that tells whether the learner's action is good or not, in other words, whether the learner in the target domain successfully mimics expert in the source domain or not. To improve behavior feature extraction, we boost the
simple adversarial learning, as the structure shown in Fig. [ by adding an additional structure from which we can check that the obtained feature well contains required information without information loss, based image reconstruction and consistency check, as shown in Fig. 3 That

---

is, we adopt two encoders in each of the source and target domains: domain encoder DEx and behavior encoder x where X encoder DFxx, where X = S or T, tries to extract domain feature (S or 7) that contains only the domain information excluding behavioral information, whereas the behavior encoder F.x tries to extract behavioral feature ( [xx E or N) that contains only the behavioral information (expert or non-expert) excluding domain information. Each of these encodings is based on the aforementioned basic adversarial learning block composed of one encoder and two accompanying discriminators: domain and behavior discriminators, as shown the left side of Fig. 3 As the outputs of the four encoders, have four output features: S and Bg from the source-domain encoders, and T and Br from the target-domain encoders.
or T. as shown the left side of Fig. 3 The domain
5.2 Dual Cycle-Consistency (Dual-CC)
We now explain dual cycle-consistency (Dual-CC), composed of image-level cycle-consistency (Image-CC CC) and feature-level cycle-consistency (Feature-CC) to improve feature extraction. Using the four output feature vectors extracted by the encoders Sec. 5.1 we apply image translation
48) That is, we combine T and By and generate an image TBs by using generator Gr. and combine S and By and generate an image SBT by using generator Gs. Each image generator (Gr or Gg) is trained based on adversarial learning with the help of the corresponding image discriminator Dr or IDs), as shown in the middle part of Fig. 3 Then, from the generated images TBs and image Sy,, we apply feature extraction again by using the same trained encoders DET, BE,, DEs, BEs used in the first-stage feature extraction, as shown in the right side of Fig. 3 Finally, using the four features T, Bs. S, Br from the second-stage feature extraction, we do apply image translation again to reconstruct the original images SBs and TT by using the same generators Gg and GT used as the first-stage image generators, as shown in the rightmost side of Fig. 3 Note that if the encoders properly extracted the corresponding features without information loss, the reconstructed images SBs and BT should be the same as the original input images SBg and TBr. We exploit this Image-CC as one of our criteria for our feature-extracting encoders.
19
25
26
However, the aforementioned adversarial learning and Image-CC loss are insufficient to yield mutually exclusive behavior and domain features, as we will see in Sec. 6.7 To overcome this limitation, we enhance the dual feature extraction by complementing Image-CC by imposing Peature-CC.. Note that our goal is not image translation but feature extraction relevant to domain-adaptive IL. The Feature- CC means that the extracted features S, Bg, T and By in the left dual feature extraction part of Fig. 3 should respectively be the same their corresponding feature vectors S, Bg, T and BT in the right cycle-consistency check part of Fig. 3 The rationale for this consistency is as follows. Consider By and By for example. Br is extracted from the target-domain image T Br,, whereas Br is extracted from a source-domain image SBT generated from source-domain image generator Gg. For the two behavior features By and By from two different domains to be the same, the domain information in BT and Br should be eliminated. Thus, the Feature-CC helps the extraction of mutually exclusive behavior and domain features while the Image-CC requires the combination of behavior and domain features to preserve full information about the input observation without any loss. Since both consistencies are not perfect, we adopt both Image- CC and Feature-CC for our feature extraction, so named Dual-CC.
5.3 Further Consistency and Stability Enhancement
In addition to the Dual-CC, our architecture allows us to verify other consistencies on the extracted features. First, enforee the image reconstruction consistency by combining the features S and Bs from DEg and BEg in the first-stage feature extraction and feeding the feature combination (S, Bs)
into image generator Gs. The output SBs should be the same as the original image SBs. The same consistency applies to the feature combination (T, Br) in the target domain with image generator Gr- Second, enforce the feature reconstruction consistency by feeding the generated source-domain image SBs described above into the encoders DEs and Bss, then we obtain domain feature Š and behavior feature Bg. These two features should be identical to the features S and Bs extracted the first-stage feature extraction. The same principle applies to the target domain.
5

---

DET
BET
TN
TE
Drew
predict
behavior label
BEg
BEy
SE
TL
Figure 4: Reward-generating discriminator Drew and its training
In addition to the key consistency idea explained above, we apply further techniques of feature vector regularization and feature vector similarity to enhance performance. Feature vector regularization maintains the L2-norm of the feature vector which helps to prevent the feature vectors from becoming too large. Feature vector similarity another form of consistency that guides the model that the domain feature vectors of observations from the same domain should have similar values and the behavior feature vectors of observations from the same behavior should have similar values.
With this additional guidance for feature extraction on top of the conventional adversarial learn- ing block, the proposed scheme significantly improves feature extraction compared to previous approaches, as we will see in Sec. 6, Once the overall structure is learned, we take the behavior encoder Er in the target domain (the block inside the blue dotted block in Fig. 3) to train the learner policy. This improved behavior feature extraction can enhance the learner's 's performance in the target domain. Note that the complexity of D3IL simply the level of widely used image translation.
5.4 Reward Generation and Learner Policy Update
Now, let us explain how to update the learner policy ne. One simply use the behavior discriminator BDg in the target domain together with the behavior encoder BEr to generate a reward for policy updates. Instead, we propose a different way of reward generation. We use Ey but do not use BDn from the feature extraction model. This is because, in our implementation. BD is trained to distinguish the expert behavior E from the non-expert behavior N, but what we need is discriminator that distinguishes the expert behavior from the learner behavior I, so that the learner policy my can receive proper rewards for imitation. Therefore, we train another network called reward- -enarrtinnn discriminator Drew for reward generation. Drew and F jointly learn in an adversarial manner like in GAIL. Drew takes a behavior feature vector from B3 ET input and learns to predict the behavior label (E or L) of the input. For stable learning, we consider GAIL with gradient penalty (GP) (24.
To train Drew. need the expert behavior feature from B well the learner behavior feature L. L is simply obtained by applying the policy output orz to BE,. The required expert behavior feature can be obtained by applying generated target-domain expert image TE to BEr, where TE is generated from the feature extraction model using domain feature from orn and behavior feature E from OSE, as shown in Fig. 4 (Or simply we can use the expert feature E obtained by applying source-domain expert image OSE to EEs based on the imposed Feature-CC.) The loss for Dreat with fixed behavior encoder B ET is given by
log Lp = =E Blorx. Dog (BEr + Eorr.Orr (lo(g Drew (BEr (OTL))
+ Min (Bt
1
where ôTg G(DE(OTN), BE(osE)) is generated image from the domain feature T of or and the behavior feature E of OSE. and the third expectation the GP term and is over the mixture of êTg and oTL with certain ratio. Drew takes a behavior feature vector as input and predicts its behavior label or L by assigning value I to expert behavior and value 0 to learner behavior. On the other hand, To learns to generate observations orL that the corresponding behavior feature vector resembles that of the expert. The learner updates its policy using SAC [17), and the estimated reward for an observation O; is defined by r(oz) log( Drem (BET(o))) log(1 - Drew (B:)():))) Appendix A provides the details of loss functions implementing D3II,, and Algorithms and 2 in Appendix B summarise our domain-adaptiv IL algorithm.

---

6 Experiments
6.1 Experimental Setup
We evaluated D3IL on various types of domain shifts in IL tasks, including changes in visual effects, the robot's degree of freedom, dynamics, and embodiment. We also tested if D3IL can solve tasks when obtaining expert demonstrations by direct RL in the target domain is challenging. Each task consists of source domain and a target domain, with base RL task either in Gym [5] or in DeepMind Contol Suite (DMCS) 41] In Gym, we used five environments: Inverted Pendulum (IP), Inverted Double Pendulum (IDP), Reacher-two (RE2), Reacher-three (RE3), and HalfCheetah (HC). In addition, we used customized U-Maze environment [20] for the experiment CartPole-Swingle and Pendulum. Appendix E contains sample observations, task configuration, and space dimensions for each environment.
in Section. 6.6] DMCS, we used three environments: CartPole-Baland
We compared the performance of D3IL with three major baselines: TPIL 37 and DisentanGAlL [6] and GWIL [9]. TPIL and DisentanGAlL are image-based IL methods that aim to extract domain-inddependet features from image observations and are most relevant to our problem setting. Additionally, we also compared D3IL with GWIL, recent state-based method. For GWILL demonstrations were provided as state-action sequences for all experiments, while for other methods demonstrations were provided image observations. Appendix Appendix[pprovides Drroviddss implementation details including network structures and hyperparameters for D3IL.
6.2 Results on IL Tasks with Changing Visual Effects
We evaluated D3IL on tasks where the domain difference is caused by visual effects on image observations. We conducted the experiment in four IL tasks: IP and IDP, with different color combinations of the pole and cart (IP-to-colored and IDP-to-colored tasks), and RE2 and RE3, with different camera angles (RE2-to-tilee tttttettteed and RE3-to-tilted tasks). Figure 17 Appendix [.]] contains sample images for each task. The average episodic return for D3IL and the baselines over seeds are shown in Fig. [8 in Appendix [F.]] As shown in Fig. 18 D3IL outperformed the baselines with large margins on tasks with changing visual effects.
6.3 Results on IL Tasks with Changing Prgree-oo-Freeddee of Robot
This includes scenarios where the learner robot should imitate an expert performing the same task, but the number of joints differs between them. We conducted four IL tasks: two based on IP (one pole) and IDP (two poles), and two based on RE2 (2-link arm) and RE3 (3-link arm). Sample observations for each IL task are shown in Figure 14 in Appendix E Fig. 5 shows the average episodic return over 5 seeds. It is seen that D3IL outperforms the baseline methods by a large margin in all the tasks. In Fig. the dotted line represents the performance of SAC trained with true rewards in the target domain. This dotted line can considered as the performance upper bound when using SAC as the control algorithm. is interesting to observe that in Figs. (c) and (d). the baselines seem to learn the task initially but yield gradually decreasing performance, whereas D3IL continues to improve its performance. Appendix
Next, we evaluated D3IL on IL tasks with varying robot's degree-of-freedee
F.2 provides further discussions on the quality of extracted features on these tasks. Appendix F3
provides additional results on IP, IDP, RE2, and RE3 with simpler settings created by (6]
TPIL
DisentanGAIL
GWIL
D3IL (ours)
Target domain expert
00
3.5
3.0
2.
30
02
04
3.6
0.3
2.0
02
0.4
06
0.
2.0
00
04
2.
(a) IDP-to-one
(b) IP-to-two
(c) RE3-to-two
(d) RE2-to-three
Figure 5: The learning curves in the target domain of the IL tasks with changing DOF
7

---

TPIL
DisentanGAIL
GWIL
D3IL (ours)
Target domain expert
260
Maat Narrepoe Wan
Vee spoot: war
kre
630
660
sre
232
200
Wevestess
205
Theesless
Teesteps
Threstess
245
(a) CoriPoleBalance-to- (h) CanPPolSSwinUp-tto (c)
Pendulum-to- (d)
Pendulum-to-
Pendulum
Pendulum
CanPoleBalance
CartPoleSwingUp
Figure 7: The learning curves for the considered IL tasks for CartPole and Pendulum environments
6.4 Results on Tasks with Changing Internal Dynamics
We examined D3IL on task with a different type of domain shift, specifically a change in the robot's dynamies. The considered II. task is http//hheeeaat--ooooeeeeeee
Wosh cotsolc rotum
The base RI task is as
8.00
follows. HalfCheetah (HC) environment, the goal to move an animal-like 2D robot forward as fast as possible by controlling six joints. Similar to (6], we also have a modified environment named as HalfChhetahh-Loceelle robot can move only four joints instead of six, as two feet joints are immobilized. Fig. [5 in Appendix E shows sample observations for both environments, where the immobilized feet are colored in red. Fig. 6 shows the average episodic return over 5 seeds. Fig. 6 shows that D3II. has superior performance to the baseline methods and can solve IL tasks by changing robot's dynamics.
(HC-LF)'. In HC-LF, the
0.0
0.2
0.4
0.6
0.8
Timestpp
DavntanGAll
D3IL
Figure 6: Result on HC-LF
6.5 Results on Tasks with Changing Robot Embodiment
We examined whether D3IL can train the agent to mimic an expert with a different embodiment. We use CartPole-Balance CartPole-Swiing Swing and Pendulum environments from DMCS [41, where sample observations are provided in Fig. 16 (see Appendix E). The goal for all environments is to keep the pole upright. The CartPole agents do by sliding the cart, while the Pendulum agent does so by adding torque on the center. The pole is initially set upright in
while it not in other environments. Domain adaptation between these tasks is challenging because not only the embodiment between agents in both domains are different but also the distributions of expert and non-expert demonstrations, and initial observations for both domains are quite different. Fig. shows the performances on IL tasks with different robot embodiment. A Although D3IL does not fully reach the upper bound, it achieved far better performance than the previous methods on tasks by changing the robot's embodiment.
6.6 Experiment AntUMaze: When Direct RL in Target Domain is Difficult
In fact, for II. tasks we have done far, the learner policy be learned directly in the target domain by RL interacting with the target-domain environment. The real power of domain-adaptive II. comes when such direct RL in the target domain is very difficult or impossible by known RL algorithms. Here, we consider one such case known as AntUMaze (20). In AntUMaze, quadruped ant robot should move the red goal position through a U-shaped maze, as shown in the second row of Fig. It can be verified that it is very difficult to solve this problem directly with current state-of-ther RL algorithm such as SAC. This is because the problem involves complicated robot manipulation and pathfinding simultaneously. In order to solve AntUMaze, we consider a simpler task in the source domain known as PointUMaze, which easily solvable by SAC. PointUMaze, the robot is simplified as point whose state is defined as the point location and point direction, shown in the first row of Fig. whereas the state of the ant robot is much more complicated. Appendix (D.3 provides a detailed explanation of the training process for the UMaze environments. Now, the goal
8

---

of this IL task (named PointUMaze-too tnt) is to solve the AntUMaze problem in the target domain with its own dynamics based on the point robot demonstrations.
Souree domain: Point robot
2000
TFL Deartart00
SAC Dawil
600
Target domain: Ant robot
400
0.0
1.0
3.5
Time
Tmedrops
.10
Figure 8: Sample images of U-Figure 9: Results on PointUMaze- to-Ant
maze
Fig. shows the average episodic return over 5 seeds. As shown in Fig. 9 the ant robot trained with D3IL reaches the goal position after about 1M timesteps, while the ant robot trained with the baseline IL algorithms failed to reach the goal position. D3IL can train the ant agent to learn desirable behaviors from point-robot expert demonstrations with totally different dynamics. We observed that SAC with reward could not solve this task. So, it is difficult to directly solve the AntUMaze with RL or to obtain expert demonstrations in AntUMaze directly. D3IL can transfer expert knowledge from an easier task to a harder task, and train the agent in the target domain to perform the AntUMaze task.
6.7 Analysis of Dual Cycle-Consistency
To analyze the impact of Dual-CC, we use t-SNE to visualize behavior features generated by our model. Fig. 10 shows the t-SNE plots of the behavior feature extracted by the model trained on the RE2-to-three task: We used the model with full Dual-CC in Fig. 10 (a), the model with only Feature-CC in Fig. 10 (b), and the model with only Image-CC in Fig, 10 (c). Fig, T0 (a) based on the Dual-CC shows the desired properties; target-domain non-expert (TN) and source-domain non-expert (SN) are well overlapped, and some of the random behavior feature vectors coincide with the source-domain expert (SE) behavior. We also plotted behavior features of generated target experts (TF) with green '+' in the figure. Since the behavior feature of TE is derived from the behavior feature of SE and we imposed the Feature-CC, the red circles and green '+' overlap quite a good portion. However, there still exists some gap at a certain region, showing the feature extraction is not perfect. Fig. (c) shows that domain information still remains when removing the Feature-CC. and the Feature-CC CC indeed helps eliminate domain information. As seen in Fig. 10 (b), only the Feature-CC is not enough to extract the desired behavior feature. Indeed, Dual-CC is essential in extracting domain- inmuiinedddeeeddent behavior features.
6.8 Ablation Study
Impact of each loss component D3IL uses multiple add-ons the basic adversar- ial behavior feature learning for consistency check. As described in Sec. the add-ons for better feature extraction are (1) image and feature reconstruction consistency with the generators in the middle part of Fig. 3 (2) Image-CC with the right part of Fig. 3 (3) feature similarity, and (4) Feature- CCC. We removed each component from the full algorithm in the order from (4) to (1) and evaluated the corresponding performance, which is shown Table 1
TN
(2)
1b1
[C]
Figure 10: T-SNE plot of behavior feature from D3IL: (a) Dual-CC (b) Feature- CC only (c) Image-CCC only
Table I each component has a benefit to performance enhancement, Especially, the consistency check for the extracted feature significantly enhances the performance. Indeed, our consistency criterion based on dual feature extraction and image reconstruction is effective.

---

Reward-generating discriminator Drew In Sec. 5.4 we explained the use of Drew for reward estimation. As explained, we train Drew and the learner policy To in an adversarial manner. Instead of using an external Drew. we could use the behavior discriminator BDp in the feature extraction model to train the policy Tg. To see how this method works, we trained xo using the behavior discriminator BDg in the feature extraction model, not the external Drew- As shown in Table 2 training a using BDg is not successful because of the reason mentioned in Sec. 5.4
Table I: Impact of each loss component
RE3-to-too
Table 2:
Reward-generating discriminator:
basic adversarial learning
-10.7 = 0.5
Drew versus BDB
image/feature reconstr. consistency
-6.0 0.7
IP-to-color
RE2-to-tilted
image-level cycle-consistency
-3.0 0.1
Use Drew
1000 = 0
-4.3 + 0.3
feature similarity
-2.8 0.1
Use BDg
293 + 324
-11.9 0.8
feature-level cycle-consistnncy
-2.6 0.1
Domain encoders As described in Sec. 5 we proposed employing both domain encoder and behavior encoder for enhanced domain-independerr behavior feature extraction. This approach was motivated by the observation that solely relying domain-related information from behavior features. To investigate the importance of domain encoders, we evaluated the performance of D3IL with and without domain encoders (DEs and DET) and their corresponding discriminators (DDd and BDp) across four tasks: IP-to-two, RE2-to-thres RE3-to-tiltee and CartPoleBalamce-to to- Mo-Pendulumm All other networks and losses remain unchanged. Fig. I shows the results. The results demonstrate the benefit conditioning generators on learned domain encoders to boost model performance.
behavior encoder was insufficient to eliminate
D3IL (with domain encoders)
D3IL (without domain encoders)
Target domain expert
30000
2003
6000
4830
Timesteps
(a) IP-to-two
(b) thethee
(c) RE--to-tlleed
(d)
CarlPoeeBalance-to
Pendulum
Figure 11: Ablation study on domain encoders
7 Discussion and Conclusion
In this paper, we have considered the problem of domain-adaptive adaptive IL with visual observation. We have proposed new architecture based on dual feature extraction and image reconstruction in order to improve behavior feature extraction, and a reward generation procedure suited to the proposed model. The proposed architecture imposes additional consistency criteria on feature extraction in addition to basic adversarial behavior feature extraction to obtain better behavior features. Numerical results show that D3IL yields superior performance to existing methods in IL tasks with various domain differences and is useful when direct RI. in the target domain is difficult.
Limitations While our method demonstrates superior performance, also has some limitations. One is the tricky tuning of loss coefficients, which involves several loss functions. However, we observed that balancing the scale of loss components is sufficient. Additionally, our feature extraction model remains fixed during policy updates. Future works could leverage the past target learner experiences to update the model further. Another limitation is the need for target environment interaction during training. characteristic shared by several Nomain-adaptive IL approaches. It will be interesting to extend our method to an offline approach future works. Moreover, our method lacks quantitative measure for domain shifts. Future research could develop mathematical way to quantify domain shifts, enabling the measurement of task difficulty and addressing more complex II. problems. Lastly, future works could extend our method to multi-task or multi-moda scenarios.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by Institute of Information & Communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) Nlo.2022-0-004400 Develop- ment of Core Technologies for Task-orienter Reinforcement Learning for Commercialization of Autonomous Drones) and in part by Institute of Information & Communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 202200022-000122,, Development of Artificial Intelligence Technology for Self-Improving Competency-
Learning Capabilities). Prof. Scungyul Han is currently with Artificial Intelligence Graduate School of UNIST, and his work partly supported by Artificial Intelligence Graduate School support (UNIST), IITP grant funded by the Korea government (MSIT) (No.2020-0-01336,
References
[1] Abbeel, P. and Ng. A. Y. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference Machine learning, pp- 1, 2004.
[2] Arjovsky, M., Chintala, S., and Bottou, I.. Wasserstein gan. arxiv 2017. arXiv preprint arXiv: 170100787, 30:4, 2017.
[3] Bain, M. and Sammut, C. A framework for behavioural cloning. In Machine Intelligence 15, Intelligent Agents SSt. Catherine's 's College, Oxford, July 1995], pp- 103-129, GBR, 1999. Oxford University. ISBN 0198538677.
[4] Brockman, G., Cheung, V., Pettersson, L.,, Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym, arXiv preprint arXiv: urXiv:1606.0/540 2016.
[5] Brockman, G., Cheung, V., Pettersson, L.,, Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym, 2016.
[6] Cetin, E. and Celiktutan, O. Domain-robust visual imitation learning with mutual information constraints. arXiv preprint arXiv:2103.05079, 2021.
[7] Chae, J., Han, S., Jung, W., Cho, M., Choi, S., and Sung, Y. Robust imitation learning against variations in environment dynamics. In International Conference on Machine Learning, pp- 2828 2852. PMLR, 2022.
[8] Edwards, A., Sahni, H., Schroecker, Y., and Isbell, C. Imitating latent policies from observation. In International conference on machine learning, pp. 1755-1763. PMLR, 2019.
[9] Fickinger, A., Cohen, S., Russell, S., and Amos, B. Cross-domain imitation learning via optimal transport. In 10th International Conference on Learning Representations, ICLR, 2022.
[10] Finn, C., Levine, S., and Abbeel, P. Guided cost learning: Deep inverse optimal control via policy optimization. In Proceedings of the 33rd International Conference International Conference on Machine Learning Volume 48, ICML' 16, pp. 49-58. JMLR.org, 2016.
[11] Franzmeyer, T., Torr, P., and Henriques, J. F. Learn what matters: cross-domain imitation learning with task raskvrelevant embeddings. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, Advances D.. Cho, K., and Oh, A. Advances in pp. 26283-262944 Curran Associates, Inc., 2022.
Neural Information Processing Systems, volume 35,
[12] Fu, J., Luo, K., and Levine, S. Learning robust rewards with adverserial inverse reinforcement learning. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenKeview.net, 2018.
[13] Gamrian, S. and Goldberg, Y. Transfer learning for related reinforcement learning tasks via image-to-image translation. In International Conference on Machine Learning, pp. 2063-2072. PMLR, 2019.
[14] Gangwani, T. and Peng. J. State-only imitation with transition dynamics mismatch. In Interna- tional Conference on Learning Representations, 2020.
11

---

[15] Ganin, Y. and Lempitsky, V. Unsupervised domain adaptation by backpropagation. In Interna- tional conference on machine learning, pp. 1180-1189. PMLR, 2015.
[16] Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
[17] Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018.
[18] Ho, J. and Ermon, S. Generative adversarial imitation learning. Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 4572 4582-4588, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 978151083881
[19] Huang, X., Liu, M.-Y., Belongie, S., and Kautz, J. Multimodal unsupervised image-to-image translation. In Proceedings of the European conference on computer vision (ECCV), pp. 172- 189, 2018.
[20] Kanagawa, Y. Mujoco-maze, 2021. URL BtRp::tt::/// http://ppw.ggiau....o///////////
[21] Karnan, H., Warnell, G., Xiao, X., and Stone, P. Voila: Visual-observatiin-nnnly imitation learning for autonomous navigation. arXiv preprint arXiv:2105.09371, 2021.
[22] Kim, K., Gu, Y., Song, J., Zhao, S., and Ermon, S. Domain adaptive imitation learning. In International Conference on Machine Learning, pp. 5286-5295, PMLR, 2020.
[23] Kingma, D. and Ba, J. Adam: method for stochastic optimization. arXiv preprint arXiv: 2014.
[24] Kostrikov, I., Agrawal, K., Dwibedi, D., Levine, S., and Tompson, Discriminator-actor critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXiv: 1809 02925, 2018.
[25] Lee, H.-Y., Tseng, H.-Y., Huang, J.,B., Singh, M., and Yang, M.-H. Diverse image-to-image translation via disentangled representations. In Proceedings of the European conference on computer vision (ECCV), pp. 35-51, 2018.
[26] Lee, H.-Y., Tseng, H.-Y., Mao, Q., Huang. J.-B., Lu, Y.-D., Singh, M., and Yang, M.-H. Drit++: Diverse image-to-image translation via disentangled representations. International Journal of 2417, 2020.
Computer Vision,
[27] Liu, F., Ling, Z., Mu, T., and Su, H. State alignment-brat based imitation learning. In International Conference on Learning Representations, 2020.
[28] Liu, Y., Gupta, A., Abbeel, P., and Levine, S. Imitation from observation: Learning imitate behaviors from raw video via context translation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1118-1125. IEEE, 2018.
[29] Ng, A. Y. and Russell, S. J. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, ICML '00, pp. 663-670, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607072.
[30] Osa, T., Pajarinen, J.. Neumann, G., Bagnell, J. A., Abbeel, and Peters. J. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics, 7(1-2):1-179 2018. doi: 10.1561/2300000053.
[31] Pomerleau, D. A. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88-97, 1991.
[32] Raychaudhuri, D. S., Paul, S., Vanbaar, J., and Roy- noy-Chowdhury, A. K. Cross-domain imitation from observations. In International Conference on Machine Learning, pp. 8902-8912. PMLR, 2021.
12

---

[33] Ross, S., Gordon, G., and Bagnell, D. A reduction of imitation learning and structured prediction no-regret online learning. In Gordon, G., Dunson, D., and Dudik, M. (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 627-635, Fort Lauderdale, FI., USA, 11-13 Apr 2011 1. PMLR.
[34] Sermanet, P., Lynch. C.,, Chebotar, Y., Hsu, J., Jang, E., Schaal, S., Levine, S., and Brain, G. cont-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference robotics and automation (ICRA), pp. 1134-1141 IEEE, 2018
[35] Sharma, P., Pathak, D., and Gupta, A. Third-person visual imitation learning via decoupled hierarchical controller. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F,, Fox, E.,, and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.. 2019.
[36] Smith, L., Dhawan, N., Zhang, M., Abbeel, P., and Levine, S. Avid: Learning multi-stage tasks via pixel-level translation of human videos, 2020.
[37] Stadie, B. C., Abbeel, P., and Sutskever, Third-person imitation learning, 2017.
[38] Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018
[39] Torabi, F., Warnell, G., and Stone, Behavioral cloning from observation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI'1, pp. 4950-4957. AAAI Press, 2018. ISBN 9780999241127.
[40] Torabi, F., Warnell, G., and Stone, P. Generative adversarial imitation from observation. arXiv preprini arXiv: /807.06/ 58, 2018.
[41] Tunyasuvunakool, S., Muldal, A., Doron, Y., Liu, S., Bohez, S., Merel, J., Erez, T., Lillicrap, T., Heess, N., and Tassa, cin_contool: Software and tasks for continuous control. Software Impacts, 6: 6:100022, 2020. ISSN 2665-9638. doi: http://o.org/1111116f......
[42] Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008
[43] Wen, C., Lin, J., Qian, J., Gao, Y., and Jayaraman, D. Keyframe-focused visual imitation learning. arXiv preprint arXiv:2106. 216.:22006004452,, 2021.
[44] Yi, Z.. Zhang, H., Tan, P., and Gong, M. Dualgan: Unsupervised dual learning for image-to- image translation. In Proceedings of the IEEE international conference computer vision, pp. 2849-2857, 2017.
[45] Zakka, K., Zeng, A., Florence, P., Tompson, J., Bohg. J., and Dwibedi, D. Xirl: Cross- embodiment inverse reinforcement learning. In Faust, A., Hsu, D., and Neumann, G. (eds.), Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pp. 537-546. PMLR, 08-11 Nov 2022.
[46] Zhang, X., Li, Y., Zhang, Z., and Zhang, Z.-L. f-gail: Learning f-divergence for generative adversarial imitation learning. Advances in neural information processing systems, 33:1 2805- 1 12815, 2020.
[47] Zheng. B., Verma, S., Zhou, J., Tsang, I., and Chen, Imitation learning: Progress, taxonomies and opportunities. arXiv preprint arXiv: 106. 02..//...200 /2177, 20117,2002 2021.
[48] Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
[49] Ziebart, B. D., Maas, A., Bagnell, J.,., and Dey, A. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence Volume 3, AAAI'08 08, pp.
AAAI Press, 2008. ISBN 9781577353683.
13

---

A The Loss Functions for the Proposed Model
In Sec. we provided the key idea of the proposed learning model. Here, we provide detailed explanation of the loss functions implementing the learning model described in Fig. 3 of the main paper. Table shows the notations for observation sets.
Table 3: The notations for observation sets
Notation
Source domain
Os OSE U UOS
Target domain
Or OTN U OTL
Expert behavior
Og OSE
Non-expert behavior
ON = Osn U UOTN
All observations
OALL OsE U Osx UOIN UOTL
The observation sets OSE. OsN. OIN and OrL defined in Sec. 3 of the main paper.
A.1 Dual Feature Extraction
The left side of Fig- 3 shows the basic adversarial learning block for dual feature extraction. That is, we adopt two encoders in each of the source and target domains: domain encoder and behavior encoder. Behavior encoders EsEs. BTT) learn to obtain behavior f vectors from images that only preserve the behavioral information and discard the domain information. Here, subscript S or T denotes the domain from which each encoder takes input. Domain encoders (DEs, DET) learn to obtain domain feuture vectors which preserve only the domain information and discard the behavioral information. The encoders learn to optimize the WGAN objective 22] [6] based on the feature discriminators: behavior discriminator BD, and domain discriminator DD,. Discriminators BD and DDg are associated with the behavior encoder BEx. where X = S or T. and Discriminators BDD and DDn are associated with the domain encoder DEx, where X S or T. BDg learns to identify the behavioral information from behavior feature vectors, and DDB learns to identify the domain information from behavior feature vectors. BDp learns to identify the behavioral information from domain feature vectors, and DDp learns to identify the domain information from domain feature vectors. For notational simplicity, we define functions BE and DE
BEs(() if o Os (o) if FoeOr Or
BE
(D) if o Os DE (o)
DE
(2)
if of
Two loss functions are defined for the dual feature extraction: feature prediction loss and feature adversarial loss.
1) Feature prediction loss: This loss makes the model retain the desired information from feature vectors and is given by
I feat
(Boco Ex (BE()))
Eoco, (DD, -Eo,
(3)
where E is the expectation over inputs o E A. The first term Eocoz D (BE())) in the right- hand side (RHS) of (3) means that if the input observation of the encoder (E is from the expert observation Og, then the discriminator BDg with the extracted feature should declare a high value. The second term - -Eveo, [ (BE())) in the right-hand side (RHS) of (3) means that if the input observation the encoder BE(-) from nonexpert observation set ON, then the discriminator BD with the extracted feature should declare a low value. The third term Eocos (DD(()(()) in the right-hand side (RHS) of (3) means that if the input observation of the encoder DF(-) is from the source-domain set Os, then the discriminator DDp with the extracted feature should declare high
14

---

value. The fourth term -oo [DDp(DE(o)]] in the right-hand side (RHS) of means that if the input observation of the encoder (E ) is from the target-domain set Or, then the discriminator DDD with the extracted feature should declare a low value. Since we define a loss function, we have the negative sign in front of the RHS of B (Note that BDB learns to assign higher values expert behavior and to assign lower values to non-expert behavior. DDD learns to assign higher values to the source domain and to assign lower values to the target domain.) The encoders (BE, DE) help discriminators (BDB, DDD) to identify the behavioral or domain information from the input, and they jointly learn minimize "feat"
2) Feature adversarial loss: This loss makes the model delete the undesired information from feature
vectors by adversarial learning between the encoders and the discriminators. The loss is given by
- Eo (DDD,()D)(BB(O))) ED L
Jade feat
+ Boo0 (BDr - EN [BDn (DE())))
(4)
In this loss function, the output of the behavior encoder is fed the domain diseriminator, and the output of the domain encoder is fed to the behavior discriminator. The learning of the encoders and the discriminators done an adversarial manner as
min max I Rdde BE,DE DD.,BD. -feat-
(5)
Recall that BD learns to assign higher values to expert behavior and to assign lower values to non-expert behavior, and DD. learns assign higher values the source domain and to assign lower values to the target domain. Thus, the discriminators try to do their best, whereas the encoders try to fool the discriminators.
The first two terms in the RHS of (4) imply that the output of the behavior encoder BE should not contain the domain information. The last two terms in the RHS of (4) imply that the output of the domain encoder DE should not contain the behavioral information.
A.2 Image Reconstruction and Associated Consistency Check
In the previous subsection, we explained the loss functions of basic adversarial learning for dual feature extraction. Now, we explain the loss functions associated with consistency checks with image reconstruction.
As shown in the middle part of Fig. for image reconstruction, we adopt a generator in each of the source and target domains: generator Gs for the source domain and generator Gr for the target domain. Generators Gs and Gy learn produce images from the feature vectors so that the generated images should resemble the original input images. Gs takes a source-domain feature vector and behavior feature vector as input and generates an image that resembles images in the source domain. On the other hand, Gr takes a target-domain feature vector and behavior feature vector as input and generates an image that resembles images the target domain.
Generators Gs and Gr learn to optimize the WGAN objective with the help of image discriminators: IDg and IDr. IDs distinguishes real and generated images in the source domain, and I Drr does so in the target domain. For notational simplicity, we define functions G(---) and ID(,)) as
((DE()) BE(() (r((DE())BB(()))
if Os if Or if 0s
(6)
(IDs(o)
ID(o) = (D)
(7)
if Or.
1) Image adversarial loss: The basic loss function for training the image generators and the associated blocks is image adversarial loss of WGAN learning [2] [6]. The image adversarial loss makes the model generate images that resemble the real ones and is given by
L ade img (SE) + 1a (SN) + (TN) (TLL).
(8)
15

---

where (x) is the adversarial WGAN loss for a real image in Oz. In more detail, the first term in the RHS of can be expressed
(9) Note that the image discriminator ID. learns to assign a high value for true image and a low value for fake image. The first term in the RHS of ( trains the image discriminator / DD)) to assign high value to a true image E OSE, whereas the second term in the RHS of (9) trains the image discriminator I D.) to assign low value fake image. The latter is because the (-) input, ie.. the generator output (o))())) (o,))) with (0z, 0y) E (OSN. Osg) inside the second term in the RHS of 9 is a fake image; for this image, the behavior feature taken from oy OSE but the domain feature is taken from O, OsN not in OSE-
( (SE) = oo - DonGLEEE (EE,
The other terms 1 (SN), 10 (TN) and 1 (TL) in the RHS of ( similarly defined as
(SNN ESEOSN [I[(o] - L (.).ss)) LDDGGDE(C)).BE(oo))))
(10)
(TNN = E. [ID())] 10 (O))
(11)
(TL) = EocorI [ID 1, (D((C(D((o.) BE((,,
(12)
where ÖSN.fak: is the set of image combinations {(o,, o,)) such that the generated fake image G(DE(0,); (E)) )) has label SN. ÓTN,fake and frke are similarly defined. Specifically,
ŌSN,fake (OsE, OsN) (OsE, Orx) U (OsN,Orr OrN)
ÖSN, fake (OSE, OSN) ) (OSE, OTN) U (00sN,OTN)
ÓTL.pavr = (OrN. OTL).
With the image adversarial loss Lonng in 8 the learning is performed in an adversarial manner. The encoders and generators learn to minimize T. while the image discriminators learn to maximize Lony- That is, the image discriminators are trained to distinguish fake images from real images, whereas the encoders and the generators are trained to fool the image discriminators ID..
Note that the image adversarial loss is basically for training the image generators and the associated blocks in WGAN adversarial manner. With the availability of the image generators Gs and GT, we can impose our first and second consistency criteria: image reconstruction consistency and feature reconstruction consistency. We define the corresponding loss functions below.
2) Image reconstruction consistency loss: This loss checks the feature extraction is properly done. When we combine the features and Bs from DEs and B Es with an input true image OsE (or input true image OSN) in the first-stage feature extraction and input the feature combination (S, Bs) into image generator Gs. the generated image should be the same as the original observation image oSE (or ogN). The same consistency applies the feature combination (T, B)) in the target domain with true input image orN or orL and the image generator Gr. Thus, the image reconstruction loss is given as
recon EDCOALL Imoc 6)]
(13)
where
= ((DD(())BBBLC)
(14)
OALL OSE UOsN UOTN UOTL. and Iyse (u, v) is the mean square error between u and v. The encoders and generators learn to minimize Liecon
3) Feature reconstruction consistency loss: This is the second self-consistency criterion. If we input the generated source-domain image SBs described in the above image reconstruction loss part into the encoders D.s and Ess, then we obtain domain feature Š and behavior feature Bg, and these two features should be the same the features S and Bs extracted in the first-stage feature extraction.
16

---

The same principle applies to the target domain. Thus, the feature reconstruction consistency loss is expressed as
feat ( (1B80)))B((()133
Eo (
(15)
where
6 = C(DE()),
(16)
|| ll2 denotes the 2-norm of vector, and OLL = OSE UsN OTN OTL. Note that is generated image from the domain and behavior features extracted from o, and D and (o) are the domain and behavior features from the generated image The encoders and generators learn to minimize L feat.
A.3
Cycle-Consistency Check
Now, let us consider our third consistency criterion: cycle-consistency check with the right side of Fig. This consistency check involves image translation in the middle part of Fig. and the image retranslation in the right part of Fig. B in the main paper, and requires that the original image and the reconstructed image of translationvieetaanlatioon should be the same for perfect feature extraction and generation. The image cycle-consistency loss expressed as
cycle img
E (o, cyate mmseloz, G(DE(O BE( BE(O(wa)))
(17)
where
(2,2) = G(DF(()), ((g.x) = G(DE(()),
(18)
(19)
and O Ogyete is the set of image pairs such that the two images in pair do not belong to the same domain, ie., one image belongs to the source domain and the other image belongs to the target domain. The explanation as follows. Consider that we apply image Og to the upper input and image oy to the lower input of the left side of Fig. Blin the main paper. Then, ((..))) = G G(DF(oxx) B (oy)) is the generated image in the lower row and (gg.x) = =GoyE(oy) (E(oz)) is the generated image in the upper row in the middle part of Fig. i the main paper. Finally, the image ()))) ((()()))) is the final reconstructed image the upper row of the right side of Fig. 3 the main paper. So, can check the cycle-consistency between Og and G G(DE ( ), B ((( )) The situation is mirrored
when we apply image o₂ the lower input and image Oy to the upper input of the left side of Fig.
in the main paper. The encoders and generators learn to minimize Leyele
Limg
The feature cycle-consistency loss is expressed
L cycle feat =R (o (y Syrle Mmsce (DE(oz), D (((())))) + B (o eyela Imnn (BE(o,), BE()))a)))
(20)
where
((z,) = G(DE(,,), BE(o,)
(21)
The explanation is as follows. For example, from the target domain real image TBr, we extract domain feature and behavior feature BT. From these features, we can reconstruct TB. Also, we extract domain feature Î (from TB) and behavior feature By (from SBy). From these features, we can reconstruct TBr. Let's assume By (behavior feature from the source domain image SBT generated by Gs) replaced by By (behavior feature from the target domain image T Br). Then from T and By we should still reconstruct TBr. That is, By (behavior feature from the source domain image SBT made by Gs) BT (behavior feature from the target domain image TB).. They are behavior features from different domains but they are equal. This implies that the behavior feature is independent of domain information. We name this by feature cycle-consistency because this constraint implicitly satisfies image cycle-consistency. if explicitly constrain BT Br and T T, then we can replace By and T by Br and T, and the image cycle-consistency is implicitly satisfied.
17

---

A.4 Other Losses
We further define losses to enhance performance.
1) Feature regularization loss: The feature regularization loss prevents the feature vector values from exploding for stable learning, and is given by
Lreg featt
Eue [II + ||
(22)
Instead, more rigorously, we can use the following regularization:
I
[(]) [
Eooos
Cnrrm, a1 )²]
"feat
Feat Eogos [(]]E}(o)]] Cnorm,b
reg.3
feat ErcOr [(])Er(()] - Cnorm
reg.
EDEOr [[]BEr(o)]] - Cnorm,b) )
"fet
I Leeg "fett I. I.ea.l ffett
+ 2. pegg 1.eg..4
(23)
Where, Crorm,d and Cnorm,b & hyperparameters with non-negative values. Appendix D.2 provides details including the value of Coorm,d and Cnorm,d for the experiment.
2) Feature similarity loss: This loss aims map observations of the same domain to similar points the feature vector space and to map observations of the same behavior to similar points in feature vector space. The feature similarity loss is defined as
aim = [[MBE(OsN) (BE(OTN))B [[ndE(Ose) - -DDEOos) + [!DEEOOT)) (DE(OTr)) 1 feat
(24)
where (BB(OO)) and (DB(O.) the means of the behavior feature vectors and the domain feature vectors over the observation set On (æ {SE, SN,, TT),, respectively.
A.5 Final Loss
Combining the above individual loss functions, have the final objective for the encoders E = (BEs, DEs, ET, DET), the generators G = (Gs,, GT)), feature discriminators FD (BDB, DDB, BDD, DDp) and image discriminators ID = (IDs, IDT) as follows:
min LFGG B.G
adv L feat + Xadu, ado L + Nony Ling Lade + Aing Ling + Xjett Lifeut
(25)
+ Xaneee Linn + cayele cyele +
Leyele
sim sim feat X LineL + reg reg feac featt feat
add min LFD,ID Xfeat prod FDJID
yadv adv yadu ado XfeatL
Ninn Ling
(26)
Cycle cycle Aaim reg where the weighting factors Xade Xadv Xadv Trecon feat Aigg feat feats feat are hyper-
fratr Arng. parameters and their values are shown in Appendix and (26) with negative sign in (26 Hence, these two terms induce adversarial learning in the overall learning process.
D Note that yeat and Lade appear in both (25)
A.6 Reward Generation and Learner Policy Update
In this section, we explain the simple method of extracting expert features and training Drew for reward generation. That is, we just the output of BEs in the source domain with expert input osg. In this case, Drew learns to minimize the following objective while behavior encoder BET is fixed:
Lo BusnEOsH Drw HF [lg((1 Drew (Be((B())
Bor
(27)
+ (1: (Be()) (BE(o)))ll 1)²).
where the third expectation in (7 is over the mixture of êTæ and orL. with certain ratio. The proposed method other than this simple method explained in Sec. 5
D Drew takes behavior feature vector input and predicts its behavior label E or L from the input
18

---

behavior feature vector. Drew learns to assign the value to the expert behavior and 0 to the learner behavior. On the other hand, To lears to generate observations OTL. so that the corresponding behavior feature vector looks like the expert. The learner updates the policy using SAC [7., and the estimated reward for an observation Or is defined by
f(o+) Drew (BET(o)))) log(1 Drem (ot()))))
(28)
which is in a similar form to that in 24
19

---

B Algorithm Pseudo Codes
Algorithm 1 Dual feature extraction and image generation
Input:
The number of epochs nepoch_its domain encoders DE (DEs. DEr), be- havior encoders BE EB BET), generators G (Gs, Gr), feature discriminators FD nBDn, DDn, BDD, DDn), image discriminators ID = (IDs, IDr), observation sets OSE, OSN, OrN of size Rddemo-
Initialize parameters for DE, BE, G, FD,, ID.
Make copy of OTN and initialize OTL, with the copy. for to Nepoch it do
Sample a minibatch of observations OSE, USN, OTN, from OSE, OSN, OTN, OT respec- tively.
for in (osg U osN UorN UOTL) do
Extract (o) and (E()).
Generate in eq. (4).
Extract ( and (E)
end for
for Oz: Oy (not in the same domain) (osg JosN Uory Uorz) do
Generate (2.,) and (y.zx) in eqs. (18 and (19)
Generate (x..)) (E (v.)))
end for
Compute LE,G and LID.ID in eqs. (2 and 26
Update DE, BE and to minmmize Lg,G- Update FD and ID to minimize LTD,ID- end for return E, FD,
Algorithm 2 Reward estimation and policy update
Input:
The number of training epochs Repoch_poo the number of discriminator updates Nupdate the number of policy updates Trupdate, e. domain encoders DE, behavior encoders BE, generators G. discriminator Drew for reward estimation, learner policy observations sets OSE. OSN, OTN of size ndeemo replay buffer B of size fbuffer-
Initialize parameters for Drews To and for 1
Repooch_po do Sample a trajectory T ztP- Store transitions (s, o, s') and observations B.
for kg = 1 to Rupdaaee, do
Sample minibatch of observations OSE OsE, OTN E OTN, oTL B.
Update Drew to minimize Lp in Sec. 5.4
end for
for ka = 1 to Rappdate,0 do
Sample minibatch of transitions (s, a, 8') and corresponding observations Compute reward f in Sec. 5.4
Update the policy ny using SAC.
end for
end for
20

---

C Third-Person Imitation Learning (TPIL)
This section summarises TPIL one the pioneering works that address the domain shift problem in IL. TPIL trains a model based on an unsupervised domain adaptation technique [15] with GAIL [8]. TPIL consists of single behavior encoder BE, domain discriminator DD, and behavior discriminator BD, as shown in Fig. [2] The input label X B in Fig. [22 means that the input is the domain with behavior Bx X can be source S or target T, and 8x can expert or non-expert N. The key idea is to train BE to extract domain-independees inputs. The trained BE can be used to tell whether the learner's action the target domain expert behavior or non-expert behavior, and predict the domain label (source or target) of the imput, and B learns to predict the behavior label (expert non-expert) of the input. Therefore, encoder BE learns to fool DD by removing domain information from the input while helping BD by preserving behavior information from the input. In 377, the behavior feature is concatenation of (E)) and (E(o::4)) however, we just simply the notation and denote the behavior feature by BE():).
behavior features from
use this evaluation to train the learner. DD learns to
The total loss for TPIL LTPTI. is defined follows:
ETPIL Σ (C b,) (DCCC((D((((((()))).)
(29)
where o, an image observation, LoE is the cross-entropy loss, de is the domain label of o (1 for source domain and 0 for target domain), b; is the behavior label of (1 for expert behavior and for non-expert behavior), Xa is hyperparameter, and G is a Gradient Reversal Layer (GRL) [5]] is defined by
((x)
dg(x)/dr I
where A is hyperparameter. GRL enables updating BE, DD, and BD simultaneously using nack-propagation. The imitation reward r for an observation o generated by the learner policy is defined by the probability that BD predicts the observation to be generated by expert policy, as determined by r, (((o)().))
inputs
behavior encoder
behavior feature
predict domain
DD
SBs
BE
By By
BD
predict behavior
TB₁
behavioral info. (expert, non-expert)
domain infa. (source, target)
Figure 12: Basic structure for domain-indeppendet behavior feature extraction: BE behavior
encoder, BD behavior discriminator, -oomain discriminator
21

---

D Implementation Details
D.1 Network Architectures
In this section, we explain the network architectures of the components of the proposed method. Thole the layers for encoders, generators, and discriminators.
Each behavior encoder (BEg or BEr) consists of 6 convolutional layers and a flattened layer. The number of output channels for each convolutional layer is (16, 16, 32, 32, 64, 64), and each channel has 3 x 3 size. Zero padding is applied before each convolutional layer, and ReLU activation is applied after each convolutional layer except the last layer. The flattening layer reshapes the input to a one-dimensional vector. The shape of the input is (N, W, H. 4C), where N is the minibatch size, W and H are the width and height of each image frame, and C is the number of channels (for RGB images, C = 3). The shape of the behavior feature vector is (N, (W/4) x (H/4) x 64).
Each domain encoder (DEs, DET) consists of 6 convolutional layers, flattened layer, and linear layer. The number of output channels for each convolutional layer is (16. 16. 32, 32, 64, 64), and each channel has 3 3 size. Zero padding is applied before each convolutional layer, and ReLU activation is applied after each convolutional layer. The shape of the input is (N, W, H. C), and the shape of the domain feature vector is ().
Each generator (Gs or GT) consists of 7 transposed convolutional layers. The number of output channels for each layer is (64, 64, 32, 32, 16, 16, 4C), and each channel has 3 x 3 size. Zero padding is applied before each layer, and ReLUU activation is applied after each layer except the last layer. The input is behavior feature vector and domain feature vector. The shape of the output is (N, W, H, 4C', which is the same as the shape of the input of behavior encoders.
Each feature discriminator (BDg, DDB, BDD. or DDD) consists of 2 fully connected layers and a linear layer. Each fully connected layer has 32 hidden units with ReLU activation. Each image discriminator (IDs or IDr) consists of 6 convolutional layers, a flattened layer, and a linear layer. Each convolutional layer has 3 x channel size. The number of output channels for each layer is (16, 16, 32, 32, 64, 64). Zero padding is applied before each layer, and ReLU activation is applied after each layer except on the last layer. The discriminator D for reward estimation consists of fully connected layers and a linear layer. Each fully connected layer has 100 hidden units with ReI.U activation. The input size is (N, (W/4) x (H/4) x 64).
For the SAC algorithm, each actor and the critic consist of 2 fully connected layers and linear layer. Each fully connected layer has 256 hidden units with ReLU activation.
D.2 Details for Training Process
Before training the model, we collected expert demonstrations OSE, and non-expert datasets OSN. OrN. OSE is obtained by the expert policy TE. which is trained for 1 million timesteps in the source domain using SAC. OSN is obtained by a policy taking uniformly random actions in the source domain, and OTN is obtained by a policy taking uniformly random actions in the target domain. For each IL task, the number of observations (i.e., the number of timesteps) for Osg, OSN, OT dedemo = 10000 except for BallCheetah-to-looked--eee
task, where Tedemo = 20000 for this task. Each observation consists of 4 RGB images, but the size of these images varies depending on the specific IL task. For IL tasks including IP, IDP, CartPole, and Pendulum, the image size is 32x32. For IL tasks including RE2 and RE3, the image size is 48x48. For IL tasks including HalfCheetah and UMaze, the image size 64x64. Note that the proposed method does not require a specific input image size.
Our proposed method has two training phases. The first phase updates domain encoders, behavior encoders, generators, feature discriminators, and image discriminators. The second phase updates the discriminator D for reward estimation and the policy лe. In the first phase, we trained the model for Repooh_it = 50000 epochs for all IL tasks except for HalfCheetah--o-lockeee
task, where Nepoch_it = 200000 for this task. In each epoch, we sampled minibatch of size from each dataset OSE. OSN, OTN, OrL. We set OT as copy of OrN. For coefficients in Eqs. (25) and (26). we set 0.01, feat = 0.1, Aady = 1, Xeim feat" 1000, Xrecon Xing
Xpreed = X feat
cyele 100000.
We used A cycle feat
= 10 for IP-to-color, IDP-to-one, IP-to-two, and PointUMaae-to--an
cyele = 100 feat
22

---

Table 4: Layers for networks in the proposed model. Conv(ne, act) denotes 2D convolutional layer with the number of output channels (nc), stride (st), and activation (act). Conv(nc, st, act) denotes 2D transposed convolutional layer with the number of output channels (nc), stride (st), and activation (act). FC(nh, act) denotes fully connected layer with the number of units (nh) and activation (act). Flatten denotes function that reshapes the input one-dimensional vector.
BEs, BET
Conv(16, 1, ReLU)
DEs, DEr
Conv(16, 1, ReLU)
Conv(16, 1, ReLU)
Conv(16, 1, ReLU)
Conv(32, 2, ReLU)
Conv(32, 2, ReLU)
Conv(32, I, ReLU)
Conv(32, 1, ReLU)
Conv(64, 2, ReLU)
Conv(64, 2, ReLU)
Conv(64, 1, Linear)
Conv(64, 1, ReLU)
Flatten
Flatten FC(8, Linear)
BDB, DDB
FC(32, ReLU) FC(32, ReLU)
BDD. DDD
FC(32, ReLU)
FC(32, ReLU)
Linear)
FC(I, Linear)
Gs. GT
Conv Transpose ReLU)
1Ds. IDT
Conv(16, 1, ReLU)
ConyTranspose 1, ReLU)
Conv(16, 1, ReLU)
ConTTranspose(32,, ReLU)
Conv(32, 2, ReLU)
ConyTranspose 1, ReLU)
Conv(32, 1, ReLU)
16, ReLU)
Conv(64, 2, ReLU)
Conv Transpose(16,, (16, 1, ReLU)
Conv(64, I, ReLU)
ConYTpssppS Linear)
Flatten
FC(I, Linear)
D
FC(100, ReLU)
FC(100, ReLU)
FC(1, Linear)
for IDP-to-color, feat 100 for RE3-to-tilted, and X Spucee 10000 for RE2-to-tilted, RE3-to-two, RE2-to-three, and HC-to-LF. We used X Xcyele fear 100 for all environments from DeepMind Control Suite. We chose these coefficients so that the scale of each loss component is balanced, which was observed to be enough to yield good performances of πT.
As mentioned in Appendix A.4 we adjusted the values of Caorm,d dt and Guorm,h in Eq. 23 of the proposed method for the experiment. For II. tasks including IP, IDP, RE2, RE3, and HalfCheetah, we set Cnorm.d = Cnorm.b = O. For II. tasks including CartPole and Pendulum, we set Grorm,d = 1 and C = 20. For UMaze task, we set Crorm,d = 1 and Cnorm,b = 40. The ratio between Goorm,d . and Crorm,b b is determined based on the input image size since the proposed method does not require a specific input image size and produces behavior features varying sizes depending on the input image size.
In the second phase, we trained for reward estimation and Ty for Repock_pol epochs, where Nepochh pol = 20 for IP-to-color, IDP-to-one tasks, Pepooh pol = 30 for CartPole and Pendulum tasks Mind Control Suite, and Nepochh_pol = 100 for other tasks. Each epoch consists of 10000 timesteps. In each epoch, the following process is repeated. We sampled a trajectory from wo, and stored state transitions (s, a, s') and observations o to the replay buffer B. The maximum size of B Teuffer 100000, which is much smaller compared to that in off- RL algorithms because of the large memory consumption when storing images B. Then, D is updated for Nupdote, times. We Nuppdate. D to be 50 times smaller than the episode length for each IL task. For example, if the episode length is 1000, then Nupdatte,D = 20. For every D update, we sampled a minibatch of size 128 from each dataset Osg and OrN. For Or L. sampled minibatch of size 64 from B and minibatch of size 64 from OrN. After D update, Ty is updated for Nupdate, e times using SAC [17). We set u to be equal to the episode length for each IL task. For example, if the episode length is 1000, then Nupdaate,0 = 1000. For every To update, we sampled minibatch of size 256 from B.
In each epoch, we evaluated To after updating D and We sampled Nevel trajectories using and computed the average return, where each return is computed based on the true reward in the target domain. We set Nevel = 10 default. For IL tasks including Reacher environments, we set
23

---

Reval 200 to reduce variance because each return highly depends on the goal position in Reacher environments. We used Adam optimizer [23] for optimizing all networks. We the learning rate Ir 0.001 and momentum parameters 31 0.9 and B2 = 0.999 for encoders, generators, and discriminators. We set ir = 0.0003, 3 = 0.9 and |32 = 0.999 for the actor and critic. For SAC, we set the discount factor 0.99, the parameter the entropy coefficient a 0.1.
0.995 for Polyak averaging the Q-network, and
The proposed method is implemented on TensorFlow 2.0 with CUDA 10.0, and we used two Intel Xeon CPUs and a TITAN Xp GPU as the main computing resources. One can use GeForce RTX 3090 GPU instead as the computing resource, which requires TensorFlow 2.5 and CUDA 11.4. Based on 32x32 images, The GPU memory consumption is about 2.5GB for the training feature extraction model and about 1.5GB during policy update. Note that these highly depend on the image size and the batch size. Our D3IL implemented based on the code provided by the authors of 66
D.3 Dataset and Training Process for PointUMaazeto-AAnt Task
For the UMaze environments, the number of observations for OSE; OSN or OTN is Ridemo = 10000. Each observation consists of 4 frames and each frame a 64x64 RGB image. During the training, the reward that the agent receives for each timestep is Ttotal TIL is the estimated reward given by D; Lreach_gool is if the robot reaches the goal and 0 otherwise; "yooa is the reward when the robot reaches the goal; and c > O is scaling coefficient. In the experiment, we chose e 100 and gggo 1, and these parameters are applied to all baselines. For the proposed method, the number of epochs, minibatch sizes, and network structures for the training feature extraction network and image reconstruction network are the same as those in Appendix [D.1 The policy is trained for Neppoch_ree poo 200 and each epoch consists of 10000 timesteps, so the total number timesteps for policy training is 2,000,000.
X (rIL + 1reach_godd X Tggoat., where
Also, we trained an agent using a vanilla RL method (SAC [7]) with access to true rewards, to show that it is difficult to obtain expert demonstrations in the target domain. For SAC, the agent receives true rewards. For true rewards, the agent receives penalty of -0,0001 for each timestep and receives 1000 when the robot reaches the goal, so the maximum true episodic return is 1000.
24

---

E RL Environment Settings and Sample Image Observations
We first describe the base RL tasks in the Gym.
Inverted pendulum (IP): The agent moves a cart to prevent the pole from falling. The episode length is 1000. Unlike the IP task in Gym [4] an episode does not terminate midway even if the pole falls. This makes the task much more challenging because each episode provides less useful observations solve the task.
Inverted double pendulum (IDP): This is harder version of the IP task where there are two poles on the cart.
Reacher-two (RE2): The agent moves two-link arm with one end fixed that the movable endpoint of the arm reaches the target point. The episode length is 50. Unlike the Reacher task in Gym, we narrowed the goal point candidates to 16 points, where the polar coordinate (r, ()) of the target point can be E {0.15, 0.2), and E {0, 7/4, ι 2 3 3T/4, 7x/4).
Reacher-three (RE3): This harder version of the RE2 task where the agent moves three-link armed robot.
Note that our IL tasks are more difficult than those considered in the experiments in the baseline algorithm references 66 37.. Our simulation setting has an episode length 1000 for the IP and IDP tasks instead of 50. The episode horizon is also component of an MDP, and the observation space of the MDP changes as the episode horizon changes. MDP with longer episode length can yield more various observations which are much different from the initial one, the observation gap between the source domain and the target domain increases. We meant this for a bigger domain gap.
If the goal position is fixed throughout the entire learning phase in RE tasks, designed in (6,37] distinguishing between expert and non-expert behavior solely depends on arm position. However, in our implementation, the goal position is randomly chosen from 16 positions for each episode. This makes distinguishing expert and non-expert behavior much more challenging because the same arm movement be considered expert non-expert behavior depending on the goal position. Therefore, extracting proper behavior features is much more difficult as the feature extraction model should catch the relationship between the arm movement and the goal position.
InvertedPendulum (IP)
InvertedDoubleeennuuue (IDP)
Time
Time
Reacher-two (RE2)
Reacher-three (RE3)
Time
Time
Figure 13: Example observations for IP, IDP, RE2, and RE3 environment
25

---

The following images are sample observations for IL tasks with changing robot's degree of freedom.
-ne-to-
IP-to-two
RE3-to-two
RE22-t- RE:-tt-ttree
Source
Target
Source
Target
Source
Target
Source
Target
Figure 14: Example image pairs for IL tasks with changing DOF
In the HalfCheetah (HC) environment, the agent controls an animal-like robot. The goal is to move the animal-like 2D robot forward as fast possible by controlling 6 joints. Similarly to [6], have a modification with immobilized feet, and this modified environment is HHalfChettb--Loeeed Feet (HC-LF)'. In the HC-LF environment, the robot can use only 4 joints instead of 6, where the immobilized feet are colored red.
HalfCheetah (movable feet)
HalfCheetah (locked feet)
FT
Time
Figure 15: Example observations of HalfCheetah environment
The following images are sample image observations of environments in the DeepMind Control Suite (DMCS). use CartPole-Balame CartPole-S UppnggU and Pendulum environments from DMCS. The goal for all environments is to keep the pole upright.
CartPol--aalon Balance:: The pole starts in an upright position and the agent should move the cart keep the pole upright while avoiding exceeding the boundaries along the x-axis.
CartPole-S CartPole-Swingup: The pole starts in downward position and the agent should move the cart to swing the pole and keep the pole in an upright position while avoiding exceeding the boundaries along the x-axis.
Pendulum: The agent should add torque on the center to swing the pole and keep the pole upright.
Domain adaptation between these tasks is challenging because not only the embodiment between agents both domains are different but also the distributions of expert and non-expert demonstrations, and initial observations for both domains are quite different.
CartPoleBalance
CartPoleSwingUp
Pendulum
Tim
Time
Time
Thme
Figure 16: Example observations of environments in DMCS
We omit sample image observations of PointUMaze and AntUMaze environments because they are already provided in the main paper. Table S shows the dimensions of state space, action space, and visual observation for RL tasks used in the Experiment section.
26

---

Table 5: State dimension, action dimension, and visual observation image size for RL tasks used in the Experiments section. Each visual observation input consists of RGB images, and the rightmost column shows the size (width height) of each image.
State dimension
Action dimension
Visual observation
RL Task
image size
Inverted Pendulum (IP)
4
32x32
Inverted Double Pendulum (IDP)
11
32x32
Reacher-two (RE2)
11
48x48
Reacher-three (RE3)
14
48x48
HalfCheetah (HC)
17
64x64
(HC-LF)
13
64x64
Cartpole
32x32
Pendulum
3
32x32
PointUMaze
64x64
AntUMaze
30
64x64
27

---

Additional Experimental Results
F.1 Results on IL Tasks with Changing Visual Effect
We evaluated D3IL on tasks where the domain difference is caused by visual effects on image observations four IL tasks: IP and IDP, with different color combinations of the pole and cart (IP-to-colored and IDP-to-colored tasks), and RE2 and RE3, with different camera angles (RE2-to- tilted and RE3-to-tilter tasks). Sample image observations are provided in Figure 17 For the IP and IDP tasks, the primary changes occur in the pixel values of the pole and the cart. Meanwhile, for the RE2 and RE3 tasks, major changes are observed in the pixel values of the background outside of the arms and the goal position. The average episodic return for D3IL and the baselines over 5 seeds are shown in Fig. 18 We only included baselines that receive demonstrations as images because changing visual effects on image observations do not affect the true state space and the action space. As shown in Fig. [8] D3IL outperformed the baselines with large margins on IL tasks with changing visual effects image observations.
IP-to-color
IDP-to-color
RE2 to tilted
RE3 to tilted
Source
Target
Source
Target
Souree
Target
Source
Target
Figure 17: Example image pairs for II. tasks with changing visual effects
TPIL
DisentanGAIL
DBIL (ours)
Target domain expert
Pear www.ggoooeeeeee..
3030
403
4030
285
(a) IP-to-color
(b) IDP-to-color
(c) RE2-to-tilted
(d) RE3-to-tilted
Figure 18: The learning curves in the target domain of the IL tasks with visual effects
28

---

F.2 The Evaluation on the Feature Quality
Fig. 19 shows some instances of the estimated rewards from sampled observations in the RE2-to- three task, corresponding to the time of I 150,000 timesteps in Fig. [5d] The left column of Fig. 19 is non-expert behavior and the right column is expert behavior. One side of the Reacher arm is fixed at
the center of the image. In the left column, the movable arm end moves away from the goal position through four time-consecutive images, whereas the movable arm end moves towards the goal position in the right column. Note that in these instances, the baseline algorithm assigns higher reward to the non-expert behavior, whereas the proposed method correctly assigns a higher reward to the expert behavior. We conjecture that this wrong reward estimation of the baseline algorithms based on poor features the cause the gradual performance degradation of the baseline algorithms in Fig. 5
In Fig. 20 we checked the quality of extracted features from conventional method, TPIL, and that of our method. Since TPIL does not consider image reconstruction but considers only the extraction of domain-independen behavior features. In order to generate an image from the domain domain-independeee independent behavior feature of TPH.., along with TPII. we also trained two generators (Gs, Gr): Gs takes behavior feature vector as input and generates image that resembles the images from the source domain. Gr takes a behavior feature vector as input and generates an image that resembles the images from the target domain. This G5 and GT training is done with two discriminators (IDs and ID)), where 1Ds distinguishes the real and generated images in the source domain, and DD does so in the target domain. As shown in Fig. 20 the reconstructed image from the behavior feature of TPIL is severely broken for the RE2-to RE2-to-tilted task. Thus, it shows that the "domain- "oomain-nndeeendent" behavior feature of TPIL does not preserve all necessary information of the original observation image. On the other hand, our method based on dual feature extraction and image reconstruction, yields image reconstruction looking like ones. This implies that the features extracted from method well preserve the necessary information, and thus reconstruction is well done.
Non-expert behavior
Expert behavior
)
D3IL.
0.520
J
TPIL
11.35
1-.93
DisentanGAIL
2.284
18581
Figure 19: RE2-tO-three: Instances of estimated rewards. Four time-consecutive images comprise one observation. The estimated reward is shown the fourth image.
29

---

Conventional IL (TPIL)
DJLL
Lask
IP It-to-two
IP-to--tw
Target
Target
Target
larget
True observations
Imaged generated different domain
Figure 20: Examples of real and generated images. The generated images generated from the feature vectors obtained by conventional TPIL and our proposed method, Here we used IP-to-two and RE2-to-three task, where each task is explained in Sec. 6.1
30

---

F.3 Experimental Results on Simpler Tasks
In the main result subsection of Sec. we mentioned that the baseline algorithms perform well in the case of easier settings. As a preliminary experiment, we verify the performance of the proposed method and the bascline IL algorithms on simpler IL tasks, which are easier variants of the IL tasks We first describe the base RI. tasks.
shown in Sections 6.2 and 6.3
Inverted pendulum short (IPshort): This is the same task as the task, but the episode length is 50.
Inverted double pendulum short (IDPshort): This is the same task as the IDP task, but the episode length is 50.
Reacher-two one goal (RE2onegoal); This is the same task as the RE2 task, but the goal point is fixed to (z, 3) (1.0, 1.0) for the entire training.
Reacher-three one goal (RE3onegoal); This is the same task as the RE3 task, but the goal point is fixed to (z, 3) (1.0, 1.0) for the entire training.
Sample visual observations for each II. task provided in Fig. 21 We followed the training process as described [6. For each II. task, we collected OSE. OSN and OrN of size 10000. The learner policy T is trained for 20 epochs. Each epoch has 1000 timesteps, the total number of timesteps is 20000. The size of the replay buffer is 10000. We used TPIL [37] and DisentanGAlL. [6] as baseline algorithms. For each IL task, trained the learner policy using our proposed method and baseline algorithms over 5 seeds. For evaluation, we generated trajectories from for every epoch and computed the average return, where the return was computed based on the true reward in the target domain.
Fig. shows the average episodic return. As shown in Fig. 2 the baseline algorithms perform well in these easy II. tasks. The II tasks shown in Fig. 21 are much easier to solve than the II. tasks in Sections 6.2 and 6.3 because the pole needs be straight up for only 50 timesteps instead of 1000 timesteps in IPshort and IDPshort, and the goal position is fixed to single point instead of randomly sampled from 16 candidates in RE2ongoal and RE3onegoal. The baselines can achieve good performance on these easy IL tasks. However, as shown in Fig. of the main paper and Fig. [8] Appendix 6.22 the baseline algorithm yields poor performance on hard IL tasks and the proposed method achieves superior performance on hard IL tasks to the baselines due to the high-quality feature extraction by the dual feature extraction and image reconstruction.
31

---

IPshort-to-color
IDPshort-to-color
RH2oneegoa-to-tiledd
RESonegoal-to-tilte
Source
Target
Source Target
Source
Target
Source
Target
IDPshort-to-one
IPshort-to-two
RE3onegoal to-two
REZonegoal three
Source
Target
Source
Target
Source
Target
Source
Target
Figure 21: Example observation pairs for each IL task. For each pair of images, the left-hand side is the image in the source domain, and the right-hand side is the image in the target domain.
DisentanGAIL
(ours)
Target domain export
Tamtaggs
Ti
Seetnge
(a) [Pshort-to-color
IDPshort-to-color
(c) RE2onegoal to-tilted
RE3onegoal-to--tin
144
(c) IDPshort-to-one
(f) [Pshort-to-two
(g) RE3onegoal-to-twor
(h) REZonegoal-to-tle
Figure 22: The learning curves for the simpler tasks the target domain
32

---

F4 Experiments Results with Single Expert Trajectory during Policy Update
In this section, we investigated whether D3IL can successfully train the target learner when a single trajectory of expert demonstration is available during policy training. For results in Section 6 multiple trajectories expert demonstrations and non-expert data are provided for D3IL. TPIL. and DisentanGAll., where the details are provided in Appendix D Since GWIL calculates the Noomov-Wasserstiin distance between expert and learner policies, GWIL can leverage only single expert trajectory. This is also true for the implementation with the code provided by the authors of [9. In contrast, the size of the replay buffer f of GWII equals the number of training timesteps, which can vary from 200,000 to 2,000,000 depending the IL task. On the other hand, fer fixed to 100,000 for D3IL and baselines that utilize image observations. This is due to the huge memory consumption for storing image observations. In this experiment, we evaluated the performance D3IL under the condition of a single trajectory of expert demonstration during policy training. To provide trajectory of SE, and similarly, we supplied SN and TNN with the same size of SE. In Reacher environments, we collected one trajectory per goal position for each SE, SN, and TN, respectively. Fig. 23 shows the average return of D3II and baselines across seeds.
TPIL
DisentanGAIL
GWIL
D3IL (ours)
Targel domain expert
1000
Mean wpiiodd raturn
8000
600
2000
-12
166
Timesteps
166
(a) IDP-to-one
(h) IP-to-two
(c) RE3-oo-two
800
Mean wptpoptt INcure
800
600
Mean epteont
400
400
200
0.0
0.2
0.6
0.8
0.2
0.4
0.6
0.8
1.0
(d) RE2-to-three
(e) HalfCheetah-oo-Lookeellll .ggs
(f) CariPoleBalance-to
Near apicoiic wauu
Meur reture
500
600
400
400
200
200
-200
0
Timesteps
Timesteps
Timesteps
(g) CarPooeSwingup-tooPenndddd Penduuuuum
(h) Pendulum-to--CarPoleeee
Pendallm-to-CCrtooleSSwigg
Figure 23: The learning curves for the considered IL tasks when using single expert trajectory
33

---

F,5 Domain Transfer between CartPole-Balance and CartPole-SwingUp
In Section 6.5 we present the results of four IL tasks that use three different environments in DMCS: CartPole- Balance, HartPole-SwingUU and Pendulum. In these tasks, the robot's 's embodiment and dynamics of both domains are the same but their task configuration and initial state distribution of both domains are different. Fig. shows the results of the other two II. tasks where both domains use the CartPole environment. We evaluated D3II. and the baselines on II. tasks for seeds. As illustrated in Fig. 24 all methods failed to solve the CartPole-Swing task using demonstrations from the CartPole- Palance task. contrast, our methods were able to successfully solve the CartPole-Balanee task using demonstrations from the CartPole-SwingU) task. This difference in performance can be attributed to the variation in expert demonstrations between the two environments. Demonstrations of the CartPole-Swing
task typically show images in which the pole is upright, with only small number of images where the pole is initially pointing downwards and begins to swing up. As a result, an agent for the CartPole-Balance task learn that expert behavior involves keeping the pole upright and preventing from falling. In contrast, demonstrations of the BaaaaneBBalncce task typically only show images in which the pole is upright, which means that agent designed for the CartPole-Swing task may not have access to information on expert behavior for swinging up the pole from downward downward-pointing position. This problem more diverse images from the GartPole-Swing/ task. Our method can also be enhanced by utilizing images generated from the target learner train our feature extraction model.
be mitigated if non-expert data contains
TPIL
DisentanGAlL
GWIL
D3IL (ours)
Target domain expert
800
Mcan apisodic return
Mcan opisodic return
1000
600
750
400
500
200
250
2
0
Timesteps
Timesteps
(a) CartPole: Swwinn
(b) CartPole: SwingUp-to-Balame
Figure 24: The learning curves for the considered IL tasks for CarPPle--enduulmm and CartPole- Up environments
34

---

F.6 Results on Acrobot Environment
In this section, we evaluated D3IL on IL tasks where the target domain is the Acrobot environment in DMCS. In Acrobot, the agent should apply torque to the joint located between a two-link pole to make the pole stand upright. The dynamics of the Acrobot environment are much different from those of other environments like CartPole and Pendulum. Although the state and action dimensions of the Acrobot environment are low, achieving high scores this task non-trivial. In our implementation, the policy using vanilla RL with SAC, with 10 million timesteps, 256 hidden units for actor and critic networks, and batch size of 256, obtained the average return of 45.49 across 5 seeds. Figure [5 shows sample observations for the Acrobot environment, where each observation consists of four 32x32 RGB images with corresponding consecutive timesteps.
We evaluated D3IL and baselines on two IL tasks, where the source domain is either CartPole- SwingUp or Pendulum, and the target domain Acrobot. PartPole-SwingU) and Pendulum are known to be solvable by SAC. We examined whether D3IL can transfer expert knowledge from simpler tasks more challenging ones. Fig. 26 shows the average episodic return of D3II. and the baselines across 5 seeds. The results demonstrate that D3IL outperforms other baselines, and achieves the average return of vanilla RL using SAC with 10 million timesteps in much smaller timesteps. Our method has the potential to improve if additional information on Acrobot's ' dynamics is provided to the model. This can be accomplished by obtaining diverse non-expert data, such as observations from the policy with medium quality or observations from the target learner policy, to train the feature extraction model.
Acrobot
Time
Figure 25: Sample observations Acrobot task.
(eurs)
Target (10 timesteps)
80
100
Moan spisodic return
Mcan opisodic retu
80
60
40
20
Timesteps
le6
Timesteps
1e6
(a) CartPoleSwingU--to Acrobot
(b) Pendulum-to- Ac-oobbbttt
Figure 26: The learning curves for the considered IL tasks with Acrobot as the target domain task
35

---

F.7 Experiment on Walker, Cheetah, and Hopper Task
To assess the effectiveness of our proposed method more challenging scenarios, we tested our method on IL tasks with domain shifts caused by variations in robot morphology. We employed Walker, Cheetah, and Hopper environments in the DeepMind Control Suite. In these environments, the objective is to propel 2D robot forward as fast possible by manipulating its joints. The difficulty arises from the significant difference in the robot morphology and dynamics between the source and target domains. Sample visual observations for each II. task are provided in Fig. 27
For all environments, million timesteps in the source domain using SAC. The number of observations is fddemo 1000 (5 trajectories and each trajectory has length of 200). Each observation consists of 4 frames and each frame is 64x64 RGB image.
set each episode length
200. The expert policy g is trained for
We evaluated our proposed method and baseline algorithm in three tasks: Walker-to Ch-hetthh, Walker- to-Hopper, and Cheetah-to- Bortahhtt-Hoooo to-Hopper (denoted as 'A-to-B' where 'A' is the source domain and 'B' is the target domain). camera generates image observations by observing the robot's movement. Rewards were assigned to the agent based on the robot's horizontal velocity, with higher velocity leading to greater rewards. Fig. 28 depiets the learning curve of method and baselines three tasks, where each curve shows the average horizontal velocity of the robot in the target domain over three runs. These results demonstrate the superiority of our method over the baselines, which leads us to anticipate that our proposed method can extend to more complicated domains.
(a) Walker
(b) Cheetah
(c) Hopper
Figure 27: Sample observations for Walker, Cheetah, and Hopper task.
TPIL
DisentanGAIL
GWIL
D3IL (ours)
Avergoo Voleity along X330
3.25
3.00
fewrage Val.n.ly 400ng
0.50
0.25
Avcrage
(a) Walker-to-Cheetah
(b) Walker-to-Hopper
(c) Choctah-to- to-hhpp
Figure 28: Results on Walker, Cheetah, and Hopper task
36

---

F.8 Visualizing Sample Trajectories for Trained Learner
In this section, we provide visualized sample trajectories of demonstration sequences in the source domain and sequences generated by the learned policy trained in the target domain for qualitative analysis on two tasks: RE2-to-three and PointUMaze-to- the source domain expert and target domain learner on the RE2-to-three task. The top row of Fig. 29 shows a sample trajectory of image observations sampled by a in the source domain using SAC for 1 million timesteps. The bottom row of Fig. 29 shows sample trajectory of image observations sampled by a trained using D3IL for million timesteps the target domain.
Fig. 29 shows sample trajectories from
Source domain: RE2 Expert demonstration
Target domain: RE3 Policy trained for 1M timesteps
Figure 29; (Top) An example trajectory of image observations generated by mh in the source domain. (Bottom) An example trajectory of image observations generated by ma trained using D3IL for million timesteps in the target domain.
tested the performance of D3IL the PointUMaze-to-Ant task. Fig. 30 shows sample trajectories from the source domain expert and target domain learner. The top row of Fig. 30 shows sample trajectory of image observations sampled by Te in the source domain using SAC for 0.1 million timesteps. The bottom row of Fig. 30 shows a sample trajectory of image observations sampled my trained using D3IL for million timesteps in the target domain. The episode lengths for the trained learner vary when running multiple evaluation trajectories, most of which have episode lengths in the range of 60 to 80. This implies that D3IL train the agent in the target domain with its dynamics by observing expert demonstrations in the source domain with a different embodiment.
In Section 6.6
Source domain: Point robot Expert demonstration (episode length 18)
Target domain: Ant robot
Learner trained for 2M timesteps (episode length-69)
Figure 30: (Top) An example trajectory of image observations generated by Th in the source domain. In this figure, the interval between two consecutive image frames timestep, and the total episode length is 18. (Bottom) An example trajectory of image observations generated by me trained using target domain. this figure, the interval between two consecutive image frames is 4 timesteps, and the total trajectory length is 69.
D3IL for million timesteps
37

---

G Further Discussions
Explanation of the performance gap with GWIL GWIL is state-based method, but having access to expert states does not necessarily ensure superior performance. In the original GWIL paper [9] the authors provided simulation results on Cendulum-o--Cartpoleevvii
(Fig. in (9), showing GWIL achieves 600 points. However, Fig. 7 in [9] assumes that the target-domain learner receives the target-domain sparse reward directly in addition to the IL reward. So, the GWIL result in our paper can be different from that in the GWIL paper. Despite our efforts to tune GWIL's hyperparameters, GWIL did not show high performance only with the IL reward without direct larget-domain sparse reward. Our approach consistently shows better performance in almost all tasks presented in this paper. As the GWIL paper indicates, GWIL can recover the optimal policy only up to isometric transformations. Given that expert demonstrations only cover fraction of the optimal policy's scope, recovering the exact optimal policy becomes more difficult. Additionally, GWIL uses Euclidean distance as metric within each space to compute confines the method to scenarios limited to rigid transformations between domains.
W distance, which
Broader Impacts The proposed learning model does not raise significant ethical concerns. However, as an II algorithm, should be applied to train models that learn tasks beneficial society.
38


Reviews:
[{"review_id": "2flyRV7qyW", "reviewer": "Reviewer_33qr", "summary": "This work introduces a method of learning a visual behavior encoding for reinforcement learning tasks, D3IL.\n\nFrom a TPIL behavior feature encoder baseline, D3IL introduces dual feature extraction (including both behavior and domain feature encoders) and dual cycle-consistency (cycle consistency on image and feature levels). From inputs of unpaired target and source images, D3IL translates across domains and then cycles back to the original domain, via a pair of generator networks which take in source images and domain targets. Further, image reconstruction consistency, feature vector similarity, and feature vector consistency is added for further loss and stability.\n\nThe domain and behavior feature encoders and target generator from above are taken and used to train a reward generating discriminator, $D_rew$, which distinguishes between expert and learner behavior (rather than expert and non-expert data used for training the previous models). This is used alongside the target behavior encoder to train SAC RL policies for the given tasks.\n\nD3IL is tested on many domain transfer tasks with variations in visual effects, degree-of-freedom, internal dynamics, and robot embodiment - D3IL demonstrates superior performance on the target domain compared to baselines of TPIL, …. It is further demonstrated on an AntUMaze task where direct RL learning in the target domain is difficult, but possible with D3IL. Ablation studies show that the contributions of each of the consistency losses across feature and images, and cycle consistency, are all important and contribute to the final performance.\n", "questions": "Some discussion on the weakness-quality section would be most helpful\n", "limitations": "Some consideration of the limitations is explained in the weaknesses-significance section.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "#### Originality\nThe work builds on top of TPIL to build a larger framework for cross-domain imitation / feature encoding, by incorporating dual feature extraction and cycle consistency, and training a further reward-generating discriminator.\n\n#### Quality\nThe experiments are detailed, covering four categories of domain change and with ablation studies on components of the loss. The experiments are run over multiple seeds and the method is compared against several baselines (TPIL, DisetanGAIL, GWIL) to present a good picture of the overall learning and domain transfer capability. There are further many more detailed experiments in the appendix to which present good coverage.\n\n#### Clarity\nThe work is generally well-written and the description of D3IL which draws from premises of prior TPIL work is understandable. The explanations of losses and model composition are detailed and the diagrams help with understanding the proposed methodology. The experiments are easy to follow. Further task explanation and experimental methodology is well explained in the appendix.\n\n\n#### Significance\nThe method seems broadly applicable across tasks and domain transfer goals in simulation.\n\nThe experiment on AntUMaze is particularly motivating as towards the practical applications of the method, demonstrating that a task which could not be solved directly could be improved by training from an imitation agent learned on a simpler task - in a sort of curriculum learning.\n", "weaknesses": "#### Quality\nThe dual feature extraction, in particular the domain encoder added from TPIL, is motivated on lines 135-136 as to “check that the obtained feature well contains the required information without information loss, based on image reconstruction and consistency check.” However, it’s still unclear why specifically it is necessary to pass observations as input to domain encoder, as the work assumes that the policy knows what environment it is operating in. Would an ideal domain encoder be able to output a one-hot vector indicating its domain to minimize all the domain-encoder relevant losses? As there are separate generators for each domain (source/target), should it be clear without any domain conditioning what the “style” of each generator output should be? For example, in CycleGAN and its successors, the domain encoding is not required.\n\nAn ablation study without using a domain encoder, but a static method of encoding the domain or skipping the domain encoding entirely may be helpful.\n\n#### Clarity\nUnfortunately many of the experimental results are left to to the appendix including all of Sections 6.2 and 6.5, and reading the appendix seems necessary to have a full understanding of the model performance.\n\nThe method is somewhat hard to initially grasp from reading the text of Section 5 as there are many similar variables in the notation which differ by one letter or modifier, with many such modifiers. However, explanations are generally helped by Figures 1 and 3.\n\nThe limitations are also only mentioned in the appendix.\n\nMisc:\n- Line 191: is another form of consistency [that] guide[s] the model so that …\n- Line 195: scheme significantly improve[s]\n- Line 211: we need [the] expert behavior feature\n\n####  Significance\nThe practical applications of the method seem best suited for cases where it is hard to train an expert in the target environment directly via RL, but having access to expert data on other tasks as observation-only would help. Many of these cases would be covered by real world scenarios, which is out-of-scope of the current work. Even on the AntUMaze problem, the learning process did not necessarily have to be done via adversarial learning and visual-only observations, as the full expert state would be available.\n\nFrom the ablation in Table 1, it appears that the large majority of the performance improvement comes from image-level consistency and the feature-level consistency was only of minor gain, especially compared to the uncertainty/error.\n\nAs mentioned in the limitations in the appendix, the paper also introduces quite the complex process with significantly many more loss terms and two pipeline stages compared to prior work, which may make it harder to apply. Adversarial learning techniques generally require more careful tuning than supervised methods, and this work introduces two adversarially-learned stages.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.\n\n**1. Answer to Quality section**\n\nWe claim that learned domain encoders are essential for enhancing the feature extraction and policy performance in the domain-adaptive IL problem with visual observations. To demonstrate this,  we conducted an experiment wherein we excluded domain encoders and their corresponding discriminators from our proposed model while keeping all other elements consistent. We then trained the policy in the target domain to evaluate its performance. The result is shown in Fig. 2 of the attached PDF of the Common Response. As depicted in the figures, our proposed model with domain encoders is superior to our model without them. To enhance domain-independent behavior feature extraction, we proposed employing both domain encoder and behavior encoder, ensuring that the resultant domain feature and behavior feature are mutually independent and contain complementary information about the input. This approach was motivated by the observation that solely relying on a behavior encoder was insufficient to eliminate domain-related information from the behavior feature.\n\n**2. Answer to Clarity section**\n\nWe made efforts to incorporate as much content as possible within the main body of the submission. Nevertheless, due to the extensive volume of the contents and the constraints of the main body, some details and results had to be left in the appendix. However, we ensure that the most crucial contents and results are present in the main paper. We will mention the discussion of limitations in the main body.\n\n**3. Experiment on more challenging domain mismatch**\n\nWe anticipate that domain-adaptive IL with visual observations will be helpful in real-world scenarios. As the reviewer mentioned, our  method was evaluated primarily on simulated environments. During this short rebuttal period, we could not apply our method to real-world scenarios, but to assess the applicability of the proposed method to more challenging situations, we conducted more experiments with more challenging domain mismatch conditions. For the results, please see  the Common Response and Fig. 1 of the attached PDF therein. These results lead us to anticipate that our proposed method has enough possibilities to extend to more complicated domains. While we presented experimental results on the simulated environments due to time constraints, we aim to address real-world scenarios in our future works.\n\n**4. Feature-level consistency**\n\nAs the reviewer  pointed out, the impact of image-level consistency on the performance of the learned policy appears to be more prominent than that of feature-level cycle-consistency. Nevertheless, it's important to emphasize that both image-level cycle-consistency and feature-level cycle-consistency are essential in extracting domain-independent behavior features. It is also worth noting that the policy achieves the best performance when all these elements are combined, and including feature-level cycle-consistency doesn't incur additional complexity to the model structure.\n\n**5. Complexity**\n\nAs discussed in the paper, our proposed framework involves a two-stage learning process that can increase complexity. Moreover, similarly to our proposed method, adversarial learning generally requires careful handling to ensure stability. However, the complexity of the proposed approach is much less than  that of widely-used image translation models because our model's network structure consists of only a few layers of convolutional neural networks and multi-layer perceptrons. Additionally, we observed that effective loss component balancing and discriminator regularization with gradient penalties are sufficient for achieving favorable results."}, {"review_id": "g9HsbvlAJY", "reviewer": "Reviewer_szho", "summary": "The paper attempts to solve the problem of domain shift in Imitation Learning with image observation. The author proposes D3IL architecture which is based on dual feature extraction (behavior and domain specific features from input) and dual cycle consistency (image level and feature level cycle consistency). The paper demonstrates effectiveness of the proposed method compared to baselines Third Person IL (TPIL), Disentan GAIL over different Gym based tasks (e.x. Reacher, Pendulum etc.) and  DeepMind Control Suite (DMCS) tasks (Maze based tasks). ", "questions": "- Have you tried interchanging source and target domains?\n- The relations to source -> expert and target -> learner is a little confusing. Maybe if you can represent in a better way that would be really good.\n\n", "limitations": "- most of them are mentioned in the paper", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem addressed in an interesting problem for Imitation Learning.\n- Good thorough evaluation different types of experiments.\n- Strong experiments results.\n- The paper is well written.\n", "weaknesses": "- The problem is well motivated from real robotics based task, but no real robotics based task is demonstrated in the paper. It would be interesting to see the application on sim-to-real based tasks.\n- It would be good to discuss work similar TCN work [1].\n- The extensions to other IL methods like IRL is a little unclear.\n- Line 90-93 are a little unclear.\n\n[1] Sermanet, Pierre, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. \"Time-contrastive networks: Self-supervised learning from video.\" In 2018 IEEE international conference on robotics and automation (ICRA), pp. 1134-1141. IEEE, 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.\n\n**1. Experiment on more challenging domain mismatch**\n\n As the reviewer mentioned, our  method was evaluated primarily on simulated environments. During this short rebuttal period, we could not apply our method to real-world scenarios, but to assess the applicability of the proposed method to more challenging situations, we conducted more experiments with more challenging domain mismatch conditions. For the results, please see  the Common Response and Fig. 1 of the attached PDF therein. These results lead us to anticipate that our proposed method has enough possibilities to extend to more complicated domains including real-world scenarios.\n\n**2. Discussion of TCN**\n\nTime-Contrastive Networks (TCNs) learn representations from visual demonstrations of different viewpoints via time-contrastive approach. TCN makes embeddings from images of the same time but different viewpoint to be similar and makes embeddings from images of different times in the same viewpoint to be dissimilar. One of the key distinctions between TCN and our method is that while TCN relies on time-synchronized demonstrations with simultaneous viewpoints for model training and reward construction, our method does not necessitate any time-synchronization among demonstrations. Also, TCN mainly addresses viewpoint mismatch between domains, while our method encompasses general domain mismatch including the difference of viewpoint, degree-of-freedom, dynamics, and embodiment. We will include the discussion in the paper.\n\n**3. Discussion to the extensions to IRL**\n\nOur method involves an adversarial IL scheme designed for domain adaptive IL with visual observations. The feature extraction model learns to extract domain-independent behavior features from observations. Subsequently, the reward-generating discriminator $D_{rew}$ generates a reward by comparing the behavior feature of the policy to that of the expert demonstration.\nAs most adversarial IL methods do, our approach also learns the reward for policy training. Therefore, we anticipate extending our approach to the IRL framework in future works, involving making a reward function itself capable of transfer across diverse domains.\n\n**4. Lines 90-93**\n\nIn addition to expert demonstrations from the source domain, we assume access to the non-optimal trajectories of image observations from both the source and target domains. These non-optimal trajectories are generated by non-expert policies. The visual observations of rollouts from a non-expert policy $\\pi_{SN}$ in the source domain are denoted by $O_{SN}$, and the visual observations of rollouts from a non-expert policy $\\pi_{TN}$ in the target domain are denoted by $O_{TN}$. Here, 'SN' refers to 'source non-expert' and 'TN' refers to 'target non-expert'. We will clarify this part in the paper.\n\n**5. Interchanging source and target domains**\n\nAs shown in the paper, we included experimental results where source and target domains were interchanged. The examples are IP-IDP, RE2-RE3 on OpenAI Gym (Fig. 5 in the main paper) and CartPole-Pendulum on DeepMind Control Suite (Fig. 7 in the main paper).\n\n**6. Relations to source-expert and target-learner**\n\nIn this paper, the source domain is where we can obtain expert demonstrations, and the target domain is where we train the agent. The expert is the one who performs a task optimally. The agent in the target domain learns the task from the expert's demonstration  in the source domain. We call the agent or its policy a learner, since in the beginning it is not an expert for the target-domain task. We will clarify this part in the paper.\n"}, {"review_id": "XChyCGX63g", "reviewer": "Reviewer_3tvh", "summary": "Imitation learning from visual observations for domain adaptation. Given expert and non expert demonstrations in a source domain, and non expert demonstrations in a target domain, learn a reward and policy in the target domain via image consistency and domain and behavior feature consistency.", "questions": "In addition to the weaknesses above, \n\n- How do the target policies compare to the source policies? E.g. in RE2 to RE3, does the policy learn to use all 3 links or does it apply a strategy with 2 links? Would suggest providing videos for qualitative analysis. \n- In Figure 10(a), why is it desired that the source and target expert behavior features overlap? Shouldn't they require different features for different domains? (E.g. number of links.)\n- Is there an explanation for the performance gap with GWIL (in Figures 5-9)? A state-based method should have more information about the environment.\n- Please state the state/action space for every environment (for completeness and clarity).\n- Suggest defining formally what good/bad actions mean.\n- Line 22: suggest replacing 'should' with 'can'.\n- Typos in lines: 55, 130, 194, 195, 355.", "limitations": "Please add to Section G the assumption re access to the target environment for training $\\pi_{\\theta}$ and in what cases will D3IL fail to learn a successful target policy.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Domain adaptation in imitation learning remains an important problem. The paper tackles this challenging problem assuming access to visual observations but not to actions, other proxy tasks or expert demonstrations in the target environment. These assumptions allow for better generality. The problem statement, setup and method are clearly presented and the quality is high.", "weaknesses": "- The paper empirically demonstrates shifts in color, number of joints that can move, and action space. What are the general assumptions about the relation between the source and target domains? (There is some similarity, otherwise transfer would not be possible..)\n- The papers uses visual observations, which is a more general setup than states. However, the proposed domains are quite simple (very few agents/objects in each, very simple backgrounds, etc.). How is the method expected to scale to more complex environments?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.\n\n**1. Assumptions about the relation between the source and target domains**\n\nIt may be difficult to express the general assumption about the relation between domains clearly and mathematically. But, we think that the general assumption about the relation between domains will be that the source-domain and target-domain environments share a common objective, such as maintaining the pole upright position, reaching to the goal point, or running forward.\n\n**2. Experiment on more challenging domain mismatch**\n\nWe anticipate that domain adaptive IL with visual observations will be helpful for addressing real-world scenarios, which include more complex situations. Application to real-world scenarios would be more challenging and intriguing, and we aim to address real-world scenarios in our future works. To assess the applicability of the proposed method in more challenging situations, we provide experimental results of experiments conducted under more challenging domain mismatch conditions. Please refer to the Common Response and Fig. 1 of the attached PDF for detailed explanations.\n\n**3. Qualitative Analysis on RE2 to RE3**\n\nIn the target domain, the agent adapts to the robot’s unique structure and dynamics to perform tasks. For example, in the RE2 to RE3 task, the agent in the target domain learns to utilize all three links for effective task execution. As a qualitative analysis, we provide visual representations of a demonstration sequence in the source domain and a sequence generated by the learned policy trained in the target domain. Please see Common Response and Fig. 3 of the attached PDF for detailed explanations.\n\n**4. Why is it desired that the source and target expert behavior features overlap**\n\nIn this paper, our aim is for behavior features to contain task-related information while excluding domain-specific information. For example, in the transition from RE2 to RE3, the number of links could be seen as domain-specific components. On the other hand, the goal position and correlation between the arm movements and the goal position can be treated as behavior-related components. Our objective is to achieve overlap between the behavior features of source and target experts, leading to domain-independent behavior features.\n\n**5. Explanation of the performance gap with GWIL**\n\nGWIL is a state-based method, but  having access to expert states does not necessarily ensure superior performance. In the original GWIL paper, the authors provided simulation results on Pendulum-to-Cartpole (Fig. 7 of their paper), showing GWIL achieves 600 points. However, Fig. 7 of their paper assumes that the target-domain learner receives target-domain sparse reward directly in addition to the reward from imitation learning. So, our GWIL result in our paper can be different from that in the GWIL paper. Despite our efforts to tune  GWIL's  hyperparameters, GWIL did not show high performance only with the reward from imitation learning without direct target-domain sparse reward.  Our approach consistently shows better performance in almost all tasks presented in this paper. As the GWIL paper indicates, GWIL can recover the optimal policy only up to isometric transformations. Given that expert demonstrations only cover a fraction of the optimal policy's scope, recovering the exact optimal policy becomes more difficult. Additionally, GWIL uses Euclidean distance as a metric within each space to compute Gromov-Wasserstein distance, which confines the method to scenarios limited to rigid transformations between domains.\n\n**6. State/action space for every environment**\n\nHere are the state/action dimensions for every environment. We will include this in the paper to improve clarity. We also include the size of observation for each environment. For the detailed explanation of visual observations, please refer to Appendix D.2 and E.\n\n- IP (state 4-dim, action 1-dim, observation 32x32 RGB)\n- IDP (state 11-dim, action 1-dim, observation 32x32 RGB)\n- RE2 (state 11-dim, action 2-dim, observation 48x48 RGB)\n- RE3 (state 14-dim, action 3-dim, observation 48x48 RGB)\n- HC (state 17-dim, action 6-dim, observation 64x64 RGB)\n- HC-LF (state 13-dim, action 4-dim, observation 64x64 RGB)\n- DMCS Cartpole (state 5-dim, action 1-dim, observation 32x32 RGB)\n- DMCS Pendulum (state 3-dim, action 1-dim, observation 32x32 RGB)\n- DMCS Acrobat (state 6-dim, action 1-dim, observation 32x32 RGB)\n- PointUMaze (state 7-dim, action 2-dim, observation 64x64 RGB)\n- AntUMaze (state 30-dim, action 8-dim, observation 64x64 RGB)\n\n**7. Meaning of good and bad actions**\n\nFrom the reinforcement learning perspective, good actions refer to ones that yield a larger return. From the imitation learning perspective, good actions refer to ones that closely resemble the expert actions in a given state. We will clarify this part in the paper.\n\n**8. Comments on Section G**\n\nAs the reviewer pointed out, the proposed method requires interaction with the target environment for training. It is a characteristic, shared by several domain adaptive IL approaches. It will be interesting to extend our method to offline approach in future works. As mentioned in Section 2, some approaches do not require target-domain interactions during training, but those methods  rely on expert demonstrations for proxy tasks in the target domain.\n\nThe success of the proposed method is affected by the similarity of the objective in the source and target domains. It will be difficult for our approach to succeed if source and target domains aim for completely different tasks. Future works could also include the extensions of our method to multi-task or multi-modal situations. We will include this in the paper."}, {"review_id": "WWD17JEt9e", "reviewer": "Reviewer_V2Bv", "summary": "This paper aims to solve the domain-adaptive imitation learning problem, where the demonstrations are collected in a source domain, and the agent needs to learn policies in a target domain. Beyond that, the tasks are with high-dimensional visual observations. The authors have proposed a new framework based on dual feature extraction and image reconstruction to achieve behavior feature extraction and reward generation.", "questions": "The experiment section is weak. The proposed approach is barely evaluated in limited simple tasks. If the authors could show that the proposed approach can work in more challenging domains, such as humanoid running, the quality of this paper could be greatly improved.", "limitations": "This paper has not discussed the limitations in the main body of this submission.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper aims to solve a very practical problem, since in real-world problems, we can hardly collect expert demonstrations in the target learning environment. Instead, collecting expert demo in a similar source domain is much easier. ", "weaknesses": "The framework in Figure 3 is too much complex. The authors are encouraged to clarify their contributions in this paper and simplify the parts in the existing literature. \n\nPrevious works study the problem of domain adaptive imitation learning, and this work aims to solve the same problem with visual observations. The proposed approach augments the previous work with an encoder to deal with the high-dimensional visual observations. This idea seems a little straightforward.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.\n\n**1. Contribution of this paper**\n\nIn summary, our contribution is that we propose a novel feature extraction model in the domain-adaptive IL with visual observations. We significantly improved the quality of behavior feature extraction by effectively eliminating domain-specific information from the behavior features. The salient components include dual feature extraction, dual cycle-consistency, and an externally located reward-generating discriminator for policy training. We will make this part more clear in the paper.\n\nWe would like to emphasize that our approach is beyond  a simple combination of existing domain-adaptive IL with encoders. Our approach merges the merits of prior IL and image translation approaches uniquely. Our proposed method enhances behavior feature extraction with domain-specific information elimination and facilitates the transfer of expert knowledge from the source domain to the target domain. Furthermore, our method outperforms existing approaches across various scenarios. We demonstrated the efficacy of our work in qualitative and quantitative evaluations, including learning curves, ablation studies, and visualization of behavior features via t-SNE.\n\n**2. Experiment on more challenging domain mismatch**\n\nAddressing domain adaptive IL with visual observations remains a remarkable challenge. Learning from visual demonstrations is not an easy problem, especially when the adaptation between the source and target domain is required. \nOur method achieves superior performances to baselines across various types of domain mismatches. It is also important to note that our proposed method tackles more intricate situations than previous approaches. Sections E and F.3 explain the distinction of the setup for IP and RE tasks. Moreover, we examined our proposed method on IL tasks with a more challenging scenario, and the results strengthened the ability of scalability of our approach to more complex domains. Please refer to Common Response and Fig. 1 of the attached PDF."}]

Author rebuttal:
Thank you for your valuable feedback. In the Common Response, we address inquiries from multiple reviewers and provide responses requiring images and plots. Please refer to the attached PDF for detailed explanations.

**Experiment on more challenging domain mismatch**

To assess the effectiveness of our proposed method in more challenging scenarios, we tested our method on IL tasks with domain shifts caused by variations in robot morphology. We employed Walker, Cheetah, and Hopper environments in the DeepMind Control Suite. In these environments, the objective is to propel a 2D robot forward as fast as possible by manipulating its joints. The difficulty arises from the significant difference in the robot morphology and dynamics between the source and target domains.

We evaluated our proposed method and baseline algorithm in three tasks: Walker-to-Cheetah, Walker-to-Hopper, and Cheetah-to-Hopper (denoted as 'A-to-B' where 'A' is the source domain and 'B' is the target domain). A camera generates image observations by observing the robot's  movement. Rewards were assigned to the agent based on the robot's horizontal velocity, with higher velocity leading to greater rewards. Fig. 1 of the attached PDF depicts the learning curve of our method and baselines on three tasks, where each curve shows the average horizontal velocity of the robot in the target domain over three runs. These results demonstrate the superiority of our method over the baselines. The results in the paper and the attached PDF lead us to anticipate that our proposed method can extend to more complicated domains. While we presented experimental results on the simulated environments due to time constraints, we aim to encompass real-world scenarios in future works.

As highlighted in the paper, domain adaptive IL with visual observations is still a challenging problem. Each task considered  in the Experiments section exhibits a distinct domain mismatch, and our proposed method significantly enhanced performances on various tasks without necessitating time-aligned data or proxy task demonstrations. Consequently, our work plays a substantial role in advancing the field of domain adaptive IL with visual observations.

**Ablation study on domain encoders (Reviewer 33qr)**

During the rebuttal period, we conducted an experiment wherein we trained our proposed feature extraction model of D3IL without involving domain encoders ($DE_S$ and $DE_T$) and their corresponding discriminators ($DD_D$ and $BD_D$). All other networks and losses remain unchanged. We then trained the policy in the target domain to assess its performance. We evaluate our method with and without domain encoders in four tasks: IP-to-two, RE2-to-three, RE3-to-tilted, and CartPoleBalance-to-Pendulum. Fig. 2 of the attached PDF shows the results. The results demonstrate the  benefit of conditioning generators on learned domain encoders to boost model performance. To enhance domain-independent behavior feature extraction, we proposed employing both domain encoder and behavior encoder, ensuring that the resultant domain feature and behavior feature are mutually independent and contain complementary information about the input. This approach was motivated by the observation that solely relying on a behavior encoder was insufficient to eliminate domain-related information from the behavior feature.

**Qualitative Analysis on RE2 to RE3 (Reviewer 3tvh)**

In the target domain, the agent adapts to the robot's unique structure and dynamics to perform tasks. For example, in the RE2 to RE3 task, the agent in the target domain learns to utilize all three links for effective task execution. As a qualitative analysis, we provide visual representations of a demonstration sequence in the source domain and a sequence generated by the learned policy trained in the target domain. In Fig. 3 of the attached PDF, we present sample trajectories of image observations. The red mark in the figure indicates the goal position. The top row corresponds to a demonstration in the source domain (RE2), while the bottom row corresponds to the policy learned in the target domain (RE3) over 1 million timesteps. A similar analysis for the UMaze environment is available in Appendix F.7.

Meta-review: Imitation learning from visual observations is an important topic within the NeurIPS community, and the authors have provided strong empirical evidence that the proposed method represents an important new entry in this space. That said, the authors are strongly encouraged to make the modifications to the paper that arose during the author-reviewer discussion paper, especially one that makes it clear that the questions from reviewers regarding how to measure/quantify domain shift remain unanswered.
Decision: Accept (poster)