Title: Domain Adaptive Imitation Learning with Visual Observation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

Deep Reinforcement Learning

Imitation Learning
Abstract: In this paper, we consider domain-adaptive imitation learning with visual observation, where an agent in a target domain learns to perform a task by observing expert demonstrations in a source domain. Domain adaptive imitation learning arises in practical scenarios where a robot, receiving visual sensory data, needs to mimic movements by visually observing other robots from different angles or observing robots of different shapes. To overcome the domain shift in cross-domain imitation learning with visual observation, we propose a novel framework for extracting domain-independent behavioral features from input observations that can be used to train the learner, based on dual feature extraction and image reconstruction. Empirical results demonstrate that our approach outperforms previous algorithms for imitation learning from visual observation with domain shift.

Domain Adaptive Imitation Learning with Visual Observation

Abstract

In this paper, we consider domain-adaptive imitation learning with visual observation, where an agent in a target domain learns to perform a task by observing expert demonstrations in a source domain. Domain adaptive imitation learning arises in practical scenarios where a robot, receiving visual sensory data, needs to mimic movements by visually observing other robots from different angles or observing robots of different shapes. To overcome the domain shift in cross-domain imitation learning with visual observation, we propose a novel framework for extracting domain-independent behavioral features from input observations that can be used to train the learner, based on dual feature extraction and image reconstruction. Empirical results demonstrate that our approach outperforms previous algorithms for imitation learning from visual observation with domain shift.

1 Introduction

Imitation learning (IL) is a framework where an agent learns behavior by mimicking an expert's demonstration without access to true rewards. The mainstream IL methods include Behavior Cloning (BC), which trains the learner with supervised learning [3, 31, 33, 39], Inverse Reinforcement Learning (IRL), which tries to find the reward function [1, 29, 49], and Adversarial Imitation Learning (AIL), which trains the learner to match its state-action visitation distribution to that of the expert via adversarial learning [10, 12, 18, 40, 46]. IL provides an effective alternative to solving reinforcement learning (RL) problems because it circumvents the difficulty in the careful design of a reward function for intended behavior [30, 47]. Despite the effectiveness, conventional IL assumes that the expert and the learner lie in the same domain, but this is a limited assumption in the real world. For example, a self-driving car should learn real-world driving skills from demonstrations in a driving simulator. In such a case, the expert and the learner do not lie in the same domain.

In this paper, we consider the challenge of domain shift in IL, i.e., an agent in a target domain learns to perform a task based on an expert's demonstration in a source domain. In particular, we focus on the case where the expert demonstrations are provided in a visual form. In many practical cases, obtaining expert demonstrations as an explicit sequence of states (such as positions and velocities) and actions (such as movement angles) can be challenging. Instead, the expert's demonstrations are often provided as image sequences. One example can be a robot mimicking a movement by visually observing human behavior. However, learning from visual observations poses a major challenge in IL since images are high-dimensional and often include only partial information about the expert's behavior or information unrelated to the expert's behavior. Even minor changes between the source and target domains can vastly affect the agent's policy and lead to unstable learning. In recent years,

several works aimed to address the challenge of domain shift in IL. Some methods show impressive performance by training a model to learn a mapping between domains [22, 32], requiring proxy tasks (i.e., additional tasks to assist in learning the target task) to provide expert demonstrations in both source and target domains. However, this approach assumes expert demonstrations for similar tasks in the target domain, which may not be practical in many realistic domain transfers where expert demonstrations in the target domain are unavailable. Other methods generate metric-based rewards that minimize the distance between the expert and the learner trajectories [9, 27], assuming the expert's state is directly available rather than requiring image observations. Our focus is on solving domain shifts with visual observations without the use of proxy tasks or direct access to expert's states. Our work is closely related to the methods that learn a model to extract domain-invariant features [6, 37]. However, removing domain information was not satisfactory in situations with a large domain gap. In contrast, we propose a novel learning method to train the learner in domain-adaptive IL with visual observation, achieving significantly improved performance in IL tasks with domain shift.

2 Related Work

IL with domain shift IL with domain shift is a challenging problem where the learner should learn from an expert in a source domain and apply the learned policy in a target domain. In recent years, there has been significant progress in addressing this problem in machine learning, RL, and robotics [6, 7, 9, 11, 14, 22, 27, 28, 32, 34-37, 45]. For example, Fickinger et al. [9] proposed utilizing Gromov-Wasserstein distance to generate reward minimizing the difference between expert and learner policies. Franzmeyer et al. [11] learned a mapping between two domains, where expert embeddings are reduced by selecting task-relevant state elements based on mutual information criterion. These methods assume to have access expert's states; in contrast, our method assumes expert demonstrations are only available as high-dimensional image sequences with partial information about the expert. Kim et al. [22] and Raychaudhuri et al. [32] trained a model that can map between two domains by using expert demonstrations for proxy tasks on both domains. In comparison, our method does not rely on expert demonstrations of proxy tasks. Sermanet et al. [34] employed a model that learns representations from visual demonstrations of different viewpoints using a time-contrastive approach. This approach mainly focuses on viewpoint mismatches between domains and relies on time-synchronized demonstrations with simultaneous viewpoints. Chae et al. [7] introduced an IL algorithm designed to train a policy that exhibits robustness against variations in environment dynamics by imitating multiple experts. In contrast, our method encompasses general domain mismatches including the differences in viewpoint, degree of freedom, dynamics, and embodiment.

We focus on solving domain shift IL with image observation. Liu et al. [28] used time-aligned expert demonstrations in multiple source domains and learned a model to transform the demonstration from one domain into another. Our work is closely related to Third-Person Imitation Learning (TPIL) [37], which trains a model based on domain-independent feature extraction [15] with GAIL [18]. Similarly, Cetin & Celiktutan [6] proposed restricting mutual information between non-expert data and domain labels. Our method improves on the idea of feature extraction from non-time-aligned visual data by applying dual feature extraction and image reconstruction with consistency checks.

IL from observations (IfO) IfO is an area of IL, where the expert state-action trajectory is not provided explicitly. In IfO, the learner should capture the behavioral information from observation. Demonstrations can be provided either as a set of state vectors without actions [8, 39] or simply as image observations [13, 21, 28, 43]. Torabi et al. [39] learned the inverse dynamics to infer the missing actions from observations in the demonstration. Edwards et al. [8] used latent actions to model the transition between two consecutive observations and relabel those latent actions to true actions. We assume that the learner knows its own state and action but expert demonstration is provided only in the form of visual observations as in other works ([6, 37].

3 System Model

Setup An Markov Decision Process (MDP) is described by a tuple ( S , A , P, R, γ, ρ 0 ) , where S is the state space, A is the action space, P : S × A × S → R + is the state transition probability, R : S × A → R is the reward function, γ ∈ (0 , 1) is the discount factor, and ρ 0 is the initial state distribution [38]. Given an MDP, a stochastic policy π : S × A → R + is a conditional probability over actions given state s ∈ S . The goal of RL is to find an optimal policy π ∗ that maximizes

J ( π ) = E π [ ∑ ∞ t =0 γ t R ( s t , a t )] , where s 0 ∼ ρ 0 , a t ∼ π ( ·| s t ) , s t +1 ∼ P ( ·| s t , a t ) . In IL, expert demonstrations are provided instead of the true R . The goal of IL with a single domain is to imitate the demonstration generated by an expert policy π E .

Problem formulation We now define IL with domain adaptation. We have two domains: source domain and target domain . The source domain is where we can obtain expert demonstrations, and the target domain is where we train the agent. Both domains are modeled as MDPs. The MDPs of the source domain (using subscript S ) and the target domain (using subscript T ) are given by M S = ( S S , A S , P S , R S , γ S , ρ 0 ,S ) and M T = ( S T , A T , P T , R T , γ T , ρ 0 ,T ) , respectively. An expert is the one who performs a task optimally. The agent in the target domain learns the task from expert demonstrations in the source domain. We call the agent a learner . The expert policy π E : S S × A S → R + is in the source domain and the learner policy π θ : S T × A T → R + is in the target domain. We assume that a set of image observations of π E are provided, which is denoted by O SE , where ' SE ' refers to 'source expert'. The set of observations generated by π θ is denoted by O TL , where ' TL ' refers to 'target learner'. In addition to expert demonstrations from the source domain, we assume access to the non-optimal trajectories of image observations from both the source and target domains. The visual observations of rollouts from a non-expert policy π SN : S S ×A S → R + in the source domain are denoted by O SN , and the visual observations of rollouts from a non-expert policy π TN : S T ×A T → R + in the target domain are denoted by O TN . Here, ' SN ' refers to 'source non-expert', and ' TN ' refers to 'target non-expert'. O SN and O TN are additional data to assist our feature extraction and image reconstruction process. π SN and π TN can be constructed in several ways, but we choose these policies as ones taking uniformly random actions. Each observation set has size n demo and each observation o t consists of 4 images corresponding to 4 consecutive timesteps. The goal is to train π θ to solve a given task in the target domain using O SE , O SN , O TN , and O TL .

4 Motivation

IL with domain adaptation presents a major challenge as the learner cannot directly mimic the expert demonstration, especially when it is given in a visual form. Also, we cannot leverage expert demonstrations for proxy tasks. To overcome domain shift, conventional methods [6, 37] extracts domain-invariant features from images. We call such feature behavior feature , capturing only expert or non-expert behavior information. One example is TPIL [37]. TPIL consists of a single behavior encoder BE , a domain discriminator DD , and a behavior discriminator BD , as shown in Fig. 1 (The label XB X in Fig. 1 means that the input is in the X domain with behavior B X . X can be source S or target T , and B X can be expert E or non-expert N . We use this notation throughout the paper. See Appendix C for more details). The key idea is that BE learns to fool DD by removing domain information from the input while helping BD by preserving behavior information from the input.

inputs

behavior

encoder

behavior

feature

DD

domain

predict

SBs

BE

Bs or BT

BD

predict

behavior

TBT

behavioralinfo.

domaininfo.

SE

SN

×

TN

3

(a)

(b)

However, the behavior feature extraction of the conventional methods does not work to our satisfaction. Fig. 2 shows the t-SNE [42] plots of the behavior features extracted by the conventional methods [6, 37] after training for the task of Reacher with two links (source domain) to three links (target domain). As seen in the figure, the behavior feature vectors of SN (orange squares) and those of TN (blue crosses) are separated in the left figure (note that when domain information is completely

←--dual feature extraction_--→--

---------image generation----

<-----cycle-consistency check -----→!

DDD

domaindiscriminator

IDT

image discriminator

domainencoder

BDD

behaviordiscriminator

fake

inputs

DEs

images

DET

S

DDB

个

behaviorencoder

GT

Gs

BDB

SBs

BEs

TBs

BET

SBs

Bs

DD

contains info. T + Bs

Bs

behavioral info.

IDs

domain info.

BDD

DET

generator

DEs

DDB

S

Gs

BDB

TBT

BET

SBT

BEs

TB

contains info. S + BT

BT

image-level cycle-consistency (lmage-CC)

removed, the behavior feature vectors of SN and TN should overlap). Furthermore, since the target non-expert performs random actions for each given state, some actions resemble the expert actions in a given state, while other actions do not. So, there should be some overlap between the blue crosses ( TN ) and the red dots ( SE ). However, the blue crosses are well separated from the red circles in the right figure. Thus, the domain information is not properly removed from the behavior feature. Extracting domain-independent behavior features needs to fully exploit available source-domain and target-domain data.

Our approach is to embed the basic feature extraction structure into a bigger model with data translation. Our main idea for this is dual feature extraction and dual cycle-consistency . First, we propose a dual feature extraction scheme to extract both the domain feature and behavior feature from the input so that these two features are independent and contain full information about the input. Second, we propose a new cycle-consistency scheme that fits domain-adaptive IL with visual observation. The proposed model aims to extract better features for reward generation rather than to generate images. Thus, solely applying image-level cycle-consistency is not sufficient to solve the problem of our interest. We circumvent this limitation by additionally applying feature-level cycle-consistency on top of image-level cycle-consistency. Third, we adopt a reward-generating discriminator outside the feature extraction model for proper reward generation by exploiting the generated target-domain expert data from the feature extraction model. Therefore, we name our approach as D3IL (Dual feature extraction and Dual cycle-consistency for Domain adaptive IL with visual observation) .

5 Proposed Methodology

5.1 Dual Feature Extraction

The behavior feature in the target domain is the required information that tells whether the learner's action is good or not, in other words, whether the learner in the target domain successfully mimics the expert in the source domain or not. To improve behavior feature extraction, we boost the simple adversarial learning, as the structure shown in Fig. 1, by adding an additional structure from which we can check that the obtained feature well contains the required information without information loss, based on image reconstruction and consistency check, as shown in Fig. 3. That

is, we adopt two encoders in each of the source and target domains: domain encoder DE X and behavior encoder BE X , where X = S or T , as shown in the left side of Fig. 3. The domain encoder DE X , where X = S or T , tries to extract domain feature ( S or T ) that contains only the domain information excluding behavioral information, whereas the behavior encoder BE X tries to extract behavioral feature ( B X = E or N ) that contains only the behavioral information (expert or non-expert) excluding domain information. Each of these encodings is based on the aforementioned basic adversarial learning block composed of one encoder and two accompanying discriminators: domain and behavior discriminators, as shown in the left side of Fig. 3. As the outputs of the four encoders, we have four output features: S and B S from the source-domain encoders, and T and B T from the target-domain encoders.

5.2 Dual Cycle-Consistency (Dual-CC)

We now explain dual cycle-consistency (Dual-CC) , composed of image-level cycle-consistency (Image-CC) and feature-level cycle-consistency (Feature-CC) to improve feature extraction. Using the four output feature vectors extracted by the encoders in Sec. 5.1, we apply image translation [19, 25, 26, 44, 48]. That is, we combine T and B S and generate an image ̂ TB S by using generator G T , and combine S and B T and generate an image ̂ SB T by using generator G S . Each image generator ( G T or G S ) is trained based on adversarial learning with the help of the corresponding image discriminator ( ID T or ID S ), as shown in the middle part of Fig. 3. Then, from the generated images ̂ TB S and image ̂ SB T , we apply feature extraction again by using the same trained encoders DE T , BE T , DE S , BE S used in the first-stage feature extraction, as shown in the right side of Fig. 3. Finally, using the four features ̂ T, ̂ B S , ̂ S, ̂ B T from the second-stage feature extraction, we do apply image translation again to reconstruct the original images ̂ SB S and ̂ TB T by using the same generators G S and G T used as the first-stage image generators, as shown in the rightmost side of Fig. 3. Note that if the encoders properly extracted the corresponding features without information loss, the reconstructed images ̂ SB S and ̂ TB T should be the same as the original input images SB S and TB T . We exploit this Image-CC as one of our criteria for our feature-extracting encoders.

However, the aforementioned adversarial learning and Image-CC loss are insufficient to yield mutually exclusive behavior and domain features, as we will see in Sec. 6.7. To overcome this limitation, we enhance the dual feature extraction by complementing Image-CC by imposing Feature-CC . Note that our goal is not image translation but feature extraction relevant to domain-adaptive IL. The Feature-CC means that the extracted features S , B S , T and B T in the left dual feature extraction part of Fig. 3 should respectively be the same as their corresponding feature vectors ˆ S , ˆ B S , ˆ T and ˆ B T in the right cycle-consistency check part of Fig. 3. The rationale for this consistency is as follows. Consider B T and ̂ B T for example. B T is extracted from the target-domain image TB T , whereas ̂ B T is extracted from a source-domain image ̂ SB T generated from source-domain image generator G S . For the two behavior features B T and ̂ B T from two different domains to be the same, the domain information in B T and ̂ B T should be eliminated. Thus, the Feature-CC helps the extraction of mutually exclusive behavior and domain features while the Image-CC requires the combination of behavior and domain features to preserve full information about the input observation without any loss. Since both consistencies are not perfect, we adopt both Image-CC and Feature-CC for our feature extraction, so named Dual-CC.

5.3 Further Consistency and Stability Enhancement

In addition to the Dual-CC, our architecture allows us to verify other consistencies on the extracted features. First, we enforce the image reconstruction consistency by combining the features S and B S from DE S and BE S in the first-stage feature extraction and feeding the feature combination ( S, B S ) into image generator G S . The output ˜ SB S should be the same as the original image SB S . The same consistency applies to the feature combination ( T, B T ) in the target domain with image generator G T . Second, we enforce the feature reconstruction consistency by feeding the generated source-domain image ˜ SB S described above into the encoders DE S and BE S , then we obtain domain feature ˜ S and behavior feature ˜ B S . These two features should be identical to the features S and B S extracted in the first-stage feature extraction. The same principle applies to the target domain.

DET

二

元

BET

口

T

E

TN

TE

predict

behaviorlabel

BEs

Tθ

BET

SE

E

TL

7

In addition to the key consistency idea explained above, we apply further techniques of feature vector regularization and feature vector similarity to enhance performance. Feature vector regularization maintains the L 2 -norm of the feature vector which helps to prevent the feature vectors from becoming too large. Feature vector similarity is another form of consistency that guides the model so that the domain feature vectors of observations from the same domain should have similar values and the behavior feature vectors of observations from the same behavior should have similar values.

With this additional guidance for feature extraction on top of the conventional adversarial learning block, the proposed scheme significantly improves feature extraction compared to previous approaches, as we will see in Sec. 6. Once the overall structure is learned, we take the behavior encoder BE T in the target domain (the block inside the blue dotted block in Fig. 3) to train the learner policy. This improved behavior feature extraction can enhance the learner's performance in the target domain. Note that the complexity of D3IL is simply at the level of widely used image translation.

5.4 Reward Generation and Learner Policy Update

Now, let us explain how to update the learner policy π θ . One can simply use the behavior discriminator BD B in the target domain together with the behavior encoder BE T to generate a reward for policy updates. Instead, we propose a different way of reward generation. We use BE T but do not use BD B from the feature extraction model. This is because, in our implementation, BD B is trained to distinguish the expert behavior E from the non-expert behavior N , but what we need is a discriminator that distinguishes the expert behavior E from the learner behavior L so that the learner policy π θ can receive proper rewards for imitation. Therefore, we train another network called reward-generating discriminator D rew for reward generation. D rew and π θ jointly learn in an adversarial manner like in GAIL. D rew takes a behavior feature vector from BE T as input and learns to predict the behavior label ( E or L ) of the input. For stable learning, we consider GAIL with gradient penalty (GP) [24].

To train D rew , we need the expert behavior feature from BE T as well as the learner behavior feature L . L is simply obtained by applying the policy output o TL to BE T . The required expert behavior feature can be obtained by applying generated target-domain expert image ̂ TE to BE T , where ̂ TE is generated from the feature extraction model using domain feature T from o TN and behavior feature E from o SE , as shown in Fig. 4. (Or simply we can use the expert feature E obtained by applying source-domain expert image o SE to BE S based on the imposed Feature-CC.) The loss for D rew with fixed behavior encoder BE T is given by

where ˆ o TE = G ( DE ( o TN ) , BE ( o SE )) is a generated image from the domain feature T of o TN and the behavior feature E of o SE , and the third expectation is the GP term and is over the mixture of ˆ o TE and o TL with a certain ratio. D rew takes a behavior feature vector as input and predicts its behavior label E or L by assigning value 1 to expert behavior and value 0 to learner behavior. On the other hand, π θ learns to generate observations o TL so that the corresponding behavior feature vector resembles that of the expert. The learner updates its policy using SAC [17], and the estimated reward for an observation o t is defined by ˆ r ( o t ) = log( D rew ( BE T ( o t ))) -log(1 -D rew ( BE T ( o t ))) . Appendix A provides the details of loss functions implementing D3IL, and Algorithms 1 and 2 in Appendix B summarise our domain-adaptive IL algorithm.

6 Experiments

6.1 Experimental Setup

We evaluated D3IL on various types of domain shifts in IL tasks, including changes in visual effects, the robot's degree of freedom, dynamics, and embodiment. We also tested if D3IL can solve tasks when obtaining expert demonstrations by direct RL in the target domain is challenging. Each IL task consists of a source domain and a target domain, with a base RL task either in Gym [5] or in DeepMind Contol Suite (DMCS) [41]. In Gym, we used five environments: Inverted Pendulum (IP), Inverted Double Pendulum (IDP), Reacher-two (RE2), Reacher-three (RE3), and HalfCheetah (HC). In addition, we used a customized U-Maze environment [20] for the experiment in Section. 6.6. In DMCS, we used three environments: CartPole-Balance, CartPole-SwingUp, and Pendulum. Appendix E contains sample observations, task configuration, and space dimensions for each environment.

We compared the performance of D3IL with three major baselines: TPIL [37] and DisentanGAIL [6], and GWIL [9]. TPIL and DisentanGAIL are image-based IL methods that aim to extract domain-independent features from image observations and are most relevant to our problem setting. Additionally, we also compared D3IL with GWIL, a recent state-based IL method. For GWIL, demonstrations were provided as state-action sequences for all experiments, while for other methods demonstrations were provided as image observations. Appendix D provides implementation details including network structures and hyperparameters for D3IL. 1000

6.2 Results on IL Tasks with Changing Visual Effects

We evaluated D3IL on tasks where the domain difference is caused by visual effects on image observations. We conducted the experiment in four IL tasks: IP and IDP, with different color combinations of the pole and cart (IP-to-colored and IDP-to-colored tasks), and RE2 and RE3, with different camera angles (RE2-to-tilted and RE3-to-tilted tasks). Figure 17 in Appendix F.1 contains sample images for each task. The average episodic return for D3IL and the baselines over 5 seeds are shown in Fig. 18 in Appendix F.1. As shown in Fig. 18, D3IL outperformed the baselines with large margins on tasks with changing visual effects. 6.3 Results on IL Tasks with Changing Degree-of-Freedom of a Robot 400 600 800 Mean episodic return

Next, we evaluated D3IL on IL tasks with varying robot's degree-of-freedom. This includes scenarios where the learner robot should imitate an expert performing the same task, but the number of joints differs between them. We conducted four IL tasks: two based on IP (one pole) and IDP (two poles), and two based on RE2 (2-link arm) and RE3 (3-link arm). Sample observations for each IL task are shown in Figure 14 in Appendix E. Fig. 5 shows the average episodic return over 5 seeds. It is seen that D3IL outperforms the baseline methods by a large margin in all the IL tasks. In Fig. 5 the dotted line represents the performance of SAC trained with true rewards in the target domain. This dotted line can be considered as the performance upper bound when using SAC as the control algorithm. It is interesting to observe that in Figs. 5 (c) and (d), the baselines seem to learn the task initially but yield gradually decreasing performance, whereas D3IL continues to improve its performance. Appendix F.2 provides further discussions on the quality of extracted features on these tasks. Appendix F.3 provides additional results on IP, IDP, RE2, and RE3 with simpler settings created by [6]. 0 1 2 3 4 Timesteps 0 200

TPIL

DisentanGAIL

GWIL

D3IL (ours)

Target domain expert

0.0

0.5

1.0

1.5

2.0

Timesteps

1e5

0

200

400

600

800

1000

Mean episodic return

(a) IDP-to-one

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e6

0

2000

4000

6000

8000

10000

Mean episodic return

(b) IP-to-two

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e6

12

10

8

6

4

2

Mean episodic return

(c) RE3-to-two

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e6

12

10

8

6

4

2

Mean episodic return

(d) RE2-to-three

5

1e6

0

1

2

Timesteps

TPIL

DisentanGAIL

GWIL

D3IL (ours)

Target domain expert

0

1

2

3

Timesteps

1e5

0

200

400

600

800

Mean episodic return

(a)

CartPoleBalance-to-

Pendulum

0

1

2

3

Timesteps

1e5

0

200

400

600

800

1000

Mean episodic return

(b)

CartPoleSwingUp-to-

Pendulum

0

1

2

3

Timesteps

1e5

200

0

200

400

600

800

1000

Mean episodic return

(c)

Pendulum-to-

CartPoleBalance

0

1

2

3

Timesteps

1e5

0

200

400

600

800

Mean episodic return

(d)

Pendulum-to-

CartPoleSwingUp

800

1000

6.4 Results on Tasks with Changing Internal Dynamics

Mean episodic return

We examined D3IL on a task with a different type of domain shift, specifically a change in the robot's dynamics. The considered IL task is 'HalfCheetah-to-LockedFeet'. The base RL task is as follows. In HalfCheetah (HC) environment, the goal is to move an animal-like 2D robot forward as fast as possible by controlling six joints. Similar to [6], we also have a modified environment named as 'HalfCheetah-LockedFeet (HC-LF)'. In HC-LF, the robot can move only four joints instead of six, as two feet joints are immobilized. Fig. 15 in Appendix E shows sample observations for both environments, where the immobilized feet are colored in red. Fig. 6 shows the average episodic return over 5 seeds. Fig. 6 shows that D3IL has superior performance to the baseline methods and can solve IL tasks by changing robot's dynamics.

6.5 Results on Tasks with Changing Robot Embodiment

We examined whether D3IL can train the agent to mimic an expert with a different embodiment. We use CartPole-Balance, CartPole-SwingUp, and Pendulum environments from DMCS [41], where sample observations are provided in Fig. 16 (see Appendix E). The goal for all environments is to keep the pole upright. The CartPole agents do so by sliding the cart, while the Pendulum agent does so by adding torque on the center. The pole is initially set upright in CartPole-Balance, while it is not in other environments. Domain adaptation between these tasks is challenging because not only the embodiment between agents in both domains are different but also the distributions of expert and non-expert demonstrations, and initial observations for both domains are quite different. Fig. 7 shows the performances on IL tasks with different robot embodiment. Although D3IL does not fully reach the upper bound, it achieved far better performance than the previous methods on tasks by changing the robot's embodiment.

6.6 Experiment on AntUMaze: When Direct RL in Target Domain is Difficult

In fact, for IL tasks we have done so far, the learner policy can be learned directly in the target domain by RL interacting with the target-domain environment. The real power of domain-adaptive IL comes when such direct RL in the target domain is very difficult or impossible by known RL algorithms. Here, we consider one such case known as AntUMaze [20]. In AntUMaze, a quadruped ant robot should move to the red goal position through a U-shaped maze, as shown in the second row of Fig. 8. It can be verified that it is very difficult to solve this problem directly with a current state-of-the-art RL algorithm such as SAC. This is because the problem involves complicated robot manipulation and pathfinding simultaneously. In order to solve AntUMaze, we consider a simpler task in the source domain known as PointUMaze, which is easily solvable by SAC. In PointUMaze, the robot is simplified as a point whose state is defined as the point location and point direction, as shown in the first row of Fig. 8, whereas the state of the ant robot is much more complicated. Appendix D.3 provides a detailed explanation of the training process for the UMaze environments. Now, the goal

600

400

200

0

3

4

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e6

0

200

400

600

800

1000

Mean episodic return

0

1

TPIL

GWIL

Target-domain expert

DisentanGAIL

D3IL (Ours)

2

5

Timesteps

1e6

3

4

of this IL task (named PointUMaze-to-Ant) is to solve the AntUMaze problem in the target domain with its own dynamics based on the point robot demonstrations.

Sourcedomain:Pointrobot

0.0

0.5

1.0

1.5

2.0

Timesteps

1e6

0

200

400

600

800

1000

Mean episodic return

TPIL

DisentanGAIL

GWIL

D3IL (Ours)

SAC (vanila RL)

Targetdomain:Antrobot

Time

Fig. 9 shows the average episodic return over 5 seeds. As shown in Fig. 9, the ant robot trained with D3IL reaches the goal position after about 1M timesteps, while the ant robot trained with the baseline IL algorithms failed to reach the goal position. D3IL can train the ant agent to learn desirable behaviors from point-robot expert demonstrations with totally different dynamics. We observed that SAC with reward could not solve this task. So, it is difficult to directly solve the AntUMaze with RL or to obtain expert demonstrations in AntUMaze directly. D3IL can transfer expert knowledge from an easier task to a harder task, and train the agent in the target domain to perform the AntUMaze task.

6.7 Analysis of Dual Cycle-Consistency

To analyze the impact of Dual-CC, we use t-SNE to visualize behavior features generated by our model. Fig. 10 shows the t-SNE plots of the behavior feature extracted by the model trained on the RE2-to-three task: We used the model with full Dual-CC in Fig. 10 (a), the model with only Feature-CC in Fig. 10 (b), and the model with only Image-CC in Fig. 10 (c). Fig. 10 (a) based on the Dual-CC shows the desired properties; target-domain non-expert ( TN ) and source-domain non-expert ( SN ) are well overlapped, and some of the random behavior feature vectors coincide with the source-domain expert ( SE ) behavior. We also plotted behavior features of generated target experts ( ̂ TE ) with green '+' in the figure. Since the behavior feature of ̂ TE is derived from the behavior feature of SE and we imposed the Feature-CC, the red circles and green '+' overlap quite a good portion. However, there still exists some gap at a certain region, showing the feature extraction is not perfect. Fig. 10 (c) shows that domain information still remains when removing the Feature-CC, and the Feature-CC indeed helps eliminate domain information. As seen in Fig. 10 (b), only the Feature-CC is not enough to extract the desired behavior feature. Indeed, Dual-CC is essential in extracting domain-independent behavior features.

6.8 Ablation Study

Impact of each loss component D3IL uses multiple add-ons to the basic adversarial behavior feature learning for consistency check. As described in Sec. 5, the add-ons for better feature extraction are (1) image and feature reconstruction consistency with the generators in the middle part of Fig. 3, (2) Image-CC with the right part of Fig. 3, (3) feature similarity, and (4) Feature-CC. We removed each component from the full algorithm in the order from (4) to (1) and

SE

SN

+企X

TN

a

(b)

(C)

evaluated the corresponding performance, which is shown in Table 1. In Table 1, each component has a benefit to performance enhancement. Especially, the consistency check for the extracted feature significantly enhances the performance. Indeed, our consistency criterion based on dual feature extraction and image reconstruction is effective.

Mean episodic return

10000

Reward-generating discriminator D rew In Sec. 5.4, we explained the use of D rew for reward estimation. As explained, we train D rew and the learner policy π θ in an adversarial manner. Instead of using an external D rew , we could use the behavior discriminator BD B in the feature extraction model to train the policy π θ . To see how this method works, we trained π θ using the behavior discriminator BD B in the feature extraction model, not the external D rew . As shown in Table 2, training π θ using BD B is not successful because of the reason mentioned in Sec. 5.4. 8000

6000

2000

Domain encoders As described in Sec. 5, we proposed employing both domain encoder and behavior encoder for enhanced domain-independent behavior feature extraction. This approach was motivated by the observation that solely relying on a behavior encoder was insufficient to eliminate domain-related information from behavior features. To investigate the importance of domain encoders, we evaluated the performance of D3IL with and without domain encoders ( DE S and DE T ) and their corresponding discriminators ( DD D and BD D ) across four tasks: IP-to-two, RE2-to-three, RE3-to-tilted, and CartPoleBalance-to-Pendulum. All other networks and losses remain unchanged. Fig. 11 shows the results. The results demonstrate the benefit of conditioning generators on learned domain encoders to boost model performance. 0.0 0.2 0.4 0.6 0.8 Timesteps 0

D3IL (with domain encoders)

D3IL (without domain encoders)

Target domain expert

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e6

0

2000

4000

6000

8000

10000

Mean episodic return

(a) IP-to-two

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e6

12

10

8

6

4

2

Mean episodic return

(b) RE2-to-three

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e6

12

10

8

6

4

2

Mean episodic return

(c) RE3-to-tilted

0

1

2

3

Timesteps

1e5

0

200

400

600

800

Mean episodic return

(d)

CartPoleBalance-to-

7 Discussion and Conclusion

In this paper, we have considered the problem of domain-adaptive IL with visual observation. We have proposed a new architecture based on dual feature extraction and image reconstruction in order to improve behavior feature extraction, and a reward generation procedure suited to the proposed model. The proposed architecture imposes additional consistency criteria on feature extraction in addition to basic adversarial behavior feature extraction to obtain better behavior features. Numerical results show that D3IL yields superior performance to existing methods in IL tasks with various domain differences and is useful when direct RL in the target domain is difficult.

Limitations While our method demonstrates superior performance, it also has some limitations. One is the tricky tuning of loss coefficients, which involves several loss functions. However, we observed that balancing the scale of loss components is sufficient. Additionally, our feature extraction model remains fixed during policy updates. Future works could leverage the past target learner experiences to update the model further. Another limitation is the need for target environment interaction during training, a characteristic shared by several domain-adaptive IL approaches. It will be interesting to extend our method to an offline approach in future works. Moreover, our method lacks a quantitative measure for domain shifts. Future research could develop a mathematical way to quantify domain shifts, enabling the measurement of task difficulty and addressing more complex IL problems. Lastly, future works could extend our method to multi-task or multi-modal scenarios.

Pendulum

1.0

1e6

Acknowledgments and Disclosure of Funding

This work was supported in part by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2022-0-00469, Development of Core Technologies for Task-oriented Reinforcement Learning for Commercialization of Autonomous Drones) and in part by Institute of Information & Communications Technology Planning &Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2022-0-00124, Development of Artificial Intelligence Technology for Self-Improving Competency-Aware Learning Capabilities). Prof. Seungyul Han is currently with Artificial Intelligence Graduate School of UNIST, and his work is partly supported by Artificial Intelligence Graduate School support (UNIST), IITP grant funded by the Korea government (MSIT) (No.2020-0-01336).

References

Abbeel, P. and Ng, A. Y. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning , pp. 1, 2004.

Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein gan. arxiv 2017. arXiv preprint arXiv:1701.07875 , 30:4, 2017.

Bain, M. and Sammut, C. A framework for behavioural cloning. In Machine Intelligence 15, Intelligent Agents [St. Catherine's College, Oxford, July 1995] , pp. 103-129, GBR, 1999. Oxford University. ISBN 0198538677.

Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym, 2016.

Cetin, E. and Celiktutan, O. Domain-robust visual imitation learning with mutual information constraints. arXiv preprint arXiv:2103.05079 , 2021.

Chae, J., Han, S., Jung, W., Cho, M., Choi, S., and Sung, Y. Robust imitation learning against variations in environment dynamics. In International Conference on Machine Learning , pp. 2828-2852. PMLR, 2022.

Edwards, A., Sahni, H., Schroecker, Y., and Isbell, C. Imitating latent policies from observation. In International conference on machine learning , pp. 1755-1763. PMLR, 2019.

Fickinger, A., Cohen, S., Russell, S., and Amos, B. Cross-domain imitation learning via optimal transport. In 10th International Conference on Learning Representations, ICLR , 2022.

Finn, C., Levine, S., and Abbeel, P. Guided cost learning: Deep inverse optimal control via policy optimization. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 , ICML'16, pp. 49-58. JMLR.org, 2016.

Franzmeyer, T., Torr, P., and Henriques, J. a. F. Learn what matters: cross-domain imitation learning with task-relevant embeddings. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems , volume 35, pp. 26283-26294. Curran Associates, Inc., 2022.

Fu, J., Luo, K., and Levine, S. Learning robust rewards with adverserial inverse reinforcement learning. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018.

Gamrian, S. and Goldberg, Y. Transfer learning for related reinforcement learning tasks via image-to-image translation. In International Conference on Machine Learning , pp. 2063-2072. PMLR, 2019.

Gangwani, T. and Peng, J. State-only imitation with transition dynamics mismatch. In International Conference on Learning Representations , 2020.

Ganin, Y. and Lempitsky, V. Unsupervised domain adaptation by backpropagation. In International conference on machine learning , pp. 1180-1189. PMLR, 2015.

Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C. Improved training of wasserstein gans. Advances in neural information processing systems , 30, 2017.

Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018.

Ho, J. and Ermon, S. Generative adversarial imitation learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems , NIPS'16, pp. 4572-4580, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.

Huang, X., Liu, M.-Y., Belongie, S., and Kautz, J. Multimodal unsupervised image-to-image translation. In Proceedings of the European conference on computer vision (ECCV) , pp. 172189, 2018.

Kanagawa, Y. Mujoco-maze, 2021. URL https://github.com/kngwyu/mujoco-maze .

Karnan, H., Warnell, G., Xiao, X., and Stone, P. Voila: Visual-observation-only imitation learning for autonomous navigation. arXiv preprint arXiv:2105.09371 , 2021.

Kim, K., Gu, Y., Song, J., Zhao, S., and Ermon, S. Domain adaptive imitation learning. In International Conference on Machine Learning , pp. 5286-5295. PMLR, 2020.

Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Kostrikov, I., Agrawal, K. K., Dwibedi, D., Levine, S., and Tompson, J. Discriminator-actorcritic: Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXiv:1809.02925 , 2018.

Lee, H.-Y., Tseng, H.-Y., Huang, J.-B., Singh, M., and Yang, M.-H. Diverse image-to-image translation via disentangled representations. In Proceedings of the European conference on computer vision (ECCV) , pp. 35-51, 2018.

Lee, H.-Y., Tseng, H.-Y., Mao, Q., Huang, J.-B., Lu, Y.-D., Singh, M., and Yang, M.-H. Drit++: Diverse image-to-image translation via disentangled representations. International Journal of Computer Vision , 128(10):2402-2417, 2020.

Liu, F., Ling, Z., Mu, T., and Su, H. State alignment-based imitation learning. In International Conference on Learning Representations , 2020.

Liu, Y., Gupta, A., Abbeel, P., and Levine, S. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pp. 1118-1125. IEEE, 2018.

Ng, A. Y. and Russell, S. J. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning , ICML '00, pp. 663-670, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607072.

Osa, T., Pajarinen, J., Neumann, G., Bagnell, J. A., Abbeel, P., and Peters, J. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics , 7(1-2):1-179, 2018. doi: 10.1561/2300000053.

Pomerleau, D. A. Efficient training of artificial neural networks for autonomous navigation. Neural computation , 3(1):88-97, 1991.

Raychaudhuri, D. S., Paul, S., Vanbaar, J., and Roy-Chowdhury, A. K. Cross-domain imitation from observations. In International Conference on Machine Learning , pp. 8902-8912. PMLR, 2021.

Ross, S., Gordon, G., and Bagnell, D. A reduction of imitation learning and structured prediction to no-regret online learning. In Gordon, G., Dunson, D., and Dudík, M. (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , volume 15 of Proceedings of Machine Learning Research , pp. 627-635, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR.

Sermanet, P., Lynch, C., Chebotar, Y., Hsu, J., Jang, E., Schaal, S., Levine, S., and Brain, G. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA) , pp. 1134-1141. IEEE, 2018.

Sharma, P., Pathak, D., and Gupta, A. Third-person visual imitation learning via decoupled hierarchical controller. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Smith, L., Dhawan, N., Zhang, M., Abbeel, P., and Levine, S. Avid: Learning multi-stage tasks via pixel-level translation of human videos, 2020.

Stadie, B. C., Abbeel, P., and Sutskever, I. Third-person imitation learning, 2017.

Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction . The MIT Press, second edition, 2018.

Torabi, F., Warnell, G., and Stone, P. Behavioral cloning from observation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence , IJCAI'18, pp. 4950-4957. AAAI Press, 2018. ISBN 9780999241127.

Torabi, F., Warnell, G., and Stone, P. Generative adversarial imitation from observation. arXiv preprint arXiv:1807.06158 , 2018.

Tunyasuvunakool, S., Muldal, A., Doron, Y., Liu, S., Bohez, S., Merel, J., Erez, T., Lillicrap, T., Heess, N., and Tassa, Y. dm_control: Software and tasks for continuous control. Software Impacts , 6:100022, 2020. ISSN 2665-9638. doi: https://doi.org/10.1016/j.simpa.2020.100022.

Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Wen, C., Lin, J., Qian, J., Gao, Y., and Jayaraman, D. Keyframe-focused visual imitation learning. arXiv preprint arXiv:2106.06452 , 2021.

Yi, Z., Zhang, H., Tan, P., and Gong, M. Dualgan: Unsupervised dual learning for image-toimage translation. In Proceedings of the IEEE international conference on computer vision , pp. 2849-2857, 2017.

Zakka, K., Zeng, A., Florence, P., Tompson, J., Bohg, J., and Dwibedi, D. Xirl: Crossembodiment inverse reinforcement learning. In Faust, A., Hsu, D., and Neumann, G. (eds.), Proceedings of the 5th Conference on Robot Learning , volume 164 of Proceedings of Machine Learning Research , pp. 537-546. PMLR, 08-11 Nov 2022.

Zhang, X., Li, Y., Zhang, Z., and Zhang, Z.-L. f-gail: Learning f-divergence for generative adversarial imitation learning. Advances in neural information processing systems , 33:1280512815, 2020.

Zheng, B., Verma, S., Zhou, J., Tsang, I., and Chen, F. Imitation learning: Progress, taxonomies and opportunities. arXiv preprint arXiv:2106.12177 , 2021.

Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision , pp. 2223-2232, 2017.

Ziebart, B. D., Maas, A., Bagnell, J. A., and Dey, A. K. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 3 , AAAI'08, pp. 1433-1438. AAAI Press, 2008. ISBN 9781577353683.

A The Loss Functions for the Proposed Model

In Sec. 5, we provided the key idea of the proposed learning model. Here, we provide a detailed explanation of the loss functions implementing the learning model described in Fig. 3 of the main paper. Table 3 shows the notations for observation sets.

The observation sets O SE , O SN , O TN and O TL are defined in Sec. 3 of the main paper.

A.1 Dual Feature Extraction

The left side of Fig. 3 shows the basic adversarial learning block for dual feature extraction. That is, we adopt two encoders in each of the source and target domains: domain encoder and behavior encoder . Behavior encoders ( BE S , BE T ) learn to obtain behavior feature vectors from images that only preserve the behavioral information and discard the domain information. Here, subscript S or T denotes the domain from which each encoder takes input. Domain encoders ( DE S , DE T ) learn to obtain domain feature vectors which preserve only the domain information and discard the behavioral information. The encoders learn to optimize the WGAN objective [2, 16] based on the feature discriminators : behavior discriminator BD ∗ and domain discriminator DD ∗ . Discriminators BD B and DD B are associated with the behavior encoder BE X , where X = S or T , and Discriminators BD D and DD D are associated with the domain encoder DE X , where X = S or T . BD B learns to identify the behavioral information from behavior feature vectors, and DD B learns to identify the domain information from behavior feature vectors. BD D learns to identify the behavioral information from domain feature vectors, and DD D learns to identify the domain information from domain feature vectors. For notational simplicity, we define functions BE and DE as

Two loss functions are defined for the dual feature extraction: feature prediction loss and feature adversarial loss .

1) Feature prediction loss: This loss makes the model retain the desired information from feature vectors and is given by

where E o ∈ A [ · ] is the expectation over inputs o ∈ A . The first term E o ∈ O E [ BD B ( BE ( o ))] in the righthand side (RHS) of (3) means that if the input observation of the encoder BE ( · ) is from the expert observation set O E , then the discriminator BD B with the extracted feature should declare a high value. The second term -E o ∈ O N [ BD B ( BE ( o ))] in the right-hand side (RHS) of (3) means that if the input observation of the encoder BE ( · ) is from nonexpert observation set O N , then the discriminator BD B with the extracted feature should declare a low value. The third term E o ∈ O S [ DD D ( DE ( o ))] in the right-hand side (RHS) of (3) means that if the input observation of the encoder DE ( · ) is from the source-domain set O S , then the discriminator DD D with the extracted feature should declare a high

value. The fourth term -E o ∈ O T [ DD D ( DE ( o ))] in the right-hand side (RHS) of (3) means that if the input observation of the encoder DE ( · ) is from the target-domain set O T , then the discriminator DD D with the extracted feature should declare a low value. Since we define a loss function, we have the negative sign in front of the RHS of (3). (Note that BD B learns to assign higher values to expert behavior and to assign lower values to non-expert behavior. DD D learns to assign higher values to the source domain and to assign lower values to the target domain.) The encoders ( BE , DE ) help discriminators ( BD B , DD D ) to identify the behavioral or domain information from the input, and they jointly learn to minimize L pred feat .

2) Feature adversarial loss: This loss makes the model delete the undesired information from feature vectors by adversarial learning between the encoders and the discriminators. The loss is given by

In this loss function, the output of the behavior encoder is fed to the domain discriminator, and the output of the domain encoder is fed to the behavior discriminator. The learning of the encoders and the discriminators is done in an adversarial manner as

Recall that BD ∗ learns to assign higher values to expert behavior and to assign lower values to non-expert behavior, and DD ∗ learns to assign higher values to the source domain and to assign lower values to the target domain. Thus, the discriminators try to do their best, whereas the encoders try to fool the discriminators.

The first two terms in the RHS of (4) imply that the output of the behavior encoder BE should not contain the domain information. The last two terms in the RHS of (4) imply that the output of the domain encoder DE should not contain the behavioral information.

A.2 Image Reconstruction and Associated Consistency Check

In the previous subsection, we explained the loss functions of basic adversarial learning for dual feature extraction. Now, we explain the loss functions associated with consistency checks with image reconstruction.

As shown in the middle part of Fig. 3, for image reconstruction, we adopt a generator in each of the source and target domains: generator G S for the source domain and generator G T for the target domain. Generators G S and G T learn to produce images from the feature vectors so that the generated images should resemble the original input images. G S takes a source-domain feature vector and a behavior feature vector as input and generates an image that resembles images in the source domain. On the other hand, G T takes a target-domain feature vector and a behavior feature vector as input and generates an image that resembles images in the target domain.

Generators G S and G T learn to optimize the WGAN objective with the help of image discriminators: ID S and ID T . ID S distinguishes real and generated images in the source domain, and ID T does so in the target domain. For notational simplicity, we define functions G ( · , · ) and ID ( · , · ) as

1) Image adversarial loss : The basic loss function for training the image generators and the associated blocks is image adversarial loss of WGAN learning [2, 16]. The image adversarial loss makes the model generate images that resemble the real ones and is given by

where l a ( x ) is the adversarial WGAN loss for a real image in O x . In more detail, the first term in the RHS of (8) can be expressed as

Note that the image discriminator ID ∗ learns to assign a high value for a true image and a low value for a fake image. The first term in the RHS of (9) trains the image discriminator ID ( · ) to assign a high value to a true image o ∈ O SE , whereas the second term in the RHS of (9) trains the image discriminator ID ( · ) to assign a low value to a fake image. The latter is because the ID ( · ) input, i.e., the generator output G ( DE ( o x ) , BE ( o y )) with ( o x , o y ) ∈ ( O SN , O SE ) inside the second term in the RHS of (9) is a fake image; for this image, the behavior feature is taken from o y ∈ O SE but the domain feature is taken from o x ∈ O SN not in O SE .

The other terms l a ( SN ) , l a ( TN ) and l a ( TL ) in the RHS of (8) are similarly defined as

where ˆ O SN,fake is the set of image combinations { ( o x , o y ) } such that the generated fake image G ( DE ( o x ) , BE ( o y )) has label SN . ˆ O TN,fake and ˆ O TL,fake are similarly defined. Specifically,

With the image adversarial loss L adv imag in (8), the learning is performed in an adversarial manner. The encoders and generators learn to minimize L adv img , while the image discriminators learn to maximize L adv img . That is, the image discriminators are trained to distinguish fake images from real images, whereas the encoders and the generators are trained to fool the image discriminators ID ∗ .

Note that the image adversarial loss is basically for training the image generators and the associated blocks in a WGAN adversarial manner. With the availability of the image generators G S and G T , we can impose our first and second consistency criteria: image reconstruction consistency and feature reconstruction consistency . We define the corresponding loss functions below.

2) Image reconstruction consistency loss : This loss checks the feature extraction is properly done. When we combine the features S and B S from DE S and BE S with an input true image o SE (or input true image o SN ) in the first-stage feature extraction and input the feature combination ( S, B S ) into image generator G S , the generated image should be the same as the original observation image o SE (or o SN ). The same consistency applies to the feature combination ( T, B )) in the target domain with a true input image o TN or o TL and the image generator G T . Thus, the image reconstruction loss is given as

O ALL = O SE ∪ O SN ∪ O TN ∪ O TL , and l mse ( u, v ) is the mean square error between u and v . The encoders and generators learn to minimize L recon img .

3) Feature reconstruction consistency loss : This is the second self-consistency criterion. If we input the generated source-domain image ˜ SB S described in the above image reconstruction loss part into the encoders DE S and BE S , then we obtain domain feature ˜ S and behavior feature ˜ B S , and these two features should be the same as the features S and B S extracted in the first-stage feature extraction.

where

The same principle applies to the target domain. Thus, the feature reconstruction consistency loss is expressed as

where

|| · || 2 denotes the 2-norm of a vector, and O ALL = O SE ∪ O SN ∪ O TN ∪ O TL . Note that ˆ o is a generated image from the domain and behavior features extracted from o , and DE (ˆ o ) and BE (ˆ o ) are the domain and behavior features from the generated image ˆ o . The encoders and generators learn to minimize L recon feat .

A.3 Cycle-Consistency Check

Now, let us consider our third consistency criterion: cycle-consistency check with the right side of Fig. 3. This consistency check involves image translation in the middle part of Fig. 3 and the image retranslation in the right part of Fig. 3 in the main paper, and requires that the original image and the reconstructed image of translation/retranslation should be the same for perfect feature extraction and generation. The image cycle-consistency loss is expressed as

where

and O 2 cycle is the set of image pairs such that the two images in a pair do not belong to the same domain, i.e., one image belongs to the source domain and the other image belongs to the target domain. The explanation is as follows. Consider that we apply image o x to the upper input and image o y to the lower input of the left side of Fig. 3 in the main paper. Then, ˆ o ( x,y ) = G ( DE ( o x ) , BE ( o y )) is the generated image in the lower row and ˆ o ( y,x ) = G ( DE ( o y ) , BE ( o x )) is the generated image in the upper row in the middle part of Fig. 3 in the main paper. Finally, the image G ( DE (ˆ o ( x,y ) ) , BE (ˆ o ( y,x ) )) is the final reconstructed image in the upper row of the right side of Fig. 3 in the main paper. So, we can check the cycle-consistency between o x and G ( DE (ˆ o ( x,y ) ) , BE (ˆ o ( y,x ) )) . The situation is mirrored when we apply image o x to the lower input and image o y to the upper input of the left side of Fig. 3 in the main paper. The encoders and generators learn to minimize L cycle img .

The feature cycle-consistency loss is expressed as

where

The explanation is as follows. For example, from the target domain real image TB T , we extract domain feature T and behavior feature B T . From these features, we can reconstruct TB T . Also, we extract domain feature ̂ T (from ̂ TB S ) and behavior feature ̂ B T (from ̂ SB T ). From these features, we can reconstruct TB T . Let's assume ̂ B T (behavior feature from the source domain image ̂ SB T generated by G S ) is replaced by B T (behavior feature from the target domain image TB T ). Then from ̂ T and B T we should still reconstruct TB T . That is, ̂ B T (behavior feature from the source domain image ̂ SB T made by G S ) = B T (behavior feature from the target domain image TB T ). They are behavior features from different domains but they are equal. This implies that the behavior feature is independent of domain information. We name this by feature cycle-consistency because this constraint implicitly satisfies image cycle-consistency. if we explicitly constrain ̂ B T = B T and ̂ T = T , then we can replace ̂ B T and ̂ T by B T and T , and the image cycle-consistency is implicitly satisfied.

A.4 Other Losses

We further define losses to enhance performance.

1) Feature regularization loss : The feature regularization loss prevents the feature vector values from exploding for stable learning, and is given by

Instead, more rigorously, we can use the following regularization:

Where, c norm,d and c norm,b are hyperparameters with non-negative values. Appendix D.2 provides details including the value of c norm,d and c norm,b for the experiment.

2) Feature similarity loss : This loss aims to map observations of the same domain to similar points in the feature vector space and to map observations of the same behavior to similar points in the feature vector space. The feature similarity loss is defined as

where µ BE ( O x ) and µ DE ( O x ) are the means of the behavior feature vectors and the domain feature vectors over the observation set O x ( x ∈ { SE,SN,TN,TL } ), respectively.

A.5 Final Loss

Combining the above individual loss functions, we have the final objective for the encoders E = ( BE S , DE S , BE T , DE T ) , the generators G = ( G S , G T ) , feature discriminators FD = ( BD B , DD B , BD D , DD D ) and image discriminators ID = ( ID S , ID T ) as follows:

where the weighting factors λ adv feat , λ adv feat , λ adv img , λ recon img , λ recon feat , λ cycle img , λ cycle feat , λ sim feat , λ reg feat are hyperparameters and their values are shown in Appendix D. Note that L adv feat and L adv img appear in both (25) and (26) with negative sign in (26). Hence, these two terms induce adversarial learning in the overall learning process.

A.6 Reward Generation and Learner Policy Update

In this section, we explain the simple method of extracting expert features and training D rew for reward generation. That is, we just use the output of BE S in the source domain with expert input o SE . In this case, D rew learns to minimize the following objective while behavior encoder BE T is fixed:

where the third expectation in (27) is over the mixture of ˆ o TE and o TL with a certain ratio. The proposed method other than this simple method is explained in Sec. 5.

D rew takes a behavior feature vector as input and predicts its behavior label E or L from the input

behavior feature vector. D rew learns to assign the value 1 to the expert behavior and 0 to the learner behavior. On the other hand, π θ learns to generate observations O TL so that the corresponding behavior feature vector looks like the expert. The learner updates the policy using SAC [17], and the estimated reward for an observation o t is defined by

which is in a similar form to that in [24].

B Algorithm Pseudo Codes

Algorithm 1 Dual feature extraction and image generation

Input: The number of epochs n epoch _ it , domain encoders DE = ( DE S , DE T ) , behavior encoders BE = ( BE S , BE T ) , generators G = ( G S , G T ) , feature discriminators FD = ( BD B , DD B , BD D , DD D ) , image discriminators ID = ( ID S , ID T ) , observation sets O SE , O SN , O TN of size n demo .

Sample a minibatch of observations o SE , o SN , o TN , o TL from O SE , O SN , O TN , O TL , respec-

Initialize parameters for DE , BE , G , FD , ID . Make a copy of O TN and initialize O TL with the copy. for k = 1 to n epoch _ it do tively. for o in ( o SE ∪ o SN ∪ o TN ∪ o TL ) do Extract DE ( o ) and BE ( o ) . Generate ˆ o in eq. (14). Extract DE (ˆ o ) and BE (ˆ o ) . end for for o x , o y (not in the same domain) in ( o SE ∪ o SN ∪ o TN ∪ o TL ) do Generate ˆ o ( x,y ) and ˆ o ( y,x ) in eqs. (18) and (19). Generate G ( DE (ˆ o ( x,y ) ) .BE (ˆ o ( y,x ) )) . end for Compute L E,G and L FD,ID in eqs. (25) and (26). Update DE , BE and G to minimize L E,G . Update FD and ID to minimize L FD,ID . end for return E,G,FD,ID

Algorithm 2 Reward estimation and policy update

Input: The number of training epochs n epoch _ pol , the number of discriminator updates n update,D , the number of policy updates n update,θ , domain encoders DE , behavior encoders BE , generators G , discriminator D rew for reward estimation, learner policy π θ , observations sets O SE , O SN , O TN of size n demo , replay buffer B of size n buffer . Initialize parameters for D rew , π θ and B . for k 1 = 1 to n epoch _ pol do Sample a trajectory τ ∼ π θ . Store transitions ( s, a, s ′ ) and observations o to B . for k 2 = 1 to n update,D do Sample minibatch of observations o SE ∈ O SE , o TN ∈ O TN , o TL ∈ B . Update D rew to minimize L D in Sec. 5.4. end for for k 3 = 1 to n update,θ do Sample minibatch of transitions ( s, a, s ′ ) and corresponding observations o . Compute reward ˆ r in Sec. 5.4. Update the policy π θ using SAC. end for end for

C Third-Person Imitation Learning (TPIL)

This section summarises TPIL [37], one of the pioneering works that address the domain shift problem in IL. TPIL trains a model based on an unsupervised domain adaptation technique [15] with GAIL [18]. TPIL consists of a single behavior encoder BE , a domain discriminator DD , and a behavior discriminator BD , as shown in Fig. 12. The input label XB X in Fig. 12 means that the input is in the X domain with behavior B X . X can be source S or target T , and B X can be expert E or non-expert N . The key idea is to train BE to extract domain-independent behavior features from inputs. The trained BE can be used to tell whether the learner's action in the target domain is expert behavior or non-expert behavior, and we can use this evaluation to train the learner. DD learns to predict the domain label (source or target) of the input, and BD learns to predict the behavior label (expert or non-expert) of the input. Therefore, encoder BE learns to fool DD by removing domain information from the input while helping BD by preserving behavior information from the input. In [37], the behavior feature is a concatenation of BE ( o t ) and BE ( o t -4 ) ; however, we just simply the notation and denote the behavior feature by BE ( o t ) .

The total loss for TPIL L TPIL is defined as follows:

where o i is an image observation, L CE is the cross-entropy loss, d i is the domain label of o i (1 for source domain and 0 for target domain), b i is the behavior label of o i (1 for expert behavior and 0 for non-expert behavior), λ d is a hyperparameter, and G is a Gradient Reversal Layer (GRL) [15] is defined by

where λ g is a hyperparameter. GRL enables updating BE , DD , and BD simultaneously using back-propagation. The imitation reward r t for an observation o t generated by the learner policy is defined by the probability that BD predicts the observation to be generated by an expert policy, as determined by r t = BD ( BE ( o t )) .

inputs

behaviorencoder

behavior

predict

feature

DD

domain

SBs

BE

Bs or Br

BD

predict

behavior

TBT

behavioral info. (expert, non-expert)

domain info. (source, target)

D Implementation Details

D.1 Network Architectures

In this section, we explain the network architectures of the components of the proposed method. Table 4 shows the layers for encoders, generators, and discriminators.

Each behavior encoder ( BE S or BE T ) consists of 6 convolutional layers and a flattened layer. The number of output channels for each convolutional layer is (16, 16, 32, 32, 64, 64), and each channel has 3 × 3 size. Zero padding is applied before each convolutional layer, and ReLU activation is applied after each convolutional layer except the last layer. The flattening layer reshapes the input to a one-dimensional vector. The shape of the input is ( N,W,H, 4 C ) , where N is the minibatch size, W and H are the width and height of each image frame, and C is the number of channels (for RGB images, C = 3 ). The shape of the behavior feature vector is ( N, ( W/ 4) × ( H/ 4) × 64) .

Each domain encoder ( DE S , DE T ) consists of 6 convolutional layers, a flattened layer, and a linear layer. The number of output channels for each convolutional layer is (16, 16, 32, 32, 64, 64), and each channel has 3 × 3 size. Zero padding is applied before each convolutional layer, and ReLU activation is applied after each convolutional layer. The shape of the input is ( N,W,H,C ) , and the shape of the domain feature vector is ( N, 8) .

Each generator ( G S or G T ) consists of 7 transposed convolutional layers. The number of output channels for each layer is (64, 64, 32, 32, 16, 16, 4 C ), and each channel has 3 × 3 size. Zero padding is applied before each layer, and ReLU activation is applied after each layer except the last layer. The input is a behavior feature vector and a domain feature vector. The shape of the output is ( N,W,H, 4 C ) , which is the same as the shape of the input of behavior encoders.

Each feature discriminator ( BD B , DD B , BD D , or DD D ) consists of 2 fully connected layers and a linear layer. Each fully connected layer has 32 hidden units with ReLU activation. Each image discriminator ( ID S or ID T ) consists of 6 convolutional layers, a flattened layer, and a linear layer. Each convolutional layer has 3 × 3 channel size. The number of output channels for each layer is (16, 16, 32, 32, 64, 64). Zero padding is applied before each layer, and ReLU activation is applied after each layer except on the last layer. The discriminator D for reward estimation consists of 2 fully connected layers and a linear layer. Each fully connected layer has 100 hidden units with ReLU activation. The input size is ( N, ( W/ 4) × ( H/ 4) × 64) .

For the SAC algorithm, each actor and the critic consist of 2 fully connected layers and a linear layer. Each fully connected layer has 256 hidden units with ReLU activation.

D.2 Details for Training Process

Before training the model, we collected expert demonstrations O SE , and non-expert datasets O SN , O TN . O SE is obtained by the expert policy π E , which is trained for 1 million timesteps in the source domain using SAC. O SN is obtained by a policy taking uniformly random actions in the source domain, and O TN is obtained by a policy taking uniformly random actions in the target domain. For each IL task, the number of observations (i.e., the number of timesteps) for O SE , O SN , O TN is n demo = 10000 except for HalfCheetah-to-locked-legs task, where n demo = 20000 for this task. Each observation consists of 4 RGB images, but the size of these images varies depending on the specific IL task. For IL tasks including IP, IDP, CartPole, and Pendulum, the image size is 32x32. For IL tasks including RE2 and RE3, the image size is 48x48. For IL tasks including HalfCheetah and UMaze, the image size is 64x64. Note that the proposed method does not require a specific input image size.

Our proposed method has two training phases. The first phase updates domain encoders, behavior encoders, generators, feature discriminators, and image discriminators. The second phase updates the discriminator D for reward estimation and the policy π θ . In the first phase, we trained the model for n epoch _ it = 50000 epochs for all IL tasks except for HalfCheetah-to-locked-legs task, where n epoch _ it = 200000 for this task. In each epoch, we sampled a minibatch of size 8 from each dataset O SE , O SN , O TN , O TL . We set O TL as a copy of O TN . For coefficients in Eqs. (25) and (26), we set λ pred feat = λ adv feat = 0 . 01 , λ reg feat = 0 . 1 , λ adv img = 1 , λ sim feat = λ recon feat = 1000 , λ recon img = λ cycle img = 100000 . We used λ cycle feat = 10 for IP-to-color, IDP-to-one, IP-to-two, and PointUMaze-to-ant, λ cycle feat = 100

for IDP-to-color, λ cycle feat = 100 for RE3-to-tilted, and λ cycle feat = 10000 for RE2-to-tilted, RE3-to-two, RE2-to-three, and HC-to-LF. We used λ cycle feat = 100 for all environments from DeepMind Control Suite. We chose these coefficients so that the scale of each loss component is balanced, which was observed to be enough to yield good performances of π θ .

As mentioned in Appendix A.4, we adjusted the values of c norm,d and c norm,b in Eq. (23) of the proposed method for the experiment. For IL tasks including IP, IDP, RE2, RE3, and HalfCheetah, we set c norm,d = c norm,b = 0 . For IL tasks including CartPole and Pendulum, we set c norm,d = 1 and c norm,b = 20 . For UMaze task, we set c norm,d = 1 and c norm,b = 40 . The ratio between c norm,d and c norm,b is determined based on the input image size since the proposed method does not require a specific input image size and produces behavior features of varying sizes depending on the input image size.

In the second phase, we trained D for reward estimation and π θ for n epoch _ pol epochs, where n epoch _ pol = 20 for IP-to-color, IDP-to-one tasks, n epoch _ pol = 30 for CartPole and Pendulum tasks in DeepMind Control Suite, and n epoch _ pol = 100 for other IL tasks. Each epoch consists of 10000 timesteps. In each epoch, the following process is repeated. We sampled a trajectory from π θ , and stored state transitions ( s, a, s ′ ) and observations o to the replay buffer B . The maximum size of B is n buffer = 100000 , which is much smaller compared to that in off-policy RL algorithms because of the large memory consumption when storing images in B . Then, D is updated for n update,D times. We n update,D to be 50 times smaller than the episode length for each IL task. For example, if the episode length is 1000, then n update,D = 20 . For every D update, we sampled a minibatch of size 128 from each dataset O SE and O TN . For O TL , we sampled a minibatch of size 64 from B and a minibatch of size 64 from O TN . After D update, π θ is updated for n update,θ times using SAC [17]. We set n update,θ to be equal to the episode length for each IL task. For example, if the episode length is 1000, then n update,θ = 1000 . For every π θ update, we sampled a minibatch of size 256 from B .

In each epoch, we evaluated π θ after updating D and π θ . We sampled n eval trajectories using π θ and computed the average return, where each return is computed based on the true reward in the target domain. We set n eval = 10 as default. For IL tasks including Reacher environments, we set

n eval = 200 to reduce variance because each return highly depends on the goal position in Reacher environments. We used Adam optimizer [23] for optimizing all networks. We set the learning rate lr = 0 . 001 and momentum parameters β 1 = 0 . 9 and β 2 = 0 . 999 for encoders, generators, and discriminators. We set lr = 0 . 0003 , β 1 = 0 . 9 and β 2 = 0 . 999 for the actor and critic. For SAC, we set the discount factor γ = 0 . 99 , the parameter ρ = 0 . 995 for Polyak averaging the Q-network, and the entropy coefficient α = 0 . 1 .

The proposed method is implemented on TensorFlow 2.0 with CUDA 10.0, and we used two Intel Xeon CPUs and a TITAN Xp GPU as the main computing resources. One can use a GeForce RTX 3090 GPU instead as the computing resource, which requires TensorFlow 2.5 and CUDA 11.4. Based on 32x32 images, The GPU memory consumption is about 2.5GB for the training feature extraction model and about 1.5GB during policy update. Note that these highly depend on the image size and the batch size. Our D3IL is implemented based on the code provided by the authors of [6].

D.3 Dataset and Training Process for PointUMaze-to-Ant Task

For the UMaze environments, the number of observations for O SE , O SN or O TN is n demo = 10000 . Each observation consists of 4 frames and each frame is a 64x64 RGB image. During the training, the reward that the agent receives for each timestep is r total = c × ( r IL + 1 reach _ goal × r goal ) , where r IL is the estimated reward given by D ; 1 reach _ goal is 1 if the robot reaches the goal and 0 otherwise; r goal is the reward when the robot reaches the goal; and c > 0 is a scaling coefficient. In the experiment, we chose c = 100 and r goal = 1 , and these parameters are applied to all baselines. For the proposed method, the number of epochs, minibatch sizes, and network structures for the training feature extraction network and image reconstruction network are the same as those in Appendix D.1. The policy is trained for n epoch _ pol = 200 and each epoch consists of 10000 timesteps, so the total number of timesteps for policy training is 2,000,000.

Also, we trained an agent using a vanilla RL method (SAC [17]) with access to true rewards, to show that it is difficult to obtain expert demonstrations in the target domain. For SAC, the agent receives true rewards. For true rewards, the agent receives a penalty of -0.0001 for each timestep and receives 1000 when the robot reaches the goal, so the maximum true episodic return is 1000.

E RL Environment Settings and Sample Image Observations

We first describe the base RL tasks in the Gym.

Inverted pendulum (IP): The agent moves a cart to prevent the pole from falling. The episode length is 1000. Unlike the IP task in Gym [4], an episode does not terminate midway even if the pole falls. This makes the task much more challenging because each episode provides less useful observations to solve the task.

Inverted double pendulum (IDP): This is a harder version of the IP task where there are two poles on the cart.

Reacher-two (RE2): The agent moves a two-link arm with one end fixed so that the movable endpoint of the arm reaches the target point. The episode length is 50. Unlike the Reacher task in Gym, we narrowed the goal point candidates to 16 points, where the polar coordinate ( r, φ ) of the target point can be r ∈ { 0 . 15 , 0 . 2 } , and φ ∈ { 0 , π/ 4 , π/ 2 , 3 π/ 4 , · · · , 7 π/ 4 } .

Reacher-three (RE3): This is a harder version of the RE2 task where the agent moves a three-link armed robot.

Note that our IL tasks are more difficult than those considered in the experiments in the baseline algorithm references [6, 37]. Our simulation setting has an episode length of 1000 for the IP and IDP tasks instead of 50. The episode horizon is also a component of an MDP, and the observation space of the MDP changes as the episode horizon changes. MDP with a longer episode length can yield more various observations which are much different from the initial one, so the observation gap between the source domain and the target domain increases. We meant this for a bigger domain gap.

If the goal position is fixed throughout the entire learning phase in RE tasks, as designed in [6, 37], distinguishing between expert and non-expert behavior solely depends on the arm position. However, in our implementation, the goal position is randomly chosen from 16 positions for each episode. This makes distinguishing expert and non-expert behavior much more challenging because the same arm movement can be considered expert or non-expert behavior depending on the goal position. Therefore, extracting proper behavior features is much more difficult as the feature extraction model should catch the relationship between the arm movement and the goal position.

InvertedPendulum (IP)

InvertedDoublePendulum (IDP)

Time

Time

Reacher-two (RE2)

Reacher-three (RE3)

Time

Time

The following images are sample observations for IL tasks with changing a robot's degree of freedom.

IDP-to-one

IP-to-two

RE3-to-two

RE2-to-three

Source

Target

Source

Target

Source

Target

Source

Target

In the HalfCheetah (HC) environment, the agent controls an animal-like 2D robot. The goal is to move the animal-like 2D robot forward as fast as possible by controlling 6 joints. Similarly to [6], we have a modification with immobilized feet, and this modified environment is 'HalfCheetah-LockedFeet (HC-LF)'. In the HC-LF environment, the robot can use only 4 joints instead of 6, where the immobilized feet are colored in red.

HalfCheetah (movable feet)

HalfCheetah (locked feet)

Time

Time

The following images are sample image observations of environments in the DeepMind Control Suite (DMCS). We use CartPole-Balance, CartPole-SwingUp, and Pendulum environments from DMCS. The goal for all environments is to keep the pole upright.

CartPole-Balance : The pole starts in an upright position and the agent should move the cart to keep the pole upright while avoiding exceeding the boundaries along the x-axis.

CartPole-Swingup : The pole starts in a downward position and the agent should move the cart to swing the pole and keep the pole in an upright position while avoiding exceeding the boundaries along the x-axis.

Pendulum : The agent should add torque on the center to swing the pole and keep the pole upright.

Domain adaptation between these tasks is challenging because not only the embodiment between agents in both domains are different but also the distributions of expert and non-expert demonstrations, and initial observations for both domains are quite different.

CartPoleBalance

CartPoleSwingUp

Pendulum

Time

Time

Time

We omit sample image observations of PointUMaze and AntUMaze environments because they are already provided in the main paper. Table 5 shows the dimensions of state space, action space, and visual observation for RL tasks used in the Experiment section.

F Additional Experimental Results

F.1 Results on IL Tasks with Changing Visual Effect 1000

We evaluated D3IL on tasks where the domain difference is caused by visual effects on image observations on four IL tasks: IP and IDP, with different color combinations of the pole and cart (IP-to-colored and IDP-to-colored tasks), and RE2 and RE3, with different camera angles (RE2-totilted and RE3-to-tilted tasks). Sample image observations are provided in Figure 17. For the IP and IDP tasks, the primary changes occur in the pixel values of the pole and the cart. Meanwhile, for the RE2 and RE3 tasks, major changes are observed in the pixel values of the background outside of the arms and the goal position. The average episodic return for D3IL and the baselines over 5 seeds are shown in Fig. 18. We only included baselines that receive demonstrations as images because changing visual effects on image observations do not affect the true state space and the action space. As shown in Fig. 18, D3IL outperformed the baselines with large margins on IL tasks with changing visual effects on image observations. 400 600 800 Mean episodic return

0

1

2

3

4

Timesteps

0

200

IP-to-color

IDP-to-color

RE2-to-tilted

RE3-to-tilted

Source

Target

Source

Target

Source

Target

Source

Target

TPIL

DisentanGAIL

D3IL (ours)

Target domain expert

0.0

0.5

1.0

1.5

2.0

Timesteps

1e5

0

200

400

600

800

1000

Mean episodic return

(a) IP-to-color

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e6

0

2000

4000

6000

8000

Mean episodic return

(b) IDP-to-color

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e6

12

10

8

6

4

Mean episodic return

(c) RE2-to-tilted

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e6

12

10

8

6

4

2

Mean episodic return

(d) RE3-to-tilted

5

1e6

F.2 The Evaluation on the Feature Quality

Fig. 19 shows some instances of the estimated rewards from sampled observations in the RE2-tothree task, corresponding to the time of 150,000 timesteps in Fig. 5d. The left column of Fig. 19 is non-expert behavior and the right column is expert behavior. One side of the Reacher arm is fixed at the center of the image. In the left column, the movable arm end moves away from the goal position through four time-consecutive images, whereas the movable arm end moves towards the goal position in the right column. Note that in these instances, the baseline algorithm assigns a higher reward to the non-expert behavior, whereas the proposed method correctly assigns a higher reward to the expert behavior. We conjecture that this wrong reward estimation of the baseline algorithms based on poor features is the cause of the gradual performance degradation of the baseline algorithms in Fig. 5.

In Fig. 20, we checked the quality of extracted features from a conventional method, TPIL, and that of our method. Since TPIL does not consider image reconstruction but considers only the extraction of domain-independent behavior features. In order to generate an image from the domain-independent behavior feature of TPIL, along with TPIL we also trained two generators ( G S , G T ) : G S takes a behavior feature vector as input and generates an image that resembles the images from the source domain. G T takes a behavior feature vector as input and generates an image that resembles the images from the target domain. This G S and G T training is done with two discriminators ( ID S and ID T ), where ID S distinguishes the real and generated images in the source domain, and ID T does so in the target domain. As shown in Fig. 20, the reconstructed image from the behavior feature of TPIL is severely broken for the RE2-to-tilted task. Thus, it shows that the "domain-independent" behavior feature of TPIL does not preserve all necessary information of the original observation image. On the other hand, our method based on dual feature extraction and image reconstruction, yields image reconstruction looking like ones. This implies that the features extracted from our method well preserve the necessary information, and thus reconstruction is well done.

Non-expert behavior

Expert behavior

D3IL

=-0.520

r=0.046

TPIL

r=-11.35

r=-13.93

DisentanGAIL

=-2.284

=-4.581

Conventional IL (TPIL)

D3IL

IL task

IP-to-two

RE2-to-tilted

IP-to-two

RE2-to-tilted

Domain

Source

Target

Source

Target

Source

Target

Source

Target

True images

observations

Imagedgenerated

toadifferentdomain

F.3 Experimental Results on Simpler Tasks

In the main result subsection of Sec. 6, we mentioned that the baseline algorithms perform well in the case of easier settings. As a preliminary experiment, we verify the performance of the proposed method and the baseline IL algorithms on simpler IL tasks, which are easier variants of the IL tasks shown in Sections 6.2 and 6.3. We first describe the base RL tasks.

Inverted pendulum - short (IPshort) : This is the same task as the IP task, but the episode length is 50.

Inverted double pendulum - short (IDPshort) : This is the same task as the IDP task, but the episode length is 50.

Reacher-two - one goal (RE2onegoal) : This is the same task as the RE2 task, but the goal point is fixed to ( x, y ) = (1 . 0 , 1 . 0) for the entire training.

Reacher-three - one goal (RE3onegoal) : This is the same task as the RE3 task, but the goal point is fixed to ( x, y ) = (1 . 0 , 1 . 0) for the entire training.

Sample visual observations for each IL task are provided in Fig. 21. We followed the training process as described in [6]. For each IL task, we collected O SE , O SN and O TN of size 10000. The learner policy π θ is trained for 20 epochs. Each epoch has 1000 timesteps, so the total number of timesteps is 20000. The size of the replay buffer is 10000. We used TPIL [37] and DisentanGAIL [6] as baseline algorithms. For each IL task, we trained the learner policy using our proposed method and baseline algorithms over 5 seeds. For evaluation, we generated 5 trajectories from π θ for every epoch and computed the average return, where the return was computed based on the true reward in the target domain.

Fig. 22 shows the average episodic return. As shown in Fig. 22, the baseline algorithms perform well in these easy IL tasks. The IL tasks shown in Fig. 21 are much easier to solve than the IL tasks in Sections 6.2 and 6.3 because the pole needs to be straight up for only 50 timesteps instead of 1000 timesteps in IPshort and IDPshort, and the goal position is fixed to a single point instead of randomly sampled from 16 candidates in RE2ongoal and RE3onegoal. The baselines can achieve good performance on these easy IL tasks. However, as shown in Fig. 5 of the main paper and Fig. 18 in Appendix 6.2, the baseline algorithm yields poor performance on hard IL tasks and the proposed method achieves superior performance on hard IL tasks to the baselines due to the high-quality feature extraction by the dual feature extraction and image reconstruction.

1000

800

0

200

400

600

Mean episodic return

IDPshort-to-color

RE2onegoal-to-tilted

RE3onegoal-to-tilted

Source

Target

Source

Target

Source

Target

IDPshort-to-one

IPshort-to-two

RE3onegoal-to-two

RE2onegoal-to-three

Source

Source

Target

Source

Target

Source

Target

0

1

2

3

4

TPIL

DisentanGAIL

D3IL (ours)

Target domain expert

0.0

0.5

1.0

1.5

2.0

Timesteps

1e4

15

20

25

30

35

40

45

50

Mean episodic return

(a) IPshort-to-color

0.0

0.5

1.0

1.5

2.0

Timesteps

1e4

200

250

300

350

400

450

Mean episodic return

(b) IDPshort-to-color

0.0

0.5

1.0

1.5

2.0

Timesteps

1e4

10

8

6

4

2

Mean episodic return

(c) RE2onegoal-to-tilted

0.0

0.5

1.0

1.5

2.0

Timesteps

1e4

8

7

6

5

4

3

2

1

Mean episodic return

(d) RE3onegoal-to-tilted

0.0

0.5

1.0

1.5

2.0

Timesteps

1e4

15

20

25

30

35

40

45

50

Mean episodic return

(e) IDPshort-to-one

0.0

0.5

1.0

1.5

2.0

Timesteps

1e4

200

250

300

350

400

450

Mean episodic return

(f) IPshort-to-two

0.0

0.5

1.0

1.5

2.0

Timesteps

1e4

8

6

4

2

Mean episodic return

(g) RE3onegoal-to-two

0.0

0.5

1.0

1.5

2.0

Timesteps

1e4

8

7

6

5

4

3

2

1

Mean episodic return

(h) RE2onegoal-to-three

5

1e6

800

F.4 Experiments Results with Single Expert Trajectory during Policy Update In this section, we investigated whether D3IL can successfully train the target learner when a single trajectory of expert demonstration is available during policy training. For results in Section 6, multiple trajectories of expert demonstrations and non-expert data are provided for D3IL, TPIL, and DisentanGAIL, where the details are provided in Appendix D. Since GWIL calculates the Gromov-Wasserstein distance between expert and learner policies, GWIL can leverage only a single expert trajectory. This is also true for the implementation with the code provided by the authors of [9]. In contrast, the size of the replay buffer n buffer of GWIL equals the number of training timesteps, which can vary from 200,000 to 2,000,000 depending on the IL task. On the other hand, n buffer is fixed to 100,000 for D3IL and baselines that utilize image observations. This is due to the huge memory consumption for storing image observations. In this experiment, we evaluated the performance of D3IL under the condition of a single trajectory of expert demonstration during policy training. To provide one trajectory of SE , and similarly, we supplied SN and TN with the same size of SE . In Reacher environments, we collected one trajectory per goal position for each SE , SN , and TN , respectively. Fig. 23 shows the average return of D3IL and baselines across 5 seeds. 0 1 2 3 4 Timesteps 0 200 400 600 Mean episodic return

TPIL

DisentanGAIL

GWIL

D3IL (ours)

Target domain expert

0.0

0.5

1.0

1.5

2.0

Timesteps

1e5

0

200

400

600

800

1000

Mean episodic return

(a) IDP-to-one

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e6

0

2000

4000

6000

8000

Mean episodic return

(b) IP-to-two

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e6

12

10

8

6

4

2

Mean episodic return

(c) RE3-to-two

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e6

12

10

8

6

4

2

Mean episodic return

(d) RE2-to-three

0.0

0.2

0.4

0.6

0.8

1.0

Timesteps

1e6

0

200

400

600

800

1000

Mean episodic return

(e) HalfCheetah-to-LockedLegs

0

1

2

3

Timesteps

1e5

0

200

400

600

800

Mean episodic return

(f) CartPoleBalance-to-Pendulum

0

1

2

3

Timesteps

1e5

0

200

400

600

800

Mean episodic return

0

1

2

3

Timesteps

1e5

200

0

200

400

600

800

1000

Mean episodic return

0

1

2

3

Timesteps

1e5

0

200

400

600

800

Mean episodic return

(g) CartPoleSwingup-to-Pendulum

(h) Pendulum-to-CartPoleBalance

(i) Pendulum-to-CartPoleSwingup

5

1e6

1000

800

F.5 Domain Transfer between CartPole-Balance and CartPole-SwingUp

In Section 6.5, we present the results of four IL tasks that use three different environments in DMCS: CartPole-Balance, CartPole-SwingUp, and Pendulum. In these tasks, the robot's embodiment and dynamics of both domains are the same but their task configuration and initial state distribution of both domains are different. Fig. 24 shows the results of the other two IL tasks where both domains use the CartPole environment. We evaluated D3IL and the baselines on IL tasks for 5 seeds. As illustrated in Fig. 24, all methods failed to solve the CartPole-SwingUp task using demonstrations from the CartPole-Balance task. In contrast, our methods were able to successfully solve the CartPole-Balance task using demonstrations from the CartPole-SwingUp task. This difference in performance can be attributed to the variation in expert demonstrations between the two environments. Demonstrations of the CartPole-SwingUp task typically show images in which the pole is upright, with only a small number of images where the pole is initially pointing downwards and begins to swing up. As a result, an agent for the CartPole-Balance task can learn that expert behavior involves keeping the pole upright and preventing it from falling. In contrast, demonstrations of the CartPole-Balance task typically only show images in which the pole is upright, which means that an agent designed for the CartPole-SwingUp task may not have access to information on expert behavior for swinging up the pole from a downward-pointing position. This problem can be mitigated if non-expert data contains more diverse images from the CartPole-SwingUp task. Our method can also be enhanced by utilizing images generated from the target learner to train our feature extraction model. 0 1 2 3 4 Timesteps 0 200 400 600 Mean episodic return

TPIL

DisentanGAIL

GWIL

D3IL (ours)

Target domain expert

0

1

2

3

Timesteps

1e5

0

200

400

600

800

Mean episodic return

(a) CartPole: Balance-to-SwingUp

0

1

2

3

Timesteps

1e5

0

250

500

750

1000

Mean episodic return

(b) CartPole: SwingUp-to-Balance

5

1e6

F.6 Results on Acrobot Environment

In this section, we evaluated D3IL on IL tasks where the target domain is the Acrobot environment in DMCS. In Acrobot, the agent should apply torque to the joint located between a two-link pole to make the pole stand upright. The dynamics of the Acrobot environment are much different from those of other environments like CartPole and Pendulum. Although the state and action dimensions of the Acrobot environment are low, achieving high scores in this task is non-trivial. In our implementation, the policy using vanilla RL with SAC, with 10 million timesteps, 256 hidden units for actor and critic networks, and a batch size of 256, obtained the average return of 45.49 across 5 seeds. Figure 25 shows sample observations for the Acrobot environment, where each observation consists of four 32x32 RGB images with corresponding consecutive timesteps. 80

We evaluated D3IL and baselines on two IL tasks, where the source domain is either CartPoleSwingUp or Pendulum, and the target domain is Acrobot. CartPole-SwingUp and Pendulum are known to be solvable by SAC. We examined whether D3IL can transfer expert knowledge from simpler tasks to more challenging ones. Fig. 26 shows the average episodic return of D3IL and the baselines across 5 seeds. The results demonstrate that D3IL outperforms other baselines, and achieves the average return of vanilla RL using SAC with 10 million timesteps in much smaller timesteps. Our method has the potential to improve if additional information on Acrobot's dynamics is provided to the model. This can be accomplished by obtaining diverse non-expert data, such as observations from the policy with medium quality or observations from the target learner policy, to train the feature extraction model. 40 60

20

0

Mean episodic return

0.5

1.0

1.5

Timesteps

Acrobot

0.0

2.0

1e6

TPIL

DisentanGAIL

GWIL

D3IL (ours)

Target domain SAC (10 million timesteps)

0.0

0.5

1.0

1.5

2.0

Timesteps

1e6

0

20

40

60

80

Mean episodic return

(a) CartPoleSwingUp-to-Acrobot

0.0

0.5

1.0

1.5

2.0

Timesteps

1e6

0

20

40

60

80

100

Mean episodic return

(b) Pendulum-to-Acrobot

F.7 Experiment on Walker, Cheetah, and Hopper Task

To assess the effectiveness of our proposed method in more challenging scenarios, we tested our method on IL tasks with domain shifts caused by variations in robot morphology. We employed Walker, Cheetah, and Hopper environments in the DeepMind Control Suite. In these environments, the objective is to propel a 2D robot forward as fast as possible by manipulating its joints. The difficulty arises from the significant difference in the robot morphology and dynamics between the source and target domains. Sample visual observations for each IL task are provided in Fig. 27. 16

For all environments, we set each episode length to be 200. The expert policy π E is trained for 3 million timesteps in the source domain using SAC. The number of observations is n demo = 1000 (5 trajectories and each trajectory has a length of 200). Each observation consists of 4 frames and each frame is a 64x64 RGB image. 12 14

We evaluated our proposed method and baseline algorithm in three tasks: Walker-to-Cheetah, Walkerto-Hopper, and Cheetah-to-Hopper (denoted as 'A-to-B' where 'A' is the source domain and 'B' is the target domain). A camera generates image observations by observing the robot's movement. Rewards were assigned to the agent based on the robot's horizontal velocity, with higher velocity leading to greater rewards. Fig. 28 depicts the learning curve of our method and baselines on three tasks, where each curve shows the average horizontal velocity of the robot in the target domain over three runs. These results demonstrate the superiority of our method over the baselines, which leads us to anticipate that our proposed method can extend to more complicated domains. 6 8 10 Mean episodic return

(a) Walker

(b) Cheetah

(c) Hopper

0.0

0.5

1.0

Timesteps

0

2

4

TPIL

DisentanGAIL

GWIL

D3IL (ours)

0

1

2

3

4

5

Timesteps

1e5

0

1

2

3

Average Velocity along x-axis

(a) Walker-to-Cheetah

0

1

2

3

4

5

Timesteps

1e5

0.25

0.00

0.25

0.50

0.75

1.00

1.25

Average Velocity along x-axis

(b) Walker-to-Hopper

0

1

2

3

4

5

Timesteps

1e5

0.2

0.0

0.2

Average Velocity along x-axis

(c) Cheetah-to-Hopper

1.5

F.8 Visualizing Sample Trajectories for Trained Learner

In this section, we provide visualized sample trajectories of demonstration sequences in the source domain and sequences generated by the learned policy trained in the target domain for qualitative analysis on two tasks: RE2-to-three and PointUMaze-to-Ant. Fig. 29 shows sample trajectories from the source domain expert and target domain learner on the RE2-to-three task. The top row of Fig. 29 shows a sample trajectory of image observations sampled by π E in the source domain using SAC for 1 million timesteps. The bottom row of Fig. 29 shows a sample trajectory of image observations sampled by π θ trained using D3IL for 1 million timesteps in the target domain.

Sourcedomain:RE2-Expertdemonstration

Target domain:RE3-Policy trainedfor1M timesteps

In Section 6.6, we tested the performance of D3IL on the PointUMaze-to-Ant task. Fig. 30 shows sample trajectories from the source domain expert and target domain learner. The top row of Fig. 30 shows a sample trajectory of image observations sampled by π E in the source domain using SAC for 0.1 million timesteps. The bottom row of Fig. 30 shows a sample trajectory of image observations sampled by π θ trained using D3IL for 2 million timesteps in the target domain. The episode lengths for the trained learner vary when running multiple evaluation trajectories, most of which have episode lengths in the range of 60 to 80. This implies that D3IL can train the agent in the target domain with its dynamics by observing expert demonstrations in the source domain with a different embodiment.

Source domain: Point robot -Expert demonstration (episode length=18)

Target domain: Ant robot - Learner trained for 2M timesteps (episodelength=69)

Figure 30: (Top) An example trajectory of image observations generated by π E in the source domain. In this figure, the interval between two consecutive image frames is 1 timestep, and the total episode length is 18. (Bottom) An example trajectory of image observations generated by π θ trained using D3IL for 2 million timesteps in the target domain. In this figure, the interval between two consecutive image frames is 4 timesteps, and the total trajectory length is 69.

G Further Discussions

Explanation of the performance gap with GWIL GWIL is a state-based method, but having access to expert states does not necessarily ensure superior performance. In the original GWIL paper [9], the authors provided simulation results on Pendulum-to-CartpoleSwingUp (Fig. 7 in [9]), showing GWIL achieves 600 points. However, Fig. 7 in [9] assumes that the target-domain learner receives the target-domain sparse reward directly in addition to the IL reward. So, the GWIL result in our paper can be different from that in the GWIL paper. Despite our efforts to tune GWIL's hyperparameters, GWIL did not show high performance only with the IL reward without direct target-domain sparse reward. Our approach consistently shows better performance in almost all tasks presented in this paper. As the GWIL paper indicates, GWIL can recover the optimal policy only up to isometric transformations. Given that expert demonstrations only cover a fraction of the optimal policy's scope, recovering the exact optimal policy becomes more difficult. Additionally, GWIL uses Euclidean distance as a metric within each space to compute Gromov-Wasserstein distance, which confines the method to scenarios limited to rigid transformations between domains.

Broader Impacts The proposed learning model does not raise significant ethical concerns. However, as an IL algorithm, it should be applied to train models that learn tasks beneficial to society.

Peer reviews:
comments: SUMMARY:
This paper aims to solve the domain-adaptive imitation learning problem, where the demonstrations are collected in a source domain, and the agent needs to learn policies in a target domain. Beyond that, the tasks are with high-dimensional visual observations. The authors have proposed a new framework based on dual feature extraction and image reconstruction to achieve behavior feature extraction and reward generation.

STRENGTHS:
This paper aims to solve a very practical problem, since in real-world problems, we can hardly collect expert demonstrations in the target learning environment. Instead, collecting expert demo in a similar source domain is much easier.

WEAKNESSES:
The framework in Figure 3 is too much complex. The authors are encouraged to clarify their contributions in this paper and simplify the parts in the existing literature. 

Previous works study the problem of domain adaptive imitation learning, and this work aims to solve the same problem with visual observations. The proposed approach augments the previous work with an encoder to deal with the high-dimensional visual observations. This idea seems a little straightforward.

QUESTIONS:
The experiment section is weak. The proposed approach is barely evaluated in limited simple tasks. If the authors could show that the proposed approach can work in more challenging domains, such as humanoid running, the quality of this paper could be greatly improved.

LIMITATIONS:
This paper has not discussed the limitations in the main body of this submission.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nImitation learning from visual observations for domain adaptation. Given expert and non expert demonstrations in a source domain, and non expert demonstrations in a target domain, learn a reward and policy in the target domain via image consistency and domain and behavior feature consistency.\n\nSTRENGTHS:\nDomain adaptation in imitation learning remains an important problem. The paper tackles this challenging problem assuming access to visual observations but not to actions, other proxy tasks or expert demonstrations in the target environment. These assumptions allow for better generality. The problem statement, setup and method are clearly presented and the quality is high.\n\nWEAKNESSES:\n- The paper empirically demonstrates shifts in color, number of joints that can move, and action space. What are the general assumptions about the relation between the source and target domains? (There is some similarity, otherwise transfer would not be possible..)\n- The papers uses visual observations, which is a more general setup than states. However, the proposed domains are quite simple (very few agents/objects in each, very simple backgrounds, etc.). How is the method expected to scale to more complex environments?\n\nQUESTIONS:\nIn addition to the weaknesses above, \n\n- How do the target policies compare to the source policies? E.g. in RE2 to RE3, does the policy learn to use all 3 links or does it apply a strategy with 2 links? Would suggest providing videos for qualitative analysis. \n- In Figure 10(a), why is it desired that the source and target expert behavior features overlap? Shouldn't they require different features for different domains? (E.g. number of links.)\n- Is there an explanation for the performance gap with GWIL (in Figures 5-9)? A state-based method should have more information about the environment.\n- Please state the state/action space for every environment (for completeness and clarity).\n- Suggest defining formally what good/bad actions mean.\n- Line 22: suggest replacing 'should' with 'can'.\n- Typos in lines: 55, 130, 194, 195, 355.\n\nLIMITATIONS:\nPlease add to Section G the assumption re access to the target environment for training $\\pi_{\\theta}$ and in what cases will D3IL fail to learn a successful target policy.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper attempts to solve the problem of domain shift in Imitation Learning with image observation. The author proposes D3IL architecture which is based on dual feature extraction (behavior and domain specific features from input) and dual cycle consistency (image level and feature level cycle consistency). The paper demonstrates effectiveness of the proposed method compared to baselines Third Person IL (TPIL), Disentan GAIL over different Gym based tasks (e.x. Reacher, Pendulum etc.) and  DeepMind Control Suite (DMCS) tasks (Maze based tasks).

STRENGTHS:
- The problem addressed in an interesting problem for Imitation Learning.
- Good thorough evaluation different types of experiments.
- Strong experiments results.
- The paper is well written.

WEAKNESSES:
- The problem is well motivated from real robotics based task, but no real robotics based task is demonstrated in the paper. It would be interesting to see the application on sim-to-real based tasks.
- It would be good to discuss work similar TCN work [1].
- The extensions to other IL methods like IRL is a little unclear.
- Line 90-93 are a little unclear.

[1] Sermanet, Pierre, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. "Time-contrastive networks: Self-supervised learning from video." In 2018 IEEE international conference on robotics and automation (ICRA), pp. 1134-1141. IEEE, 2018.

QUESTIONS:
- Have you tried interchanging source and target domains?
- The relations to source -> expert and target -> learner is a little confusing. Maybe if you can represent in a better way that would be really good.

LIMITATIONS:
- most of them are mentioned in the paper
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a method of learning a visual behavior encoding for reinforcement learning tasks, D3IL.\n\nFrom a TPIL behavior feature encoder baseline, D3IL introduces dual feature extraction (including both behavior and domain feature encoders) and dual cycle-consistency (cycle consistency on image and feature levels). From inputs of unpaired target and source images, D3IL translates across domains and then cycles back to the original domain, via a pair of generator networks which take in source images and domain targets. Further, image reconstruction consistency, feature vector similarity, and feature vector consistency is added for further loss and stability.\n\nThe domain and behavior feature encoders and target generator from above are taken and used to train a reward generating discriminator, $D_rew$, which distinguishes between expert and learner behavior (rather than expert and non-expert data used for training the previous models). This is used alongside the target behavior encoder to train SAC RL policies for the given tasks.\n\nD3IL is tested on many domain transfer tasks with variations in visual effects, degree-of-freedom, internal dynamics, and robot embodiment - D3IL demonstrates superior performance on the target domain compared to baselines of TPIL, …. It is further demonstrated on an AntUMaze task where direct RL learning in the target domain is difficult, but possible with D3IL. Ablation studies show that the contributions of each of the consistency losses across feature and images, and cycle consistency, are all important and contribute to the final performance.\n\nSTRENGTHS:\n#### Originality\nThe work builds on top of TPIL to build a larger framework for cross-domain imitation / feature encoding, by incorporating dual feature extraction and cycle consistency, and training a further reward-generating discriminator.\n\n#### Quality\nThe experiments are detailed, covering four categories of domain change and with ablation studies on components of the loss. The experiments are run over multiple seeds and the method is compared against several baselines (TPIL, DisetanGAIL, GWIL) to present a good picture of the overall learning and domain transfer capability. There are further many more detailed experiments in the appendix to which present good coverage.\n\n#### Clarity\nThe work is generally well-written and the description of D3IL which draws from premises of prior TPIL work is understandable. The explanations of losses and model composition are detailed and the diagrams help with understanding the proposed methodology. The experiments are easy to follow. Further task explanation and experimental methodology is well explained in the appendix.\n\n\n#### Significance\nThe method seems broadly applicable across tasks and domain transfer goals in simulation.\n\nThe experiment on AntUMaze is particularly motivating as towards the practical applications of the method, demonstrating that a task which could not be solved directly could be improved by training from an imitation agent learned on a simpler task - in a sort of curriculum learning.\n\nWEAKNESSES:\n#### Quality\nThe dual feature extraction, in particular the domain encoder added from TPIL, is motivated on lines 135-136 as to “check that the obtained feature well contains the required information without information loss, based on image reconstruction and consistency check.” However, it’s still unclear why specifically it is necessary to pass observations as input to domain encoder, as the work assumes that the policy knows what environment it is operating in. Would an ideal domain encoder be able to output a one-hot vector indicating its domain to minimize all the domain-encoder relevant losses? As there are separate generators for each domain (source/target), should it be clear without any domain conditioning what the “style” of each generator output should be? For example, in CycleGAN and its successors, the domain encoding is not required.\n\nAn ablation study without using a domain encoder, but a static method of encoding the domain or skipping the domain encoding entirely may be helpful.\n\n#### Clarity\nUnfortunately many of the experimental results are left to to the appendix including all of Sections 6.2 and 6.5, and reading the appendix seems necessary to have a full understanding of the model performance.\n\nThe method is somewhat hard to initially grasp from reading the text of Section 5 as there are many similar variables in the notation which differ by one letter or modifier, with many such modifiers. However, explanations are generally helped by Figures 1 and 3.\n\nThe limitations are also only mentioned in the appendix.\n\nMisc:\n- Line 191: is another form of consistency [that] guide[s] the model so that …\n- Line 195: scheme significantly improve[s]\n- Line 211: we need [the] expert behavior feature\n\n####  Significance\nThe practical applications of the method seem best suited for cases where it is hard to train an expert in the target environment directly via RL, but having access to expert data on other tasks as observation-only would help. Many of these cases would be covered by real world scenarios, which is out-of-scope of the current work. Even on the AntUMaze problem, the learning process did not necessarily have to be done via adversarial learning and visual-only observations, as the full expert state would be available.\n\nFrom the ablation in Table 1, it appears that the large majority of the performance improvement comes from image-level consistency and the feature-level consistency was only of minor gain, especially compared to the uncertainty/error.\n\nAs mentioned in the limitations in the appendix, the paper also introduces quite the complex process with significantly many more loss terms and two pipeline stages compared to prior work, which may make it harder to apply. Adversarial learning techniques generally require more careful tuning than supervised methods, and this work introduces two adversarially-learned stages.\n\nQUESTIONS:\nSome discussion on the weakness-quality section would be most helpful\n\nLIMITATIONS:\nSome consideration of the limitations is explained in the weaknesses-significance section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}