SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To train  $D_{rew}$ , we need the expert behavior feature from  $BE_T$  as well as the learner behavior feature  $L$ .  $L$  is simply obtained by applying the policy output  $o_{TL}$  to  $BE_T$ . The required expert behavior feature can be obtained by applying generated target-domain expert image  $\widehat{TE}$  to  $BE_T$ , where  $\widehat{TE}$  is generated from the feature extraction model using domain feature  $T$  from  $o_{TN}$  and behavior feature  $E$  from  $o_{SE}$ , as shown in Fig. 4. (Or simply we can use the expert feature  $E$  obtained by applying source-domain expert image  $o_{SE}$  to  $BE_S$  based on the imposed Feature-CC.) The loss for  $D_{rew}$  with fixed behavior encoder  $BE_T$  is given by
$$
\begin{array}{l} L _ {D} = \mathbb {E} _ {\left(o _ {T N}, o _ {S E}\right) \in \left(O _ {T N}, O _ {S E}\right)} \left[ \log \left(D _ {r e w} \left(B E _ {T} \left(\hat {o} _ {T E}\right)\right)\right) \right] + \mathbb {E} _ {o _ {T L} \in O _ {T L}} \left[ \log \left(1 - D _ {r e w} \left(B E _ {T} \left(o _ {T L}\right)\right)\right) \right] \\ + \mathbb {E} _ {o _ {x} \in M i x (\delta_ {T E}, o _ {T L})} [ (| | \nabla D _ {r e w} (B E _ {T} (o _ {x})) | | _ {2} - 1) ^ {2} ] \\ \end{array}
$$
where  $\hat{o}_{TE} = G(DE(o_{TN}), BE(o_{SE}))$  is a generated image from the domain feature  $T$  of  $o_{TN}$  and the behavior feature  $E$  of  $o_{SE}$ , and the third expectation is the GP term and is over the mixture of  $\hat{o}_{TE}$  and  $o_{TL}$  with a certain ratio.  $D_{rew}$  takes a behavior feature vector as input and predicts its behavior label  $E$  or  $L$  by assigning value 1 to expert behavior and value 0 to learner behavior. On the other hand,  $\pi_{\theta}$  learns to generate observations  $o_{TL}$  so that the corresponding behavior feature vector resembles that of the expert. The learner updates its policy using SAC [17], and the estimated reward for an observation  $o_t$  is defined by  $\hat{r}(o_t) = \log(D_{rew}(BE_T(o_t))) - \log(1 - D_{rew}(BE_T(o_t)))$ . Appendix A provides the details of loss functions implementing D3IL, and Algorithms 1 and 2 in Appendix B summarise our domain-adaptive IL algorithm.
We evaluated D3IL on various types of domain shifts in IL tasks, including changes in visual effects, the robot's degree of freedom, dynamics, and embodiment. We also tested if D3IL can solve tasks when obtaining expert demonstrations by direct RL in the target domain is challenging. Each IL task consists of a source domain and a target domain, with a base RL task either in Gym [5] or in DeepMind Contol Suite (DMCS) [41]. In Gym, we used five environments: Inverted Pendulum (IP), Inverted Double Pendulum (IDP), Reacher-two (RE2), Reacher-three (RE3), and HalfCheetah (HC). In addition, we used a customized U-Maze environment [20] for the experiment in Section. 6.6. In DMCS, we used three environments: CartPole-Balance, CartPole-SwingUp, and Pendulum. Appendix E contains sample observations, task configuration, and space dimensions for each environment.
We compared the performance of D3IL with three major baselines: TPIL [37] and DisentanGAIL [6], and GWIL [9]. TPIL and DisentanGAIL are image-based IL methods that aim to extract domain-independent features from image observations and are most relevant to our problem setting. Additionally, we also compared D3IL with GWIL, a recent state-based IL method. For GWIL, demonstrations were provided as state-action sequences for all experiments, while for other methods demonstrations were provided as image observations. Appendix D provides implementation details including network structures and hyperparameters for D3IL.
6.2 Results on IL Tasks with Changing Visual Effects
We evaluated D3IL on tasks where the domain difference is caused by visual effects on image observations. We conducted the experiment in four IL tasks: IP and IDP, with different color combinations of the pole and cart (IP-to-colored and IDP-to-colored tasks), and RE2 and RE3, with different camera angles (RE2-to-tilted and RE3-to-tilted tasks). Figure 17 in Appendix F.1 contains sample images for each task. The average episodic return for D3IL and the baselines over 5 seeds are shown in Fig. 18 in Appendix F.1. As shown in Fig. 18, D3IL outperformed the baselines with large margins on tasks with changing visual effects.
6.3 Results on IL Tasks with Changing Degree-of-Freedom of a Robot
Next, we evaluated D3IL on IL tasks with varying robot's degree-of-freedom. This includes scenarios where the learner robot should imitate an expert performing the same task, but the number of joints differs between them. We conducted four IL tasks: two based on IP (one pole) and IDP (two poles), and two based on RE2 (2-link arm) and RE3 (3-link arm). Sample observations for each IL task are shown in Figure 14 in Appendix E. Fig. 5 shows the average episodic return over 5 seeds. It is seen that D3IL outperforms the baseline methods by a large margin in all the IL tasks. In Fig. 5 the dotted line represents the performance of SAC trained with true rewards in the target domain. This dotted line can be considered as the performance upper bound when using SAC as the control algorithm. It is interesting to observe that in Figs. 5 (c) and (d), the baselines seem to learn the task initially but yield gradually decreasing performance, whereas D3IL continues to improve its performance. Appendix F.2 provides further discussions on the quality of extracted features on these tasks. Appendix F.3 provides additional results on IP, IDP, RE2, and RE3 with simpler settings created by [6].
Figure 5: The learning curves in the target domain of the IL tasks with changing DOF
Figure 7: The learning curves for the considered IL tasks for CartPole and Pendulum environments
6.4 Results on Tasks with Changing Internal Dynamics
We examined D3IL on a task with a different type of domain shift, specifically a change in the robot's dynamics. The considered IL task is 'HalfCheetah-to-LockedFeet'. The base RL task is as follows. In HalfCheetah (HC) environment, the goal is to move an animal-like 2D robot forward as fast as possible by controlling six joints. Similar to [6], we also have a modified environment named as 'HalfCheetah-LockedFeet (HC-LF)'. In HC-LF, the robot can move only four joints instead of six, as two feet joints are immobilized. Fig. 15 in Appendix E shows sample observations for both environments, where the immobilized feet are colored in red. Fig. 6 shows the average episodic return over 5 seeds. Fig. 6 shows that D3IL has superior performance to the baseline methods and can solve IL tasks by changing robot's dynamics.
Figure 6: Result on HC-LF
6.5 Results on Tasks with Changing Robot Embodiment
We examined whether D3IL can train the agent to mimic an expert with a different embodiment. We use CartPole-Balance, CartPole-SwingUp, and Pendulum environments from DMCS [41], where sample observations are provided in Fig. 16 (see Appendix E). The goal for all environments is to keep the pole upright. The CartPole agents do so by sliding the cart, while the Pendulum agent does so by adding torque on the center. The pole is initially set upright in CartPole-Balance, while it is not in other environments. Domain adaptation between these tasks is challenging because not only the embodiment between agents in both domains are different but also the distributions of expert and non-expert demonstrations, and initial observations for both domains are quite different. Fig. 7 shows the performances on IL tasks with different robot embodiment. Although D3IL does not fully reach the upper bound, it achieved far better performance than the previous methods on tasks by changing the robot's embodiment.
6.6 Experiment on AntUMaze: When Direct RL in Target Domain is Difficult
In fact, for IL tasks we have done so far, the learner policy can be learned directly in the target domain by RL interacting with the target-domain environment. The real power of domain-adaptive IL comes when such direct RL in the target domain is very difficult or impossible by known RL algorithms. Here, we consider one such case known as AntUMaze [20]. In AntUMaze, a quadruped ant robot should move to the red goal position through a U-shaped maze, as shown in the second row of Fig. 8. It can be verified that it is very difficult to solve this problem directly with a current state-of-the-art RL algorithm such as SAC. This is because the problem involves complicated robot manipulation and pathfinding simultaneously. In order to solve AntUMaze, we consider a simpler task in the source domain known as PointUMaze, which is easily solvable by SAC. In PointUMaze, the robot is simplified as a point whose state is defined as the point location and point direction, as shown in the first row of Fig. 8, whereas the state of the ant robot is much more complicated. Appendix D.3 provides a detailed explanation of the training process for the UMAze environments. Now, the goal
of this IL task (named PointUMaze-to-Ant) is to solve the AntUMaze problem in the target domain with its own dynamics based on the point robot demonstrations.
Source domain: Point robot
Target domain: Ant robot
Tim
Figure 8: Sample images of U-Figure 9: Results on PointUMazemaze to-Ant
Fig. 9 shows the average episodic return over 5 seeds. As shown in Fig. 9, the ant robot trained with D3IL reaches the goal position after about 1M timesteps, while the ant robot trained with the baseline IL algorithms failed to reach the goal position. D3IL can train the ant agent to learn desirable behaviors from point-robot expert demonstrations with totally different dynamics. We observed that SAC with reward could not solve this task. So, it is difficult to directly solve the AntUMaze with RL or to obtain expert demonstrations in AntUMaze directly. D3IL can transfer expert knowledge from an easier task to a harder task, and train the agent in the target domain to perform the AntUMaze task.
6.7 Analysis of Dual Cycle-Consistency
To analyze the impact of Dual-CC, we use t-SNE to visualize behavior features generated by our model. Fig. 10 shows the t-SNE plots of the behavior feature extracted by the model trained on the RE2-to-three task: We used the model with full Dual-CC in Fig. 10 (a), the model with only Feature-CC in Fig. 10 (b), and the model with only Image-CC in Fig. 10 (c). Fig. 10 (a) based on the Dual-CC shows the desired properties; target-domain non-expert  $(TN)$  and source-domain non-expert  $(SN)$  are well overlapped, and some of the random behavior feature vectors coincide with the source-domain expert  $(SE)$  behavior. We also plotted behavior features of generated target experts  $(\widehat{TE})$  with green  $+$  in the figure. Since the behavior feature of  $\widehat{TE}$  is derived from the behavior feature of  $SE$  and we imposed the Feature-CC, the red circles and green  $+$  overlap quite a good portion. However, there still exists some gap at a certain region, showing the feature extraction is not perfect. Fig. 10 (c) shows that domain information still remains when removing the Feature-CC, and the Feature-CC indeed helps eliminate domain information. As seen in Fig. 10 (b), only the Feature-CC is not enough to extract the desired behavior feature. Indeed, Dual-CC is essential in extracting domain-independent behavior features.
6.8 Ablation Study
Impact of each loss component D3IL
uses multiple add-ons to the basic adversarial behavior feature learning for consistency check. As described in Sec. 5, the add-ons for better feature extraction are (1) image and feature reconstruction consistency with the generators in the middle part of Fig. 3, (2) Image-CC with the right part of Fig. 3, (3) feature similarity, and (4) Feature-CC. We removed each component from the full algorithm in the order from (4) to (1) and
evaluated the corresponding performance, which is shown in Table 1. In Table 1, each component has a benefit to performance enhancement. Especially, the consistency check for the extracted feature significantly enhances the performance. Indeed, our consistency criterion based on dual feature extraction and image reconstruction is effective.
Figure 10: T-SNE plot of behavior feature from D3IL: (a) Dual-CC (b) Feature-CC only (c) Image-CC only
Reward-generating discriminator  $D_{rew}$  In Sec. 5.4, we explained the use of  $D_{rew}$  for reward estimation. As explained, we train  $D_{rew}$  and the learner policy  $\pi_{\theta}$  in an adversarial manner. Instead of using an external  $D_{rew}$ , we could use the behavior discriminator  $BD_{B}$  in the feature extraction model to train the policy  $\pi_{\theta}$ . To see how this method works, we trained  $\pi_{\theta}$  using the behavior discriminator  $BD_{B}$  in the feature extraction model, not the external  $D_{rew}$ . As shown in Table 2, training  $\pi_{\theta}$  using  $BD_{B}$  is not successful because of the reason mentioned in Sec. 5.4.
Table 1: Impact of each loss component
RE3-to-two
basic adversarial learning	-10.7 ± 0.5
+ image/feature reconstr. consistency	-6.0 ± 0.7
+ image-level cycle-consistency	-3.0 ± 0.1
+ feature similarity	-2.8 ± 0.1
+ feature-level cycle-consistency	-2.6 ± 0.1
Table 2: Reward-generating discriminator:  $D_{rew}$  versus  $BD_{B}$
IP-to-color	RE2-to-tilted
Use Drew	1000 ± 0	-4.3 ± 0.3
Use BDB	293 ± 324	-11.9 ± 0.8
Domain encoders As described in Sec. 5, we proposed employing both domain encoder and behavior encoder for enhanced domain-independent behavior feature extraction. This approach was motivated by the observation that solely relying on a behavior encoder was insufficient to eliminate domain-related information from behavior features. To investigate the importance of domain encoders, we evaluated the performance of D3IL with and without domain encoders  $(DE_{S}$  and  $DE_{T})$  and their corresponding discriminators  $(DD_{D}$  and  $BD_{D})$  across four tasks: IP-to-two, RE2-to-three, RE3-to-tilted, and CartPoleBalance-to-Pendulum. All other networks and losses remain unchanged. Fig. 11 shows the results. The results demonstrate the benefit of conditioning generators on learned domain encoders to boost model performance.
Figure 11: Ablation study on domain encoders
In this paper, we have considered the problem of domain-adaptive IL with visual observation. We have proposed a new architecture based on dual feature extraction and image reconstruction in order to improve behavior feature extraction, and a reward generation procedure suited to the proposed model. The proposed architecture imposes additional consistency criteria on feature extraction in addition to basic adversarial behavior feature extraction to obtain better behavior features. Numerical results show that D3IL yields superior performance to existing methods in IL tasks with various domain differences and is useful when direct RL in the target domain is difficult.
Limitations While our method demonstrates superior performance, it also has some limitations. One is the tricky tuning of loss coefficients, which involves several loss functions. However, we observed that balancing the scale of loss components is sufficient. Additionally, our feature extraction model remains fixed during policy updates. Future works could leverage the past target learner experiences to update the model further. Another limitation is the need for target environment interaction during training, a characteristic shared by several domain-adaptive IL approaches. It will be interesting to extend our method to an offline approach in future works. Moreover, our method lacks a quantitative measure for domain shifts. Future research could develop a mathematical way to quantify domain shifts, enabling the measurement of task difficulty and addressing more complex IL problems. Lastly, future works could extend our method to multi-task or multi-modal scenarios.
This work was supported in part by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2022-0-00469, Development of Core Technologies for Task-oriented Reinforcement Learning for Commercialization of Autonomous Drones) and in part by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2022-0-00124, Development of Artificial Intelligence Technology for Self-Improving Competency-Aware Learning Capabilities). Prof. Seungyul Han is currently with Artificial Intelligence Graduate School of UNIST, and his work is partly supported by Artificial Intelligence Graduate School support (UNIST), IITP grant funded by the Korea government (MSIT) (No.2020-0-01336).
[1] Abbeel, P. and Ng, A. Y. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 1, 2004.
[2] Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein gan. arxiv 2017. arXiv preprint arXiv:1701.07875, 30:4, 2017.
[3] Bain, M. and Sammut, C. A framework for behavioural cloning. In Machine Intelligence 15, Intelligent Agents [St. Catherine's College, Oxford, July 1995], pp. 103-129, GBR, 1999. Oxford University. ISBN 0198538677.
[4] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
[5] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym, 2016.
[6] Cetin, E. and Celiktutan, O. Domain-robust visual imitation learning with mutual information constraints. arXiv preprint arXiv:2103.05079, 2021.
[7] Chae, J., Han, S., Jung, W., Cho, M., Choi, S., and Sung, Y. Robust imitation learning against variations in environment dynamics. In International Conference on Machine Learning, pp. 2828-2852. PMLR, 2022.
[8] Edwards, A., Sahni, H., Schroecker, Y., and Isbell, C. Imitating latent policies from observation. In International conference on machine learning, pp. 1755-1763. PMLR, 2019.
[9] Fickinger, A., Cohen, S., Russell, S., and Amos, B. Cross-domain imitation learning via optimal transport. In 10th International Conference on Learning Representations, ICLR, 2022.
[10] Finn, C., Levine, S., and Abbeel, P. Guided cost learning: Deep inverse optimal control via policy optimization. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML'16, pp. 49-58. JMLR.org, 2016.
[11] Franzmeyer, T., Torr, P., and Henriques, J. a. F. Learn what matters: cross-domain imitation learning with task-relevant embeddings. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 26283-26294. Curran Associates, Inc., 2022.
[12] Fu, J., Luo, K., and Levine, S. Learning robust rewards with adversarial inverse reinforcement learning. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[13] Gamrian, S. and Goldberg, Y. Transfer learning for related reinforcement learning tasks via image-to-image translation. In International Conference on Machine Learning, pp. 2063-2072. PMLR, 2019.
[14] Gangwani, T. and Peng, J. State-only imitation with transition dynamics mismatch. In International Conference on Learning Representations, 2020.
A The Loss Functions for the Proposed Model
In Sec. 5, we provided the key idea of the proposed learning model. Here, we provide a detailed explanation of the loss functions implementing the learning model described in Fig. 3 of the main paper. Table 3 shows the notations for observation sets.
Table 3: The notations for observation sets
Notation
Source domain	OS=OSE∪OSN
Target domain	OT=OTN∪OTL
Expert behavior	OE=OSE
Non-expert behavior	ON=OSN∪OTN
All observations	OALL=OSE∪OSN∪OTN∪OTL
The observation sets  $O_{SE}, O_{SN}, O_{TN}$  and  $O_{TL}$  are defined in Sec. 3 of the main paper.
A.1 Dual Feature Extraction
The left side of Fig. 3 shows the basic adversarial learning block for dual feature extraction. That is, we adopt two encoders in each of the source and target domains: domain encoder and behavior encoder. Behavior encoders  $(BE_{S}, BE_{T})$  learn to obtain behavior feature vectors from images that only preserve the behavioral information and discard the domain information. Here, subscript  $S$  or  $T$  denotes the domain from which each encoder takes input. Domain encoders  $(DE_{S}, DE_{T})$  learn to obtain domain feature vectors which preserve only the domain information and discard the behavioral information. The encoders learn to optimize the WGAN objective [2, 16] based on the feature discriminators: behavior discriminator  $BD_{*}$  and domain discriminator  $DD_{*}$ . Discriminators  $BD_{B}$  and  $DD_{B}$  are associated with the behavior encoder  $BE_{X}$ , where  $X = S$  or  $T$ , and Discriminators  $BD_{D}$  and  $DD_{D}$  are associated with the domain encoder  $DE_{X}$ , where  $X = S$  or  $T$ .  $BD_{B}$  learns to identify the behavioral information from behavior feature vectors, and  $DD_{B}$  learns to identify the domain information from behavior feature vectors.  $BD_{D}$  learns to identify the behavioral information from domain feature vectors, and  $DD_{D}$  learns to identify the domain information from domain feature vectors. For notational simplicity, we define functions  $BE$  and  $DE$  as
$$
B E (o) = \left\{ \begin{array}{l l} B E _ {S} (o) & \text {i f} o \in O _ {S} \\ B E _ {T} (o) & \text {i f} o \in O _ {T} \end{array} \right. \tag {1}
$$
$$
D E (o) = \left\{ \begin{array}{l l} D E _ {S} (o) & \text {i f} o \in O _ {S} \\ D E _ {T} (o) & \text {i f} o \in O _ {T}. \end{array} \right. \tag {2}
$$
Two loss functions are defined for the dual feature extraction: feature prediction loss and feature adversarial loss.
1) Feature prediction loss: This loss makes the model retain the desired information from feature vectors and is given by
$$
\begin{array}{l} L _ {f e a t} ^ {p r e d} = - \left(\mathbb {E} _ {o \in O _ {E}} [ B D _ {B} (B E (o)) ] - \mathbb {E} _ {o \in O _ {N}} [ B D _ {B} (B E (o)) ] \right. \\ \left. + \mathbb {E} _ {o \in O _ {S}} [ D D _ {D} (D E (o)) ] - \mathbb {E} _ {O \in O _ {T}} [ D D _ {D} (D E (o)) ]\right) \tag {3} \\ \end{array}
$$
where  $\mathbb{E}_{o\in A}[\cdot ]$  is the expectation over inputs  $o\in A$ . The first term  $\mathbb{E}_{o\in O_E}[BD_B(BE(o))]$  in the right-hand side (RHS) of (3) means that if the input observation of the encoder  $BE(\cdot)$  is from the expert observation set  $O_{E}$ , then the discriminator  $BD_{B}$  with the extracted feature should declare a high value. The second term  $-\mathbb{E}_{o\in O_N}[BD_B(BE(o))]$  in the right-hand side (RHS) of (3) means that if the input observation of the encoder  $BE(\cdot)$  is from nonexpert observation set  $O_{N}$ , then the discriminator  $BD_{B}$  with the extracted feature should declare a low value. The third term  $\mathbb{E}_{o\in O_S}[DD_D(DE(o))]^\dagger$  in the right-hand side (RHS) of (3) means that if the input observation of the encoder  $DE(\cdot)$  is from the source-domain set  $O_{S}$ , then the discriminator  $DD_{D}$  with the extracted feature should declare a high
value. The fourth term  $-\mathbb{E}_{o\in O_T}[DD_D(DE(o))]'$  in the right-hand side (RHS) of (3) means that if the input observation of the encoder  $DE(\cdot)$  is from the target-domain set  $O_{T}$ , then the discriminator  $DD_{D}$  with the extracted feature should declare a low value. Since we define a loss function, we have the negative sign in front of the RHS of (3). (Note that  $BD_{B}$  learns to assign higher values to expert behavior and to assign lower values to non-expert behavior.  $DD_{D}$  learns to assign higher values to the source domain and to assign lower values to the target domain.) The encoders  $(BE, DE)$  help discriminators  $(BD_{B}, DD_{D})$  to identify the behavioral or domain information from the input, and they jointly learn to minimize  $L_{feat}^{pred}$ .
2) Feature adversarial loss: This loss makes the model delete the undesired information from feature vectors by adversarial learning between the encoders and the discriminators. The loss is given by