SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sungho Choi $^{1}$   
Jongseong Chae $^{1}$
Seungyul Han $^{2*}$   
Whiyoung Jung $^{4}$
Woojun Kim<sup>3</sup>  
Youngchul Sung<sup>1</sup>
$^{1}$ KAIST  $^{2}$ UNIST  $^{3}$ Carnegie Mellon University  $^{4}$ LG AI Research  $\{sungho.choi,jongseong.chae,ycsung\} @$ kaist.ac.kr, syhan@unist.ac.kr, woojunk@andrew.cmu.edu, whiyoung.jung@lgresearch.ai
In this paper, we consider domain-adaptive imitation learning with visual observation, where an agent in a target domain learns to perform a task by observing expert demonstrations in a source domain. Domain adaptive imitation learning arises in practical scenarios where a robot, receiving visual sensory data, needs to mimic movements by visually observing other robots from different angles or observing robots of different shapes. To overcome the domain shift in cross-domain imitation learning with visual observation, we propose a novel framework for extracting domain-independent behavioral features from input observations that can be used to train the learner, based on dual feature extraction and image reconstruction. Empirical results demonstrate that our approach outperforms previous algorithms for imitation learning from visual observation with domain shift.
Imitation learning (IL) is a framework where an agent learns behavior by mimicking an expert's demonstration without access to true rewards. The mainstream IL methods include Behavior Cloning (BC), which trains the learner with supervised learning [3, 31, 33, 39], Inverse Reinforcement Learning (IRL), which tries to find the reward function [1, 29, 49], and Adversarial Imitation Learning (AIL), which trains the learner to match its state-action visitation distribution to that of the expert via adversarial learning [10, 12, 18, 40, 46]. IL provides an effective alternative to solving reinforcement learning (RL) problems because it circumvents the difficulty in the careful design of a reward function for intended behavior [30, 47]. Despite the effectiveness, conventional IL assumes that the expert and the learner lie in the same domain, but this is a limited assumption in the real world. For example, a self-driving car should learn real-world driving skills from demonstrations in a driving simulator. In such a case, the expert and the learner do not lie in the same domain.
In this paper, we consider the challenge of domain shift in IL, i.e., an agent in a target domain learns to perform a task based on an expert's demonstration in a source domain. In particular, we focus on the case where the expert demonstrations are provided in a visual form. In many practical cases, obtaining expert demonstrations as an explicit sequence of states (such as positions and velocities) and actions (such as movement angles) can be challenging. Instead, the expert's demonstrations are often provided as image sequences. One example can be a robot mimicking a movement by visually observing human behavior. However, learning from visual observations poses a major challenge in IL since images are high-dimensional and often include only partial information about the expert's behavior or information unrelated to the expert's behavior. Even minor changes between the source and target domains can vastly affect the agent's policy and lead to unstable learning. In recent years,
several works aimed to address the challenge of domain shift in IL. Some methods show impressive performance by training a model to learn a mapping between domains [22, 32], requiring proxy tasks (i.e., additional tasks to assist in learning the target task) to provide expert demonstrations in both source and target domains. However, this approach assumes expert demonstrations for similar tasks in the target domain, which may not be practical in many realistic domain transfers where expert demonstrations in the target domain are unavailable. Other methods generate metric-based rewards that minimize the distance between the expert and the learner trajectories [9, 27], assuming the expert's state is directly available rather than requiring image observations. Our focus is on solving domain shifts with visual observations without the use of proxy tasks or direct access to expert's states. Our work is closely related to the methods that learn a model to extract domain-invariant features [6, 37]. However, removing domain information was not satisfactory in situations with a large domain gap. In contrast, we propose a novel learning method to train the learner in domain-adaptive IL with visual observation, achieving significantly improved performance in IL tasks with domain shift.
IL with domain shift IL with domain shift is a challenging problem where the learner should learn from an expert in a source domain and apply the learned policy in a target domain. In recent years, there has been significant progress in addressing this problem in machine learning, RL, and robotics [6, 7, 9, 11, 14, 22, 27, 28, 32, 34-37, 45]. For example, Fickinger et al. [9] proposed utilizing Gromov-Wasserstein distance to generate reward minimizing the difference between expert and learner policies. Franzmeyer et al. [11] learned a mapping between two domains, where expert embeddings are reduced by selecting task-relevant state elements based on mutual information criterion. These methods assume to have access expert's states; in contrast, our method assumes expert demonstrations are only available as high-dimensional image sequences with partial information about the expert. Kim et al. [22] and Raychaudhuri et al. [32] trained a model that can map between two domains by using expert demonstrations for proxy tasks on both domains. In comparison, our method does not rely on expert demonstrations of proxy tasks. Sermanet et al. [34] employed a model that learns representations from visual demonstrations of different viewpoints using a time-contrastive approach. This approach mainly focuses on viewpoint mismatches between domains and relies on time-synchronized demonstrations with simultaneous viewpoints. Chae et al. [7] introduced an IL algorithm designed to train a policy that exhibits robustness against variations in environment dynamics by imitating multiple experts. In contrast, our method encompasses general domain mismatches including the differences in viewpoint, degree of freedom, dynamics, and embodiment.
We focus on solving domain shift IL with image observation. Liu et al. [28] used time-aligned expert demonstrations in multiple source domains and learned a model to transform the demonstration from one domain into another. Our work is closely related to Third-Person Imitation Learning (TPIL) [37], which trains a model based on domain-independent feature extraction [15] with GAIL [18]. Similarly, Cetin & Celiktutan [6] proposed restricting mutual information between non-expert data and domain labels. Our method improves on the idea of feature extraction from non-time-aligned visual data by applying dual feature extraction and image reconstruction with consistency checks.
IL from observations (IfO) IfO is an area of IL, where the expert state-action trajectory is not provided explicitly. In IfO, the learner should capture the behavioral information from observation. Demonstrations can be provided either as a set of state vectors without actions [8, 39] or simply as image observations [13, 21, 28, 43]. Torabi et al. [39] learned the inverse dynamics to infer the missing actions from observations in the demonstration. Edwards et al. [8] used latent actions to model the transition between two consecutive observations and relabel those latent actions to true actions. We assume that the learner knows its own state and action but expert demonstration is provided only in the form of visual observations as in other works ([6, 37]).
3 System Model
Setup An Markov Decision Process (MDP) is described by a tuple  $(S, \mathcal{A}, P, R, \gamma, \rho_0)$ , where  $S$  is the state space,  $\mathcal{A}$  is the action space,  $P: S \times \mathcal{A} \times S \to \mathbb{R}^+$  is the state transition probability,  $R: S \times \mathcal{A} \to \mathbb{R}$  is the reward function,  $\gamma \in (0,1)$  is the discount factor, and  $\rho_0$  is the initial state distribution [38]. Given an MDP, a stochastic policy  $\pi: S \times \mathcal{A} \to \mathbb{R}^+$  is a conditional probability over actions given state  $s \in S$ . The goal of RL is to find an optimal policy  $\pi^*$  that maximizes
$J(\pi) = \mathbb{E}_{\pi}[\sum_{t=0}^{\infty} \gamma^{t} R(s_{t}, a_{t})]$ , where  $s_{0} \sim \rho_{0}, a_{t} \sim \pi(\cdot | s_{t}), s_{t+1} \sim P(\cdot | s_{t}, a_{t})$ . In IL, expert demonstrations are provided instead of the true  $R$ . The goal of IL with a single domain is to imitate the demonstration generated by an expert policy  $\pi_{E}$ .
Problem formulation We now define IL with domain adaptation. We have two domains: source domain and target domain. The source domain is where we can obtain expert demonstrations, and the target domain is where we train the agent. Both domains are modeled as MDPs. The MDPs of the source domain (using subscript  $S$ ) and the target domain (using subscript  $T$ ) are given by  $\mathcal{M}_S = (\mathcal{S}_S, \mathcal{A}_S, P_S, R_S, \gamma_S, \rho_{0,S})$  and  $\mathcal{M}_T = (\mathcal{S}_T, \mathcal{A}_T, P_T, R_T, \gamma_T, \rho_{0,T})$ , respectively. An expert is the one who performs a task optimally. The agent in the target domain learns the task from expert demonstrations in the source domain. We call the agent a learner. The expert policy  $\pi_E : S_S \times \mathcal{A}_S \to \mathbb{R}^+$  is in the source domain and the learner policy  $\pi_\theta : S_T \times \mathcal{A}_T \to \mathbb{R}^+$  is in the target domain. We assume that a set of image observations of  $\pi_E$  are provided, which is denoted by  $O_{SE}$ , where 'SE' refers to 'source expert'. The set of observations generated by  $\pi_\theta$  is denoted by  $O_{TL}$ , where 'TL' refers to 'target learner'. In addition to expert demonstrations from the source domain, we assume access to the non-optimal trajectories of image observations from both the source and target domains. The visual observations of rollouts from a non-expert policy  $\pi_{SN} : S_S \times \mathcal{A}_S \to \mathbb{R}^+$  in the source domain are denoted by  $O_{SN}$ , and the visual observations of rollouts from a non-expert policy  $\pi_{TN} : S_T \times \mathcal{A}_T \to \mathbb{R}^+$  in the target domain are denoted by  $O_{TN}$ . Here, 'SN' refers to 'source non-expert', and 'TN' refers to 'target non-expert'.  $O_{SN}$  and  $O_{TN}$  are additional data to assist our feature extraction and image reconstruction process.  $\pi_{SN}$  and  $\pi_{TN}$  can be constructed in several ways, but we choose these policies as ones taking uniformly random actions. Each observation set has size  $n_{demo}$  and each observation  $o_t$  consists of 4 images corresponding to 4 consecutive timesteps. The goal is to train  $\pi_\theta$  to solve a given task in the target domain using  $O_{SE}$ ,  $O_{SN}$ ,  $O_{TN}$ , and  $O_{TL}$ .
4 Motivation
IL with domain adaptation presents a major challenge as the learner cannot directly mimic the expert demonstration, especially when it is given in a visual form. Also, we cannot leverage expert demonstrations for proxy tasks. To overcome domain shift, conventional methods [6, 37] extracts domain-invariant features from images. We call such feature behavior feature, capturing only expert or non-expert behavior information. One example is TPIL [37]. TPIL consists of a single behavior encoder  $BE$ , a domain discriminator  $DD$ , and a behavior discriminator  $BD$ , as shown in Fig. 1 (The label  $XB_{X}$  in Fig. 1 means that the input is in the  $X$  domain with behavior  $B_{X}$ .  $X$  can be source  $S$  or target  $T$ , and  $B_{X}$  can be expert  $E$  or non-expert  $N$ . We use this notation throughout the paper. See Appendix C for more details). The key idea is that  $BE$  learns to fool  $DD$  by removing domain information from the input while helping  $BD$  by preserving behavior information from the input.
Figure 1: Basic structure for domain-independent behavior feature extraction:  $BE$  - behavior encoder,  $BD$  - behavior discriminator,  $DD$  - domain discriminator
Figure 2: T-SNE plots of features from (a) TPIL and (b) DisentanGAIL. Each point represents a behavior feature: Red circle  $(SE)$ , orange square  $(SN)$  and blue cross  $(TN)$
However, the behavior feature extraction of the conventional methods does not work to our satisfaction. Fig. 2 shows the t-SNE [42] plots of the behavior features extracted by the conventional methods [6, 37] after training for the task of Reacher with two links (source domain) to three links (target domain). As seen in the figure, the behavior feature vectors of  $SN$  (orange squares) and those of  $TN$  (blue crosses) are separated in the left figure (note that when domain information is completely
Figure 3: The structure of the learning model: Inputs in source and target domains have features  $SB_{S}$  and  $TB_{T}$ , respectively.  $B_{S}$  can be  $E$  (expert) and  $N$  (non-expert), and  $B_{T}$  can be  $N$  (non-expert) and  $L$  (learner).  $DE$  is domain encoder (blue),  $BE$  is behavior encoder (orange) and  $G$  is generator (green).  $DD, BD$  and  $ID$  with gray color are domain discriminator, behavior discriminator, and image discriminator, respectively. Small long rectangles represent features. The sub-block in the blue dotted line in the lower left corner corresponds to the basic structure shown in Fig. 1.
removed, the behavior feature vectors of  $SN$  and  $TN$  should overlap). Furthermore, since the target non-expert performs random actions for each given state, some actions resemble the expert actions in a given state, while other actions do not. So, there should be some overlap between the blue crosses  $(TN)$  and the red dots  $(SE)$ . However, the blue crosses are well separated from the red circles in the right figure. Thus, the domain information is not properly removed from the behavior feature. Extracting domain-independent behavior features needs to fully exploit available source-domain and target-domain data.
Our approach is to embed the basic feature extraction structure into a bigger model with data translation. Our main idea for this is dual feature extraction and dual cycle-consistency. First, we propose a dual feature extraction scheme to extract both the domain feature and behavior feature from the input so that these two features are independent and contain full information about the input. Second, we propose a new cycle-consistency scheme that fits domain-adaptive IL with visual observation. The proposed model aims to extract better features for reward generation rather than to generate images. Thus, solely applying image-level cycle-consistency is not sufficient to solve the problem of our interest. We circumvent this limitation by additionally applying feature-level cycle-consistency on top of image-level cycle-consistency. Third, we adopt a reward-generating discriminator outside the feature extraction model for proper reward generation by exploiting the generated target-domain expert data from the feature extraction model. Therefore, we name our approach as D3IL (Dual feature extraction and Dual cycle-consistency for Domain adaptive IL with visual observation).
5 Proposed Methodology
5.1 Dual Feature Extraction
The behavior feature in the target domain is the required information that tells whether the learner's action is good or not, in other words, whether the learner in the target domain successfully mimics the expert in the source domain or not. To improve behavior feature extraction, we boost the simple adversarial learning, as the structure shown in Fig. 1, by adding an additional structure from which we can check that the obtained feature well contains the required information without information loss, based on image reconstruction and consistency check, as shown in Fig. 3. That
is, we adopt two encoders in each of the source and target domains: domain encoder  $DE_{X}$  and behavior encoder  $BE_{X}$ , where  $X = S$  or  $T$ , as shown in the left side of Fig. 3. The domain encoder  $DE_{X}$ , where  $X = S$  or  $T$ , tries to extract domain feature ( $S$  or  $T$ ) that contains only the domain information excluding behavioral information, whereas the behavior encoder  $BE_{X}$  tries to extract behavioral feature ( $B_{X} = E$  or  $N$ ) that contains only the behavioral information (expert or non-expert) excluding domain information. Each of these encodings is based on the aforementioned basic adversarial learning block composed of one encoder and two accompanying discriminators: domain and behavior discriminators, as shown in the left side of Fig. 3. As the outputs of the four encoders, we have four output features:  $S$  and  $B_{S}$  from the source-domain encoders, and  $T$  and  $B_{T}$  from the target-domain encoders.
5.2 Dual Cycle-Consistency (Dual-CC)
We now explain dual cycle-consistency (Dual-CC), composed of image-level cycle-consistency (Image-CC) and feature-level cycle-consistency (Feature-CC) to improve feature extraction. Using the four output feature vectors extracted by the encoders in Sec. 5.1, we apply image translation [19, 25, 26, 44, 48]. That is, we combine  $T$  and  $B_{S}$  and generate an image  $\widehat{TB_S}$  by using generator  $G_{T}$ , and combine  $S$  and  $B_{T}$  and generate an image  $\widehat{SB_T}$  by using generator  $G_{S}$ . Each image generator  $(G_{T}$  or  $G_{S})$  is trained based on adversarial learning with the help of the corresponding image discriminator  $(ID_{T}$  or  $ID_{S})$ , as shown in the middle part of Fig. 3. Then, from the generated images  $\widehat{TB_S}$  and image  $\widehat{SB_T}$ , we apply feature extraction again by using the same trained encoders  $DE_{T}, BE_{T}, DE_{S}, BE_{S}$  used in the first-stage feature extraction, as shown in the right side of Fig. 3. Finally, using the four features  $\widehat{T}, \widehat{B}_{S}, \widehat{S}, \widehat{B}_{T}$  from the second-stage feature extraction, we do apply image translation again to reconstruct the original images  $\widehat{SB_S}$  and  $\widehat{TB_T}$  by using the same generators  $G_{S}$  and  $G_{T}$  used as the first-stage image generators, as shown in the rightmost side of Fig. 3. Note that if the encoders properly extracted the corresponding features without information loss, the reconstructed images  $\widehat{SB_S}$  and  $\widehat{TB_T}$  should be the same as the original input images  $SB_{S}$  and  $TB_{T}$ . We exploit this Image-CC as one of our criteria for our feature-extracting encoders.
However, the aforementioned adversarial learning and Image-CC loss are insufficient to yield mutually exclusive behavior and domain features, as we will see in Sec. 6.7. To overcome this limitation, we enhance the dual feature extraction by complementing Image-CC by imposing Feature-CC. Note that our goal is not image translation but feature extraction relevant to domain-adaptive IL. The Feature-CC means that the extracted features  $S$ ,  $B_{S}$ ,  $T$  and  $B_{T}$  in the left dual feature extraction part of Fig. 3 should respectively be the same as their corresponding feature vectors  $\hat{S}$ ,  $\hat{B}_{S}$ ,  $\hat{T}$  and  $\hat{B}_{T}$  in the right cycle-consistency check part of Fig. 3. The rationale for this consistency is as follows. Consider  $B_{T}$  and  $\widehat{B_T}$  for example.  $B_{T}$  is extracted from the target-domain image  $TB_{T}$ , whereas  $\widehat{B_T}$  is extracted from a source-domain image  $\widehat{SB_T}$  generated from source-domain image generator  $G_{S}$ . For the two behavior features  $B_{T}$  and  $\widehat{B_T}$  from two different domains to be the same, the domain information in  $B_{T}$  and  $\widehat{B_T}$  should be eliminated. Thus, the Feature-CC helps the extraction of mutually exclusive behavior and domain features while the Image-CC requires the combination of behavior and domain features to preserve full information about the input observation without any loss. Since both consistencies are not perfect, we adopt both Image-CC and Feature-CC for our feature extraction, so named Dual-CC.
5.3 Further Consistency and Stability Enhancement
In addition to the Dual-CC, our architecture allows us to verify other consistencies on the extracted features. First, we enforce the image reconstruction consistency by combining the features  $S$  and  $B_{S}$  from  $DE_{S}$  and  $BE_{S}$  in the first-stage feature extraction and feeding the feature combination  $(S, B_{S})$  into image generator  $G_{S}$ . The output  $\widetilde{SB_{S}}$  should be the same as the original image  $SB_{S}$ . The same consistency applies to the feature combination  $(T, B_{T})$  in the target domain with image generator  $G_{T}$ . Second, we enforce the feature reconstruction consistency by feeding the generated source-domain image  $\widetilde{SB_{S}}$  described above into the encoders  $DE_{S}$  and  $BE_{S}$ , then we obtain domain feature  $\tilde{S}$  and behavior feature  $\tilde{B}_{S}$ . These two features should be identical to the features  $S$  and  $B_{S}$  extracted in the first-stage feature extraction. The same principle applies to the target domain.
Figure 4: Reward-generating discriminator  $D_{rew}$  and its training
In addition to the key consistency idea explained above, we apply further techniques of feature vector regularization and feature vector similarity to enhance performance. Feature vector regularization maintains the  $L_{2}$ -norm of the feature vector which helps to prevent the feature vectors from becoming too large. Feature vector similarity is another form of consistency that guides the model so that the domain feature vectors of observations from the same domain should have similar values and the behavior feature vectors of observations from the same behavior should have similar values.
With this additional guidance for feature extraction on top of the conventional adversarial learning block, the proposed scheme significantly improves feature extraction compared to previous approaches, as we will see in Sec. 6. Once the overall structure is learned, we take the behavior encoder  $BE_{T}$  in the target domain (the block inside the blue dotted block in Fig. 3) to train the learner policy. This improved behavior feature extraction can enhance the learner's performance in the target domain. Note that the complexity of D3IL is simply at the level of widely used image translation.
5.4 Reward Generation and Learner Policy Update
Now, let us explain how to update the learner policy  $\pi_{\theta}$ . One can simply use the behavior discriminator  $BD_{B}$  in the target domain together with the behavior encoder  $BE_{T}$  to generate a reward for policy updates. Instead, we propose a different way of reward generation. We use  $BE_{T}$  but do not use  $BD_{B}$  from the feature extraction model. This is because, in our implementation,  $BD_{B}$  is trained to distinguish the expert behavior  $E$  from the non-expert behavior  $N$ , but what we need is a discriminator that distinguishes the expert behavior  $E$  from the learner behavior  $L$  so that the learner policy  $\pi_{\theta}$  can receive proper rewards for imitation. Therefore, we train another network called reward-generating discriminator  $D_{rew}$  for reward generation.  $D_{rew}$  and  $\pi_{\theta}$  jointly learn in an adversarial manner like in GAIL.  $D_{rew}$  takes a behavior feature vector from  $BE_{T}$  as input and learns to predict the behavior label ( $E$  or  $L$ ) of the input. For stable learning, we consider GAIL with gradient penalty (GP) [24].