SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Each behavior encoder  $(BE_{S}$  or  $BE_{T})$  consists of 6 convolutional layers and a flattened layer. The number of output channels for each convolutional layer is (16, 16, 32, 32, 64, 64), and each channel has  $3\times 3$  size. Zero padding is applied before each convolutional layer, and ReLU activation is applied after each convolutional layer except the last layer. The flattening layer reshapes the input to a one-dimensional vector. The shape of the input is  $(N,W,H,4C)$ , where  $N$  is the minibatch size,  $W$  and  $H$  are the width and height of each image frame, and  $C$  is the number of channels (for RGB images,  $C = 3$ ). The shape of the behavior feature vector is  $(N,(W / 4)\times (H / 4)\times 64)$ .
Each domain encoder  $(DE_S, DE_T)$  consists of 6 convolutional layers, a flattened layer, and a linear layer. The number of output channels for each convolutional layer is (16, 16, 32, 32, 64, 64), and each channel has  $3 \times 3$  size. Zero padding is applied before each convolutional layer, and ReLU activation is applied after each convolutional layer. The shape of the input is  $(N, W, H, C)$ , and the shape of the domain feature vector is  $(N, 8)$ .
Each generator  $(G_S$  or  $G_{T})$  consists of 7 transposed convolutional layers. The number of output channels for each layer is (64, 64, 32, 32, 16, 16,  $4C$ ), and each channel has  $3 \times 3$  size. Zero padding is applied before each layer, and ReLU activation is applied after each layer except the last layer. The input is a behavior feature vector and a domain feature vector. The shape of the output is  $(N, W, H, 4C)$ , which is the same as the shape of the input of behavior encoders.
Each feature discriminator  $(BD_{B},DD_{B},BD_{D},$  or  $DD_{D})$  consists of 2 fully connected layers and a linear layer. Each fully connected layer has 32 hidden units with ReLU activation. Each image discriminator  $(ID_S$  or  $ID_T)$  consists of 6 convolutional layers, a flattened layer, and a linear layer. Each convolutional layer has  $3\times 3$  channel size. The number of output channels for each layer is (16, 16, 32, 32, 64, 64). Zero padding is applied before each layer, and ReLU activation is applied after each layer except on the last layer. The discriminator  $D$  for reward estimation consists of 2 fully connected layers and a linear layer. Each fully connected layer has 100 hidden units with ReLU activation. The input size is  $(N,(W / 4)\times (H / 4)\times 64)$
For the SAC algorithm, each actor and the critic consist of 2 fully connected layers and a linear layer. Each fully connected layer has 256 hidden units with ReLU activation.
D.2 Details for Training Process
Before training the model, we collected expert demonstrations  $O_{SE}$ , and non-expert datasets  $O_{SN}$ ,  $O_{TN}$ .  $O_{SE}$  is obtained by the expert policy  $\pi_E$ , which is trained for 1 million timesteps in the source domain using SAC.  $O_{SN}$  is obtained by a policy taking uniformly random actions in the source domain, and  $O_{TN}$  is obtained by a policy taking uniformly random actions in the target domain. For each IL task, the number of observations (i.e., the number of timesteps) for  $O_{SE}$ ,  $O_{SN}$ ,  $O_{TN}$  is  $n_{demo} = 10000$  except for HalfCheetah-to-locked-legs task, where  $n_{demo} = 20000$  for this task. Each observation consists of 4 RGB images, but the size of these images varies depending on the specific IL task. For IL tasks including IP, IDP, CartPole, and Pendulum, the image size is 32x32. For IL tasks including RE2 and RE3, the image size is 48x48. For IL tasks including HalfCheetah and UMaze, the image size is 64x64. Note that the proposed method does not require a specific input image size.
Our proposed method has two training phases. The first phase updates domain encoders, behavior encoders, generators, feature discriminators, and image discriminators. The second phase updates the discriminator  $D$  for reward estimation and the policy  $\pi_{\theta}$ . In the first phase, we trained the model for  $n_{epoch\_it} = 50000$  epochs for all IL tasks except for HalfCheetah-to-locked-legs task, where  $n_{epoch\_it} = 200000$  for this task. In each epoch, we sampled a minibatch of size 8 from each dataset  $O_{SE}, O_{SN}, O_{TN}, O_{TL}$ . We set  $O_{TL}$  as a copy of  $O_{TN}$ . For coefficients in Eqs. (25) and (26), we set  $\lambda_{feat}^{pred} = \lambda_{feat}^{adv} = 0.01$ ,  $\lambda_{feat}^{reg} = 0.1$ ,  $\lambda_{feat}^{adv} = 1$ ,  $\lambda_{feat}^{sim} = \lambda_{feat}^{recon} = 1000$ ,  $\lambda_{img}^{recon} = \lambda_{img}^{cycle} = 100000$ . We used  $\lambda_{feat}^{cycle} = 10$  for IP-to-color, IDP-to-one, IP-to-two, and PointUMaze-to-ant,  $\lambda_{feat}^{cycle} = 100$
Table 4: Layers for networks in the proposed model. Conv(nc, st, act) denotes a 2D convolutional layer with the number of output channels (nc), stride (st), and activation (act). Conv(nc, st, act) denotes a 2D transposed convolutional layer with the number of output channels (nc), stride (st), and activation (act). FC(nh, act) denotes a fully connected layer with the number of units (nh) and activation (act). Flatten denotes a function that reshapes the input to a one-dimensional vector.
\(B E_{S}, B E_{T}\)	Conv(16, 1, ReLU)Conv(16, 1, ReLU)Conv(32, 2, ReLU)Conv(32, 1, ReLU)Conv(64, 2, ReLU)Conv(64, 1, Linear)Flatten	\(D E_{S}, D E_{T}\)	Conv(16, 1, ReLU)Conv(16, 1, ReLU)Conv(32, 2, ReLU)Conv(32, 1, ReLU)Conv(64, 2, ReLU)Conv(64, 1, ReLU)FlattenFC(8, Linear)
\(B D_{B}, D D_{B}\)	FC(32, ReLU)FC(32, ReLU)FC(1, Linear)	\(B D_{D}, D D_{D}\)	FC(32, ReLU)FC(32, ReLU)FC(1, Linear)
\(G_{S}, G_{T}\)	ConvTranspose(64, 1, ReLU)ConvTranspose(64, 1, ReLU)ConvTranspose(32, 2, ReLU)ConvTranspose(32, 1, ReLU)ConvTranspose(16, 2, ReLU)ConvTranspose(16, 1, ReLU)ConvTranspose(4C, 1, Linear)	\(I D_{S}, I D_{T}\)	Conv(16, 1, ReLU)Conv(16, 1, ReLU)Conv(32, 2, ReLU)Conv(32, 1, ReLU)Conv(64, 2, ReLU)Conv(64, 1, ReLU)FlattenFC(1, Linear)
D	FC(100, ReLU)FC(100, ReLU)FC(1, Linear)
for IDP-to-color,  $\lambda_{\text{feat}}^{\text{cycle}} = 100$  for RE3-to-tilted, and  $\lambda_{\text{feat}}^{\text{cycle}} = 10000$  for RE2-to-tilted, RE3-to-two, RE2-to-three, and HC-to-LF. We used  $\lambda_{\text{feat}}^{\text{cycle}} = 100$  for all environments from DeepMind Control Suite. We chose these coefficients so that the scale of each loss component is balanced, which was observed to be enough to yield good performances of  $\pi_{\theta}$ .
As mentioned in Appendix A.4, we adjusted the values of  $c_{norm,d}$  and  $c_{norm,b}$  in Eq. (23) of the proposed method for the experiment. For IL tasks including IP, IDP, RE2, RE3, and HalfCheetah, we set  $c_{norm,d} = c_{norm,b} = 0$ . For IL tasks including CartPole and Pendulum, we set  $c_{norm,d} = 1$  and  $c_{norm,b} = 20$ . For UMaze task, we set  $c_{norm,d} = 1$  and  $c_{norm,b} = 40$ . The ratio between  $c_{norm,d}$  and  $c_{norm,b}$  is determined based on the input image size since the proposed method does not require a specific input image size and produces behavior features of varying sizes depending on the input image size.
In the second phase, we trained  $D$  for reward estimation and  $\pi_{\theta}$  for  $n_{\text{epoch\_pol}}$  epochs, where  $n_{\text{epoch\_pol}} = 20$  for IP-to-color, IDP-to-one tasks,  $n_{\text{epoch\_pol}} = 30$  for CartPole and Pendulum tasks in DeepMind Control Suite, and  $n_{\text{epoch\_pol}} = 100$  for other IL tasks. Each epoch consists of 10000 timesteps. In each epoch, the following process is repeated. We sampled a trajectory from  $\pi_{\theta}$ , and stored state transitions  $(s, a, s')$  and observations  $o$  to the replay buffer  $B$ . The maximum size of  $B$  is  $n_{\text{buffer}} = 100000$ , which is much smaller compared to that in off-policy RL algorithms because of the large memory consumption when storing images in  $B$ . Then,  $D$  is updated for  $n_{\text{update},D}$  times. We  $n_{\text{update},D}$  to be 50 times smaller than the episode length for each IL task. For example, if the episode length is 1000, then  $n_{\text{update},D} = 20$ . For every  $D$  update, we sampled a minibatch of size 128 from each dataset  $O_{SE}$  and  $O_{TN}$ . For  $O_{TL}$ , we sampled a minibatch of size 64 from  $B$  and a minibatch of size 64 from  $O_{TN}$ . After  $D$  update,  $\pi_{\theta}$  is updated for  $n_{\text{update},\theta}$  times using SAC [17]. We set  $n_{\text{update},\theta}$  to be equal to the episode length for each IL task. For example, if the episode length is 1000, then  $n_{\text{update},\theta} = 1000$ . For every  $\pi_{\theta}$  update, we sampled a minibatch of size 256 from  $B$ .
In each epoch, we evaluated  $\pi_{\theta}$  after updating  $D$  and  $\pi_{\theta}$ . We sampled  $n_{eval}$  trajectories using  $\pi_{\theta}$  and computed the average return, where each return is computed based on the true reward in the target domain. We set  $n_{eval} = 10$  as default. For IL tasks including Reacher environments, we set
$n_{eval} = 200$  to reduce variance because each return highly depends on the goal position in Reacher environments. We used Adam optimizer [23] for optimizing all networks. We set the learning rate  $lr = 0.001$  and momentum parameters  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.999$  for encoders, generators, and discriminators. We set  $lr = 0.0003$ ,  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.999$  for the actor and critic. For SAC, we set the discount factor  $\gamma = 0.99$ , the parameter  $\rho = 0.995$  for Polyak averaging the Q-network, and the entropy coefficient  $\alpha = 0.1$ .
The proposed method is implemented on TensorFlow 2.0 with CUDA 10.0, and we used two Intel Xeon CPUs and a TITAN Xp GPU as the main computing resources. One can use a GeForce RTX 3090 GPU instead as the computing resource, which requires TensorFlow 2.5 and CUDA 11.4. Based on  $32 \times 32$  images, The GPU memory consumption is about 2.5GB for the training feature extraction model and about 1.5GB during policy update. Note that these highly depend on the image size and the batch size. Our D3IL is implemented based on the code provided by the authors of [6].
D.3 Dataset and Training Process for PointUMaze-to-Ant Task
For the UMaze environments, the number of observations for  $O_{SE}, O_{SN}$  or  $O_{TN}$  is  $n_{demo} = 10000$ . Each observation consists of 4 frames and each frame is a 64x64 RGB image. During the training, the reward that the agent receives for each timestep is  $r_{total} = c \times (r_{IL} + \mathbf{1}_{reach\_goal} \times r_{goal})$ , where  $r_{IL}$  is the estimated reward given by  $D$ ;  $\mathbf{1}_{reach\_goal}$  is 1 if the robot reaches the goal and 0 otherwise;  $r_{goal}$  is the reward when the robot reaches the goal; and  $c > 0$  is a scaling coefficient. In the experiment, we chose  $c = 100$  and  $r_{goal} = 1$ , and these parameters are applied to all baselines. For the proposed method, the number of epochs, minibatch sizes, and network structures for the training feature extraction network and image reconstruction network are the same as those in Appendix D.1. The policy is trained for  $n_{epoch\_pol} = 200$  and each epoch consists of 10000 timesteps, so the total number of timesteps for policy training is 2,000,000.
Also, we trained an agent using a vanilla RL method (SAC [17]) with access to true rewards, to show that it is difficult to obtain expert demonstrations in the target domain. For SAC, the agent receives true rewards. For true rewards, the agent receives a penalty of -0.0001 for each timestep and receives 1000 when the robot reaches the goal, so the maximum true episodic return is 1000.
E RL Environment Settings and Sample Image Observations
We first describe the base RL tasks in the Gym.
Inverted pendulum (IP): The agent moves a cart to prevent the pole from falling. The episode length is 1000. Unlike the IP task in Gym [4], an episode does not terminate midway even if the pole falls. This makes the task much more challenging because each episode provides less useful observations to solve the task.
Inverted double pendulum (IDP): This is a harder version of the IP task where there are two poles on the cart.
Reacher-two (RE2): The agent moves a two-link arm with one end fixed so that the movable endpoint of the arm reaches the target point. The episode length is 50. Unlike the Reacher task in Gym, we narrowed the goal point candidates to 16 points, where the polar coordinate  $(r,\varphi)$  of the target point can be  $r\in \{0.15,0.2\}$ , and  $\varphi \in \{0,\pi /4,\pi /2,3\pi /4,\dots ,7\pi /4\}$ .
Reacher-three (RE3): This is a harder version of the RE2 task where the agent moves a three-link armed robot.
Note that our IL tasks are more difficult than those considered in the experiments in the baseline algorithm references [6, 37]. Our simulation setting has an episode length of 1000 for the IP and IDP tasks instead of 50. The episode horizon is also a component of an MDP, and the observation space of the MDP changes as the episode horizon changes. MDP with a longer episode length can yield more various observations which are much different from the initial one, so the observation gap between the source domain and the target domain increases. We meant this for a bigger domain gap.
If the goal position is fixed throughout the entire learning phase in RE tasks, as designed in [6, 37], distinguishing between expert and non-expert behavior solely depends on the arm position. However, in our implementation, the goal position is randomly chosen from 16 positions for each episode. This makes distinguishing expert and non-expert behavior much more challenging because the same arm movement can be considered expert or non-expert behavior depending on the goal position. Therefore, extracting proper behavior features is much more difficult as the feature extraction model should catch the relationship between the arm movement and the goal position.
Figure 13: Example observations for IP, IDP, RE2, and RE3 environment
The following images are sample observations for IL tasks with changing a robot's degree of freedom.
Figure 14: Example image pairs for IL tasks with changing DOF
In the HalfCheetah (HC) environment, the agent controls an animal-like 2D robot. The goal is to move the animal-like 2D robot forward as fast as possible by controlling 6 joints. Similarly to [6], we have a modification with immobilized feet, and this modified environment is 'HalfCheetah-Locked-Feet (HC-LF)'. In the HC-LF environment, the robot can use only 4 joints instead of 6, where the immobilized feet are colored in red.
Figure 15: Example observations of HalfCheetah environment
The following images are sample image observations of environments in the DeepMind Control Suite (DMCS). We use CartPole-Balance, CartPole-SwingUp, and Pendulum environments from DMCS. The goal for all environments is to keep the pole upright.
CartPole-Balance: The pole starts in an upright position and the agent should move the cart to keep the pole upright while avoiding exceeding the boundaries along the x-axis.
CartPole-Swingup: The pole starts in a downward position and the agent should move the cart to swing the pole and keep the pole in an upright position while avoiding exceeding the boundaries along the x-axis.
Pendulum: The agent should add torque on the center to swing the pole and keep the pole upright.
Domain adaptation between these tasks is challenging because not only the embodiment between agents in both domains are different but also the distributions of expert and non-expert demonstrations, and initial observations for both domains are quite different.
Figure 16: Example observations of environments in DMCS
We omit sample image observations of PointUMaze and AntUMaze environments because they are already provided in the main paper. Table 5 shows the dimensions of state space, action space, and visual observation for RL tasks used in the Experiment section.
Table 5: State dimension, action dimension, and visual observation image size for RL tasks used in the Experiments section. Each visual observation input consists of 4 RGB images, and the rightmost column shows the size (width x height) of each image.
RL Task	State dimension	Action dimension	Visual observation image size
Inverted Pendulum (IP)	4	1	32x32
Inverted Double Pendulum (IDP)	11	1	32x32
Reacher-two (RE2)	11	2	48x48
Reacher-three (RE3)	14	3	48x48
HalfCheetah (HC)	17	6	64x64
HalfCheetah-Locked-Feet (HC-LF)	13	4	64x64
Cartpole	5	1	32x32
Pendulum	3	1	32x32
PointUMaze	7	2	64x64
AntUMaze	30	8	64x64
F.1 Results on IL Tasks with Changing Visual Effect
We evaluated D3IL on tasks where the domain difference is caused by visual effects on image observations on four IL tasks: IP and IDP, with different color combinations of the pole and cart (IP-to-colored and IDP-to-colored tasks), and RE2 and RE3, with different camera angles (RE2-to-tilted and RE3-to-tilted tasks). Sample image observations are provided in Figure 17. For the IP and IDP tasks, the primary changes occur in the pixel values of the pole and the cart. Meanwhile, for the RE2 and RE3 tasks, major changes are observed in the pixel values of the background outside of the arms and the goal position. The average episodic return for D3IL and the baselines over 5 seeds are shown in Fig. 18. We only included baselines that receive demonstrations as images because changing visual effects on image observations do not affect the true state space and the action space. As shown in Fig. 18, D3IL outperformed the baselines with large margins on IL tasks with changing visual effects on image observations.
IP-to-color
Figure 17: Example image pairs for IL tasks with changing visual effects
IDP-to-color
RE2-to-tilted
RE3-to-tilted
(a) IP-to-color
Figure 18: The learning curves in the target domain of the IL tasks with visual effects
(b) IDP-to-color
(c) RE2-to-tilted
(d) RE3-to-tilted
F.2 The Evaluation on the Feature Quality
Fig. 19 shows some instances of the estimated rewards from sampled observations in the RE2-to-three task, corresponding to the time of 150,000 timesteps in Fig. 5d. The left column of Fig. 19 is non-expert behavior and the right column is expert behavior. One side of the Reacher arm is fixed at the center of the image. In the left column, the movable arm end moves away from the goal position through four time-consecutive images, whereas the movable arm end moves towards the goal position in the right column. Note that in these instances, the baseline algorithm assigns a higher reward to the non-expert behavior, whereas the proposed method correctly assigns a higher reward to the expert behavior. We conjecture that this wrong reward estimation of the baseline algorithms based on poor features is the cause of the gradual performance degradation of the baseline algorithms in Fig. 5.
In Fig. 20, we checked the quality of extracted features from a conventional method, TPIL, and that of our method. Since TPIL does not consider image reconstruction but considers only the extraction of domain-independent behavior features. In order to generate an image from the domain-independent behavior feature of TPIL, along with TPIL we also trained two generators  $(G_{S}, G_{T})$ :  $G_{S}$  takes a behavior feature vector as input and generates an image that resembles the images from the source domain.  $G_{T}$  takes a behavior feature vector as input and generates an image that resembles the images from the target domain. This  $G_{S}$  and  $G_{T}$  training is done with two discriminators  $(ID_{S}$  and  $ID_{T})$ , where  $ID_{S}$  distinguishes the real and generated images in the source domain, and  $ID_{T}$  does so in the target domain. As shown in Fig. 20, the reconstructed image from the behavior feature of TPIL is severely broken for the RE2-to-tilted task. Thus, it shows that the "domain-independent" behavior feature of TPIL does not preserve all necessary information of the original observation image. On the other hand, our method based on dual feature extraction and image reconstruction, yields image reconstruction looking like ones. This implies that the features extracted from our method well preserve the necessary information, and thus reconstruction is well done.
Figure 19: RE2-to-three: Instances of estimated rewards. Four time-consecutive images comprise one observation. The estimated reward is shown in the fourth image.
Figure 20: Examples of real and generated images. The generated images are generated from the feature vectors obtained by conventional TPIL and our proposed method. Here we used IP-to-two and RE2-to-three task, where each task is explained in Sec. 6.1.
F.3 Experimental Results on Simpler Tasks
In the main result subsection of Sec. 6, we mentioned that the baseline algorithms perform well in the case of easier settings. As a preliminary experiment, we verify the performance of the proposed method and the baseline IL algorithms on simpler IL tasks, which are easier variants of the IL tasks shown in Sections 6.2 and 6.3. We first describe the base RL tasks.
Inverted pendulum - short (IPshort): This is the same task as the IP task, but the episode length is 50.
Inverted double pendulum - short (IDPshort): This is the same task as the IDP task, but the episode length is 50.
Reacher-two - one goal (RE2onegoal): This is the same task as the RE2 task, but the goal point is fixed to  $(x,y) = (1.0,1.0)$  for the entire training.
Reacher-three - one goal (RE3onegoal): This is the same task as the RE3 task, but the goal point is fixed to  $(x,y) = (1.0,1.0)$  for the entire training.
Sample visual observations for each IL task are provided in Fig. 21. We followed the training process as described in [6]. For each IL task, we collected  $O_{SE}$ ,  $O_{SN}$  and  $O_{TN}$  of size 10000. The learner policy  $\pi_{\theta}$  is trained for 20 epochs. Each epoch has 1000 timesteps, so the total number of timesteps is 20000. The size of the replay buffer is 10000. We used TPIL [37] and DisentanGAIL [6] as baseline algorithms. For each IL task, we trained the learner policy using our proposed method and baseline algorithms over 5 seeds. For evaluation, we generated 5 trajectories from  $\pi_{\theta}$  for every epoch and computed the average return, where the return was computed based on the true reward in the target domain.
Fig. 22 shows the average episodic return. As shown in Fig. 22, the baseline algorithms perform well in these easy IL tasks. The IL tasks shown in Fig. 21 are much easier to solve than the IL tasks in Sections 6.2 and 6.3 because the pole needs to be straight up for only 50 timesteps instead of 1000 timesteps in IPshort and IDPshort, and the goal position is fixed to a single point instead of randomly sampled from 16 candidates in RE2ongoal and RE3onegoal. The baselines can achieve good performance on these easy IL tasks. However, as shown in Fig. 5 of the main paper and Fig. 18 in Appendix 6.2, the baseline algorithm yields poor performance on hard IL tasks and the proposed method achieves superior performance on hard IL tasks to the baselines due to the high-quality feature extraction by the dual feature extraction and image reconstruction.
Figure 21: Example observation pairs for each IL task. For each pair of images, the left-hand side is the image in the source domain, and the right-hand side is the image in the target domain.
Figure 22: The learning curves for the simpler IL tasks in the target domain
F.4 Experiments Results with Single Expert Trajectory during Policy Update