SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} L _ {f e a t} ^ {a d v} = \mathbb {E} _ {o \in O _ {S}} [ D D _ {B} (B E (o)) ] - \mathbb {E} _ {o \in O _ {T}} [ D D _ {B} (B E (o)) ] \\ + \mathbb {E} _ {o \in O _ {E}} [ B D _ {D} (D E (o)) ] - \mathbb {E} _ {o \in O _ {N}} [ B D _ {D} (D E (o)) ]. \tag {4} \\ \end{array}
$$
In this loss function, the output of the behavior encoder is fed to the domain discriminator, and the output of the domain encoder is fed to the behavior discriminator. The learning of the encoders and the discriminators is done in an adversarial manner as
$$
\min  _ {B E, D E} \max  _ {D D _ {*}, B D _ {*}} L _ {\text {f e a t}} ^ {\text {a d v}}. \tag {5}
$$
Recall that  $BD_{*}$  learns to assign higher values to expert behavior and to assign lower values to non-expert behavior, and  $DD_{*}$  learns to assign higher values to the source domain and to assign lower values to the target domain. Thus, the discriminators try to do their best, whereas the encoders try to fool the discriminators.
The first two terms in the RHS of (4) imply that the output of the behavior encoder  $BE$  should not contain the domain information. The last two terms in the RHS of (4) imply that the output of the domain encoder  $DE$  should not contain the behavioral information.
A.2 Image Reconstruction and Associated Consistency Check
In the previous subsection, we explained the loss functions of basic adversarial learning for dual feature extraction. Now, we explain the loss functions associated with consistency checks with image reconstruction.
As shown in the middle part of Fig. 3, for image reconstruction, we adopt a generator in each of the source and target domains: generator  $G_{S}$  for the source domain and generator  $G_{T}$  for the target domain. Generators  $G_{S}$  and  $G_{T}$  learn to produce images from the feature vectors so that the generated images should resemble the original input images.  $G_{S}$  takes a source-domain feature vector and a behavior feature vector as input and generates an image that resembles images in the source domain. On the other hand,  $G_{T}$  takes a target-domain feature vector and a behavior feature vector as input and generates an image that resembles images in the target domain.
Generators  $G_{S}$  and  $G_{T}$  learn to optimize the WGAN objective with the help of image discriminators:  $ID_{S}$  and  $ID_{T}$ .  $ID_{S}$  distinguishes real and generated images in the source domain, and  $ID_{T}$  does so in the target domain. For notational simplicity, we define functions  $G(\cdot, \cdot)$  and  $ID(\cdot, \cdot)$  as
$$
G \left(D E (o), B E \left(o ^ {\prime}\right)\right) = \left\{ \begin{array}{l l} G _ {S} \left(D E (o), B E \left(o ^ {\prime}\right)\right) & \text {i f} o \in O _ {S} \\ G _ {T} \left(D E (o), B E \left(o ^ {\prime}\right)\right) & \text {i f} o \in O _ {T} \end{array} \right. \tag {6}
$$
$$
I D (o) = \left\{ \begin{array}{l l} I D _ {S} (o) & \text {i f} o \in O _ {S} \\ I D _ {T} (o) & \text {i f} o \in O _ {T}. \end{array} \right. \tag {7}
$$
1) Image adversarial loss: The basic loss function for training the image generators and the associated blocks is image adversarial loss of WGAN learning [2, 16]. The image adversarial loss makes the model generate images that resemble the real ones and is given by
$$
L _ {i m g} ^ {a d v} = l ^ {a} (S E) + l ^ {a} (S N) + l ^ {a} (T N) + l ^ {a} (T L). \tag {8}
$$
where  $l^a (x)$  is the adversarial WGAN loss for a real image in  $O_{x}$ . In more detail, the first term in the RHS of (8) can be expressed as
$$
l ^ {a} (S E) = \mathbb {E} _ {o \in O _ {S E}} [ I D (o) ] - \mathbb {E} _ {\left(o _ {x}, o _ {y}\right) \in \left(O _ {S N}, O _ {S E}\right)} [ I D (G (D E \left(o _ {x}\right), B E \left(o _ {y}\right)) ]. \tag {9}
$$
Note that the image discriminator  $ID_{*}$  learns to assign a high value for a true image and a low value for a fake image. The first term in the RHS of (9) trains the image discriminator  $ID(\cdot)$  to assign a high value to a true image  $o \in O_{SE}$ , whereas the second term in the RHS of (9) trains the image discriminator  $ID(\cdot)$  to assign a low value to a fake image. The latter is because the  $ID(\cdot)$  input, i.e., the generator output  $G(DE(o_x), BE(o_y))$  with  $(o_x, o_y) \in (O_{SN}, O_{SE})$  inside the second term in the RHS of (9) is a fake image; for this image, the behavior feature is taken from  $o_y \in O_{SE}$  but the domain feature is taken from  $o_x \in O_{SN}$  not in  $O_{SE}$ .
The other terms  $l^a (SN), l^a (TN)$  and  $l^a (TL)$  in the RHS of (8) are similarly defined as
$$
l ^ {a} (S N) = \mathbb {E} _ {o \in O _ {S N}} [ I D (o) ] - \mathbb {E} _ {\left(o _ {x}, o _ {y}\right) \in \hat {O} _ {S N, f a k e}} [ I D (G (D E \left(o _ {x}\right), B E \left(o _ {y}\right))) ] \tag {10}
$$
$$
l ^ {a} (T N) = \mathbb {E} _ {o \in O _ {T N}} [ I D (o) ] - \mathbb {E} _ {\left(o _ {x}, o _ {y}\right) \in \hat {O} _ {T N, f a k e}} [ I D (G (D E \left(o _ {x}\right), B E \left(o _ {y}\right))) ] \tag {11}
$$
$$
l ^ {a} (T L) = \mathbb {E} _ {o \in O _ {T L}} [ I D (o) ] - \mathbb {E} _ {\left(o _ {x}, o _ {y}\right) \in \hat {O} _ {T L, f a k e}} [ I D (G (D E \left(o _ {x}\right), B E \left(o _ {y}\right))) ] \tag {12}
$$
where  $\hat{O}_{SN,fake}$  is the set of image combinations  $\{(o_x,o_y)\}$  such that the generated fake image  $G(DE(o_x),BE(o_y))$  has label  $SN$ .  $\hat{O}_{TN,fake}$  and  $\hat{O}_{TL,fake}$  are similarly defined. Specifically,
$$
\hat {O} _ {S N, f a k e} = \left(O _ {S E}, O _ {S N}\right) \cup \left(O _ {S E}, O _ {T N}\right) \cup \left(O _ {S N}, O _ {T N}\right)
$$
$$
\hat {O} _ {T L, p a i r} = (O _ {T N}, O _ {T L}).
$$
With the image adversarial loss  $L_{imag}^{adv}$  in (8), the learning is performed in an adversarial manner. The encoders and generators learn to minimize  $L_{img}^{adv}$ , while the image discriminators learn to maximize  $L_{img}^{adv}$ . That is, the image discriminators are trained to distinguish fake images from real images, whereas the encoders and the generators are trained to fool the image discriminators  $ID_*$ .
Note that the image adversarial loss is basically for training the image generators and the associated blocks in a WGAN adversarial manner. With the availability of the image generators  $G_{S}$  and  $G_{T}$ , we can impose our first and second consistency criteria: image reconstruction consistency and feature reconstruction consistency. We define the corresponding loss functions below.
2) Image reconstruction consistency loss: This loss checks the feature extraction is properly done. When we combine the features  $S$  and  $B_{S}$  from  $DE_{S}$  and  $BE_{S}$  with an input true image  $o_{SE}$  (or input true image  $o_{SN}$ ) in the first-stage feature extraction and input the feature combination  $(S, B_{S})$  into image generator  $G_{S}$ , the generated image should be the same as the original observation image  $o_{SE}$  (or  $o_{SN}$ ). The same consistency applies to the feature combination  $(T, \bar{B})$  in the target domain with a true input image  $o_{TN}$  or  $o_{TL}$  and the image generator  $G_{T}$ . Thus, the image reconstruction loss is given as
$$
L _ {i m g} ^ {\text {r e c o n}} = \mathbb {E} _ {o \in O _ {A L L}} \left[ l _ {m s e} (o, \hat {o}) \right] \tag {13}
$$
$$
\hat {o} = G (D E (o), B E (o)), \tag {14}
$$
$O_{ALL} = O_{SE}\cup O_{SN}\cup O_{TN}\cup O_{TL}$ , and  $l_{mse}(u,v)$  is the mean square error between  $u$  and  $v$ . The encoders and generators learn to minimize  $L_{img}^{recon}$ .
3) Feature reconstruction consistency loss: This is the second self-consistency criterion. If we input the generated source-domain image  $\widetilde{SB_S}$  described in the above image reconstruction loss part into the encoders  $DE_{S}$  and  $BE_{S}$ , then we obtain domain feature  $\tilde{S}$  and behavior feature  $\tilde{B}_{S}$ , and these two features should be the same as the features  $S$  and  $B_{S}$  extracted in the first-stage feature extraction.
The same principle applies to the target domain. Thus, the feature reconstruction consistency loss is expressed as
$$
L _ {f e a t} ^ {r e c o n} = \mathbb {E} _ {o \in O _ {A L L}} [ \| B E (o) - B E (\hat {o}) \| _ {2} ^ {2} ] + \mathbb {E} _ {o \in O _ {A L L}} [ \| D E (o) - D E (\hat {o}) \| _ {2} ^ {2} ] \tag {15}
$$
$$
\hat {o} = G (D E (o), B E (o)), \tag {16}
$$
$\| \cdot \|_2$  denotes the 2-norm of a vector, and  $O_{ALL} = O_{SE} \cup O_{SN} \cup O_{TN} \cup O_{TL}$ . Note that  $\hat{o}$  is a generated image from the domain and behavior features extracted from  $o$ , and  $DE(\hat{o})$  and  $BE(\hat{o})$  are the domain and behavior features from the generated image  $\hat{o}$ . The encoders and generators learn to minimize  $L_{feat}^{recon}$ .
A.3 Cycle-Consistency Check
Now, let us consider our third consistency criterion: cycle-consistency check with the right side of Fig. 3. This consistency check involves image translation in the middle part of Fig. 3 and the image retranslation in the right part of Fig. 3 in the main paper, and requires that the original image and the reconstructed image of translation/retranslation should be the same for perfect feature extraction and generation. The image cycle-consistency loss is expressed as
$$
L _ {i m g} ^ {c y c l e} = \mathbb {E} _ {\left(o _ {x}, o _ {y}\right) \in O _ {c y c l e} ^ {2}} \left[ l _ {m s e} \left(o _ {x}, G \left(D E \left(\hat {o} _ {(x, y)}\right), B E \left(\hat {o} _ {(y, x)}\right)\right)\right) \right] \tag {17}
$$
$$
\hat {o} _ {(x, y)} = G \left(D E \left(o _ {x}\right), B E \left(o _ {y}\right)\right) \tag {18}
$$
$$
\hat {o} _ {(y, x)} = G \left(D E \left(o _ {y}\right), B E \left(o _ {x}\right)\right) \tag {19}
$$
and  $O_{cycle}^{2}$  is the set of image pairs such that the two images in a pair do not belong to the same domain, i.e., one image belongs to the source domain and the other image belongs to the target domain. The explanation is as follows. Consider that we apply image  $o_x$  to the upper input and image  $o_y$  to the lower input of the left side of Fig. 3 in the main paper. Then,  $\hat{o}_{(x,y)} = G(DE(o_x), BE(o_y))$  is the generated image in the lower row and  $\hat{o}_{(y,x)} = G(DE(o_y), BE(o_x))$  is the generated image in the upper row in the middle part of Fig. 3 in the main paper. Finally, the image  $G(DE(\hat{o}_{(x,y)}), BE(\hat{o}_{(y,x)}))$  is the final reconstructed image in the upper row of the right side of Fig. 3 in the main paper. So, we can check the cycle-consistency between  $o_x$  and  $G(DE(\hat{o}_{(x,y)}), BE(\hat{o}_{(y,x)}))$ . The situation is mirrored when we apply image  $o_x$  to the lower input and image  $o_y$  to the upper input of the left side of Fig. 3 in the main paper. The encoders and generators learn to minimize  $L_{img}^{cycle}$ .
The feature cycle-consistency loss is expressed as
$$
L _ {\text {f e a t}} ^ {\text {c y c l e}} = \mathbb {E} _ {\left(o _ {x}, o _ {y}\right) \in O _ {\text {c y c l e}} ^ {2}} \left[ l _ {\text {m s e}} \left(D E \left(o _ {x}\right), D E \left(\hat {o} _ {(x, y)}\right)\right) \right] + \mathbb {E} _ {\left(o _ {x}, o _ {y}\right) \in O _ {\text {c y c l e}} ^ {2}} \left[ l _ {\text {m s e}} \left(B E \left(o _ {y}\right), B E \left(\hat {o} _ {(y, x)}\right)\right) \right] \tag {20}
$$
$$
\hat {o} _ {(x, y)} = G \left(D E \left(o _ {x}\right), B E \left(o _ {y}\right)\right) \tag {21}
$$
The explanation is as follows. For example, from the target domain real image  $TB_{T}$ , we extract domain feature  $T$  and behavior feature  $B_{T}$ . From these features, we can reconstruct  $TB_{T}$ . Also, we extract domain feature  $\widehat{T}$  (from  $\widehat{TB_S}$ ) and behavior feature  $\widehat{B_T}$  (from  $\widehat{SB_T}$ ). From these features, we can reconstruct  $TB_{T}$ . Let's assume  $\widehat{B_T}$  (behavior feature from the source domain image  $\widehat{SB_T}$  generated by  $G_{S}$ ) is replaced by  $B_{T}$  (behavior feature from the target domain image  $TB_{T}$ ). Then from  $\widehat{T}$  and  $B_{T}$  we should still reconstruct  $TB_{T}$ . That is,  $\widehat{B_T}$  (behavior feature from the source domain image  $\widehat{SB_T}$  made by  $G_{S}$ ) =  $B_{T}$  (behavior feature from the target domain image  $TB_{T}$ ). They are behavior features from different domains but they are equal. This implies that the behavior feature is independent of domain information. We name this by feature cycle-consistency because this constraint implicitly satisfies image cycle-consistency. If we explicitly constrain  $\widehat{B_T} = B_T$  and  $\widehat{T} = T$ , then we can replace  $\widehat{B_T}$  and  $\widehat{T}$  by  $B_{T}$  and  $T$ , and the image cycle-consistency is implicitly satisfied.
A.4 Other Losses
We further define losses to enhance performance.
1) Feature regularization loss: The feature regularization loss prevents the feature vector values from exploding for stable learning, and is given by
$$
L _ {\text {f e a t}} ^ {\text {r e g}} = \mathbb {E} _ {o \in O _ {A L L}} [ \| B E (o) \| _ {2} ^ {2} + \| D E (o) \| _ {2} ^ {2} ]. \tag {22}
$$
Instead, more rigorously, we can use the following regularization:
$$
L _ {f e a t} ^ {r e g, 1} = \mathbb {E} _ {o \in O _ {S}} [ (\left\| D E _ {S} (o) \right\| - c _ {n o r m, d}) ^ {2} ]
$$
$$
L _ {f e a t} ^ {r e g, 2} = \mathbb {E} _ {o \in O _ {S}} [ (\| B E _ {S} (o) \| - c _ {n o r m, b}) ^ {2} ]
$$
$$
L _ {f e a t} ^ {r e g, 3} = \mathbb {E} _ {o \in O _ {T}} \left[ \left(\left\| D E _ {T} (o) \right\| - c _ {\text {n o r m}, d}\right) ^ {2} \right]
$$
$$
L _ {f e a t} ^ {r e g, 4} = \mathbb {E} _ {o \in O _ {T}} [ (\left\| B E _ {T} (o) \right\| - c _ {n o r m, b}) ^ {2} ]
$$
$$
L _ {\text {f e a t}} ^ {\text {r e g}} = L _ {\text {f e a t}} ^ {\text {r e g}, 1} + L _ {\text {f e a t}} ^ {\text {r e g}, 2} + L _ {\text {f e a t}} ^ {\text {r e g}, 3} + L _ {\text {f e a t}} ^ {\text {r e g}, 4}. \tag {23}
$$
Where,  $c_{norm,d}$  and  $c_{norm,b}$  are hyperparameters with non-negative values. Appendix D.2 provides details including the value of  $c_{norm,d}$  and  $c_{norm,b}$  for the experiment.
2) Feature similarity loss: This loss aims to map observations of the same domain to similar points in the feature vector space and to map observations of the same behavior to similar points in the feature vector space. The feature similarity loss is defined as
$$
L _ {\text {f e a t}} ^ {\text {s i m}} = \left\| \mu_ {B E \left(O _ {S N}\right)} - \mu_ {B E \left(O _ {T N}\right)} \right\| _ {2} ^ {2} + \left\| \mu_ {D E \left(O _ {S E}\right)} - \mu_ {D E \left(O _ {S N}\right)} \right\| _ {2} ^ {2} + \left\| \mu_ {D E \left(O _ {T N}\right)} - \mu_ {D E \left(O _ {T L}\right)} \right\| _ {2} ^ {2}, \tag {24}
$$
where  $\mu_{BE(O_x)}$  and  $\mu_{DE(O_x)}$  are the means of the behavior feature vectors and the domain feature vectors over the observation set  $O_x$  ( $x \in \{SE, SN, TN, TL\}$ ), respectively.
A.5 Final Loss
Combining the above individual loss functions, we have the final objective for the encoders  $E = (BE_{S},DE_{S},BE_{T},DE_{T})$ , the generators  $G = (G_{S},G_{T})$ , feature discriminators  $FD = (BD_{B},DD_{B},BD_{D},DD_{D})$  and image discriminators  $ID = (ID_{S},ID_{T})$  as follows:
$$
\begin{array}{l} \min  _ {E, G} L _ {E, G} = \lambda_ {\text {f e a t}} ^ {\text {a d v}} L _ {\text {f e a t}} ^ {\text {p r e d}} + \lambda_ {\text {f e a t}} ^ {\text {a d v}} L _ {\text {f e a t}} ^ {\text {a d v}} + \lambda_ {\text {i m g}} ^ {\text {a d v}} L _ {\text {i m g}} ^ {\text {a d v}} + \lambda_ {\text {i m g}} ^ {\text {r e c o n}} L _ {\text {i m g}} ^ {\text {r e c o n}} + \lambda_ {\text {f e a t}} ^ {\text {r e c o n}} L _ {\text {f e a t}} ^ {\text {r e c o n}} \tag {25} \\ + \lambda_ {i m g} ^ {c y c l e} L _ {i m g} ^ {c y c l e} + \lambda_ {f e a t} ^ {c y c l e} L _ {f e a t} ^ {c y c l e} + \lambda_ {f e a t} ^ {s i m} L _ {f e a t} ^ {s i m} + \lambda_ {f e a t} ^ {r e g} L _ {f e a t} ^ {r e g} \\ \end{array}
$$
$$
\min  _ {F D, I D} L _ {F D, I D} = \lambda_ {f e a t} ^ {a d v} L _ {f e a t} ^ {p r e d} - \lambda_ {f e a t} ^ {a d v} L _ {f e a t} ^ {a d v} - \lambda_ {i m g} ^ {a d v} L _ {i m g} ^ {a d v} \tag {26}
$$
where the weighting factors  $\lambda_{feat}^{adv}, \lambda_{feat}^{adv}, \lambda_{img}^{adv}, \lambda_{img}^{recon}, \lambda_{feat}^{recon}, \lambda_{feat}^{cycle}, \lambda_{feat}^{cycle}, \lambda_{feat}^{sim}, \lambda_{feat}^{reg}$  are hyperparameters and their values are shown in Appendix D. Note that  $L_{feat}^{adv}$  and  $L_{img}^{adv}$  appear in both (25) and (26) with negative sign in (26). Hence, these two terms induce adversarial learning in the overall learning process.
A.6 Reward Generation and Learner Policy Update
In this section, we explain the simple method of extracting expert features and training  $D_{rew}$  for reward generation. That is, we just use the output of  $BE_{S}$  in the source domain with expert input  $o_{SE}$ . In this case,  $D_{rew}$  learns to minimize the following objective while behavior encoder  $BE_{T}$  is fixed:
$$
\begin{array}{l} L _ {D} = \mathbb {E} _ {o _ {S E} \in O _ {S E}} [ \log (D _ {r e w} (B E (o _ {S E}))) ] + \mathbb {E} _ {o _ {T L} \in O _ {T L}} [ \log (1 - D _ {r e w} (B E (o _ {T L}))) ] \tag {27} \\ + \mathbb {E} _ {o _ {x} \in M i x (\hat {o} _ {T E}, o _ {T L})} [ (\| \nabla D _ {r e w} (B E (o _ {x})) \| _ {2} - 1) ^ {2} ]. \\ \end{array}
$$
where the third expectation in (27) is over the mixture of  $\hat{o}_{TE}$  and  $o_{TL}$  with a certain ratio. The proposed method other than this simple method is explained in Sec. 5.
$D_{rew}$  takes a behavior feature vector as input and predicts its behavior label  $E$  or  $L$  from the input
behavior feature vector.  $D_{rew}$  learns to assign the value 1 to the expert behavior and 0 to the learner behavior. On the other hand,  $\pi_{\theta}$  learns to generate observations  $O_{TL}$  so that the corresponding behavior feature vector looks like the expert. The learner updates the policy using SAC [17], and the estimated reward for an observation  $o_t$  is defined by
$$
\hat {r} \left(o _ {t}\right) = \log \left(D _ {r e w} \left(B E _ {T} \left(o _ {t}\right)\right)\right) - \log \left(1 - D _ {r e w} \left(B E _ {T} \left(o _ {t}\right)\right)\right), \tag {28}
$$
which is in a similar form to that in [24].
B Algorithm Pseudo Codes
Algorithm 1 Dual feature extraction and image generation
Input: The number of epochs  $n_{\text{epoch\_it}}$ , domain encoders  $DE = (DE_S, DE_T)$ , behavior encoders  $BE = (BES, BE_T)$ , generators  $G = (G_S, G_T)$ , feature discriminators  $FD = (BD_B, DD_B, BD_D, DD_D)$ , image discriminators  $ID = (ID_S, ID_T)$ , observation sets  $OSE, OSN, OTN$  of size  $n_{\text{demo}}$ . Initialize parameters for  $DE, BE, G, FD, ID$ . Make a copy of  $OTN$  and initialize  $OTL$  with the copy.  
for  $k = 1$  to  $n_{\text{epoch\_it}}$  do  
Sample a minibatch of observations  $oSE, oSN, oTN, oTL$  from  $OSE, OSN, OTN, OTL$ , respectively.  
for  $o$  in  $(oSE \cup oSN \cup oTN \cup oTL)$  do  
Extract  $DE(o)$  and  $BE(o)$ .  
Generate  $\hat{o}$  in eq. (14).  
Extract  $DE(\hat{o})$  and  $BE(\hat{o})$ .  
end for  
for  $o_x, o_y$  (not in the same domain) in  $(oSE \cup oSN \cup oTN \cup oTL)$  do  
Generate  $\hat{o}_{(x,y)}$  and  $\hat{o}_{(y,x)}$  in eqs. (18) and (19).  
Generate  $G(DE(\hat{o}_{(x,y)}).BE(\hat{o}_{(y,x)}))$ .  
end for  
Compute  $LE_G$  and  $LFD_ID$  in eqs. (25) and (26).  
Update  $DE, BE$  and  $G$  to minimize  $LE_G$ .  
Update  $FD$  and  $ID$  to minimize  $LFD_ID$ .  
end for  
return  $E, G, FD, ID$
Algorithm 2 Reward estimation and policy update
Input: The number of training epochs  $n_{\text{epoch\_pol}}$ , the number of discriminator updates  $n_{\text{update},D}$ , the number of policy updates  $n_{\text{update},\theta}$ , domain encoders  $DE$ , behavior encoders  $BE$ , generators  $G$ , discriminator  $D_{\text{rew}}$  for reward estimation, learner policy  $\pi_{\theta}$ , observations sets  $O_{SE}, O_{SN}, O_{TN}$  of size  $n_{\text{demo}}$ , replay buffer  $B$  of size  $n_{\text{buffer}}$ . Initialize parameters for  $D_{\text{rew}}$ ,  $\pi_{\theta}$  and  $B$ .  
for  $k_1 = 1$  to  $n_{\text{epoch\_pol}}$  do  
Sample a trajectory  $\tau \sim \pi_{\theta}$ .  
Store transitions  $(s, a, s')$  and observations  $o$  to  $B$ .  
for  $k_2 = 1$  to  $n_{\text{update},D}$  do  
Sample minibatch of observations  $o_{SE} \in O_{SE}$ ,  $o_{TN} \in O_{TN}$ ,  $o_{TL} \in B$ .  
Update  $D_{\text{rew}}$  to minimize  $L_D$  in Sec. 5.4.  
end for  
for  $k_3 = 1$  to  $n_{\text{update},\theta}$  do  
Sample minibatch of transitions  $(s, a, s')$  and corresponding observations  $o$ .  
Compute reward  $\hat{r}$  in Sec. 5.4.  
Update the policy  $\pi_{\theta}$  using SAC.  
end for  
end for
C Third-Person Imitation Learning (TPIL)
This section summarises TPIL [37], one of the pioneering works that address the domain shift problem in IL. TPIL trains a model based on an unsupervised domain adaptation technique [15] with GAIL [18]. TPIL consists of a single behavior encoder  $BE$ , a domain discriminator  $DD$ , and a behavior discriminator  $BD$ , as shown in Fig. 12. The input label  $XB_{X}$  in Fig. 12 means that the input is in the  $X$  domain with behavior  $B_{X}$ .  $X$  can be source  $S$  or target  $T$ , and  $B_{X}$  can be expert  $E$  or non-expert  $N$ . The key idea is to train  $BE$  to extract domain-independent behavior features from inputs. The trained  $BE$  can be used to tell whether the learner's action in the target domain is expert behavior or non-expert behavior, and we can use this evaluation to train the learner.  $DD$  learns to predict the domain label (source or target) of the input, and  $BD$  learns to predict the behavior label (expert or non-expert) of the input. Therefore, encoder  $BE$  learns to fool  $DD$  by removing domain information from the input while helping  $BD$  by preserving behavior information from the input. In [37], the behavior feature is a concatenation of  $BE(o_{t})$  and  $BE(o_{t-4})$ ; however, we just simply the notation and denote the behavior feature by  $BE(o_{t})$ .
The total loss for TPIL  $\mathcal{L}_{TPIL}$  is defined as follows:
$$
\mathcal {L} _ {T P I L} = \sum_ {o _ {i}} \mathcal {L} _ {C E} (B D (B E (o _ {i})), b _ {i}) + \lambda_ {d} \mathcal {L} _ {C E} (D D (\mathcal {G} (B E (o _ {i})), d _ {i})) \tag {29}
$$
where  $o_i$  is an image observation,  $\mathcal{L}_{CE}$  is the cross-entropy loss,  $d_{i}$  is the domain label of  $o_i$  (1 for source domain and 0 for target domain),  $b_{i}$  is the behavior label of  $o_i$  (1 for expert behavior and 0 for non-expert behavior),  $\lambda_d$  is a hyperparameter, and  $\mathcal{G}$  is a Gradient Reversal Layer (GRL) [15] is defined by
$$
\mathcal {G} (x) = x
$$
$$
d \mathcal {G} (x) / d x = - \lambda_ {g} I
$$
where  $\lambda_{g}$  is a hyperparameter. GRL enables updating  $BE$ ,  $DD$ , and  $BD$  simultaneously using back-propagation. The imitation reward  $r_t$  for an observation  $o_t$  generated by the learner policy is defined by the probability that  $BD$  predicts the observation to be generated by an expert policy, as determined by  $r_t = BD(BE(o_t))$ .
Figure 12: Basic structure for domain-independent behavior feature extraction:  $BE$  - behavior encoder,  $BD$  - behavior discriminator,  $DD$  - domain discriminator
D.1 Network Architectures
In this section, we explain the network architectures of the components of the proposed method. Table 4 shows the layers for encoders, generators, and discriminators.