SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pengfei Wei
AI Lab, ByteDance
pengfei.wei@bytedance.com
Lingdong Kong
lingdong@comp.nus.edu.sg
Xinghua Qu
quxinghua17@gmail.com
ren.yi@bytedance.com
Zhiqiang Xu
zhiqiang.xu@mbzuai.ac.ae
Jing Jiang
Jing.Jiang@uts.edu.au
Xiang Yin
yinxiang.stechen@bytedance.com
Unsupervised video domain adaptation is a practical yet challenging task. In this work, for the first time, we tackle it from a disentanglement view. Our key idea is to handle the spatial and temporal domain divergence separately through disentanglement. Specifically, we consider the generation of cross-domain videos from two sets of latent factors, one encoding the static information and another encoding the dynamic information. A Transfer Sequential VAE (TranSVAE) framework is then developed to model such generation. To better serve for adaptation, we propose several objectives to constrain the latent factors. With these constraints, the spatial divergence can be readily removed by disentangling the static domain-specific information out, and the temporal divergence is further reduced from both frame- and video-levels through adversarial learning. Extensive experiments on the UCF-HMDB, Jester, and Epic-Kitchens datasets verify the effectiveness and superiority of TranSVAE compared with several state-of-the-art approaches. $^{1}$
Over the past decades, unsupervised domain adaptation (UDA) has attracted extensive research attention [41]. Numerous UDA methods have been proposed and successfully applied to various real-world applications, e.g., object recognition [38, 42, 47], semantic segmentation [51, 19, 33, 24], and object detection [3, 15, 45]. However, most of these methods and their applications are limited to the image domain, while much less attention has been devoted to video-based UDA, where the latter is undoubtedly more challenging.
Compared with image-based UDA, the source and target domains also differ temporally in video-based UDA. Images are spatially well-structured data, while videos are sequences of images with both spatial and temporal relations. Existing image-based UDA methods can hardly achieve satisfactory performance on video-based UDA tasks as they fail to consider the temporal dependency of video frames in handling the domain gaps. For instance, in video-based cross-domain action recognition tasks, domain gaps are presented by not only the actions of different persons in different scenarios but also the actions that appear at different timestamps or last at different time lengths.
$^{1}$ Code is publicly available at: https://github.com/ldkong1205/TranSVAE
Figure 1: Conceptual comparisons between the traditional all-in-one view and the proposed disentanglement view. Prior works often seek to compress implicit domain information to obtain domain-indistinguishable representations; while in this work, we pursue explicit decouplings of domain-specific information from other information via generative modeling.
Recently, few works have been proposed for video-based UDA. The key idea is to achieve domain alignment by aligning both frame- and video-level features through adversarial learning [7, 26], contrastive learning [37, 31], attention [10], or combination of these mechanisms, e.g., adversarial learning with attention [29, 8]. Though they have advanced video-based UDA, there is still room for improvement. Generally, existing methods follow an all-in-one way, where both spatial and temporal domain divergence are handled together, for adaptation (Fig. 1,  $\square$ ). However, cross-domain videos are highly complex data containing diverse mixed-up information, e.g., domain, semantic, and temporal information, which makes the simultaneous elimination of spatial and temporal divergence insufficient. This motivates us to handle the video-based UDA from a disentanglement perspective (Fig. 1,  $\square$ ) so that the spatial and temporal divergence can be well handled separately.
To achieve this goal, we first consider the generation process of cross-domain videos, as shown in Fig. 2, where a video is generated from two sets of latent factors: one set consists of a sequence of random variables, which are dynamic and incline to encode the semantic information for downstream tasks, e.g., action recognition; another set is static and introduces some domain-related spatial information to the generated video, e.g., style or appearance. Specifically, the blue / red nodes are the observed source / target videos  $\mathbf{x}^S / \mathbf{x}^T$ , respectively, over  $t$  timestamps. Static latent variables  $\mathbf{z}_d^S$  and  $\mathbf{z}_d^T$  follow a joint distribution and combining either of them with dynamic latent variables  $\mathbf{z}_t$  constructs one video data of a domain.
With the above generative model, we develop a Transfer Sequential Variational AutoEncoder (TranSVAE) for video-based UDA. TranSVAE handles the cross-domain divergence in two levels, where the first level removes the spatial divergence by disentangling  $\mathbf{z}_d$  from  $\mathbf{z}_t$ ; while the second level eliminates the temporal divergence of  $\mathbf{z}_t$ . To achieve this, we leverage appropriate constraints to ensure that the disentanglement indeed serves the adaptation purpose. Firstly, we enable a good decoupling of the two sets of latent factors by
Figure 2: Graphical illustrations of the proposed generative (1st figure) and inference (2nd figure) models for video domain disentanglement.
minimizing their mutual dependence. This encourages these two latent factor sets to be mutually independent. We then consider constraining each latent factor set. For  $\mathbf{z}_d^{\mathcal{D}}$  with  $\mathcal{D} \in \{\mathcal{S}, \mathcal{T}\}$ , we propose a contrastive triplet loss to make them static and domain-specific. This makes us readily handle spatial divergence by disentangling  $\mathbf{z}_d^{\mathcal{D}}$  out. For  $\mathbf{z}_t$ , we propose to align them across domains at both frame and video levels through adversarial learning so as to further eliminate the temporal divergence. Meanwhile, as downstream tasks use  $\mathbf{z}_t$  as input, we also add the task-specific supervision on  $\mathbf{z}_t$  extracted from source data (w/ ground-truth).
To the best of our knowledge, this is the first work that tackles the challenging video-based UDA from a domain disentanglement view. We conduct extensive experiments on popular benchmarks (UCF-HMDB, Jester, Epic-Kitchens) and the results show that TranSVAE consistently outperforms previous state-of-the-art methods by large margins. We also conduct comprehensive ablation studies
and disentanglement analyses to verify the effectiveness of the latent factor decoupling. The main contribution of the paper is summarized as follows:
- We provide a generative perspective on solving video-based UDA problems. We develop a generative graphical model for the cross-domain video generation process and propose to utilize the sequential VAE as the base generative model.
- Based on the above generative view, we propose a TranSVAE framework for video-based UDA. By developing four constraints on the latent factors to enable disentanglement to benefit adaptation, the proposed framework is capable of handling the cross-domain divergence from both spatial and temporal levels.
- We conduct extensive experiments on several benchmark datasets to verify the effectiveness of TranSVAE. A comprehensive ablation study also demonstrates the positive effect of each loss term on video domain adaptation.
Unsupervised Video Domain Adaptation. Despite the great progress in image-based UDA, only a few methods have recently attempted video-based UDA. In [7], a temporal attentive adversarial adaptation network  $(\mathrm{TA}^3\mathrm{N})$  is proposed to integrate a temporal relation module for temporal alignment. Choi et al. [10] proposed a SAVA method using self-supervised clip order prediction and clip attention-based alignment. Based on a cross-domain co-attention mechanism, the temporal co-attention network TCoN [29] focused on common key frames across domains for better alignment. Luo et al. [26] pay more attention to the domain-agnostic classifier by using a network topology of the bipartite graph to model the cross-domain correlations. Instead of using adversarial learning, Sahoo et al. [31] developed an end-to-end temporal contrastive learning framework named CoMix with background mixing and target pseudo-labels. Recently, Chen et al. [8] learned multiple domain discriminators for multi-level temporal attentive features to achieve better alignment, while Turrisi et al. [37] exploited two-headed deep architecture to learn a more robust target classifier by the combination of cross-entropy and contrastive losses. Although these approaches have advanced video-based UDA tasks, they all adopted to align features with diverse information mixed up from a compression perspective, which leaves room for further improvements.
Multi-Modal Video Adaptation. Most recently, there are also a few works integrating multiple modality data for video-based UDA. Although we only use the single modality RGB features, we still discuss this multi-modal research line for a complete literature review. The very first work exploring the multi-modal nature of videos for UDA is MM-SADA [28], where the correspondence of multiple modalities was exploited as a self-supervised alignment in addition to adversarial alignment. A later work, spatial-temporal contrastive domain adaptation (STCDA) [34], utilized a video-based contrastive alignment as the multi-modal domain metric to measure the video-level discrepancy across domains. [18] proposed cross-modal and cross-domain contrastive losses to handle feature spaces across modalities and domains. [43] leveraged both cross-modal complementary and cross-modal consensus to learn the most transferable features through a CIA framework. In [12], the authors proposed to generate noisy pseudo-labels for the target domain data using the source pre-trained model and select the clean samples in order to increase the quality of the pseudo-labels. Lin et al. [23] developed a cycle-based approach that alternates between spatial and spatiotemporal learning with knowledge transfer. Generally, all the above methods utilize the flow as the auxiliary modality input. Recently, there are also methods exploring other modalities, for instance, A3R [48] with audiios and MixDANN [44] with wild data. It is worth noting that the proposed TranSVAE – although only uses single modality RGB features – surprisingly achieves better UDA performance compared with most current state-of-the-art multi-modal methods, which highlights our superiority.
Disentanglement. Feature disentanglement is a wide and hot research topic. We only focus on works that are closely related to ours. In the image domain, some works consider adaptation from a generative view. [4] learned a disentangled semantic representation across domains. A similar idea is then applied to graph domain adaptation [5] and domain generalization [16]. [13] proposed a novel informative feature disentanglement, equipped with the adversarial network or the metric discrepancy model. Another disentanglement-related topic is sequential data generation. To generate videos, existing works [22, 50, 1] extended VAE to a recurrent form with different recursive structures. In
Figure 3: TranSVAE overview. The input videos are fed into an encoder to extract the visual features, followed by an LSTM to explore the temporal information. Two groups of mean and variance networks are then applied to model the posterior of the latent factors, i.e.,  $q(\mathbf{z}_t^{\mathcal{D}}|\mathbf{x}_{<t}^{\mathcal{D}})$  and  $q(\mathbf{z}_d^{\mathcal{D}}|\mathbf{x}_{1:T}^{\mathcal{D}})$ . The new representations  $\mathbf{z}_1^{\mathcal{D}}, \ldots, \mathbf{z}_T^{\mathcal{D}}$  and  $\mathbf{z}_d^{\mathcal{D}}$  are sampled, and then concatenated and passed to a decoder for reconstruction. Four constraints are proposed to regulate the latent factors for adaptation.
this paper, we present a VAE-based structure to generate cross-domain videos. We aim at tackling video-based UDA from a new perspective: sequential domain disentanglement and transfer.
3 Technical Approach
Formally, for a typical video-based UDA problem, we have a source domain  $S$  and a target domain  $\mathcal{T}$ . Domain  $S$  contains sufficient labeled data, denoted as  $\{(\mathbf{V}_i^S,y_i^S)\}_{i = 1}^{N_S}$ , where  $\mathbf{V}_i^S$  is a video sequence and  $y_{i}^{S}$  is the class label. Domain  $\mathcal{T}$  consists of unlabeled data, denoted as  $\{\mathbf{V}_i^T\}_{i = 1}^{N_T}$ . For a sequence  $\mathbf{V}_i^D$  from domain  $\mathcal{D}\in \{\mathcal{S},\mathcal{T}\}$ , it contains  $T$  frames  $\{\mathbf{x}_{i\_1}^{\mathcal{D}},\dots,\mathbf{x}_{i\_T}^{\mathcal{D}}\}$  in total². We further denote  $N = N^S +N^\mathcal{T}$ . Domains  $S$  and  $\mathcal{T}$  are of different distributions but share the same label space. The objective is to utilize both  $\{(\mathbf{V}_i^S,y_i^S)\}_{i = 1}^{N_S}$  and  $\{\mathbf{V}_i^T\}_{i = 1}^{N_T}$  to train a good classifier for domain  $\mathcal{T}$ . We present a table to list the notations in the Appendix.
**Framework Overview.** We adopt a VAE-based structure to model the cross-domain video generation process and propose to regulate the two sets of latent factors for adaptation purposes. The reconstruction process is based on the two sets of latent factors, i.e.,  $\mathbf{z}_1^{\mathcal{D}}$ , ...,  $\mathbf{z}_T^{\mathcal{D}}$  and  $\mathbf{z}_d^{\mathcal{D}}$  that are sampled from the posteriors  $q(\mathbf{z}_t^{\mathcal{D}}|\mathbf{x}_{<t}^{\mathcal{D}})$  and  $q(\mathbf{z}_d^{\mathcal{D}}|\mathbf{x}_{1:T}^{\mathcal{D}})$ , respectively. The overall architecture consists of five segments including the encoder, LSTM, latent spaces, sampling, and decoder, as shown in Fig. 3. The vanilla structure only gives an arbitrary disentanglement of latent factors. To make the disentanglement facilitate adaptation, we carefully constrain the latent factors as follows.
Video Sequence Reconstruction. The overall architecture of TranSVAE follows a VAE-based structure [22, 50, 1] with two sets of latent factors  $\mathbf{z}_1^D, \dots, \mathbf{z}_T^D$  and  $\mathbf{z}_d^D$ . The generative and inference graphical models are presented in Fig. 2. Similar to the conventional VAE, we use a standard Gaussian distribution for static latent factors. For dynamic ones, we use a sequential prior  $\mathbf{z}_t^D | \mathbf{z}_{<t}^D \sim \mathcal{N}(\boldsymbol{\mu}_t, \text{diag}(\boldsymbol{\sigma}_t^2))$ , that is, the prior distribution of the current dynamic factor is conditioned on the historical dynamic factors. The distribution parameters can be re-parameterized as a recurrent network, e.g., LSTM, with all previous dynamic latent factors as the input. Denoting  $\mathbf{Z}^D = \{\mathbf{z}_1^D, \dots, \mathbf{z}_T^D\}$  for simplification, we then get the prior as follows:
$$
p \left(\mathbf {z} _ {d} ^ {\mathcal {D}}, \mathbf {Z} ^ {\mathcal {D}}\right) = p \left(\mathbf {z} _ {d} ^ {\mathcal {D}}\right) \prod_ {t = 1} ^ {T} p \left(\mathbf {z} _ {t} ^ {\mathcal {D}} \mid \mathbf {z} _ {<   t} ^ {\mathcal {D}}\right). \tag {1}
$$
2Without confusion, we omit the subscript  $i$  for  $\mathbf{V}^{\mathcal{D}}$  and the corresponding notations, e.g.,  $\{\mathbf{x}_1^{\mathcal{D}},\dots ,\mathbf{x}_T^{\mathcal{D}}\}$
Following Fig. 2 (the 1st subfigure),  $\mathbf{x}_t^{\mathcal{D}}$  is generated from  $\mathbf{z}_d^{\mathcal{D}}$  and  $\mathbf{z}_t^{\mathcal{D}}$ , and we thus model  $p(\mathbf{x}_t^{\mathcal{D}}|\mathbf{z}_d^{\mathcal{D}},\mathbf{z}_t^{\mathcal{D}}) = \mathcal{N}(\boldsymbol{\mu}_t',diag(\boldsymbol{\sigma}_t'^2))$ . The distribution parameters are re-parameterized by the decoder which can be flexible networks like the deconvolutional neural network. Using  $\mathbf{V}^{\mathcal{D}} = \{\mathbf{x}_1^{\mathcal{D}},\dots,\mathbf{x}_T^{\mathcal{D}}\}$ , the generation can be formulated as follows:
$$
p \left(\mathbf {V} ^ {\mathcal {D}}\right) = p \left(\mathbf {z} _ {d} ^ {\mathcal {D}}\right) \prod_ {t = 1} ^ {T} p \left(\mathbf {x} _ {t} ^ {\mathcal {D}} \mid \mathbf {z} _ {d} ^ {\mathcal {D}}, \mathbf {z} _ {t} ^ {\mathcal {D}}\right) p \left(\mathbf {z} _ {t} ^ {\mathcal {D}} \mid \mathbf {z} _ {<   t} ^ {\mathcal {D}}\right). \tag {2}
$$
Following Fig. 2 (the 2nd subfigure), we model the posterior distributions of the latent factors as another two Gaussian distributions, i.e.,  $q(\mathbf{z}_d^{\mathcal{D}}|\mathbf{V}^{\mathcal{D}}) = \mathcal{N}(\boldsymbol{\mu}_d, \text{diag}(\boldsymbol{\sigma}_d^2))$  and  $q(\mathbf{z}_t^{\mathcal{D}}|\mathbf{x}_{<t}^{\mathcal{D}}) = \mathcal{N}(\boldsymbol{\mu}_t^{\prime\prime}, \text{diag}(\boldsymbol{\sigma}_t^{\prime\prime 2}))$ . The parameters of these two distributions are re-parameterized by the encoder, which can be a convolutional or LSTM module. However, the network of the static latent factors uses the whole sequence as the input while that of the dynamic latent factors only uses previous frames. Then the inference can be factorized as:
$$
q \left(\mathbf {z} _ {d} ^ {\mathcal {D}}, \mathbf {Z} ^ {\mathcal {D}} \mid \mathbf {V} ^ {\mathcal {D}}\right) = q \left(\mathbf {z} _ {d} ^ {\mathcal {D}} \mid \mathbf {V} ^ {\mathcal {D}}\right) \prod_ {t = 1} ^ {T} q \left(\mathbf {z} _ {t} ^ {\mathcal {D}} \mid \mathbf {x} _ {<   t} ^ {\mathcal {D}}\right). \tag {3}
$$
Combining the above generation and inference, we obtain the VAE-related objective function as:
$$
\mathcal {L} _ {\mathrm {s v a e}} = \mathbb {E} _ {q \left(\mathbf {z} _ {d} ^ {\mathcal {D}}, \mathbf {Z} ^ {\mathcal {D}} \mid \mathbf {V} ^ {\mathcal {D}}\right)} \left[ - \sum_ {t = 1} ^ {T} \log p \left(\mathbf {x} _ {t} ^ {\mathcal {D}} \mid \mathbf {z} _ {d} ^ {\mathcal {D}}, \mathbf {z} _ {t} ^ {\mathcal {D}}\right) \right] + \tag {4}
$$
$$
\mathrm {K L} (q (\mathbf {z} _ {d} ^ {\mathcal {D}} | \mathbf {V} ^ {\mathcal {D}}) | | p (\mathbf {z} _ {d} ^ {\mathcal {D}})) + \sum_ {t = 1} ^ {T} \mathrm {K L} (q (\mathbf {z} _ {t} ^ {\mathcal {D}} | \mathbf {x} _ {<   t} ^ {\mathcal {D}}) | | p (\mathbf {z} _ {t} ^ {\mathcal {D}} | \mathbf {z} _ {<   t} ^ {\mathcal {D}})),
$$
which is a frame-wise negative variational lower bound. Only using the above vanilla VAE-based loss cannot guarantee that the disentanglement serves for adaptation, and thus we propose additional constraints on the two sets of latent factors.
Mutual Dependence Minimization (Fig. 3, ). We first consider explicitly enforcing the two sets of latent factors to be mutually independent. To do so, we introduce the mutual information [2] loss  $\mathcal{L}_{\mathrm{mi}}$  to regulate the two sets of latent factors. Thus, we obtain:
$$
\mathcal {L} _ {\mathrm {m i}} \left(\mathbf {z} _ {d} ^ {\mathcal {D}}, \mathbf {Z} ^ {\mathcal {D}}\right) = \sum_ {t = 1} ^ {T} \operatorname {K L} \left(q \left(\mathbf {z} _ {d} ^ {\mathcal {D}}, \mathbf {z} _ {t} ^ {\mathcal {D}}\right) | | q \left(\mathbf {z} _ {d} ^ {\mathcal {D}}\right) q \left(\mathbf {z} _ {t} ^ {\mathcal {D}}\right)\right) = \sum_ {t = 1} ^ {T} \left[ H \left(\mathbf {z} _ {d} ^ {\mathcal {D}}\right) + H \left(\mathbf {z} _ {t} ^ {\mathcal {D}}\right) - H \left(\mathbf {z} _ {d} ^ {\mathcal {D}}, \mathbf {z} _ {t} ^ {\mathcal {D}}\right) \right]. \tag {5}
$$
To calculate Eq. (5), we need to estimate the densities of  $\mathbf{z}_d^{\mathcal{D}},\mathbf{z}_t^{\mathcal{D}}$  and  $(\mathbf{z}_d^{\mathcal{D}},\mathbf{z}_t^{\mathcal{D}})$ . Following the non-parametric way in [9], we use the mini-batch weighted sampling as follows:
$$
H (\mathbf {z}) = - \mathbb {E} _ {q (\mathbf {z})} [ \log q (\mathbf {z}) ] \approx - \log \frac {1}{M} \sum_ {i = 1} ^ {M} [ \log \frac {1}{M N} \sum_ {j = 1} ^ {M} q \left(\mathbf {z} \left(\mathbf {x} _ {i}\right) | \mathbf {x} _ {j}\right) ], \tag {6}
$$
where  $\mathbf{z}$  is  $\mathbf{z}_d^{\mathcal{D}},\mathbf{z}_t^{\mathcal{D}}$  or  $(\mathbf{z}_d^{\mathcal{D}},\mathbf{z}_t^{\mathcal{D}})$ ,  $N$  denotes the data size and  $M$  is the mini-batch size.
Domain Specificity & Static Consistency (Fig. 3,  $\square$ ). A characteristic of the domain specificity, e.g., the video style or the objective appearance, is its static consistency over dynamic frames. With this observation, we enable the static and domain-specific latent factors so that we can remove the spatial divergence by disentangling them out. Mathematically, we hope that  $\mathbf{z}_d^{\mathcal{D}}$  does not change a lot when  $\mathbf{z}_t^{\mathcal{D}}$  varies over time. To achieve this, given a sequence, we randomly shuffle the temporal order of frames to form a new sequence. The static latent factors disentangled from the original sequence and the shuffled sequence should be ideally equal or be very close at least. This motivates us to minimize the distance between these two static factors. Meanwhile, to further enhance the domain specificity, we enforce the dynamic latent factors from different domains to have a large distance. To this end, we propose the following contrastive triplet loss:
$$
\mathcal {L} _ {\mathrm {c t c}} = \max  \left(D \left(\mathbf {z} _ {d} ^ {\mathcal {D} ^ {+}}, \widetilde {\mathbf {z}} _ {d} ^ {\mathcal {D} ^ {+}}\right) - D \left(\mathbf {z} _ {d} ^ {\mathcal {D} ^ {+}}, \mathbf {z} _ {d} ^ {\mathcal {D} ^ {-}}\right) + \mathbf {m}, 0\right), \tag {7}
$$
where  $D(\cdot, \cdot)$  is Euclidean distance,  $\mathbf{m}$  is a margin set to 1 in the experiments,  $\mathbf{z}_d^{\mathcal{D}^+}$ ,  $\widetilde{\mathbf{z}}_d^{\mathcal{D}^+}$ , and  $\mathbf{z}_d^{\mathcal{D}^-}$  are static latent factors of the anchor sequence from domain  $\mathcal{D}^+$ , the shuffled sequence, and a randomly selected sequence from domain  $\mathcal{D}^-$ , respectively.  $\mathcal{D}^+$  and  $\mathcal{D}^-$  represent two different domains.
Domain Temporal Alignment (Fig. 3,  $\square$ ). We now consider to reduce the temporal divergence of the dynamic latent factors. There are several ways to achieve this, and in this paper, we take advantage of the most popular adversarial-based idea [14]. Specifically, we build a domain classifier to discriminate whether the data is from  $S$  or  $T$ . When back-propagating the gradients, a gradient reversal layer (GRL) is adopted to invert the gradients. Like existing video-based UDA methods, we also conduct both frame-level and video-level alignments. Moreover, as  $\mathrm{TA}^3\mathrm{N}$  [7] does, we exploit the temporal relation network (TRN) [49] to discover the temporal relations among different frames, and then aggregate all the temporal relation features into the final video-level features. This enables another level of alignment on the temporal relation features. Thus, we have:
$$
\mathcal {L} _ {f} = \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {1}{T} \sum_ {t = 1} ^ {T} C E \left[ G _ {f} \left(\mathbf {z} _ {i _ {-} t} ^ {\mathcal {D}}\right), d _ {i} \right], \tag {8}
$$
$$
\mathcal {L} _ {r} = \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {1}{T - 1} \sum_ {n = 2} ^ {T} C E \left[ G _ {r} \left(T r N _ {n} \left(\mathbf {Z} _ {i} ^ {\mathcal {D}}\right)\right), d _ {i} \right], \tag {9}
$$
$$
\mathcal {L} _ {v} = \frac {1}{N} \sum_ {i = 1} ^ {N} C E \left[ G _ {v} \left(\frac {1}{T - 1} \sum_ {n = 2} ^ {T} T r N _ {n} \left(\mathbf {Z} _ {i} ^ {\mathcal {D}}\right)\right), d _ {i} \right], \tag {10}
$$
where  $d_{i}$  is the domain label,  $CE$  denotes the cross-entropy loss function,  $\mathbf{Z}_i^{\mathcal{D}} = \{\mathbf{z}_{i\_ 1}^{\mathcal{D}},\dots,\mathbf{z}_{i\_ T}^{\mathcal{D}}\}$ ,  $TrN_{i}$  is the  $n$ -frame temporal relation network,  $G_{f}, G_{r}$ , and  $G_{v}$  are the frame feature level, the temporal relation feature level, and the video feature level domain classifiers, respectively. To this end, we obtain the domain adversarial loss by summing up Eqs. (8-10):
$$
\mathcal {L} _ {\mathrm {a d v}} = \mathcal {L} _ {f} + \mathcal {L} _ {r} + \mathcal {L} _ {v}. \tag {11}
$$