SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We assign equal importance to these three levels of losses to reduce the overhead of the hyperparameter search. To this end, with  $\mathcal{L}_{\mathrm{mi}}$ ,  $\mathcal{L}_{\mathrm{ctc}}$ , and  $\mathcal{L}_{\mathrm{adv}}$ , the learned dynamic latent factors are expected to be domain-invariant (the three constraints are interactive and complementary to each other for obtaining the domain-invariant dynamic latent factor), and then can be used for downstream UDA tasks. In this paper, we specifically focus on action recognition as the downstream task.
Task Specific Supervision (Fig. 3, ). We further encourage the dynamic latent factors to carry the semantic information. Considering that the source domain has sufficient labels, we accordingly design the task supervision as the regularization imposed on  $\mathbf{z}_t^S$ . This gives us:
$$
\mathcal {L} _ {\mathrm {c l s}} = \frac {1}{N ^ {S}} \sum_ {i = 1} ^ {N ^ {S}} \mathcal {L} \left(\mathcal {F} \left(\mathbf {Z} _ {i} ^ {S}\right), y _ {i} ^ {S}\right), \tag {12}
$$
where  $\mathcal{F}(\cdot)$  is a feature transformer mapping the frame-level features to video-level features, specifically a TRN in this paper, and  $\mathcal{L}(\cdot,\cdot)$  is either cross-entropy or mean squared error loss according to the targeted task.
Although the dynamic latent factors are constrained to be domain-invariant, we do not completely rely on source semantics to learn features discriminative for the target domain. We propose to incorporate target pseudo-labels in task-specific supervision. During the training, we use the prediction network obtained in the previous epoch to generate the target pseudo-labels of the unlabelled target training data for the current epoch. However, to increase the reliability of target pseudo-labels, we let the prediction network be trained only on the source supervision for several epochs and then integrate the target pseudo-labels in the following training epochs. Meanwhile, a confidence threshold is set to determine whether to use the target pseudo-labels or not. Thus, we have the final task-specific supervision as follows:
$$
\mathcal {L} _ {\mathrm {c l s}} = \frac {1}{N} \left(\sum_ {i = 1} ^ {N ^ {\mathcal {S}}} \mathcal {L} \left(\mathcal {F} \left(\mathbf {Z} _ {i} ^ {\mathcal {S}}\right), y _ {i} ^ {\mathcal {S}}\right) + \sum_ {i = 1} ^ {N ^ {\mathcal {T}}} \mathcal {L} \left(\mathcal {F} \left(\mathbf {Z} _ {i} ^ {\mathcal {T}}\right), \widetilde {y} _ {i} ^ {\mathcal {T}}\right)\right), \tag {13}
$$
where  $\widetilde{y}_i^{\mathcal{T}}$  is the pseudo-label of  $\mathbf{Z}_i^{\mathcal{T}}$ .
Summary. To this end, we reach the final objective function of our TranSVAE framework as follows:
$$
\mathcal {L} ^ {\prime} = \mathcal {L} _ {\mathrm {s v a e}} + \lambda_ {1} \mathcal {L} _ {\mathrm {m i}} + \lambda_ {2} \mathcal {L} _ {\mathrm {a d v}} + \lambda_ {3} \mathcal {L} _ {\mathrm {c t c}} + \lambda_ {4} \mathcal {L} _ {\mathrm {c l s}}, \tag {14}
$$
where  $\lambda_{i}$  with  $i = 1,2,3,4$  denotes the loss balancing weight.
In this section, we conduct extensive experimental studies on popular video-based UDA benchmarks to verify the effectiveness of the proposed TranSVAE framework.
UCF-HMDB is constructed by collecting the relevant and overlapping action classes from  $\mathrm{UCF}_{101}$  [35] and  $\mathrm{HMDB}_{51}$  [20]. It contains 3,209 videos in total with 1,438 training videos and 571 validation videos from  $\mathrm{UCF}_{101}$ , and 840 training videos and 360 validation videos from  $\mathrm{HMDB}_{51}$ . This in turn establishes two video-based UDA tasks:  $\mathbf{U} \rightarrow \mathbf{H}$  and  $\mathbf{H} \rightarrow \mathbf{U}$ .
Jester [27] consists of 148,092 videos of humans performing hand gestures. Pan et al. [29] constructed a large-scale cross-domain benchmark with seven gesture classes, and form a single transfer task  $\mathbf{J}_S\rightarrow \mathbf{J}_{\mathcal{T}}$  , where  $\mathbf{J}_S$  and  $\mathbf{J}_{\mathcal{T}}$  contain 51,498 and 51,415 video clips, respectively.
Epic-Kitchens [11] is a challenging egocentric dataset consisting of videos capturing daily activities in kitchens. [28] constructs three domains across the eight largest actions. They are  $\mathbf{D}_1$ ,  $\mathbf{D}_2$ , and  $\mathbf{D}_3$  corresponding to P08, P01, and P22 kitchens of the full dataset, resulting in six cross-domain tasks.
Sprites [22] contains sequences of animated cartoon characters with 15 action categories. The appearances of characters are fully controlled by four attributes, i.e., body, top wear, bottom wear, and hair. We construct two domains,  $\mathbf{P}_1$  and  $\mathbf{P}_2$ .  $\mathbf{P}_1$  uses the human body with attributes randomly selected from 3 top wears, 4 bottom wears, and 5 hairs, while  $\mathbf{P}_2$  uses the alien body with attributes randomly selected from 4 top wears, 3 bottom wears, and 5 hairs. The attribute pools are non-overlapping across domains, resulting in completely heterogeneous  $\mathbf{P}_1$  and  $\mathbf{P}_2$ . Each domain has 900 video sequences, and each sequence is with 8 frames.
Architecture. Following the latest works [31, 37], we use I3D [6] as the backbone<sup>3</sup>. However, different from CoMix which jointly trains the backbone, we simply use the pretrained I3D model on Kinetics [17], provided by [6], to extract RGB features. For the first three benchmarks, RGB features are used as the input of TranSVAE. ForSprites, we use the original image as the input, for the purpose of visualizing the reconstruction and disentanglement results. We use the shared encoder and decoder structures across the source and target domains. For RGB feature inputs, the encoder and decoder are fully connected layers. For original image inputs, the encoder and decoder are the convolution and deconvolution layers (from DCGAN [46]), respectively. For the TRN model, we directly use the one provided by [7]. Other details on this aspect are placed in Appendix.
Configurations. Our TranSVAE is implemented with PyTorch [30]. We use Adam with a weight decay of  $1e^{-4}$  as the optimizer. The learning rate is initially set to be  $1e^{-3}$  and follows a commonly used decreasing strategy in [14]. The batch size and the learning epoch are uniformly set to be 128 and 1,000, respectively, for all the experiments. We uniformly set 100 epochs of training under only source supervision and involved the target pseudo-labels afterward. Following the common protocol in video-based UDA [37], we perform hyperparameter selection on the validation set. The specific hyperparameters used for each task can be found in the Appendix. NVIDIA A100 GPUs are used for all experiments. Kindly refer
Table 1: UDA performance comparisons on UCF-HMDB.
Method & Year	Backbone	U → H	H → U	Average ↑
DANN (JMLR'16)	ResNet-101	75.28	76.36	75.82
JAN (ICML'17)	ResNet-101	74.72	76.69	75.71
AdaBN (PR'18)	ResNet-101	72.22	77.41	74.82
MCD (CVPR'18)	ResNet-101	73.89	79.34	76.62
TA3N (ICCV'19)	ResNet-101	78.33	81.79	80.06
ABG (MM'20)	ResNet-101	79.17	85.11	82.14
TCoN (AAAI'20)	ResNet-101	87.22	89.14	88.18
MA2L-TD (WACV'22)	ResNet-101	85.00	86.59	85.80
Source-only (Sonly)	I3D	80.27	88.79	84.53
DANN (JMLR'16)	I3D	80.83	88.09	84.46
ADDA (CVPR'17)	I3D	79.17	88.44	83.81
TA3N (ICCV'19)	I3D	81.38	90.54	85.96
SAVA (ECCV'20)	I3D	82.22	91.24	86.73
CoMix (NeurIPS'21)	I3D	86.66	93.87	90.22
CO2A (WACV'22)	I3D	87.78	95.79	91.79
TranSVAE (Ours)	I3D	87.78	98.95	93.37
Supervised-target (Tsup)	I3D	95.00	96.85	95.93
3The existing widely adopted backbones for video-based UDA include ResNet-101 and I3D. However, more recent backbones, e.g. Transformer-based or VideoMAE-based [36, 40], are promising to be used in this task.
Table 2: UDA performance comparisons to approaches using multi-modality data as the input.
Task	\( S_{only} \)	MM-SADA	STCDA	CMCD	A3R	CleanAdapt	CycDA	MixDANN	CIA	TranSVAE
\( \mathbf{U} \rightarrow \mathbf{H} \)	86.1	84.2	83.1	84.7	/	89.8	88.1	82.2	88.3	87.8 (+1.7)
\( \mathbf{H} \rightarrow \mathbf{U} \)	92.5	91.1	92.1	92.8	/	99.2	98.0	92.8	94.1	99.0 (+6.5)
Average ↑	89.3	87.7	87.6	88.8	/	94.5	93.1	87.5	91.2	93.4 (+4.1)
\( \mathbf{D}_1 \rightarrow \mathbf{D}_2 \)	43.2	49.5	52.0	50.3	53.2	52.7	/	56.0	52.5	50.5 (+7.3)
\( \mathbf{D}_1 \rightarrow \mathbf{D}_3 \)	42.5	44.1	45.5	46.3	52.1	47.0	/	47.3	47.8	50.3 (+7.8)
\( \mathbf{D}_2 \rightarrow \mathbf{D}_1 \)	43.0	48.2	49.0	49.5	51.9	46.2	/	50.3	49.8	50.3 (+7.3)
\( \mathbf{D}_2 \rightarrow \mathbf{D}_3 \)	48.0	52.7	52.5	52.0	55.5	52.7	/	52.4	53.2	58.6 (+10.6)
\( \mathbf{D}_3 \rightarrow \mathbf{D}_1 \)	43.0	50.9	52.6	48.7	51.5	47.8	/	51.0	52.2	48.0 (+5.0)
\( \mathbf{D}_3 \rightarrow \mathbf{D}_2 \)	55.5	56.1	55.6	56.3	63.2	54.4	/	54.7	57.6	58.0 (+2.5)
Average ↑	45.9	50.3	51.2	51.0	54.1	50.3	/	52.0	52.2	52.6 (+6.7)
Competitors. We compared methods from three lines. We first consider the source-only  $(S_{\mathrm{only}})$  and supervised-target  $(T_{\mathrm{sup}})$  which uses only labeled source data and only labeled target data, respectively. These two baselines serve as the lower and upper bounds for our tasks. Secondly, we consider five popular image-based UDA methods by simply ignoring temporal information, namely DANN [14], JAN [25], ADDA [38], AdaBN [21], and MCD [32]. Lastly and most importantly, we compare recent SoTA video-based UDA methods, including TA $^3$ N [7], SAVA [10], TCoN [29], ABG [26], CoMix [31], CO $^2$ A [37], and MA $^2$ L-TD [8]. All these methods use single modality features. We directly quote numbers reported in published papers whenever possible. There exist recent works conducting video-based UDA using multi-modal data, e.g. RGB + Flow. Although TranSVAE solely uses RGB features, we still take this set of methods into account. Specifically, we consider MM-SADA [28], STCDA [34], CMCD [18], A3R [48], CleanAdapt [12], CycDA [23], MixDANN [44] and CIA [43].
4.3 Comparative Study
Results on UCF-HMDB. Tab. 1 shows comparisons of TranSVAE with baselines and SoTA methods on UCF-HMDB. The best result among all the baselines is highlighted using bold. Overall, methods using the I3D backbone [6] achieve better results than those using ResNet-101. Our TranSVAE consistently outperforms all previous methods. In particular, TranSVAE achieves  $93.37\%$  average accuracy, improving the best competitor  $\mathrm{CO}^2\mathrm{A}$  [37], with the same I3D backbone [6], by  $1.38\%$ . Surprisingly, we observe that TranSVAE even yields better results (by a  $2.1\%$  improvement) than the supervised-target ( $\mathcal{T}_{\mathrm{sup}}$ ) baseline. This is because the  $\mathbf{H} \rightarrow \mathbf{U}$  task already has a good performance without adaptation, i.e.,  $88.79\%$  for the source-only ( $S_{\mathrm{only}}$ ) baseline, and thus the target pseudo-labels used in TranSVAE are almost correct. By further aligning domains and equivalently augmenting training data, TranSVAE outperforms  $\mathcal{T}_{\mathrm{sup}}$  which is only trained with target data.
Results on Jester & Epic-Kitchens. Tab. 3 shows the comparison results on the Jester and Epic-Kitchens benchmarks. We can see that our TranSVAE is the clear winner among all the methods on all the tasks. Specifically, TranSVAE achieves a  $1.4\%$  improvement and a  $9.4\%$  average improvement over the runner-up baseline CoMix [31] on Jester and Epic-Kitchens, re
Table 3: Comparison results on Jester and Epic-Kitchens.
Task	\( S_{only} \)	DANN	ADDA	\( TA^3N \)	CoMix	TranSVAE	\( T_{sup} \)
\( J_S \rightarrow J_T \)	51.5	55.4	52.3	55.5	64.7	66.1 (+14.6)	95.6
\( D_1 \rightarrow D_2 \)	32.8	37.7	35.4	34.2	42.9	50.5 (+17.7)	64.0
\( D_1 \rightarrow D_3 \)	34.1	36.6	34.9	37.4	40.9	50.3 (+16.2)	63.7
\( D_2 \rightarrow D_1 \)	35.4	38.3	36.3	40.9	38.6	50.3 (+14.9)	57.0
\( D_2 \rightarrow D_3 \)	39.1	41.9	40.8	42.8	45.2	58.6 (+19.5)	63.7
\( D_3 \rightarrow D_1 \)	34.6	38.8	36.1	39.9	42.3	48.0 (+13.4)	57.0
\( D_3 \rightarrow D_2 \)	35.8	42.1	41.4	44.2	49.2	58.0 (+22.2)	64.0
Average↑	35.3	39.2	37.4	39.9	43.2	52.6 (+17.3)	61.5
spectively. This verifies the superiority of TranSVAE over others in handling video-based UDA. However, we also notice that the accuracy gap between CoMix and  $\mathcal{T}_{\mathrm{sup}}$  is still significant on Jester. This is because the large-scale Jester dataset contains highly heterogeneous data across domains, e.g., the source domain contains videos of the rolling hand forward, while the target domain only consists of videos of the rolling hand backward. This leaves much room for improvement in the future.
Compare to Multi-Modal Methods. We further compare with four recent video-based UDA methods that use multi-modalities, e.g. RGB features, and optical flows, although TranSVAE only uses RGB features. Surprisingly, TranSVAE achieves better average results than seven out of eight multi-modal methods and is only worse than CleanAdapt [12] on UCF-HMDB and A3R [48] on Epic-Kitchens. Considering TranSVAE only uses single-modality data, we are confident that there exists great potential for further improvements of TranSVAE with multi-modal data taken into account.
Table 5: Loss separation study on different video-based UDA tasks by dropping each loss sequentially.
L_svae	L_cls	L_adv	L_mi	L_cic	PL	U→H	H→U	JS→JT	D1→D2	D1→D3	D2→D1	D2→D3	D3→D1	D3→D2	Avg.
✓	✓	✓	✓	✓	18.61	26.62	22.92	34.00	30.29	33.79	30.49	28.51	34.27	28.83
✓	✓	✓	✓	✓	83.06	93.52	48.07	40.93	43.33	43.91	51.13	41.84	52.67	55.38
✓	✓	✓	✓	✓	85.83	93.52	65.12	46.67	48.56	49.43	55.34	45.52	54.53	60.60
✓	✓	✓	✓	✓	83.89	95.80	64.89	48.53	48.25	48.96	54.21	45.52	55.73	60.64
✓	✓	✓	✓	✓	87.22	94.40	64.64	49.87	48.25	49.66	56.47	47.59	55.07	61.46
✓	✓	✓	✓	✓	✓	87.78	98.95	66.10	50.53	50.31	50.34	58.62	48.04	58.00	63.19
Figure 4: Domain disentanglement and transfer examples. Left: Video sequence inputs for  $\mathcal{D} = \mathbf{P}_1$  ("Human",  $\square$ ) and  $\mathcal{D} = \mathbf{P}_2$  ("Alien",  $\square$ ). Middle: Reconstructed sequences ( $\square$ ) with  $\mathbf{z}_1^{\mathcal{D}}, \ldots, \mathbf{z}_T^{\mathcal{D}}$ . Right: Domain transferred sequences with exchanged  $\mathbf{z}_d^{\mathcal{D}}$ .
4.4 Property Analysis
Disentanglement Analysis. We analyze the disentanglement effect of  $TranSVAE$  onSprites [22] and show results in Fig. 4. The left subfigure shows the original sequences of the two domains. The "Human" and "Alien" are completely different appearances and the former is casting spells while the latter is slashing. The middle subfigure shows the sequences reconstructed only using  $\{\mathbf{z}_1^D, \dots, \mathbf{z}_T^D\}$ . It can be clearly seen that the two sequences keep the same action as the corresponding original ones. However, if we only focus on the appearance characteristics, it is difficult to distinguish the domain to which the sequences belong. This indicates that  $\{\mathbf{z}_1^D, \dots, \mathbf{z}_T^D\}$  are indeed domain-invariant and well encode the semantic information. The right subfigure shows the sequences reconstructed by exchanging  $\mathbf{z}_d^D$ , which results in two sequences with the same actions but exchanged appearance. This verifies that  $\mathbf{z}_d^D$  is representing the appearance information, which is actually the domain-related information in this example. This property study sufficiently supports that  $TranSVAE$  can successfully disentangle the domain information from other information, with the former embedded in  $\mathbf{z}_d^D$  and the latter embedded in  $\{\mathbf{z}_1^D, \dots, \mathbf{z}_T^D\}$ .
Complexity Analysis. We further conduct complexity analysis on our TranSVAE. Specifically, we compare the number of trainable parameters, multiply-accumulate operations (MACs), floating-point operations (FLOPs), and inference frame-per-second (FPS) with existing baselines including  $\mathrm{TA}^3\mathrm{N}$  [7],  $\mathrm{CO}^2\mathrm{A}$  [37], and CoMix
Table 4: Comparison results on model complexity.
Methods	Trainable Params	MACs	FLOPs	FPS
TA3N	7.6880 M	18.2318 G	36.4636 G	0.0134 s
CoMix	30.3688 M	18.5640 G	37.1280 G	0.0157 s
CO2A	23.6720 M	18.1884 G	36.3768 G	0.0127 s
TranSVAE	12.7419 M	18.2657 G	36.5314 G	0.0133 s
[31]. All the comparison results are shown in Tab. 4. From the table, we observe that  $\text{TranSVAE}$  requires less trainable parameters than  $\text{CO}^2\text{A}$  and CoMix. Although more trainable parameters are used than  $\text{TA}^3\text{N}$ ,  $\text{TranSVAE}$  achieves significant adaptation performance improvement than  $\text{TA}^3\text{N}$  (see Tab. 1 and Tab. 3). Moreover, the MACs, FLOPs, and FPS are competitive among different methods. This is reasonable since all these approaches adopt the same I3D backbone.
Ablation Study. We now analyze the effectiveness of each loss term in Eq. (14). We compare with four variants of  $TranSVAE$ , each removing one loss term by equivalently setting the weight  $\lambda$  to 0. The ablation results on UCF-HMDB, Jester, and Epic-Kitchens are shown in Tab. 5. As can be seen, removing  $\mathcal{L}_{\mathrm{cls}}$  significantly reduces the transfer performance in all the tasks. This is reasonable as  $\mathcal{L}_{\mathrm{cls}}$  is used to discover the discriminative features. Removing any of  $\mathcal{L}_{\mathrm{adv}}$ ,  $\mathcal{L}_{\mathrm{mi}}$ , and  $\mathcal{L}_{\mathrm{ctc}}$  leads to an inferior result than the full  $TranSVAE$  setup, and removing  $\mathcal{L}_{\mathrm{adv}}$  is the most influential. This is because  $\mathcal{L}_{\mathrm{adv}}$  is used to explicitly reduce the temporal domain gaps. All these results demonstrate that every proposed loss matters in our framework.
We further conduct another ablation study by sequentially integrating  $\mathcal{L}_{\mathrm{cls}}$ ,  $\mathcal{L}_{\mathrm{adv}}$ ,  $\mathcal{L}_{\mathrm{mi}}$ , and  $\mathcal{L}_{\mathrm{ctc}}$  into our sequential VAE structure using UCF-HMDB. We use this integration order based on the average positive improvement that a loss brings to TranSVAE as shown in Tab. 5. We also take advantage of t-SNE [39] to visualize the features learned by these different variants. We plot two sets of t-SNE
Figure 5: Loss integration studies on  $\mathbf{U} \rightarrow \mathbf{H}$ . Left: The t-SNE plots for class-wise (top row) and domain (bottom row, red source & blue target) features. Right: Ablation results (\%) by adding each loss sequentially, i.e., row (a) - row (e).
Figure 6: Loss integration studies on  $\mathbf{H} \rightarrow \mathbf{U}$ . Left: The t-SNE plots for class-wise (top row) and domain (bottom row, red source & blue target) features. Right: Ablation results (\%) by adding each loss sequentially, i.e., row (a) - row (e).
figures, one using the class-wise label and another using the domain label. Fig. 5 and Fig. 6 show the visualization and the quantitative results. As can be seen from the t-SNE feature visualizations, adding a new component improves both the domain and semantic alignments, and the best alignment is achieved when all the components are considered. The quantitative results further show that the transfer performance gradually increases with the sequential integration of the four components, which again verifies the effectiveness of each component in TranSVAE. More ablation study results can be found in the Appendix.
In this paper, we proposed a TranSVAE framework for video-based UDA tasks. Our key idea is to explicitly disentangle the domain information from other information during the adaptation. We developed a novel sequential VAE structure with two sets of latent factors and proposed four constraints to regulate these factors for adaptation purposes. Note that disentanglement and adaptation are interactive and complementary. All the constraints serve to achieve a good disentanglement effect with the two-level domain divergence minimization. Extensive empirical studies clearly verify that TranSVAE consistently offers performance improvements compared with existing SoTA video-based UDA methods. We also find that TranSVAE outperforms those multi-modal UDA methods, although it only uses single-modality data. Comprehensive property analysis further shows that TranSVAE is an effective and promising method for video-based UDA.
We further discuss the limitations of the TranSVAE framework. These are also promising future directions. Firstly, the empirical evaluation is mainly on the action recognition task. The performance of TranSVAE on other video-related tasks, e.g. video segmentation, is not tested. Secondly, TranSVAE is only evaluated on the typical two-domain transfer scenario. The multi-source transfer case is not considered but is worthy of further study. Thirdly, although TranSVAE exhibits better performance than multi-modal transfer methods, its current version does not consider multi-modal data functionally and structurally. An improved TranSVAE with the capacity of using multi-modal data is expected to further boost the adaptation performance. Fourthly, current empirical evaluations are mainly based on the I3D backbone, more advanced backbones, e.g., [36, 40], are expected to be explored for further improvement. Lastly, TranSVAE handles the spatial divergence by disentangling the static domain-specific latent factors out. However, it may happen that spatial divergence is not completely captured by the static latent factors due to insufficient disentanglement.
- Sec. A provides additional implementation details to assist reproductions.
- Sec. B contains additional comparative and ablation results of the TranSVAE framework.
Sec. C elaborates on the broader impact of this work.
- Sec. D acknowledges the public resources used during the course of this work.
In this section, we provide more implementation details including the I3D feature extraction procedure, the concrete model architecture, and the hyperparameter selection in our proposed TranSVAE framework. We also provide detailed instructions for our live demo.
We present Tab. A to summarize the notations used in this paper.
A.2 I3D Feature Extractions
We extract the I3D RGB features following the routine described in SAVA [10]. Given a video sequence, 16 frames along clips are sampled by sliding a temporal window with a temporal stride of 1. Specifically, for each frame in the video, the temporal window consists of its previous seven frames and the following eight frames. Zero padding is used for the beginning and the end of the video. We then feed the sliding windows to the I3D backbone to extract features, which results in a 1024-dimensional feature vector for each frame of the video.
A.3 Model Architecture
We now provide the detailed model architecture of our TranSVAE. In Fig. A, we show the model with the image as the input, where the encoder and decoder are more complex convolutional and deconvolutional layers. For the model with the RGB features as the input, we can simply replace the encoder and decoder with fully connected linear layers. Note that the dimensionality of all the modules shown above is uniformly applied in all the experiments.
A.4 Hyperparameter Selection
There are several hyperparameters used in TranSVAE, including the balancing weights  $\lambda_{1}$ ,  $\lambda_{2}$ ,  $\lambda_{3}$ ,  $\lambda_{4}$ , the number of the video frames  $T$ , and the confidence threshold  $\eta$  for generating target pseudo-labels. For  $\lambda_{1}$  to  $\lambda_{4}$ , we select from the value set  $\{1e^{-3}, 1e^{-2}, 1e^{-1}, 0.5, 1, 5, 10, 50, 100, 1000\}$ . For  $T$ , we select from  $\{5, 6, 7, 8, 9, 10\}$ . For  $\eta$ , we set its value range from 0.9 to 1.0 with a step of 0.01.
We set a high-value range of  $\eta$  to ensure a high confidence score on the correctness of the target pseudo-labels. Following the common protocol used in video-based UDA, we conduct an extensive grid search regarding these hyperparameters on the validation set of each transfer task. Tab. B summarizes the exact used values of these hyperparameters for the UCF-HMDB [35, 20], Jester [27], and Epic-Kitchens [11] UDA benchmarks. For theSprites [22] dataset, we do not do a hyperparameter search as the data is quite simple. We simply set  $T$  as 8, which is the original length of the video sequence. The confidence threshold is set to be 0.99, and  $\lambda_{1}$  to  $\lambda_{4}$  are all set to be 1.
A.5 Demo Instruction