Unsupervised Video Domain Adaptation for Action Recognition: A Disentanglement Perspective
Authors: Pengfei Wei
Lingdong Kong
Xinghua Qu
Yi Ren
zhiqiang xu
Jing Jiang
Xiang Yin
Venue: NeurIPS 2023 poster (2023)
Keywords: action recognition
unsupervised domain adaptation
video analysis
TL;DR: We present TranSVAE to tackle video domain adaptation tasks from a domain disentanglement aspect.

Abstract: Unsupervised video domain adaptation is a practical yet challenging task. In this work, for the first time, we tackle it from a disentanglement view. Our key idea is to handle the spatial and temporal domain divergence separately through disentanglement. Specifically, we consider the generation of cross-domain videos from two sets of latent factors, one encoding the static information and another encoding the dynamic information. A Transfer Sequential VAE (TranSVAE) framework is then developed to model such generation. To better serve for adaptation, we propose several objectives to constrain the latent factors. With these constraints, the spatial divergence can be readily removed by disentangling the static domain-specific information out, and the temporal divergence is further reduced from both frame- and video-levels through adversarial learning. Extensive experiments on the UCF-HMDB, Jester, and Epic-Kitchens datasets verify the effectiveness and superiority of TranSVAE compared with several state-of-the-art approaches.

Unsupervised Video Domain Adaptation for Action
Recognition: A Disentanglement Perspective
Pengfei Wei Al Lab, ByteDance pengfei. www.eeeeieeeeeeecoom com
Lingdong Kong National University of Singapore lingdong@comp nus edu. sg
Xinghua Qu
Yi Ren AI Lab. ByteDance
Zhiqiang Xu
Al Lab, ByteDance quxinghual77@gaiil com
MBZUAI
ren het.yyyyyyteddcccc com
zhiqiang wwwmbzuai.ac.ccommmmmmmommaao ae
Jing Jiang University of Technology Sydney Jing. Jianguuts. edu. au
Xiang Yin Al Lab, ByteDance
y sinciaang.stephe@@yyteaaaaa
Abstract
Unsupervised video domain adaptation is practical yet challenging task. In this work, for the first time, we tackle it from a disentanglement view. Our key idea is handle the spatial and temporal domain divergence separately through disen- tanglement. Specifically, we consider the generation of cross-domain videos from two sets of latent factors, encoding the static information and another encoding the dynamic information. A Transfer Sequential VAE (TranSVAE) framework is then developed model such generation. To better serve for adaptation, we propose several objectives to constrain the latent factors. With these constraints, the spatial divergence can be readily removed by disentangling the static domain- specific information out, and the temporal divergence is further reduced from both frame- and video-levels through adversarial learning. Extensive experiments on the UCF-HMDB, Jester, and Epic- DitchKKithenns datasets verify the effectiveness and superiority of TranSVAE compared with several state-of-the-ar approaches.'
Introduction
Over the past decades, unsupervised domain adaptation (UDA) has attracted extensive research attention [41]. Numerous UDA methods have been proposed and successfully applied various real-world applications, e.g., object recognition [38, 42, 47], semantic segmentation [51, 19, 33, 24], and object detection [3, 15, 45]. However, most of these methods and their applications are limited to the image domain, while much less attention has been devoted to video-based UDA, where the latter undoubtedly more challenging.
Compared with image- image-based UDA, the source and target domains also differ temporally in video- based UDA. Images are spatially well-structured data, while videos are sequences of images with both spatial and temporal relations. Existing image-base UDA methods can hardly achieve satisfactory performance on video-based UDA tasks as they fail to consider the temporal dependency of video frames in handling the domain gaps. For instance, in video-based cross-domain action recognition tasks, domain gaps are presented by not only the actions of different persons in different scenarios but also the actions that appear at different timestamps last at different time lengths.
Code is publicly available at: https: htt///tttttbthh/// hot.s//oocom111011125555505 TranSVAE.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

All-in-One View
Disentanglement View (Ours)
Source Domain
Target Domain
Source-Related Info
Barget-Related Info
Semantic-Related Info
Figure 1: Conceptual comparisons between the traditional all-in-one view and the proposed dis- entanglement view. Prior works often seek compress implicit domain information to obtain domain-indiisinguishaan representations; while this work, we pursue explicit decouplings of domain-specific information from other information via generative modeling.
Recently, few works have been proposed for video-based UDA. The key idea is to achieve domain alignment by aligning both frame- and video-level features through adversarial learning [7, 26], contrastive learning [37, 31J, attention [10], combination of these mechanisms, e-g., adversarial learning with attention [29, 8). Though they have advanced video-based UDA, there is still room for improvement. Generally, existing methods follow an all-in-one way, where both spatial and temporal domain divergence are handled together, for adaptation (Fig. 1, [). However, cross- doms-nn videos are highly complex data containing diverse mixed-up information, e.g., domain, semantic, and temporal information, which makes the simultaneous elimination of spatial and temporal divergence insufficient. This motivates us to handle the video-based UDA from a disentanglement perspective (Fig. 1, ) so that the spatial and temporal divergence can be well handled separately.
To achieve this goal, we first consider the generation process of cross-domain videos, as shown in Fig. 2, where a video is generated from two sets of latent factors: one set consists of a sequence of random variables, which are dynamic and incline to encode the semantic information for downstream tasks, e.g., action recognition: another set is static and introduces some domain-related spatial information to the generated video, e.g., style or appearance. Specifically, the blue / red nodes are the observed source target videos x /xT, respectively, over timestamps. Static latent variables z and z! follow a joint distribution and combining either of them with dynamic latent variables 2, constructs one video data of a domain.
Z
Z₁
2
Z
Z
With the above generative model, we develop a Transfer Sequential Variational AutoEncoder (TranSVAE) for video- based UDA. TranSVAE handles the cross-domain divergence in two levels, where the first level removes the spatial diver- gence by disentangling zu from 2₁; while the second level eliminates the temporal divergence of z, To achieve this, we leverage appropriate constraints to ensure that the disentan- glement indeed serves the adaptation purpose. Firstly, we enable a good decoupling of the two sets of latent factors by minimizing their mutual dependence. This encourages these two latent factor sets to be mutually independent. We then consider constraining each latent factor set. For z? with D E {S,T] we propose contrastive triplet loss to make them static and domain-speciffe handle spatial divergence by disentangling 2 out. For Z, we propose to align them across domains at both frame and video levels through adversarial learning divergence. Meanwhile, as downstream tasks use 2, as input, we also add the task-speciftt supervision on z₁ extracted from source data (w/ ground-truth).
Z
X
X
Figure 2: Graphical illustrations of the proposed generative (Ist figure) and inference (2nd figure) models for video domain disentanglement.
This makes us readily
as to further climinate the temporal
To the best of our knowledge, this is the first work that tackles the challenging video-based UDA from domain disentanglement view. We conduct extensive experiments on popular benchmarks (UCF-HMDB, Jester, Epic- Kitchesth and the results show that TranSVAE consistently outperforms previous state-of-the-- the methods by large margins. We also conduct comprehensive ablation studies
2

---

and disentanglement analyses to verify the effectiveness of the latent factor decoupling. The main contribution of the paper is summarized as follows:
We provide a generative perspective on solving video-based UDA problems. We develop a generative graphical model for the cross-domain video generation process and propose to utilize the sequential VAE as the base generative model.
Based the above generative view, we propose a TranSVAE framework for video-based UDA, By developing four constraints on the latent factors to enable disentanglement to ben- efit adaptation, the proposed framework capable of handling the cross-domain divergence from both spatial and temporal levels.
We conduct extensive experiments on several benchmark datasets to verify the effectiveness of TranSVAE. A comprehensive ablation study also demonstrates the positive effect of each loss term on video domain adaptation.
2 Related Work
Unsupervised Video Domain Adaptation. Despite the great progress in image-based UDA, only a few methods have recently attempted vidco-based UDA. In [7]. a temporal attentive adversarial adaptation network (TA is proposed to integrate temporal relation module for temporal alignment. Choi et al. [10] proposed a SAVA method using self-supervised clip order prediction and clip attention- based alignment. Based
cross-domain co-attention mechanism, the temporal co-attention network TCoN [29] focused on common key frames across domains for better alignment. Luo et al. [26] pay more attention to the domain-agnostic classifier by using a network topology of the bipartite graph to model the cross-domain correlations. Instead of using adversarial learning, Sahoo et al. [31] developed an end-to-end temporal contrastive learning framework named CoMix with background mixing and target pseudo-labels. Recently, Chen el al. [8] learned multiple domain discriminators for multi-level temporal attentive features to achieve better alignment, while Turrisi al. [37] exploited two-headed deep architecture to learn a more robust target classifier by the combination of cross- entropy and contrastive losses. Although these approaches have advanced video- bdsed UDA tasks, they all adopted to align features with diverse information mixed up from a compression perspective, which leaves room for further improvements.
Multi-Modal Video Adaptation. Most recently, there are also a few works integrating multiple modality data for iideo-based UDA. Although we only use the single modality RGB features, we still discuss this multi-modal research line for a complete literature review. The very first work exploring the multi-modal nature of videos for UDA is MM-SADA [28], where the correspondence of multiple modalities was exploited as a self-supervised alignment in addition to adversarial alignment. A later work. spatial-temporee contrastive domain adaptation (STCDA) [34], utilized a video-based contrastive alignment as the multi- modal domain metric to measure the video-level discrepancy across domains. [18] proposed cross-modal and cross-domain contrastive losses to handle feature spaces across modalities and domains. [43] leveraged both cross-modal complementary and cross-modal consensus to learn the most transferable features through a CIA framework. In [12], the authors proposed to generate noisy pseudo-labell labels for the target domain data using the source pre-trained model and select the clean samples in order to increase the quality of the pseudo-labels, Lin al. [23] developed cycle-based approach that alternates between spatial and spatiotemporal learning with knowledge transfer. Generally, all the above methods utilize the flow as the auxiliary modality input. Recently, there are also methods exploring other modalities, for instance, A3R [48] with audios and MixDANN [44] with wild data. It is worth noting that the proposed TranSVAE although only uses single modality RGB features- surprisingly achieves better UDA performance compared with most current state-of-the-an modti-modal methods, which highlights our superiority.
Disentanglement. Feature disentanglement is a wide and hot research topic. We only focus on works that are closely related to ours. In the image domain, some works consider adaptation from a generative view. [4] learned disentangled semantic representation across domains. A similar idea is then applied to graph domain adaptation [5] and domain generalization [16]. [13] proposed a novel informative feature disentanglement, equipped with the adversarial network or the metric discrepancy model. Another eeeentangemeent-ellttee topic is sequential data generation. To generate videos,
existing works [22, 50, I] extended VAE to recurrent form with different recursive structures. In
3

---

Input
Encoder
BI-STT
Lalent Space
Sample Predict
Decodor
Domsinsmnarrnn Sequental
Clasily
mima
( INLT
Domain-Static Factor
Channel Concal
Independent Eq.
Adversaiial (10)
Contraslive Triplet Eq. (11)
Task Supervision Eg. (15)
Figure 3: TranSVAE overview. The input videos are fed into an encoder extract the visual features, followed by an LSTM to explore the temporal information. Two groups of mean and variance networks are then applied to model the posterior of the latent factors, i.e., q(z, [x and qz xxPrr) The new representations 2 .... zi and z are sampled, and then concatenated and passed decoder for reconstruction. Four constraints are proposed to regulate the latent factors for adaptation.
this paper, we present a VAE-based structure to generate cross-domain videos. We aim at tackling video-based UDA from perspective: sequential domain disentanglement and transfer.
3 Technical Approach
Formally, for typical video-based UDA problem, we have source domain S and a target domain T. Domain S contains sufficient labeled data, denoted as {(V,, y I where V is video sequence and v is the class label. Domain T consists of unlabeled data, denoted as (V? J.r. For sequence VP from domain D E T. it contains T frames {x_1 i'....... x in total?. We further denote N = N NT. Domains S and T are of different distributions but share the same label space. The objective is to utilize both ((V v( } and (VT 1 to train good classifier for domain T. We present a table to list the notations in the Appendix.
We adopt a VAE-based structure to model the cross-domain video generation process and propose to regulate the two sets of latent factors for adaptation purposes. The reconstruc- tion process is based on the two sets of latent factors, i.e., 2p ..... z.7 and z? that are sampled from the posteriors (z, x )1) and g D x respectively. The overall architecture consists of five segments including the encoder, LSTM, latent spaces, sampling, and decoder, as shown in Fig. The vanilla structure only gives an arbitrary disentanglement of latent factors. To make the disentanglement facilitate adaptation, we carefully constrain the latent factors as follows.
Framework Overview.
Video Sequence Reconstruction. The overall architecture of TranSVAE follows a VAF-bbaed structure [22, 50, 1] with two sets of latent factors zP, ..., zi and z The generative and inference graphical models are presented in Fig. 2. Similar to the conventional VAE, we use a standard Gaussian distribution for static latent factors. For dynamic ones, we use sequential prior zi z N diig(()))))) that is, the prior distribution of the current dynamic factor is conditioned on the historical dynamic factors. The distribution parameters can be re-parameterized as a recurrent network, e.g., LSTM, with all previous dynamic latent factors as the input. Denoting Z {z? zF) for simplification, we then get the prior as follows:
p 2P) p(Z2) II 1
(1)
Withoott confusion, omit the subscript for V and the corresponding notations, egg., (xP ,..,, ...,...] xP).
(xP,...,

---

Following Fig. 2 (the 1st subfigure), x; is generated from zy and z and we thus model p(x? z x Nu )) The distribution parameters are re-parameterized by the decoder which can be flexible networks like the deconvolutional neural network. Using V {x] ,T.,,,,,,, the generation can be formulated as follows:
P(V)) = p(z II p(x
22 24 (2
Following Fig. 2 (the 2nd subfigure), we model the posterior distributions of the latent factors as another two Gaussian distributions, i.e., (<z)/V)) (H diag(o?)) and q(z, xx+x) = Nip tiag((o"")) The parameters of these which can be convolutional or LSTM module. However, the network of the static latent factors uses the whole sequence as the input while that of the dynamic latent factors only uses previous frames. Then the inference can be factorized as:
distributions
ee-parameterized by the encoder,
T II (
(
x2))
(3)
Combining the above generation and inference, obtain the VAE-related objective function as:
Lyvae By(a By(ep -
logp(x? 12, 22)+
(4)
(L(g D V ((22))
Σ [
which is a frame-wise negative variational lower bound. Only using the above vanilla VAE-based loss cannot guarantee that the disentanglement serves for adaptation, and thus we propose additional constraints on the two sets of latent factors.
Mutual Dependence Minimization (Fig. 3, We first consider explicitly enforcing the two of latent factors to be mutually independent. To do so, we introduce the mutual information [2] loss Lmi regulate the two sets of latent factors. Thus, we obtain:
T Σ KL (g(z) (l)((()))) = > (((2))+((Z)) (=
Cii 2 2
+ H(z?) (2 2 (5)
To calculate Eq. (5), need estimate the densities of 2P x? and (z. 2.). Following the non-parametric way in [9], we use the mini-batch weighted sampling as follows:
1 M
Hz)) (x [og x log
Mlg
MN E
(6)
where 2F x (z) 22), N denotes the data size and M is the mini-batch size.
Domain Specificity & Static Consistency (Fig. 3, ). characteristic the domain specificity, e.g.. the video style the objective appearance, is its static consistency over dynamic frames. With this observation, we enable the static and commain-specife latent factors so that we can remove the spatial divergence by disentangling them out. Mathematically,
hope that z does not change lot when zP varies over time. To achieve this, given sequence, we randomly shuffle the temporal order of frames to form a new sequence. The static latent factors disentangled from the original sequence and the shuffled sequence should be ideally equal or be very close at least. This motivates us minimize the distance between these two static factors. Meanwhile, to further enhance the domain specificity, enforce the dynamic latent factors from different domains have large distance. To this end, we propose the following contrastive triplet loss:
Lec = max (D(Z)' 23 D(2z 22 ) ) +mm.0) m.m).
where .)..)) .) is Euclidean distance, is margin set to 1 in the experiments, xp" 22 and 2 are static latent factors of the anchor sequence from domain D+, the shuffled sequence, and a randomly selected sequence from domain D respecctively. D+ and D- represent two different domains.
5

---

Domain Temporal Alignment (Fig. 3, We now consider to reduce the temporal divergence of the dynamic latent factors. There are several ways to achieve this, and in this paper, we take advantage of the most popular adversarial-baasee
idea [14]. Specifically, we build domain classifier to discriminate whether the data from or TT When back propppappggg the gradients, a gradient reversal layer (GRL) is adopted to invert the gradients. Like existing video-based UDA methods, we also conduct both frame-level and video-level alignments. Moreover, as TA [7] does, we exploit the temporal relation network (TRN) [49] to discover the temporal relations among different frames, and then aggregate all the temporal relation features into the final video-level features. This enables another level of alignment the temporal relation features. Thus, we have:
T
Cj
CE (G)())))
(8)
C,
1
Σ n=
CE
Gr (TrNa ,d.]:
T
T Σ n=
1 N 2 CE G T T-
(TNN, .d
C.
(10)
where de is the domain label, CE denotes the cross-entropy loss function. z,P = {z, ,,.,, 2.. _ }
TrN; is the n-frame temporal relation network, Gy. Gr. and G, are the frame feature level, the temporal relation feature level, and the video feature level domain classifiers, respectively. To this end, we obtain the domain adversarial loss by summing up Eqs. (8-10):
Lad Lj + C, + L...
(11)
We assign equal importance to these three levels of losses to reduce the overhead of the hyperparameter search. To this end, with Lmu. Lete and Ladv, the learned dynamic latent factors are expected to be domain-invann invariant (the three constraints are interactive and complementary to each other for obtaining the domain-invariann dynamic latent factor), and then can be used for downstream UDA tasks. this paper, we specifically focus action recognition as the downstream task.
Task Specific Supervision (Fig.
We further encourage the dynamic latent factors carry the semantic information. Considering that the source domain has sufficient labels, we accordingly design the task supervision as the regularization imposed on z This gives us:
N Las NS EC(F()).
(12)
where F(-) is feature transformer mapping the frame-level features to video-level features, specifi- cally TRN in this paper, and -(:--) is either cross- cross-entropy or mean squared error loss according to the targeted task.
Although the dynamic latent factors are constrained to be domain-invariant we do not completely rely on source semanties to learn features discriminative for the target domain. We propose to incorporate target pseudo-labels in task-specific supervision. During the training, use the prediction network obtained in the previous epoch to generate the target pscudo-labels of the unlabelled target training data for the current epoch. However, to increase the reliability of target pseudo-labels, we let the prediction network be trained only on the source supervision for several epochs and then integrate the target pseudo-labele in the following training epochs. Meanwhile, confidence threshold is set to determine whether to use the target pseudo-labels or not. Thus, we have the final ssecifpe supervision follows:
NS CCFES) + + EFFF)))
NT
Lots N
(13)
where o/7 is the pseudo-label of ZT.
Summary. To this end, we reach the final objective function of our TranSVAE framework follows: L' = Lyvae + Cmi + >2Cadv + Xa + Aa
(14)
where X; with i 2, 3, 4 denotes the loss balancing weight.

---

4 Experiments
In this section, we conduct extensive experimental studies on popular video-based UDA benchmarks to verify the effectiveness of the proposed TranSVAE framework.
4.1 Datasets
UCF-HMDB is constructed by collecting the relevant and overlapping action classes from UCF 101 [35] and HMDB51 [20]. It contains 3,209 videos in total with 1,438 training videos and 571 validation videos from UCF101, and 840 training videos and 360 validation videos from HMDB51- This in turn establishes two aideo-based UDA tasks: U - H and H U.
Jester [27] consists of 148,092 videos of humans performing hand gestures. Pan et al. [29] constructed a large-scale cross-domain benchmark with seven gesture classes, and form a single transfer task Js Jr, where Jg and J contain 51,498 and 51,415 video clips, respectively.
Epic-Kitchens [11] is a challenging egocentric dataset consisting of videos capturing daily activities in kitchens. [28] constructs three domains across the eight largest actions. They are D,, D2. and D, corresponding to PO8, P01, and P22 kitchens of the full dataset, resulting in six cross-domain tasks.
Sprites [22] contains sequences of animated cartoon characters with 15 action categories. The appearances of characters are fully controlled by four attributes, i.e., body, top wear, bottom wear, and hair. We construct two domains, P1 and P2. Pi uses the human body with attributes randomly selected from top wears, bottom wears, and hairs, while P2 uses the alien body with attributes randomly selected from 4 top wears, 3 bottom wears, and 5 hairs. The attribute pools are non- overlapping across domains, resulting in completely heterogeneous P1 and P2. Each domain has 900 video sequences, and each sequence is with 8 frames.
4.2 Implementation Details
Architecture. Following the latest works [31, 37J, we use I3D [6] as the backbone". However, different from CoMix which jointly trains the backbone, on Kinetics [17], provided by [6]. to extract RGB features. For the first three benchmarks, RGB features are used as the input of TranSVAE. For Sprites, we use the original image as the input, for the purpose of visualizing the reconstruction and disentanglement results. We use the shared encoder and decoder structures across the source and target domains. For RGB feature inputs. the encoder and decoder are fully connected layers. For original image inputs, the encoder and decoder are the convolution and deconvolution layers (from DCGAN [46]), respectively. For the TRN model, we directly use the one provided by [7]. Other details on this aspect are placed in Appendix.
simply use the pretrained I3D model
Configurations. Our TranSVAE is im- plemented with PyTorch [30]. We use
Table I: UDA performance comparisons on UCF-HMDB.
Adam with weight decay of le as the optimizer. The learning rate is ini-
Method & Year
Backhone
U - H
H U
Average
DANN (JMLR 16)
ResNet-101
75.28
76.36
75.82
tially to be - and follows com- monly used decreasing strategy in [14]. The batch size and the learning epoch are uniformly set to be 128 and 1,000, respectively. for all the experiments. We uniformly set 100 epochs of train-
JAN (ICML 17)
ResNet-101
71.72
76.69
75.71 71
AdaBN (PR 18)
ResNet-101
72.22
77.41
74.82
MCD (CVPR'I 18)
ResNet-101
73.89
79.31
76.62
TA'N (ICCV119)
R11
78.33
81.79
80.06
ABG (MM'20)
ResNet-101
79.17
85
82, 14
TCoN (AAAI'2))
ResNet-101
87.22
89.14
88.18
MA*L-TD (WACV'22)
ResNet-101
85.00
86.59
85.80
Source-only (Smit)
B3D
80.27
$8.77 19.79
81.33
ing under only source supervision and involved the target pseudo-labed labels after- ward. Following the common protocol in aideo-based UDA [37], we perform hyperparameter selection on the valida- tion set. The specific hyperparameters used for each task can be found in the Appendix. NVIDIA A100 GPUs are used for all experiments. Kindly refer to our Appendix for all other details.
DANN (JMLR' 16)
BDD
80.83
88 00
84.46
ADDA (CVPR'17 17)
79.17
83.81
TA'N (ICCV119)
I3D
81.38
90.5-
85.96
SAVA (BCCV'
B3D
82.22
91.21
86.73
CoMix (NeurIPS'21)
13D
86.66
93.87
90.22
CO²A (WACV'22)
13D
87.78
95.79
91.79
TranSVAE (Ours)
L3D
87,
98 9,
93.37
Supervised-target (Tap)
B3D
95.00
96.85
95.93
'The existing widely adopted backbones for video-basee UDA include ResNet- 101 and I3D. However, more recent backbones, e.g. Transformer- based or AideoMAE-based [36, 40J. promising to be used in this task.
7

---

Table 2: UDA performance comparisons to approaches using multi-modality data as the input.
Task
Sudy
MM-SADA
STCDA
CMCD
A3R
CleanAdapt
CycDA
MixDANN
CIA
TranSVAF
U + H
86.1
84.2
83.1
84.7
/
89.8
88.1
82.2
88.3
87.8 (+1.7)
H ++U U
92.5
91.1
92.1
92.8
/
99.2
98.0
92.8
94.1
99.0 (40.5)
Average
89.3
87.7
87.6
88.3
/
94.5
93.1
87.5
91.2
93.4 (+4..))
D + D2
43.2
19.5
62.0
50.3
53.2 2
52.7
56.0
52.5
50.5 (--..))
D1 D3
42.5
44.1
45.5
46.3
52.
47.0
47.3
47.8
50.3 (17.8)
D2 D1
43.0
48.2
49.0
49.5
51.9
46.2
50.3
49.8
50.3
D2 + Dy
48.0
52.7
52.5
52.0
55.5
52.7
52.4
53.2
58. 6 (4-10.0)
Ds Ds
43.0
50.9
52.6
48.7
51.5
47.8
51.0
52.2
48. (1.00))
D3 D2
55.5
56.1
55.6
56.3
63.22
54.4
54.7
57.6
588.0
Average 45.9
50.3
51.2
51.0
54.1
50.3
52.0
52.2
52.6 (46.7)
Competitors. We compared methods from three lines. We first consider the source-only (Sonly) and anpervised-targer (Twu)) which uses only labeled source data and only labeled target data, respectively. These two baselines serve the lower and upper bounds for our tasks. Secondly, we consider five popular image-based UDA methods by simply ignoring temporal information, namely DANN [14], JAN [25], ADDA [38], AdaBN [21], and MCD [32]. Lastly and most importantly, we compare recent SoTA video- basio UDA methods, including T [7]. SAVA [10]. TCoN [29]. ABG [26], CoMix [31], CO? A [37], and LA [8]. All these methods use single modality features. We directly quote numbers reported in published papers whenever possible. There cxist recent works conducting video-based UDA using multi-modal data, e.g. RGB + Flow. Although TranSVAE soiely uses RGB features, we still take this set of methods into account. Specifically, we consider MM-SADA [28], STCDA [34], CMCD [18], A3R [48], CleanAdapt [12], CycDA [23], MixDANN [44] and CIA [43].
4.3 Comparative Study
Results on UCF-HMDB. Tab. 1 shows comparisons of TranSVAE with baselines and SoTA methods on UCF-HMDB. The best result among all the baselines is highlighted using bold. Overall, methods using the 13D backbone [6] achieve better results than those using ResNet-101. Our TranSVAE consistently outperforms all previous methods. particular. TranSVAE achieves 93.37% average accuracy, improving the best competitor CO²A A [37], with the same I3D backbone [6]. by 1.38%. Surprisingly, we observe that TranSVAE even yields better results (by 2.1% improvement) than the supervised-targee (Truр) baseline. This is because the H - U task already has a good performance without adaptation, i.e., 88. 79% for the source-only (Sanly) baseline, and thus the target pseudo- used in TranSVAE are almost correct. By further aligning domains and equivalently augmenting training data, TranSVAE outperforms Toup which only trained with target data.
Results on Jester & Eiic-Kitchens. Tab. 3 shows the comparison results on the Jester and KitcheKs bench- marks. We can see that our TranSVAE is the clear winner among all the meth- ods on all the tasks. Specifically, TranSVAE achieves 1.4% improve-
Table 3: Comparison results Jester and BoiccKiichenn.
Task
Swiy
DANN
ADDA
TA
CoMix
TranSVAE
Tap
J8 + Jr
31.5
55.4
52.3
55.3
61.7
66.1 (:2..))
93.6
D. D2
32.8
37.7
35.4
34.2
42.9
50.5
64.0
D. + D:
34.1
36.6
34.9
37.4
40.9
50.3 (:.0.))
63.7
D2
35.4
38.3
36.3
40.9
38.6
50.3
57.0
ingeeerroooo ment and 9.4% average improvement over the runner-up baseline CoMix [31] on Jester and Epic-Kitchens, re- spectively. This verifies the superiority of TranSVAE over others in handling video-based UDA. However, we also notice that the accuracy gap between CoMix and Tsup is still significant on Jester. This is because the large-scale Jester dataset contains highly heterogeneous data across domains, e.g., the source domain contains videos of the rolling hand forward, while the target domain only consists of videos of the rolling hand backward. This leaves much room for improvement in the future.
D; ++ D;
39.1
41.9
40.8
42.8
45.2
58.6 (:30.)
63.7
D. D.
31.6
38.8
36.1
39.9
42.3
48.0 (438.4)
57.0
D. ++ D.
35.8
42.1
41.4
44.2
49.2
58 (4222.2
64.0
Average 35.3
30.2
37.4
39.9
43.2
52.6 (137.91
61.5
Compare to Multi-Modal Methods. We further compare with four recent video- based UDA methods that use multi- modaliimddliiees, e.g. RGB features, and optical flows, although TranSVAE only uses RGB features. Surprisingly, TranSVAE achieves better average results than seven out eight multi-modal methods and is only worse than CleanAdapt [12] on UCF-HMDB and A3R [48] on SSpic-Kitched Considering TranSVAF only uses single-modality data, we are confident that there exists great potential for further improvements of TVanSVAE with multi-modal data taken into account.
8

---

Table 5: Loss separation study different video-based UDA tasks by dropping each loss sequentially.
C.
Lin
Lix
C.
C.
PL
U
11
I1
U
J. Jy
D.
D:
D.
D:
D. D.
D: D.
D. D.
D:
D:
Arg.
25.62
22.92
30.29
33.79
30:49
34,27
83.05
93.52
48.07
40.93
43.33
43.91
51.13
41.84
12.67
03.08
03.22
65.12
49:43
55.34
43.52
54.53
V
63,89
85.00
64.50
08.05
48.25
15. 06
54.21
65.52
30.73
80.65
36:47
4
87.78
98 95
66.10
50.53
50.31
50.34
58.02
48.04
58.00
63.19
Figure 4: Domain disentanglement and transfer examples. Left: Video sequence inputs for D P ("Human", and D P2 ("Alien", D. Middle: Reconstructed sequences (M) with 2P: :..., 21.
Right: Domain transferred sequences with exchanged z?-
4.4 Property Analysis
Disentanglement Analysis. We analyze the disentanglement effect of TranSVAE on Sprites [22] and show results in Fig. 4. The left subfigure shows the original sequences of the two domains. The "Human" and "Alien" are completely different appearances and the former is casting spells while the latter is slashing. The middle subfigure shows the sequences reconstructed only using {zP,... ..... zP) can be clearly seen that the two sequences keep the same action as the corresponding original ones. However, if we only focus on the appearance characteristics, it is difficult to distinguish the domain to which the sequences belong. This indicates that {zp ...., zP] are indeed domain-invarian and well encode the semantic information. The right subfigure shows the sequences reconstructed by exchanging z!, which results in two sequences with the same actions but exchanged appearance. This verifies that z is representing the appearance information, which is actually the comain-relatte information in this example. This property study sufficiently supports that TranSVAE can successfully disentangle the domain information from other information, with the former embedded in z and the latter embedded in {zp ,..,...,22
....,,?! zP).
Complexity Analysis. We further con- duct complexity analysis on our TranSVAE. Specifically, we compare the number of trainable parameters, multiply-accumulate operations (MACs), floating-point opera- tions (FI.Oss,, and inference frame-per- second (FPS) with existing baselines in- cluding T [7], COºA [37], and CoMix [31]. All the comparison results are shown in Tab. 4. From the table, we observe that TranSVAE requires less trainable parameters than CO²A A and CoMix. Although more trainable parameters are used than TA N, TVA achieves significant adaptation performance improvement than TA (see Tab. 1 and Tab. 3). Moreover, the MACs, FLOPs, and FPS are competitive among different methods. This is reasonable since all these approaches adopt the same 13D backbone.
Table 4: Comparison results on model complexity.
Methods
Trainuble Params
MACS
FLOPs
FPS
TA'N
7,6830 M
18.2318 G
36.4638 G
0.0134
CoMix
30.3658 M
18.5640 G
37. G
0.0157
CORA
23.6720 M
18.1884 G
36.3768 G
0.0127 s
TranSVAE
12.7419 M
18.2657 G
36.5311 G
0.0133 s
Ablation Study. We now analyze the effectiveness of each loss term in Eq. (14). We compare with four variants of TrARSVAF, each removing one loss term by equivalently setting the weight À to O. The ablation results on UCF-HMDB, Jester, and Epic-Kinug Kitchess are shown in Tab. 5. As can be seen, removing Lets significantly reduces the transfer performance in all the tasks. This is reasonable as Lets is used to discover the discriminative features. Removing any of Ladv. Cmis and Leve leads to an inferior result than the full TranSVAE setup, and removing Ladv is the most influential. This is because Ludy is used to explicitly reduce the temporal domain gaps. All these results demonstrate that every proposed loss matters in our framework.
We further conduct another ablation study by sequentially integrating Lessy Ladvs Lmis and Lase into our sequential VAE structure using UCF-HMDB. We use this integration order based on the average positive improvement that loss brings to TranSVAF as shown in Tab. We also take advantage of 1-SNE [39] to visualize the features learned by these different variants. We plot two sets of t-SNE

---

(0)
Fress beta kuti leee Aire
Accurccy (74)
Kosty
v
82 50
(b)
V
J
of
22
(6)
of
Fepp
Figure 5: Loss integration studies on U + H. Left: The t-SNE plots for class-wise (top row) and domain (bottom row, red source & blue target) features. Right: Ablation results (%) by adding each loss sequentially, i.e., row (a) row (e).
(c)
Feo: fygs
Fat
PL
Accurscy (%)
Susly
Bi 70
(5)
J
4 of
9t 70
(4)
V
o
d
Tesp
96.85 B5
Figure 6: Loss integration studies on H U. Left: The t-SNE plots for class-wise (top row) and domain (bottom row, red source & blue target) features. Right: Ablation results (%) by adding each loss sequentially, i.e., row (a) row (e).
figures, one using the class-wise label and another using the domain label. Fig. 5 and Fig. 6 show the visualization and the quantitative results. As can be seen from the t-SNE feature visualizations, adding new component improves both the domain and semantic alignments, and the best alignment is achieved when all the components are considered. The quantitative results further show that the transfer performance gradually increases with the sequential integration of the four components, which again verifies the effectiveness of each component in TranSVAE. More ablation study results can be found in the Appendix.
5 Conclusion and Limitation
In this paper, we proposed a TranSVAE framework for video-based UDA tasks. Our key idea is to explicitly disentangle the domain information from other information during the adaptation. We developed a novel sequential VAE structure with two sets of latent factors and proposed four constraints to regulate these factors for adaptation purposes. Note that disentanglement and adaptation are interactive and complementary. All the constraints serve to achieve a good disentanglement effect with the two-level domain divergence minimization. Extensive empirical studies clearly verify that TrAnSVAE consistently offers performance improvements compared with existing SoTA video-basee UDA methods. We also find that TranSVAE outperforms those multi-modal UDA methods, although it only uses single- modality data. Comprehensive property analysis further shows that TranSVAE is an effective and promising method for video- based UDA.
We further discuss the limitations of the TranSVAE framework. These are also promising future directions. Firstly, the empirical evaluation is mainly the action recognition task. The performance of TranSVAE on other video-related tasks, e.g. video segmentation. is not tested. Secondly, TranSVAE only evaluated on the typical two-domain transfer scenario. The multi-source transfer case is not considered but is worthy of further study. Thirdly, although TranSVAL exhibits better performance than multi-modal transfer methods, its current version does not consider multi mooii data functionally and structurally. An improved TranSVAE with the capacity of using multi-moda. data is expected to further boost the adaptation performance. Fourthly, current empirical evaluations are mainly based on the 13D backbone, more advanced backbones, e.g., [36, 40], are expected to be explored for further improvement. Lastly, TranSVAF handles the spatial divergence by disentangling the static domain-specine latent factors out. However, it may happen that spatial divergence is not completely captured by the static latent factors due to insufficient disentanglement.
10

---

References
[1] Junwen Bai, Weiran Wang. and Carla P Gomes. Contrastively disentangled sequential variational autoen- coder. In Advances in Neural Information Processing Systems, volume 34, pages 10105-10118, 2021. 3,
[2] Lejla Batina, Benedikt Gierlichs, Emmanuel Prouff, Matthieu Rivain, François-Xavice Standaert, and Nicolas Veyrat-Charvillon Mutual information analysis: a comprehensive study. Journal of Cryptology, 24(2)269-291, 2012, 2011.
[3] Qi Cai, Yingwei Pan, Chong- Wah Ngo, Xinmei Tian, Lingyu Duan, and Ting Yao. Exploring object relation in mean teacher for cross-domain detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11457-11466 2019.
[4] Ruichu Cai, Zijian Li, Pengfei Wei, Jie Qiao, Kun Zhang, and Zhifeng Hao. Learning disentangled semantic representation for domain adaptation. In International Joint Conferencee on Artificial Intelligence, volume 2019, page 2060, 2019. 3
[5] Ruichu Cai, Fengzhu Wu, Zijian Li. Pengfei Wei, Lingling Yi, and Kun Zhang. Graph domain adaptation: A generative view. arXiv preprint arXiv:2106.0744 074482 2021.
[6] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6299 6308, 2017. 7,
[7] Min-Hung Chen, Zsolt Kira, Ghassan AlRegib. Jackwon Yoo, Ruxin Chen, and Jian Zheng. Temporal attentive alignment for large-scale video domain adaptation. In /EEE/CVF International Conference on 8, Computer Vision, pages 6321 22330, 2019, 3, 6, 9, 17
[8] Peipeng Chen, Yuan Gao, and Andy Ma. Multi-level attentive adversarial learning with temporal dilation for unsupervised video domain adaptation. In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1259-1268, 2022. 2,
[9] Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems, volume 31, pages 1-11, 2018. 5
[10] Jinwoo Choi, Gaurav Sharma, Samuel Schulter, and Jia-Bin Huang. Shuffle and attend: Video domain adaptation. In European Conference on Computer Vision, pages 678-695. Springer, 2020. 2, 3, 8, 14
[11] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In European Conference on Computer Vision, pages 720-736. Springer, 2018.
7, 14
[12] Avijit Dasgupta, CV Jawahar, and Karteek Alahari. Overcoming label noise for source-free unsupervised video domain adaptation. In Proceedings of the Thirteenth Indian Conference Computer Vision, Graphics and Image Processing, pages 1-9, 2022. 3,8
[13] Wanxia Deng, Lingjun Zhao, Qing Liao, Deke Guo, Gangyao Kuang, Dewen Hu, Matti Pietikäinen, and Li Liu, Informative feature disentanglement for unsupervised domain adaptation. IEEE Transactions on Multimedia, 24:2407-2421, 2021. 3
[14] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training neural networks. Journal of Machine Learning Research, 17(1):2096-2300, 2016. 6, 7,8
[15] Dayan Guan, Jiaxing Huang, Aoran Xiao, Shijian Lu, and Yanpeng Cao. Uncertainty-aware unsupervised domain adaptation in object detection. IEEE Transactions on Multimedia, 24:2502-255 2021. 1
[16] Maximilian Iise, Jakub Tomezak, Christos Louizos, and Max Welling. Diva: Domain invariant variational autoencoders. In Medical Imaging with Deep Learning, pages 322-348. PMLR, 2020. 3
[17] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier. Sudheendra Vjjaanarasimmhnn Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705,0000 06950, 2017. 7
[18] Donghyun Kim, Yi- Hsuan Tsai, Bingbing Zhuang, Xiang Yu, Stan Sclaroff, Kate Saenko, and Manmohan Chandraker. Learning cross- cross-modal contrastive features for video domain adaptation. In IEEE/CVF International Conference Computer Vision, pages 13618 13627, 2021. 3,
11

---

[19] Lingdong Kong, Niamul Quader, and Venice Erin Liong, Conda: Unsupervised domain adaptation for lidar segmentation via regularized domain concatenation. In IEEE International Conference on Robotics and Automation, pages 9338-9345, 2023.
[20] Hildegard Kuchne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre Hmdb: large video datahase for human motion recognition. In IEEF/CVF International Conference on Computer Vision, pages 2556-2563. TEEE, 2011. 7,
[21] Yanghao Li, Naiyan Wang. Jianping Shi, Xiaodi Hou, and Jiaying Liu. Adaptive batch normalization for practical domain adaptation. Pattern Recognition, 80:109-10 (109-117,70 2018. 8
[22] Yingzhen Li and Stephan Mandt. Disentangled sequential autoencoder. In International Conference on Machine Learning, pages 5670-5679, PMLR, 2018, 3, 4, 7, 9, 14, 19
[23] Wei I. Lin, Anna Kukleva, Kunyang Sun, Horst Possegger, Hilde Kuehne, and Horst Bischof. Cyeda: Unsupervised cycle domain adaptation learn from image video. In European Conference on Computer Vision, pages 698-715. Springer, 2022. 8.8
[24] Youquan Liu, Lingdong Kong, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, and Ziwei Liu. Segment any point cloud sequences by distilling vision foundation models. arXiv preprint (.iiv:23366099444 2023.
[25] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adaptation networks. In International Conference on Machine Learning, pages 2208-2217. PMLR, 2017.
[26] Yadan Luo, Zi Huang, Zijian Wang. Zheng Zhang, and Mahsa Baktashmotlagh, Adversarial bipartite graph learning for video domain adaptation. In ACM International Conference on Multimedia, pages 19-27,
2020. 2, 3 8
[27] Joanna Materzynska, Guillaume Berger, Ingo Bax, and Roland Mcmisevic. The jester dataset: large-scale vidco dataset of human gestures. In /EEE/CVF International Conference on Computer Vision Workshops, pages 1- 2019. 7, 14
[28] Jonathan Munro and Dima Damen. Multi-modal domain adaptation for fine-grained action recognition. In JEEE/CVF Conference Computer Vision and Pattern Recognition, pages 122 2020. 3, 7, 8
[29] Boxiao Pan, Zhangjic Cao, Ehsan Adeli, and Juan Carlos Nicbles. Adversarial cross cross-domain action recognition with co-attention. In AAAI Conference on Artificial Intelligence, pages 11815-11822, 2020. 2, 3, 7, 8
[30] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary Veto Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high performance deep learingg library. In Advances in Neural Information Processing Systems, 2019. 7
[31] Aadarsh Sahoo, Rutav Shah, Rameswar Panda, Kate Sacnko, and Abir Das. Contrast and mix: Tempo- contrastive video domain adaptation with background mixing. In Advances in Neural Information Processing Systems, volume 34, pages 23386-23400, 2021. 2, 8,
[32] Kuniaki Saito. Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrep- ancy for unsupervised domain adaptation. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3723-3732, 2018. 8
[33] Antoine Saporta, Arthur Douillard, Tuan-Hung Vu, Patrick Pérez, and Matthiuu Cord. Multi-head distilla- tion for continual unsupervised domain adaptation in semantic segmentation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9751-37661 2022. 1
[34] Xiaolin Song, Sicheng Zhao, Jingyu Yang, Huanjing Yue, Pengfei Xu, Runbo Hu, and Hua Chai. Spatio- temporal contrastive domain adaptation for action recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9787-9795, 2021. 3,
[35] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprini 122.0222 2012. 7, 14
[36] Zhan Tong. Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self superviedd video pre- pre-training. Advances in Neural Information Processing Systems. 15:10078-1000 2022. 10
12

---

[37] Victor G Turrisi, Giacomo Zara, Paolo Rota, Thiago Oliveira-Santos Nicu Sebe, Vittorio Murino, and Elisa Ricci. Dual-head contrastive domain adaptation for video action recognition. In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2234-2243, 2022. 2,
[38] Eric Tzeng, Judy Hoffiman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7167-7176, 2017. 1,8
[39] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(11), 2008. 9
[40] Limin Wang, Bingkun Huang. Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang. Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14549-14560, 2023. 7, 10
[41] Garrett Wilson and Diane Cook. A survey unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems and Technology, 11(5):1 16),)-44222200 2020.
[42] Xiao and Lei Zhang. Dynamic weighted learning for unsupervised domain adaptation. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15242-15251, 2021.
[43] Lijin Yang, Yifei Huang, Yusuke Sugano. Yoichi Sato. Interact before align: Leveraging cross-mos moosl knowledge for domain adaptive action recognition, In /EEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14722-14732, 2022. 3,
[44] Yuehao Yin, Bin Zhu, Jingjing Chen, Lechao Cheng, and Yu-Gang Jiang. Mix-dann and dynamic- modal-distillation for video domain adaptation. In ACM International Conference on Multimedia, pages 3224 3233, 2022. 8
[45] Fuxun Yu, Di Wang. Yinpeng Chen, Nikolaos Karianakis, Tong Shen, Pei Yu, Dimitrios Lymberopoulos. Sidi Lu, Weisong Shi, and Xiang Chen. Sc-uda: Style and content gaps aware unsupervised domain adaptation for object detection. IEEE/CVF Winter Conference on Applications of Computer Vision, pages 382-391, 2022.
[46] Yang Yu. Zhiqiang Gong, Ping Zhong, and Jiaxin Shan. Unsupervised representation learning with deep convolutional neural network for remote sensing images. International Conference on Image and Graphics, pages 97-108. Springer, 2017.
[47] Jingyi Zhang. Jiaxing Huang, Zichen Tian, and Shijian Lu. Spectral unsupervised domain adaptation for visual recognition. In IREF/CVF Conference on Computer Vision and Pattern Recognition, pages 9829-9840, 2022.
[48] Yunhua Zhang, Hazel Doughty, Ling Shao, and Cees GM Snock. Audio-adaptive activity recognition across video domains. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13791-13800, 2022. 3,
[49] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In European Conference Computer Vision, pages 803-818. Springer, 2018. 6
[50] Yizhe Zhu, Martin Renqiang Min, Asim Kadav, and Hans Peter Graf. S3vae: Self-supervised sequential for representation disentanglement and data generation. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6538-6547, 2020. 3,
[51] Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. European Conference on Computer Vision, pages 289-305. Springer, 2018.
13

---

Appendix
In this appendix, we supplement the following materials to support the findings and conclusions drawn in the main body of this paper:
Sec. A provides additional implementation details to assist reproductions.
Sec, B contains additional comparative and ablation results of the TranSVAE framework.
Sec, C elaborates on the broader impact of this work.
Sec, D acknowledges the public resources used during the course of this work.
A
Additional Implementation Details
In this section, we provide more implementation details including the I3D feature extraction proce- dure, the concrete model architecture, and the hyperparameter selection in our proposed TranSVAE framework. We also provide detailed instructions for our live demo.
A.1
Notations
present Tab. A to summarize the notations used this paper.
A.2 I3D Feature Extractions
We extract the I3D RGB features following the routine described in SAVA [10]. Given video sequence, 16 frames along clips are sampled by sliding a temporal window with a temporal stride of 1. Specifically, for each frame in the video, the temporal window consists of its previous seven frames and the following eight frames. Zero padding is used for the beginning and the end of the video. We then feed the sliding windows to the 13D backbone extract features, which results in 1024-dimensional feature vector for each frame of the video.
A.3 Model Architecture
now provide the detailed model architecture of our TranSVAE. In Fig. A, we show the model with the image as the input, where the encoder and decoder are more complex convolutional and deconvolutional layers. For the model with the RGB features as the input, we can simply replace the encoder and decoder with fully connected linear layers. Note that the dimensionality of all the modules shown above is uniformly applied in all the experiments.
A.4 Hyperparameter Selection
There several hyperparameters used in TranSVAF, including the balancing weights λ₃ A2. Àg. Ay the number of the video frames T, and the confidence threshold for generating target pscudo-labels. For A₁ to AĄ. we select from the value set [le-1 3, 1e le 0.5, 10, 50, 100, 1000}. For T, we select from (5, 6, 7, 8, 9, 10). For we set its value range from 0.9 to 1.0 with step of 0.01.
We high-value range of 77 ensure high confidence score the correctness of the target pseudo-labels. Following the common protocol used in video-based UDA, we conduct an extensive grid search regarding these hyperparameters summarizes the exact used values of these hyperparameters for the UCF-HMDB [35, 20). Jester [27], and Epic-Kite Kith-Kit [1] UDA benchmarks. For the Sprites [22] dataset, search as the data is quite simple. We simply set as 8, which the original length of the video sequence. The confidence threshold set to be 0.99, and x to Xa are all set be 1.
the validation of each transfer task. Tab. B
do not do hyperparameter
A.5 Demo Instruction
As mentioned in the main body, we include live demo for our TranSVAE framework. This demo can be accessed at: https: haap::://uggggaaaaa detailed instructions for playing with this demo.
aaraaces//ddong/rranntee Here we include the
14

---

Table A: Summary of notations used in this paper.
Notation
Description
D
Domain
S/T
Source domain / Target domain
V
video sequence from domain D
VP.y.
The i-th video sequence and the corresponding action label of domain D
xx , x7
T frames of images the video sequence VD
2i y... 27
The dynamic latent factors of the video sequence from domain D
z
The static latent factors of the video sequence from domain D
X2t
The input sequence before timestamp t
X1.T
The full input sequence from f 1 to t = T.
Table B: Summary of the best-possible hyperparameter values of the TranSVAE framework for each UDA task in our experiments after an extensive grid search.
Task
Backbone
T
17
X
X2
A
A
U +
H
I3D
8
0.93
50
1
1
1
H + U
I3D
0.96
0.3
0.1
10
1
Js +J
13D
0.95
0.001
10
100
10
D1 + D2
B3D
0.96
50
10
5
100
D1 D3
13D
1
0.5
0.1
100
D2 Dr
T3D
0.93
100
10
5
100
D2 D3
I3D
10
0.91
0.5
10
50
100
D3 + Di
13D
8
0.91
100
10
50
100
Dg + D2
I3D
9
0.91
1000
1
0.1
100
This demo is built upon Hugging Face Spaces", which provides concise and easy-to-use live demo interfaces. Our demo consists of one input interface and one output interface as shown Fig. B. Specifically, the appearances of the Sprites avatars fully controlled by four attributes, i.e., body, hair color, top wear, and bottom wear. We construct two domains, P1 and P2. P1 uses the "Human" body while P1 uses the "Alien" body. The attribute pools of "Human" and "Alien" non-overlapping across domains, resulting in completely heterogeneous P, and P2. Each video sequence contains eight frames in total.
For conducting domain disentanglement and transfer with TranSVAE, users are free to choose the action and the appearance of the avatars the left-hand side of the interface. Next, simply click the "Submit" button and the adaptation results will display on the right-hand side of the interface a few seconds. The outputs include:
The 1st column: The original input of the "Human" and "Alien" avatars, i.e., (xP P1 : .... x xP)
and (xP'
x
Pa]:
The 2nd column: The reconstructed "Human" and "Alien" avatars {xP) Pt y... x and
{xP= ...., x
The 3rd column: The reconstructed "Human" and "Alien" avatars using only (zp,.., :-., z
D E P2). which are domain-invariant;
'nttp::/hhugiingaao..co
15

---

TranSVAE
Sequentill
K0: 2om 54, kernelaazeeis 4), strdenc2, 2), padding-i,, 10
(1: Butch Nemzale4 upo-le momentuneo affires True trecurunn ng.attttu Trae)
(2: LuiggetUinnggtrr ploggee inplece-True1)
k2b dopan connt (mam)) Sequentalt
Ki Cone20154 128, ke nol nelebe 4, st stideel2, 2. soldingetl, in
(11) Betor Nomzzolzz. 25,22, epse e-05, memertuneey 1, officcTTuee wankkuunninggg.oo
(201 LeskyRet Upreggtteee Sope=02 2, inplac--True))
1:30 drgan, coost (naim Sequercall
101: Con1200128 220, harrel pizuel, 4) striee12 25 piddi-gell, 1o
0.1,
wonkk.unningg..... state
EttLannghttiinnggnnn cope-02, inplace-Trune 12
1441: dogan, const Sequencalt
80 Con221256
dzesid, strte-12 21.rdiingetL.
(2:: Linthlhh plope--0 implece- 1r
1:5: Seguretta
10: 2on220::2 1024, kemme od. strident,
(1). Buso 20L0024 upo-le 05, monentum--.... affineeTrus, wong.tttinno
(2: Tanht
decoder: ceosder.neikip
1pp:2: Saquentak
(00. 2anpsss 512, simees 40,00 kluwi2, LB
(D) 22ch2o spo- 05, pomimumee.. L, iffine-Truo, track wodo//urrnnggusssTiue
(2) Look Ulnegaiieeeeeeeeeeem whwp:ieee.pp......... Truuas-TTue))
hpp22 degan upcomt (msit) Seguentitt
speajA, striee padsingetl. 10 purnieggnteeeeee True))
tpc
linegarive
k:p:3) drgannuppennt (main) Sequentist
cond 20 120 ternel www.usze-1A... di sinceeig 3), padsing A0
nichilorn2e120...om spu- le namantum-O. aftine-True track/urring.ttttt
Lewi Unwgeeiee pww.poopp........ 2. irplaca--rr))
lucck uegar_ tmain: Sezuential
(0h Comel nanspes 64, keme Siueets 4) du-12, 21 poldingel), inl
(1) Butchttor 220154, qus-ie 65, norrentumeC.... offneeea waw..uuiigggaasstooao
(2) Looki Uimegativee shoppe. 2, in Trpllo)-Tuee)))
Sp: Sequentisk
(l0 Cow/Fressposr21054 3. ke steest, Stride 23, pnddinget), 10
SigmoldD
nekul: Lookyyuu www.gauiiiittttttiieeooooe 13
dropout 7. Drepouti--0 Faphc
dfopout.v: Dropoutp-0.5 inplice
Jtt LSTMCx01522 51.0
Jt (5TMC 5125
neeni: Unnamiis.fesssreeaaa Jestureee522 seat/ blas- Truel
ouprrar, logver Lineaiin S122 out ooLJurrurss-e boseTruo)
atme L1100022 5:2, petthnttt-rrus aedirctional True) (Lmmeanc L neanorrhaaturee 1004, sut, wothhaturee---1 5513, blve tran
I.Jogvar: Uinarrnn feature==1224 out feotures- 512, bas-True) (aymet RNN02224 512, Trahh
(Lmmsam Linearte, Ieltarp-5122 out /osuusss Trus)True
C, logortt Linarin Jeccoreue522 out featursssss usedi,, blusTTuu
(C, feature, foama
(fc classiffer dramiin Unnarree feotar 51-2 out fosturiss2, blass Trus)
(TRN)
epselede monercum-0 afffne--Tuee, running stats-Truel
trn,
ranning
dr wwt.ire.dommii.. hotree.oominssiiee videer lntwrin features 256, nut frocurec 356, bascrTrue)
ite ww.rlessfter.dom side) Liteaiis featuree 256, out Praturee Truu))
(relation wwt.eodmmaanneeeeeee dasssftrrrrrrei Notulelne
(i Sequential.
Linearinneeeee /ne/eetureea-eee 201. bluseTrue) True) (Ae helli
(2) Linearinn Ainoarinn//stturee22tt out ( biss-ruel)
(2): Sezuential
000 Lirea (in 2566 out ooL.eoorrsss25e biossTrusk
(D) PeLLi
Lirea (in, festures=25,, out feoturese2, Truel)
Sequentiak
feonures blasiTrury
Taue))
Sequentiall
wet.i///tttttteeeeee out Sea.frrrreee-25 True) RelLig
121 Limearinnneeaae wtt.r//ueeeaeeeeee.... out feattraaaa biss Truid)
Sequential
(0). Linartin 236. out 2550 bloseTruek Trush
(1) Rullo
12r Limeartin Lit-Lieatttteetteeeee desturee- 236, out 2ewuureee bas Truet)
(51 Sequential
(t Linearin fentures: 256, nut feotures 250, blass Truve)
Lineartin/eettee 256. out feotoreae2, blsse Trui))
Seauentiak
RelLi
256 2eonures bisssTrue))
(pred, cosstfer video innerge featuree 25A, nin, features Trui) df. feturre domsir betent: Lnnwnith tranunee nut fosturee 511, tise Trun Iee cta.tetteerrrc doante, Jetent:: Uneende, 15aturee-512, 2 bree-Trun)
Figure A: The neural network structure (left) and a Netron graph (right) of the proposed TranSVAE framework. Zoom-ed in for the details.
16

---

Huran Seacee Rotor
Smaak Strttnggee
Ddggal
Recondmmtted
Gatrul
Pethange
salload
wilk
fal Joarc
giir
Irowe
wir
whlk
Aler
pdin
Figure B: The input (left) and output (right) interfaces of our live demo. Users are free to customize the actions and appearances of the source and target inputs (i.e., the "Human" and "Alien" avatars) in left-hand side and use them for disentanglement and transfer as shown in the right-hand side.
Table C: Ablation results without and with disentanglement term.
Lovne
Leta
Ladv
Lemi
Leve
U +
H
H
U
/
81.67
90.54
79.44
90.37
81.11
90.37
82.22
91.07
84.44
93.52
83.06
93.70
83.61
91.42
87.78
98.95
The 4th column: The reconstructed "Human" and "Alien" avatars by exchanging zP. which results in two sequences with the same actions but exchanged appearance, i.e., domain disentanglement and transfer.
B Additional Experimental Results
In this section, we provide additional quantitative and qualitative results for our TranSVAE framework.
B.1 Additional Ablation Studies
Component Integration Study. We further analyze the effect of each loss term without disentan- glement, i.e,, not adding Lyvae, on the UCF-HMDB dataset, as shown in Tab. C. Without Lyvae the framework is to learn representations with corresponding constraints many existing video-based UDA methods do. Precisely, we have the following remarks for the top-half table:
Only with Lade. the framework is equivalent to TA [7], which learns the domain- invaarinn temporal representation. We can see the adaptation results are similar to that [7].
Lmi term specifically designed for disentanglement purposes. Thus, the effect of using Lmi alone for adaptation is random verified in the above table, achieving worse results
17

---

Table D: Ablation results for the frame number T.
Task
5
7
8
9
10
12
14
16
20
U
84.44
86.11
84.17
87.78
84.72
85.28
83.89
84.44
82.78
85.00
-++U
96.85
94.22
98.42
94.75
98.95
93.70
93.87
94.40
91.05
91.40
Table E: Ablation results for the pseudo-label threshold v).
Task
w/o
0.90
0.91
0.92
0.93
0.94
0.95
0.96
0.97
0.98
0.99
U H
87.22
0.50)
86.94
87.22
87.50
87.78
86.11
86.67
86.94
86.39
86.11
86.39
H
U
94.40
(-4.55)
98.42
97.37
98.77
98.60
98.25
98.25
98.95
97.90
97.72
95.27
in the U task and better results in the tasks compared with the source-only baseline.
Lete enforces the learned static representation to be domain-specific and brings indirect benefit to adaptation, thus obtaining slightly better results than the source-only baseline.
In sum, without Lwae, the adaptation performance of each loss term alone is far from optimal. Moreover, combining all the terms can slightly improve the adaptation performance over the individual ones, however, is still obviously worse than TranSVAE results. The above ablation study shows the necessity of the disentanglement in the TranSVAE framework and proves that Lwve is an indispensable loss term.
Regarding the bottom half table showing the results with Ladv, we have the following remarks:
Each individual term with Ladv achieves better adaptation results than the source-only baseline.
Each individual term with Lat consistently outperforms the corresponding ones without Ladv-
The final TranSVAE framework combined all the loss terms yields much better results than each individual case.
This set of ablation studies indicates that 1) each term in TranSVAE brings positive effects to adapta- tion, and 2) disentanglement and adaptation interactive and complementary in framework to obtain the best possible UDA results.
Number of Frames T'. Tab. D shows the transfer performance with the variation of the number of frames 7 on the UCF-HMDB dataset. As can be seen, the two tasks achieve the optimal performance with different T, specifically T for U + H and T 9 for H U. Based on this observation, we apply grid search on the validation set to obtain the optimal T for each task in our experiments.
Target Pseudo-Lan Label Threshold We show the sensitivity analyses of the transfer performance with respect to the target pseudo label threshold v on the UCF-HMDB dataset in Tab. E. The results show that different tasks yield the best transfer result with different y), specifically 1 0.93 for U + and 17 = 0.96 for H + U. Thus, we also apply the grid search on the validation set to obtain the optimal for each task.
B.2
Additional Qualitative Results
For those who cannot access our live demo, we have included more qualitative examples for domain disentanglement and transfer in Fig. C. We also provide links for GIFs demonstrating various disentanglement and reconstruction results.
In the GIFs link, we show two cases, the first two columns for the first one and the last two columns for the second one. Each case contains source and target cartoon character performing an action. For each row, we have the following remark:
phttps://gitee wttp:thww...ccec/ooooooo0/////
18

---

Input Sequrnce
Lett "Itrmar..aaal
Rigit "Aker" wallsf
Reconstraction
Letz "Humar" daa,
Righe 'Kied walles
Domain Discntennlement
Lats nour Nall
Domain Trunder
Tall "Aker, slals
Imput Sequence "Ihenan"
Reconstraction
sption, Righ "Aber, deak
000
Domsin Disestangkeneat
Right
Fransder
Rige
Figure C: Additional qualitative results for illustrating the domain disentanglement and transfer properties in our TranSVAE framework.
The second row shows the reconstructed results of TranSVAE, i.e. z and zp As can be seen, TranSVAE reconstructs the image with a high quality.
The third row shows the reconstructed results only using the static latent factors, i.e., 2 and 0,,, where we replace x, with zero vectors. As can be seen, the reconstructed results are basically static containing the appearance of the character which is the main domain difference in the Sprites dataset [22]. Specifically, we find they generally lack arms. This is reasonable as the target action is slashing or spelling cards with arms moving, and such dynamic information on the arm is captured by zi
The fourth row shows the reconstructed results only using the dynamic latent factors, i.e., 0g and z,P where we replace z? with zero vectors. As can be seen, the reconstructed results are performing the right action but the appearance is mixed up. This shows that z, are indeed domain-invanne invariant and contain the semantic information.
The last row shows the reconstructed results of exchanging the dynamic latent factors between domains, i.e., (0 z, ) and (07 xf) As can be seen, the reconstructed results are with the original appearance but perform the transferred action. This indicates the potential of TranSVAE for some style-transfer tasks.
C Broader Impact
This paper provides novel transfer method to use cross-domain video data, which effectively helps reduce the annotation efforts in related video applications. Although the main empirical evaluation is on the video action recognition task, the model structure proposed in this paper is also applicable to other video-related tasks, such as action segmentation, video semantic segmentation, etc. More generally, the idea of disentangling domain information sheds light on other data modality style transfer tasks, e.g., voice conversion. The negative impacts of this work are difficult to predict. However, as a deep model, our method shares some common pitfalls of the standard deep learning models, e.g., demand for powerful computing resources, and vulnerability to adversarial attacks.
19

---

D Public Resources Used
We acknowledge the use of the following public resources, during the course this work:
UCF
Unknown
HMDB51
CC BY 4.0
Jester
Unknown
Epic- Epic-KKtchens'
CC BY-NC
Sprites 10
CC-BY-SA-3.0
I3D
Apache License 2.0
TRN12
BSD 2- 2-Clause License
CoMix
Apache License 2.0
TA
MIT License
1S
Unknown
PyTorch 16
PyTorch Custom License
Netron
MIT License
popsscouuneerpytoorch"
MIT License
database
hhttpp///gthhbl..caa/rurrss/s//
20


Reviews:
[{"review_id": "aIfyYIbGN5", "reviewer": "Reviewer_JyS2", "summary": "The paper tackles the unsupervised domain adaptation (UDA) problem by viewing it from a disentanglement perspective. It suggests separating two sets of latent factors, one of which is responsible for domain-related spatial information and another encodes dynamic temporal information. As a result, a new model for UDA called Transfer Sequential VAE (TranSVAE) is introduced. To achieve better disentanglement and better learning the paper considers additional constraints: minimizing mutual dependence, task-specific supervision, static consistency, and temporal domain alignment through adversarial-based idea [31]. The experiment set includes 4 dataset settings: UCF-HMDB, Jester, Epic-Kitchens, and Sprites. The proposed method outperforms related works on all these settings visually showing strong disentanglement properties on the Sprites dataset.  ", "questions": "I have no additional questions except the Weakness section discussion. ", "limitations": "The paper adequately addressed the limitations and potential negative social impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1) The paper is well-written with a good structure and useful visualizations. The motivation of the method is clear and easy to understand and follow.\n2) The method achieves good performance on several datasets including real-world domains (UCF-HMDB, Jester, Epic-Kithens) and synthetic domains (Sprites). Additionally, it is visually shown a good disentanglement property on the Sprites dataset and analysis of different loss parts to study the domain and semantic alignment of the features. \n3) The ablation study extensively analysis the impact of different proposed constraints showing that incorporating all of the losses achieves the best results. \n4) The proposed method is also performing on par with multi-modal methods that are using other modalities such as optical flow. Sometimes it even outperforms most of them. ", "weaknesses": "I would like to see more discussions about related works in the disentanglement area and more outlined explanations of differences of the proposed TranSVAE from them, especially with the papers for video generation [26, 27, 28]. Basically, they also consider the separation of static and dynamic latent factors and considers the minimization of mutual information between them. So it seems that the main difference lies in UDA setting for video action recognition itself (task-specific supervision) and other constraints in consideration (static consistency and temporal domain alignment). The temporal domain alignment is based on the existing ideas of [31], so the paper looks to me mostly as a smart combination of the existing approaches not as a fully novel model. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer JyS2 for devoting time to this review and drawing valuable comments.\n\n---\n> ***Q:** \"More discussions about related works in the disentanglement area and more outlined explanations of differences of the proposed TranSVAE from them, especially with the papers for video generation [26, 27, 28], which also consider the separation of static and dynamic latent factors and consider the minimization of mutual information between them.\"*\n\n**A:** We would like to thank the reviewer’s comment. Our response to this question is as follows:\n- Firstly, we highlight that the primary idea of this paper is to provide a new perspective on solving video-based UDA problems, that is from a *generative view of cross-domain data*. To the best of our knowledge, this is the first study in video-based UDA. Using the disentanglement framework, Sequential VAE is just one way of achieving the generative process of cross-domain videos as shown in Figure 2. \n- Secondly, disentanglement methods like [26, 27, 28] aim to learn latent feature representation from input, where each component of learned features, usually including dynamic and static ones, refers to a semantically meaningful concept. However, disentanglement can be arbitrary, and the proposed TranSVAE is designed to obtain a specific disentanglement beneficial for the adaptation tasks. As shown in Figures 5 and 6, just using SVAE [26] for disentanglement, i.e. results (a), does not yield the optimal adaptation performance in video-based UDA.\n- Thirdly, in the TranSVAE framework, disentanglement and adaptation are interactive and complementary.  All the constraints serve to achieve a good disentanglement effect with the two-level domain divergence minimization. We would like to re-emphasize that the proposed contrastive triple loss ($L_{ctc}$) not only boosts the static consistency but also achieves good domain-specific latent factors by pulling close the inter-domain latent and pushing away the intra-domain ones, which contribute to the spatial domain alignment. We agree with the reviewer that some techniques, e.g. the $L_{mi}$ and $L_{adv}$ losses, are standard in their fields. However, the overall TranSVAE framework is not trivial as each component is not isolated but complementary for the adaptation purpose. The best adaptation performance is achieved with the interaction of all the loss terms as verified. The reviewer may refer to Table 3, Figures 5 and 6 in the paper for the ablation study of each loss term. More ablation analyses are also included in our response to other reviewers. Please kindly refer to the answer to Reviewer 3nXx's Q1, thanks.\n\n---\n**References:**\n- [26] Y. Li and S. Mandt. Disentangled sequential autoencoder. ICML, 2018.\n- [27] Y. Zhu, et al. S3VAE: Self-supervised sequential VAE for representation disentanglement and data generation. CVPR, 2020.\n- [28] J. Bai, et al. Contrastively disentangled sequential variational autoencoder. NeurIPS, 2021.\n- [31] Y. Ganin, et al. Domain-adversarial training of neural networks. JMLR, 2016.\n"}, {"review_id": "HZGI9Yd7II", "reviewer": "Reviewer_3nXx", "summary": "This paper focuses on video domain adaptation. The authors' primary focus is to eliminate the temporal invariant component of the features initially, followed by the implementation of an adversarial-based domain adaptation technique. To achieve disentanglement, they employ a method of temporally random shuffling of the video, thus identifying domain-specific and static latent factors that remain unaffected by temporal shifts. These domain static factors, along with the Domain-Invariant Sequential Factors, are subsequently utilized separately for domain adaptation. The authors provide an extensive evaluation of their proposed TranSVAE method by conducting rigorous experiments on widely recognized datasets such as UCF-HMDB, Jester, and Epic-Kitchens. The results demonstrate that the TranSVAE model surpasses several state-of-the-art techniques, including those employing multi-modal approaches.\n \n\n", "questions": "1. I hope the authors can provide better ablation study for better highlighting the core technical part of this work. In my opinion, the use of $L_{svae}$, $L_{cls}$, $L_{adv}$, and PL are not the main technical contribution of this work. \n2. I hope the authors can provide a comparative analysis on training time, inference time, and model parameters. \n3. The EPIC-Kitchens100 dataset should be a better playground for evaluating this method.", "limitations": "Limitations are discussed by the authors in the paper.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The proposed idea of handling spatial and temporal domain divergence separately through disentanglement for domain adaptation is interesting. \n2. The idea of using shuffling to disentangle the domain static and dynamic latent factors is interesting.\n2. The method can outperform sota with only RGB modality on multiple datasets.", "weaknesses": "1. The proposed model contains many components but no proper ablation study is conducted to evaluate the effect of each component. In my opinion, the core part should be whether to add $L_{mi}$ and/or $L_{ctc}$ into the model. The ablation study in Figures 5 and 6 is not sufficient. The authors should add ablation study to specifically evaluate this core part, without the influence of other components of the model, especially $L_{svae}$.\n2. While this method outperforms previous works, it apparently uses much more parameters because of the use of a decoder. An analysis of training and inference time and model parameters is needed. \n3. The EPIC-Kithens dataset has a new official split for domain adaptation, which is a larger and better split with an online evaluation server for the test set. This split is used in a lot of previous works. I wonder why the authors do not perform experiments on this set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 3nXx for devoting time to this review and drawing valuable comments.\n\n---\n> ***Q1:** \"Add an ablation study to specifically evaluate $L_{mi}$ and/or $L_{ctc}$, without the influence of other components, especially $L_{svae}$.\"*\n\n**A:** Thanks for your suggestion. We highlight that the primary idea of this work is providing a generative perspective on solving video-based UDA problems. Using Sequential VAE is one way of achieving the generative process of cross-domain videos as shown in Figure 2, and thus $L_{svae}$ is indispensable in the TranSVAE framework. Moreover, in our framework, disentanglement and adaptation are interactive and complementary. All the losses work together to obtain a specific disentanglement beneficial for adaptation.\n\nWe agree with the reviewer that more ablation analyses will further solidify our work, and thus we conduct the following experiments. Note that here $L_{cls}$ is always needed as it is used to train the final classification model.\n\n1. Analyzing the effect of each loss term w/o disentanglement, i.e., not adding $L_{svae}$: \n|-|$L_{svae}$|$L_{adv}$|$L_{mi}$|$L_{ctc}$|Acc (U->H)|Acc (H->U)\n|-|-|-|-|-|:-:|:-:|\n|$\\mathcal{S}_{only}$|||||80.27|88.79\n|-||✓|||81.67|90.54\n|-|||✓||79.44|90.37\n|-||||✓|81.11|90.37\n|-||✓|✓|✓|82.22|91.07\n|-|✓|✓|✓|✓| 87.78|98.95\n- Without $L_{svae}$, the framework is to learn representations with corresponding constraints as many existing video-based UDA methods do. Precisely, we have the following remarks:\n  - Only with $L_{adv}$, the framework is equivalent to TA3N [11], which learns the domain-invariant temporal representation. We can see the adaptation results are similar to TA3N. \n  - $L_{mi}$ is a term specifically designed for disentanglement purposes. Thus, the effect of using $L_{mi}$ alone for adaptation is random as verified in the above table, achieving worse results in the U -> H task and better results in the H -> U tasks compared with the source-only baseline.\n  - $L_{ctc}$ enforces the learned static representation to be domain-specific and brings indirect benefit to adaptation, thus obtaining slightly better results than the source-only baseline. \n- In sum, without $L_{svae}$, the adaptation performance of each loss term alone is far from optimal. Moreover, combining all the terms can slightly improve the adaptation performance over the individual ones, however, is still obviously worse than TranSVAE results. The above ablation study shows the necessity of the disentanglement in the TranSVAE framework and proves that $L_{svae}$ is an indispensable loss term.\n\n2. Analyzing the effect of each loss term w/ disentanglement:\n|-|$L_{svae}$|$L_{adv}$|$L_{mi}$|$L_{ctc}$|Acc (U->H)|Acc (H->U)\n|-|-|-|-|-|:-:|:-:|\n|$\\mathcal{S}_{only}$|||||80.27|88.79\n|-|✓|✓|||84.44|93.52\n|-|✓||✓||83.06|93.70\n|-|✓|||✓|83.61|91.42\n|-|✓|✓|✓|✓| 87.78|98.95\n- From these results, we have the following remarks:\n  - Each individual term w/ $L_{svae}$ achieves better adaptation results than the source-only baseline. \n  - Each individual term w/ $L_{svae}$ consistently outperforms the corresponding ones w/o $L_{svae}$.\n  - The final TranSVAE combined all the loss terms yields much better results than each individual case.\n- This set of ablation studies indicates that 1) each term in TranSVAE brings positive effects to adaptation; 2) disentanglement and adaptation are interactive and complementary in our framework to obtain the best possible UDA results.\n\n3. Analyzing the effect of removing each loss term:\n|-|$L_{svae}$|$L_{adv}$|$L_{mi}$|$L_{ctc}$|Acc (U->H)|Acc (H->U)\n|-|-|-|-|-|:-:|:-:|\n|$\\mathcal{S}_{only}$|||||80.27|88.79\n|-|✓||✓|✓|83.06|93.52\n|-|✓|✓||✓|85.83|93.52\n|-|✓|✓|✓||83.89|95.80\n|-|✓|✓|✓|✓| 87.78|98.95\n- From the above results, we can see that removing any of the terms leads to inferior results than the full TranSVAE, and removing $L_{adv}$ is the most influential. This is reasonable as $L_{adv}$ is used to explicitly reduce the temporal domain gaps. All these results show that each loss term matters in the proposed TranSVAE framework.\n- As suggested, we have included all these ablation results and analyses in the revision.\n\n---\n> ***Q2:** \"An analysis of training and inference time and model parameters is needed.\"*\n\n**A:** Thanks for the comment. We compare the number of trainable parameters, multiply-accumulate operations (MACs), floating-point operations (FLOPs), and inference frame-per-second (FPS) with prior arts as follows:\n|Method|Trainable Params|MACs||FLOPs||FPS\n|-|:-:|-|-|-|-|-\n|CoMix|30.3688 M|18.5640 G||37.1280 G||0.0157 s\n|CO2A|23.6720 M|18.1884 G||36.3768 G||0.0127 s\n|TranSVAE|12.7419 M|18.2657 G||36.5314 G||0.0133 s\n\nWe observe that TranSVAE requires less trainable parameters than prior arts. Since these approaches adopt the same I3D backbone, the MACs, FLOPs, and FPS are competitive among them.\n\n---\n> ***Q3:** \"Perform experiments on the new official split of EPIC-Kithens domain adaptation.\"*\n\n**A:** We thank the reviewer for informing us about the new version of this dataset. In this work, we adopt the benchmark EPIC-Kithens dataset and splits as widely used in the existing video-based UDA methods, e.g. [2,11,14,18], as well as some most recent works [R1,R2]. As suggested, we will test this new split and report the results in the revised version.\n\n---\n**References:**\n- [2] E. Tzeng, et al. Adversarial discriminative domain adaptation. CVPR, 2017.\n- [11] M.-H. Chen, et al. Temporal attentive alignment for large-scale video domain adaptation. ICCV, 2019.\n- [14] A. Sahoo, et al. Contrast and mix: Temporal contrastive video domain adaptation with background mixing. NeurIPS, 2021.\n- [18] J. Munro and D. Damen. Multi-modal domain adaptation for fine-grained action recognition. CVPR, 2020.\n- [R1] A Dasgupta, et al. Overcoming label noise for source-free unsupervised video domain adaptation. ICVGIP, 2022.\n- [R2] Y. Zhang, et al. Audio-adaptive activity recognition across video domains. CVPR, 2022.\n\n\n\n\n"}, {"review_id": "EljOAKBG8e", "reviewer": "Reviewer_cyvj", "summary": "The paper proposes a Transfer Sequential VAE (TranSVAE) to model temporal and spatial domain divergence through feature disentanglement for unsupervised domain adaptation for action recognition from videos. The architecture consists of 5 components - encoder, LSTM, latent spaces, sampling and decoder. Video is generated using a VAE based structure from two sets of latent factors: Dynamic Random Variable to encode the semantic information for downstream tasks and Static Random Variable for domain-related spatial information. To make disentanglement facilitate adaptation, the latent factors are constrained following specific criteria. The approach is tested on UCF-HMDB, Jester and Epic Kitchens dataset and outperforms the concurrent related work by a significant margin.", "questions": "●\tLine 115 Framework Overview is not clear. Z_1 , ..., z_T and z_d. What do they mean. Whats ”d” here? What does x_{<t} and x_{1:T} mean at LINE 118. Line 123 again uses it without explaining what do they mean/represent/ are obtained?\n\n●\tLSTM module is unclear. From Figure 3, LSTM (yellow) seems to indicate all the frames (from both videos) are aware of one another and module looks like Fully connected network, while from description 127-129 It seems like the same LSTM applies separately to the source and target videos.\n\n●\t[Section 3] Since there is no mention of reconstruction loss, how is the decoder getting trained?\n\n●\tIn most of the cases (Tab 1,2) the accuracy of supervised targets (T_sup) is higher than the TranSVAE, so does that mean the target pseudo-labels used in TranSVAE for all the datasets are almost correct?\n\n●\tWhy some of the recent results missing? TRN [32],  CIA [21]. for H->U, and U->H. CIA [21] for epic kitchen.", "limitations": "Yes, provided.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "●\tThe approach of using static and dynamic random variable to encode semantic and domain-related spatial information using the TranSVAE architecture is interesting\n\n●\tResults are comprehensive, and well elaborated on multiple datasets, consistent performance improvements over previous methods of single and multi-modal data. \n\n●\tComprehensive ablation study shows the utility of the loss functions deployed.", "weaknesses": "●\tSummarizing the main contributions point by point would make the paper a better read\n\n●\t[Section 3] A more detailed description of the architecture and input, output of each component would have made it easier to understand the architecture\n\n●\tMethodology needs to rewritten for better explainability. (See Questions: 1 LINE 115)\n\n●\t[Section 4.3] Providing quantitative results with ResNet-101 as backbone would have proved the effectiveness of using I3D over ResNet-101 on the TranSVAE architecture\n\n●\tAblation experiment without disentanglement would justify the necessity of disentanglement.\n \n●\tRecent results are missing A3R [1] for Epic-Kitchens. CleanAdapt [2], CycDA [3] for U-> H H->U.\n\n[1]: Audio-adaptive activity recognition across video domains. CVPR’22. \n[2]: Overcoming Label Noise for Source-free Unsupervised Video Domain Adaptation. ICVGIP’22, \n[3]: CycDA: Unsupervised Cycle Domain Adaptation to Learn from Image to Video, ECCV’22\n\nMinor Weakness: static variable z_d & dynamic z_t should be written in Video Sequence Reconstruction Line 124-125.\nEquation 1,2,3 can be explained much better if the formulas were derived.\n\n**Post rebuttal**\nThank you authors, the response satisfactory answers the questions I had.\nThe rebuttal shows more results (lot of comparisons missing in original submission, Recent SOTA comparison, and FLOPs comparison). Hopefully these will be added to revised paper as well. Better expiation of methodology in order to understand the pipeline (more annotations in the figure, table explaining symbols, etc.), has been provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer cyvj for devoting time to this review and drawing valuable comments.\n\n---\n> ***Q1:** \"Summarizing the main contributions point by point would make the paper a better read.\"*\n\n**A:** We have included the following summarization in the revision. The main contribution can be summarized as follows:\n- We provide a new perspective on solving video-based UDA problems, that is from a generative view of cross-domain videos. We develop a generative graphical model for the cross-domain video generation process and propose to utilize the sequential VAE as the base generative model. \n- Based on the above generative view, we propose a TranSVAE framework for video-based UDA. By developing four constraints on the latent factors to enable disentanglement to benefit adaptation, the proposed framework is capable of handling the cross-domain divergence from both spatial and temporal levels.\n- We conduct extensive experiments on several benchmark datasets to verify the effectiveness of TranSVAE. Comprehensive ablation study also demonstrates the positive effect of each loss term on adaptation.\n\n---\n> ***Q2:** \"A more detailed description of the input and output of each component would have made it easier to understand the architecture.\"*\n\n**A:**  We have included the following description in the revision.\n- The input videos are fed into an encoder to extract the visual features, followed by an LSTM to explore the temporal information.\n- Two groups of *mean* and *variance* networks are then applied to model the posterior of the latent factors, i.e., $q(z_t^{D}|x_{<t}^{D})$ and $q(z_d^{D}|x_{1:T}^{D})$.\n- The new representations $z_1^D,...,z_T^D$ and $z_d^{D}$ are sampled, and then concatenated and passed to a decoder for reconstruction. Four constraints are proposed to regulate the latent factors for adaptation.\n\n---\n> ***Q3:** \"Better explainability of lines 115, 118, and 123.\"*\n\n**A:** We give the meaning of basic notations in the Introduction, lines 46-49. As suggested, we have added a table for better clarification of all the notations. $z_1, ..., z_T$ are the dynamic latent representations. $z_d$ is the static latent representation, where \"$d$\" is the abbreviation of \"domain\" as it is expected to be domain-specific. $x_{<t}$ represents the input sequence before time $t$, and $x_{1:T}$ represents the full sequence from $t=1$ to $t=T$.\n\n---\n> ***Q4:** \"Results with ResNet-101 as backbone.\"*\n\n**A:** We have included the TransVAE results of ResNet-101 in the revision.\n|Backbone|U->H|H->U|Avg|\n|-|-|-|-|\n|Res101|81.94|88.44|85.19\n|I3D|87.78|98.95|93.37\n\n---\n> ***Q5:** \"Ablation experiment without disentanglement would justify the necessity of disentanglement.\"*\n\n**A:** Thanks for the comment. We present the analysis of each loss term w/o disentanglement, i.e., not adding $L_{svae}$, as follows: \n|-|$L_{svae}$|$L_{adv}$|$L_{mi}$|$L_{ctc}$|Acc (U->H)|Acc (H->U)\n|-|-|-|-|-|:-:|:-:|\n|$\\mathcal{S}_{only}$|||||80.27|88.79\n|-||✓|||81.67|90.54\n|-|||✓||79.44|90.37\n|-||||✓|81.11|90.37\n|-||✓|✓|✓|82.22|91.07\n|-|✓|✓|✓|✓| 87.78|98.95\n\nWithout $L_{svae}$, the framework is to learn representations with corresponding constraints as many existing video-based UDA methods do. Due to the space limit, please kindly refer to Reviewer 3nXx's Q1 for more detailed explanations of the different disentanglement effects.\n\n---\n> ***Q6:** \"Recent results of A3R for Epic-Kitchens, CleanAdapt and CycDA for U-H.\"*\n\n**A:** Thanks for providing the reference works. We highlight the difference between these works from TranSVAE. All three works utilize *extra data sources* except for RGB, specifically audio in A3R, flow in CleanAdapt, and web data in CycDA, where current TranSVAE only uses a single RGB modality. We have included these methods for a more comprehensive comparison in the revision.\n|Method|D1->D2|D1->D3|D2->D1|D2->D3|D3->D1|D3->D2|Avg\n|-|-|-|-|-|-|-|-\n|A3R|53.2|52.1|51.9|55.5|48.7|63.2|54.1\n|CleanAdapt (2-stream)|52.7|47.0|46.2|52.7|47.8|54.4|50.3\n|TranSVAE|50.5|50.3|50.3|58.6|48.0|58.0|52.6\n\n|Method|U->H|H->U|Avg\n|-|-|-|-\n|CleanAdapt|86.1|96.1|91.1\n|CleanAdapt (2-stream)|89.8|99.2|94.5\n|CycDA w/o web data|83.3|80.4|81.9\n|CycDA w/ web data|88.1|98.0|93.1\n|TranSVAE|87.78|98.95|93.37\n\nFrom the results of U/H tasks, we can see that TranSVAE is better than CleanAdapt and CycDA if only a single RGB modality is used in these corresponding methods. TranSVAE is even better than CycDA and is around 1.1% worse than CleanAdapt if multi-modality is used. \n\nFor Epic-Kitchens, TranSVAE is better than CleanAdapt and is 1.5% worse than A3R where both baselines use multi-modality data. Considering TranSVAE only uses a single modality, it shows the effectiveness of TranSVAE for video-based UDA. Moreover, we believe introducing multi-modality data into TranSVAE is a promising direction and will try it during the revision.\n\n---\n> ***Q7:** \"$z_d$ & $z_t$ should be written in line 124-125. Derived Eq. 1,2,3.\"*\n\n**A:** Thanks for the suggestion. We have revised them accordingly.\n\n---\n> ***Q8:** \"LSTM module in Fig.3 is unclear.\"*\n\n**A:** The LSTM module is shared by two domains. Fig. 3 is only for visualization and we show the detailed structure in Section 1.2 of the Appendix.\n\n---\n> ***Q9:** \"How is the decoder getting trained?\"*\n\n**A:** The reconstruction loss is already included in $L_{svae}$, i.e. Eq. 4.\n\n---\n> ***Q10:** \"$T_{sup}$ is higher than TranSVAE, does that mean the target pseudo-labels used in TranSVAE for all the datasets are almost correct?\"*\n\n**A:** $T_{sup}$ means directly using target labeled data for training, and thus can be taken as the upper bound of the adaptation performance. This is a conventional baseline widely used in existing video-based UDA methods, e.g. [11-16].\n\n---\n> ***Q11:** Results of TRN and CIA are missing.\"*\n\n**A:** Thanks for the comment. TRN is not a video UDA method and thus is not compared. CIA is a video UDA method using multi-modality data, we already included the results of CIA in Tab. 4. \n\n\n\n"}, {"review_id": "f8gWAwzqWt", "reviewer": "Reviewer_AobY", "summary": "This paper proposes an unsupervised domain adaptive approach for video de-entangling based on unraveling spatial and temporal domain discrepancies. The proposed TranSVAE framework models the problem and optimizes the latent factors through multiple objective constraints. Experimental results demonstrate the effectiveness and superiority of this method in behavior recognition tasks.", "questions": "1 Although the novelty of the proposed method in this paper is average, the experiments and ablation studies are comprehensive, and the work is solid.\n\n2 I am particularly interested in the author's response to the second and third points raised in the weaknesses. If I am satisfied with the explanations, I may consider raising  score.\n", "limitations": "No needed.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper focuses on the unsupervised domain adaptation of videos, which is more challenging compared to images, as it considers not only spatial structures but also temporal relationships.\n\n2. The motivation and writing of this paper are well done.\n\n3. The experimental results show improvement compared to the baseline method.\n", "weaknesses": "1. It is mentioned that the approach of disentanglement for video tasks is relatively mature. Although the authors applied it to handle the domain adaptation problem by separating spatial and temporal domain differences, this approach may not be considered as novel.\n2. It is noted that there is a curiosity about why the authors did not use the same backbone as CoMix for comparison experiments. It is suggested that using the same backbone for comparison would provide a fairer evaluation.\n3. The baselines compared by the authors are considered outdated, as they include methods that are even two years old. It is recommended to include a comparison with more advanced state-of-the-art methods as following and analyze the differences.\n\n[a] Yin, Yuehao, et al. \"Mix-DANN and Dynamic-Modal-Distillation for Video Domain Adaptation.\" Proceedings of the 30th ACM International Conference on Multimedia. 2022.\n\n[b] Lin, Wei, et al. \"CycDA: Unsupervised Cycle Domain Adaptation to Learn from Image to Video.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n4. Some minor:\n(1) The clarity of Figure 3 is mentioned as not being sufficient to intuitively understand the overview of the authors' method.\n\n(2) It would be beneficial to provide information about the inference time or FLOPs (floating-point operations) of the model for a more comprehensive understanding.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer AobY for devoting time to this review and drawing valuable comments.\n\n---\n> ***Q1:** \"The approach of disentanglement for video tasks is relatively mature.\"*\n\n**A:** We would like to thank the reviewer’s comment.\n- Firstly, we highlight that the primary idea of this paper is to provide a new perspective on solving video-based UDA problems, that is from a generative view of cross-domain data. To the best of our knowledge, this is the first study in video-based UDA. Using the disentanglement framework, Sequential VAE is just one way of achieving the generative process of cross-domain videos as shown in Figure 2.\n- Secondly, disentanglement methods like [26, 27, 28] aim to learn latent feature representation from input, where each component of learned features, usually including dynamic and static ones, refers to a semantically meaningful concept. However, disentanglement can be arbitrary, and the proposed TranSVAE is designed to obtain a specific disentanglement beneficial for adaptation tasks. As shown in Figures 5 and 6, just using SVAE [26] for disentanglement, results (a), does not yield the optimal adaptation performance.\n- Thirdly, in the TranSVAE framework, disentanglement and adaptation are interactive and complementary. All the constraints serve to achieve good disentanglement with the two-level domain divergence minimization. We re-emphasize that the proposed CTC loss not only boosts the static consistency but also achieves good domain-specific latent factors by pulling close the inter-domain latent and pushing away the intra-domain ones, which contribute to the spatial domain alignment. We agree that some techniques, e.g. MI and ADV losses, are standard in their fields. However, the overall TranSVAE framework is not trivial as each component is not isolated but complementary for the adaptation purpose. The best adaptation performance is achieved with the interaction of all the loss terms. The reviewer may refer to Table 3, Figures 5 and 6 in the paper for the ablation study of each loss term. More ablation analyses are also included in our response to the Reviewer 3nXx.\n\n---\n>  ***Q2:** \"Why did not use the same backbone as CoMix for comparison experiments? It is suggested that using the same backbone for comparison would provide a fairer evaluation.\"*\n\n**A:** Thanks for the comment. We are actually using the exact same backbone I3D as CoMix. However, CoMix jointly trains the I3D network by first using the published pre-trained I3D parameters as a warm start and fine-tuning the whole weights; while our TranSVAE directly uses the generated features from published pre-trained I3D model without fine-tuning it. \n\n---\n> ***Q3:** \"It is recommended to include a comparison with more advanced state-of-the-art methods (Mix-DANN and CycDA) and analyze the differences.\"*\n\n**A:** Thanks for providing the reference works. We highlight the difference between Mix-DANN [R1] and CycDA [R2] from TranSVAE as follows. Both works utilize extra data sources except for RGB; flow in MixDANN and web data in CycDA, whereas TranSVAE only uses a single RGB modality. We have included these methods in the revision. \n|Method|D1->D2|D1->D3|D2->D1|D2->D3|D3->D1|D3->D2|Avg\n|-|-|-|-|-|-|-|-\n|MixDANN w/ flow|56.0|47.3|50.3|52.4|51.0|54.7|52.0\n|TranSVAE|50.5|50.3|50.3|58.6|48.0|58.0|52.6\n\n|Method|U->H|H->U|Avg\n|-|-|-|-\n|MixDANN w/o flow|77.5|86.5|82.0\n|MixDANN w/ flow|82.2|92.8|87.5\n|CycDA w/o web data|83.3|80.4|81.9\n|CycDA w/ web data|88.1|98.0|93.1\n|TranSVAE|87.78|98.95|93.37\n\nFrom the results of the U-H tasks, we can see that TranSVAE is better than MixDANN and CycDA if only a single RGB modality is used. TranSVAE is also better than the two methods even when multi-modality is used.\n\nFor Epic-Kitchens, TranSVAE yields slightly better results than MixDANN with flow. Considering we only use a single modality, it shows the effectiveness of our framework for video UDA. Moreover, we believe introducing multi-modality data into the current framework is promising and worthy of further study. We will try to consolidate this in the revised paper.\n\n---\n>***Q4:** \"The clarity of Figure 3 is not sufficient to intuitively understand the overview of the method.\"*\n\n**A:** Thanks for the comment. We have included the following description of the TranSVAE framework in the revision.\n- The input videos are fed into an encoder to extract the visual features, followed by an LSTM to explore the temporal information.\n- Two groups of *mean* and *variance* networks are then applied to model the posterior of the latent factors, i.e., $q(z_t^{D}|x_{<t}^{D})$ and $q(z_d^{D}|x_{1:T}^{D})$.\n- The new representations $z_1^D,...,z_T^D$ and $z_d^{D}$ are sampled, and then concatenated and passed to a decoder for reconstruction. Four constraints are proposed to regulate the latent factors for adaptation.\n\n---\n>***Q5:** \"It would be beneficial to provide information about the inference time or FLOPs of the model for a more comprehensive understanding.\"*\n\n**A:** Thanks for the comment. We compare the model complexity with prior arts as follows:\n|Method|Trainable Params|MACs||FLOPs||FPS\n|-|:-:|-|-|-|-|-\n|CoMix|30.3688 M|18.5640 G||37.1280 G||0.0127 s\n|CO2A|23.6720 M|18.1884 G||36.3768 G||0.0157 s\n|TranSVAE|12.7419 M|18.2657 G||36.5314 G||0.0133 s\n\nWe observe that TranSVAE requires less trainable parameters than prior arts. Since these approaches adopt the same I3D backbone, the MACs, FLOPs, and FPS are competitive among them.\n\n---\n**References:**\n- [26] Disentangled sequential autoencoder. ICML 2018.\n- [27] S3VAE: Self-supervised sequential VAE for representation disentanglement and data generation. CVPR 2020.\n- [28] Contrastively disentangled sequential variational autoencoder. NeurIPS 2021.\n- [31] Domain-adversarial training of neural networks. JMLR 2016.\n- [R1] Mix-DANN and dynamic-modal-distillation for video domain adaptation. MM 2022.\n- [R2] CycDA: Unsupervised cycle domain adaptation to learn from image to video. ECCV 2022."}, {"review_id": "IlPvDJdlfO", "reviewer": "Reviewer_qCdJ", "summary": "The proposed method in this paper handles the spatial and temporal domain divergence separately via disentanglement.\n\nTwo sets of latent factors are considered (i) static information encoding and (ii) dynamic information encoding.\n\nThe transfer sequential VAE framework is proposed to model the generation of cross-domain videos and better serve for the adaptation process.\n\nThe proposed method facilitates the spatial divergence removal and the temporal divergence reduction through adversarial learning.\n\nThey show good performance on several action recognition benchmarks.", "questions": "Please see my comments above.\n\nI would consider raising scores if the authors are able to provide satisfactory responses to the raised concerns. .  ", "limitations": "Discussed in either the main submission or supplementary material.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "+ Unsupervised video domain adaptation presented in this paper is an interesting topic.\n\n+ In general, the paper is well-written, well-organized.\n\n+ Extensive experiments show the effectiveness of the proposed model.\n\n+ There are also some nice visualizations and comparisons presented in the paper.", "weaknesses": "Major:\n\n- The proposed method seems more like a combination of modules/blocks from existing works, the reviewer is not very clear to the following aspects (i) the complexity analysis of the proposed model w.r.t. existing unsupervised domain adaptation methods in action recognition (ii) the major difference / contribution compared to existing works (iii) why bother to design such a complicated model? The novelty is below the acceptance bar for this venue although the reviewer does like the experiments, comparisons and visualizations presented in the paper.\n\n- Is the proposed model trained in an end-to-end way? Fig. 3 shows that there are many different components and different losses / regularizations, it is still not very clear to reviewer even though the final loss presented in Eq. (14) is a sum of different loss components.\n\n- How do you choose the hyper-parameters for e.g., T, $\\eta$, and a series of $\\lambda$s for different loss components. The supplementary material only provides the parameters used in the experiments on different evaluations of different datasets, but (i) how to choose them is not mentioned (ii) analysis of the parameters is not presented and discussed, e.g., different magnitudes of the $\\lambda$ for different sets of experiments presented in Table A of supplementary material (why for $\\lambda_4$ the values are generally large).\n\n- The choose of backbone is limited to I3D, which is published in CVPR'17, ~6 years ago, why not choosing the newest models for analysis? Many popular pipelines such as masked modeling (MAE for videos) and transformer-based video models are much more popular, are there any specific reasons of choosing I3D as backbone?\n\n- Table 4 is not referenced in the paper? Table 4 is not being thoroughly analyzed and discussed. The reasons why existing methods achieve better performance on some experiments are not discussed in the paper. The reasons why other methods outperform the proposed model should be analyzed and discussed in the paper, e.g., with multi-modal signals, will the proposed model outperform the method CIA? Also it is suggested to conduct a complexity analysis among the proposed model versus the existing SOTA methods, e.g., the number of model parameters, processing speeds, FLOPs, etc.\n\nMinor:\n\n- It is suggested to have a notation section for the maths symbols used in the paper to make them clearer to readers. For example, regular fonts are scalars; vectors are denoted by lowercase boldface letters, matrices by the uppercase boldface, etc. \n\n- Are there specific reasons to show both Fig. 5 and 6? They look almost the same and the only difference is: Fig. 5 for $\\textbf{U} \\rightarrow \\textbf{H}$ and Fig. 6 is $\\textbf{H} \\rightarrow \\textbf{U}$\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer qCdJ for devoting time to this review and drawing valuable comments.\n\n---\n> ***Q1:** \"(i) Complexity analysis w.r.t. existing UDA methods in action recognition; (ii) the major difference/contribution compared to existing works; (iii) why bother to design such a complicated model?\"*\n\n**A:** Thanks for your comment. We include a complexity analysis with existing video UDA methods on the number of trainable parameters, multiply-accumulate operations (MACs), floating-point operations (FLOPs), and inference frame-per-second (FPS) as follows.\n|Method|Trainable Params|MACs||FLOPs||FPS\n|-|:-:|-|-|-|-|-\n|CoMix|30.3688 M|18.5640 G||37.1280 G||0.0127 s\n|CO2A|23.6720 M|18.1884 G||36.3768 G||0.0157 s\n|TranSVAE|12.7419 M|18.2657 G||36.5314 G||0.0133 s\n\nWe observe that TranSVAE requires less trainable parameters than prior arts. Since these approaches adopt the same I3D backbone, the MACs, FLOPs, and FPS are competitive among them.\n\nAs for the differences to other models:\n- Firstly, we highlight that the primary idea of this paper is to provide a new perspective on solving video-based UDA problems, that is from a *generative view of cross-domain data*. To the best of our knowledge, this is the first study in video-based UDA. Using the disentanglement framework, Sequential VAE is just one way of achieving the generative process of cross-domain videos as shown in Figure 2. \n- Secondly, disentanglement methods like [26, 27, 28] aim to learn latent feature representation from input, where each component of learned features, usually including dynamic and static ones, refers to a semantically meaningful concept. However, disentanglement can be arbitrary, and the proposed TranSVAE is designed to obtain a specific disentanglement beneficial for the adaptation tasks. As shown in Figures 5 and 6, just using SVAE [26] for disentanglement, i.e. results (a), does not yield the optimal adaptation performance in video-based UDA.\n- Thirdly, in the TranSVAE framework, disentanglement and adaptation are interactive and complementary.  All the constraints serve to achieve a good disentanglement effect with the two-level domain divergence minimization. We would like to re-emphasize that the proposed contrastive triple loss ($L_{ctc}$) not only boosts the static consistency but also achieves good domain-specific latent factors by pulling close the inter-domain latent and pushing away the intra-domain ones, which contribute to the spatial domain alignment. We agree with the reviewer that some techniques, e.g. the $L_{mi}$ and $L_{adv}$ losses, are standard in their fields. However, the overall TranSVAE framework is not trivial as each component is not isolated but complementary for the adaptation purpose. The best adaptation performance is achieved with the interaction of all the loss terms as verified. The reviewer may refer to Table 3, Figures 5 and 6 in the paper for the ablation study of each loss term. More ablation analyses are also included in our response to other reviewers. Please kindly refer to the answer to Reviewer 3nXx's Q1, thanks.\n\n---\n> ***Q2:** \"Is the proposed model trained in an end-to-end way?\"*\n\n**A:** Yes, TranSVAE is trained in an end-to-end way. The training process is quite standard as the conventional sequential VAE training. The difference is that we also need to a backward w.r.t. the losses specifically designed in this framework. \n\n---\n> ***Q3:** \"How do you choose the hyper-parameters for $T$, $\\eta$, and a series of $\\lambda$s for different loss components?\"*\n\n**A:** Following existing video-based UDA methods [11,12,13,14,15,16,17], we use the grid search on the validation set to choose the hyper-parameters. An example of $T$ can be found in Table 2 of the Appendix. The essence of hyper-parameters is to balance the magnitudes of each loss term in the overall objective, and it is highly dependent on the training data. For instance, $\\lambda_4$ is for the cross-entropy loss which is usually in the range of 1e-4 to 1e-1, and thus it is usually assigned with a large value to highlight the importance of the cross-entropy loss.\n\n---\n> ***Q4:** \"Are there any specific reasons for choosing I3D as backbone? Why not choose the newest models for analysis?\"*\n\n**A:** We use I3D as the backbone following existing methods [11,12,13,14,15,16,17]; it is proven that I3D is powerful and efficient in handling different video-based UDA tasks. We agree with the reviewer that using the newest model as a backbone is a promising direction worthy of further study.\n\n---\n> ***Q5:** \"Table 4 is not being thoroughly analyzed and discussed. Also, it is suggested to conduct a complexity analysis among the proposed model versus the existing SOTA methods.\"* \n\n**A:** We are sorry for the non-reference of Table 4. It should be referenced in *\"Section 4.3 - Compared to Mulit-Modal Methods\"*. \n- Basically, it is not strictly fair to directly compare TranSVAE with multi-modality methods as we only use the single RGB as the input. We include the comparisons of Table 4 to show the promising results of TranSVAE even compared with those methods using more information, instead of aiming to beat them.\n- Currently, it is non-trivial to include multi-modality data in TranSVAE, and we take this as a promising future direction. However, from the current results, i.e., TranSVAE achieves comparable results with CIA, we believe it is promising to further boost the transfer performance if multi-modality data is considered in TranSVAE.\n- For more comparisons, kindly refer to Reviewer AobY's Q3, thanks. \n\n---\n> ***Q6:** \"It is suggested to have a notation section for the maths symbols used in the paper.\"*\n\n**A:** Thanks a lot for your suggestion. We have added such a table in our revised paper for better clarification of all the notations used. \n\n---\n> ***Q7:** \"Are there specific reasons to show both Fig. 5 and 6?\"*\n\n**A:** Thanks for asking. This is to show the results of the ablation studies on both U -> H and H -> U tasks."}]

Author rebuttal:
We sincerely thank the PCs, ACs, and all the reviewers for devoting time and effort to this review.

---
We are glad to see that the reviewers are acknowledging that:
- *"The motivation of the method is clear and easy to understand and follow"* (Reviewer JyS2)
- *"The idea of handling spatial and temporal domain divergence separately through disentanglement is interesting"* (Reviewer 3nXx)
- *"Results are comprehensive and well elaborated on multiple datasets, consistent performance improvements over previous methods of single and multi-modal data"* (Reviewer cyvj)
- *"The motivation and writing of this paper are well done"* (Reviewer AobY)
- *"Extensive experiments show the effectiveness of the proposed model"* (Reviewer qCdJ)

---
We have polished the paper, added the experiments and references, and made the clarifications in the revised version. Specifically, we have revised our manuscript to include the following changes according to the reviewers’ insightful comments:
- We have supplemented more ablation experiments, including TranSVAE w/ and w/o disentanglement, to verify the effectiveness of each objective term in our framework. 
- We have compared with more SOTA baselines that use multi-modality data sources for video-based UDA, including A3R, CleanAdapt, CycDA, and MixDANN. All the comparison results show that our TransVAE is still a competitive method even using only a single RGB modality.  
- We have added complexity analyses on the number of trainable parameters, multiply-accumulate operations, floating-point operations, and inference frame-per-second for our TranSVAE as well as for prior arts.
- For better readability, we have supplemented a notation section for the maths symbols used in the paper.
- We have added missing references suggested by the reviewers.
- We have polished and improved the elaboration of this work.
- We have carefully addressed other comments from the reviewers point by point.

---
We would like to re-emphasize the novelty and technical contributions of this work:
- We provide a new perspective on solving video-based UDA problems, that is from a generative view of cross-domain videos. To the best of our knowledge, this is the first work that tackles the challenging video-based UDA from a domain disentanglement view.
- We develop a generative graphical model for the cross-domain video generation process and propose to utilize the sequential VAE as the base generative model.
- Based on the above generative view, we propose a TranSVAE framework for video-based UDA. By developing four constraints on the latent factors to enable disentanglement to benefit adaptation, the proposed framework is capable of handling the cross-domain divergence from both spatial and temporal levels.
- We conduct extensive experiments on several benchmark datasets to verify the effectiveness of TranSVAE. Comprehensive ablation study also demonstrates the positive effect of each loss term on adaptation.

---
We will actively participate in the Author-Reviewer discussion session. Please don't hesitate to let us know of any additional comments on the manuscript or the changes.

---
Last but not least, we thank the PCs, ACs, and all the reviewers again for the time and effort devoted to this review.


Meta-review: This paper receives four borderline accepts and one weak accept. Initially, the reviewers have common concerns for combinational novelty, insufficient ablation experiment, backbone selection, and recent model result comparison. The rebuttal answers the questions from the reviewers, and provides more results, e.g., a lot of missing comparisons in original submission, recent SOTA comparison, and FLOPs comparison, etc. After the rebuttal, the reviewers' concerns are resolved. AC gives the recommendation of accept (poster) based on the authors' and reviewers' response.
Decision: Accept (poster)