Title: Unsupervised Video Domain Adaptation for Action Recognition: A Disentanglement Perspective
Venue: NeurIPS 2023
Accepted: yes
Keywords: action recognition

unsupervised domain adaptation

video analysis
Abstract: Unsupervised video domain adaptation is a practical yet challenging task. In this work, for the first time, we tackle it from a disentanglement view. Our key idea is to handle the spatial and temporal domain divergence separately through disentanglement. Specifically, we consider the generation of cross-domain videos from two sets of latent factors, one encoding the static information and another encoding the dynamic information. A Transfer Sequential VAE (TranSVAE) framework is then developed to model such generation. To better serve for adaptation, we propose several objectives to constrain the latent factors. With these constraints, the spatial divergence can be readily removed by disentangling the static domain-specific information out, and the temporal divergence is further reduced from both frame- and video-levels through adversarial learning. Extensive experiments on the UCF-HMDB, Jester, and Epic-Kitchens datasets verify the effectiveness and superiority of TranSVAE compared with several state-of-the-art approaches.

Unsupervised Video Domain Adaptation for Action Recognition: A Disentanglement Perspective

Abstract

Unsupervised video domain adaptation is a practical yet challenging task. In this work, for the first time, we tackle it from a disentanglement view. Our key idea is to handle the spatial and temporal domain divergence separately through disentanglement. Specifically, we consider the generation of cross-domain videos from two sets of latent factors, one encoding the static information and another encoding the dynamic information. A Tran sfer S equential VAE (TranSVAE) framework is then developed to model such generation. To better serve for adaptation, we propose several objectives to constrain the latent factors. With these constraints, the spatial divergence can be readily removed by disentangling the static domainspecific information out, and the temporal divergence is further reduced from both frame- and video-levels through adversarial learning. Extensive experiments on the UCF-HMDB, Jester, and Epic-Kitchens datasets verify the effectiveness and superiority of TranSVAE compared with several state-of-the-art approaches. 1

1 Introduction

Over the past decades, unsupervised domain adaptation (UDA) has attracted extensive research attention [41]. Numerous UDA methods have been proposed and successfully applied to various real-world applications, e.g. , object recognition [38, 42, 47], semantic segmentation [51, 19, 33, 24], and object detection [3, 15, 45]. However, most of these methods and their applications are limited to the image domain, while much less attention has been devoted to video-based UDA, where the latter is undoubtedly more challenging.

Compared with image-based UDA, the source and target domains also differ temporally in videobased UDA. Images are spatially well-structured data, while videos are sequences of images with both spatial and temporal relations. Existing image-based UDA methods can hardly achieve satisfactory performance on video-based UDA tasks as they fail to consider the temporal dependency of video frames in handling the domain gaps. For instance, in video-based cross-domain action recognition tasks, domain gaps are presented by not only the actions of different persons in different scenarios but also the actions that appear at different timestamps or last at different time lengths.

1

Code is publicly available at:

https://github.com/ldkong1205/TranSVAE

All-in-OneView

Disentanglement View (Ours)

SourceDomain

Source-RelatedInfo

Target-RelatedInfo

Semantic-RelatedInfo

Recently, few works have been proposed for video-based UDA. The key idea is to achieve domain alignment by aligning both frame- and video-level features through adversarial learning [7, 26], contrastive learning [37, 31], attention [10], or combination of these mechanisms, e.g. , adversarial learning with attention [29, 8]. Though they have advanced video-based UDA, there is still room for improvement. Generally, existing methods follow an all-in-one way, where both spatial and temporal domain divergence are handled together, for adaptation (Fig. 1, □ ■ ). However, cross-domain videos are highly complex data containing diverse mixed-up information, e.g. , domain, semantic, and temporal information, which makes the simultaneous elimination of spatial and temporal divergence insufficient. This motivates us to handle the video-based UDA from a disentanglement perspective (Fig. 1, □ ■ ) so that the spatial and temporal divergence can be well handled separately.

To achieve this goal, we first consider the generation process of cross-domain videos, as shown in Fig. 2, where a video is generated from two sets of latent factors: one set consists of a sequence of random variables, which are dynamic and incline to encode the semantic information for downstream tasks, e.g. , action recognition; another set is static and introduces some domain-related spatial information to the generated video, e.g. , style or appearance. Specifically, the blue / red nodes are the observed source / target videos x S / x T , respectively, over t timestamps. Static latent variables z S d and z T d follow a joint distribution and combining either of them with dynamic latent variables z t constructs one video data of a domain.

With the above generative model, we develop a Tran sfer S equential V ariational A uto E ncoder ( TranSVAE ) for videobased UDA. TranSVAE handles the cross-domain divergence in two levels, where the first level removes the spatial divergence by disentangling z d from z t ; while the second level eliminates the temporal divergence of z t . To achieve this, we leverage appropriate constraints to ensure that the disentanglement indeed serves the adaptation purpose. Firstly, we enable a good decoupling of the two sets of latent factors by

X

x

Za

Z1

Zt

Z'a

T

x

X

Z1

Zt

za

x2

minimizing their mutual dependence . This encourages these two latent factor sets to be mutually independent. We then consider constraining each latent factor set. For z D d with D ∈ {S , T } , we propose a contrastive triplet loss to make them static and domain-specific. This makes us readily handle spatial divergence by disentangling z D d out. For z t , we propose to align them across domains at both frame and video levels through adversarial learning so as to further eliminate the temporal divergence. Meanwhile, as downstream tasks use z t as input, we also add the task-specific supervision on z t extracted from source data ( w/ ground-truth).

To the best of our knowledge, this is the first work that tackles the challenging video-based UDA from a domain disentanglement view. We conduct extensive experiments on popular benchmarks (UCF-HMDB, Jester, Epic-Kitchens) and the results show that TranSVAE consistently outperforms previous state-of-the-art methods by large margins. We also conduct comprehensive ablation studies

and disentanglement analyses to verify the effectiveness of the latent factor decoupling. The main contribution of the paper is summarized as follows:

We provide a generative perspective on solving video-based UDA problems. We develop a generative graphical model for the cross-domain video generation process and propose to utilize the sequential V AE as the base generative model.

Based on the above generative view, we propose a TranSVAE framework for video-based UDA. By developing four constraints on the latent factors to enable disentanglement to benefit adaptation, the proposed framework is capable of handling the cross-domain divergence from both spatial and temporal levels.

We conduct extensive experiments on several benchmark datasets to verify the effectiveness of TranSVAE . A comprehensive ablation study also demonstrates the positive effect of each loss term on video domain adaptation.

2 Related Work

Unsupervised Video Domain Adaptation . Despite the great progress in image-based UDA, only a few methods have recently attempted video-based UDA. In [7], a temporal attentive adversarial adaptation network (TA 3 N) is proposed to integrate a temporal relation module for temporal alignment. Choi et al. [10] proposed a SAVA method using self-supervised clip order prediction and clip attentionbased alignment. Based on a cross-domain co-attention mechanism, the temporal co-attention network TCoN [29] focused on common key frames across domains for better alignment. Luo et al. [26] pay more attention to the domain-agnostic classifier by using a network topology of the bipartite graph to model the cross-domain correlations. Instead of using adversarial learning, Sahoo et al. [31] developed an end-to-end temporal contrastive learning framework named CoMix with background mixing and target pseudo-labels. Recently, Chen et al. [8] learned multiple domain discriminators for multi-level temporal attentive features to achieve better alignment, while Turrisi et al. [37] exploited two-headed deep architecture to learn a more robust target classifier by the combination of crossentropy and contrastive losses. Although these approaches have advanced video-based UDA tasks, they all adopted to align features with diverse information mixed up from a compression perspective, which leaves room for further improvements.

Multi-Modal Video Adaptation . Most recently, there are also a few works integrating multiple modality data for video-based UDA. Although we only use the single modality RGB features, we still discuss this multi-modal research line for a complete literature review. The very first work exploring the multi-modal nature of videos for UDA is MM-SADA [28], where the correspondence of multiple modalities was exploited as a self-supervised alignment in addition to adversarial alignment. A later work, spatial-temporal contrastive domain adaptation (STCDA) [34], utilized a video-based contrastive alignment as the multi-modal domain metric to measure the video-level discrepancy across domains. [18] proposed cross-modal and cross-domain contrastive losses to handle feature spaces across modalities and domains. [43] leveraged both cross-modal complementary and cross-modal consensus to learn the most transferable features through a CIA framework. In [12], the authors proposed to generate noisy pseudo-labels for the target domain data using the source pre-trained model and select the clean samples in order to increase the quality of the pseudo-labels. Lin et al. [23] developed a cycle-based approach that alternates between spatial and spatiotemporal learning with knowledge transfer. Generally, all the above methods utilize the flow as the auxiliary modality input. Recently, there are also methods exploring other modalities, for instance, A3R [48] with audios and MixDANN [44] with wild data. It is worth noting that the proposed TranSVAE - although only uses single modality RGB features - surprisingly achieves better UDA performance compared with most current state-of-the-art multi-modal methods, which highlights our superiority.

Disentanglement . Feature disentanglement is a wide and hot research topic. We only focus on works that are closely related to ours. In the image domain, some works consider adaptation from a generative view. [4] learned a disentangled semantic representation across domains. A similar idea is then applied to graph domain adaptation [5] and domain generalization [16]. [13] proposed a novel informative feature disentanglement, equipped with the adversarial network or the metric discrepancy model. Another disentanglement-related topic is sequential data generation. To generate videos, existing works [22, 50, 1] extended VAE to a recurrent form with different recursive structures. In

𝐱

!

𝒮

, 𝐱

#

𝒮

, … , 𝐱

$

𝒮

𝐱

!

𝒯

, 𝐱

#

𝒯

, … , 𝐱

$

𝒯

Input

Encoder

Bi-LSTM

Sample & Predict

𝒛

!

𝒮

, … , 𝒛

$

𝒮

𝒛

!

𝒯

, … , 𝒛

$

𝒯

% 𝐱

!

𝒮

,

% 𝐱

#

𝒮

, … , % 𝐱

$

𝒮

% 𝐱

!

𝒯

,

% 𝐱

#

𝒯

, … , % 𝐱

$

𝒯

Decoder

Reconstruct

ℒ

ℒ

ℒ

mean

var

Domain-Static Factor

𝑞(𝒛

&

𝒟

|𝐱

!:$

𝒟

)

mean

var

Domain-Invariant

Sequential Factor

𝑞(𝒛

)

𝒟

|𝐱

*)

𝒟

)

𝒛

&

𝒮

𝒛

&

𝒯

ℒ

Classify

/

: Channel Concat

ℒ

Domain Adversarial

Eq. (10)

ℒ

Task Supervision

Eq. (13)

ℒ

Contrastive Triplet

Eq. (11)

ℒ

Mutual Independent

Eq. (5)

this paper, we present a V AE-based structure to generate cross-domain videos. We aim at tackling video-based UDA from a new perspective: sequential domain disentanglement and transfer.

3 Technical Approach

Formally, for a typical video-based UDA problem, we have a source domain S and a target domain T . Domain S contains sufficient labeled data, denoted as { ( V S i , y S i ) } N S i =1 , where V S i is a video sequence and y S i is the class label. Domain T consists of unlabeled data, denoted as { V T i } N T i =1 . For a sequence V D i from domain D ∈ {S , T } , it contains T frames { x D i _ 1 , ..., x D i _ T } in total 2 . We further denote N = N S + N T . Domains S and T are of different distributions but share the same label space. The objective is to utilize both { ( V S i , y S i ) } N S i =1 and { V T i } N T i =1 to train a good classifier for domain T . We present a table to list the notations in the Appendix.

Framework Overview . We adopt a VAE-based structure to model the cross-domain video generation process and propose to regulate the two sets of latent factors for adaptation purposes. The reconstruction process is based on the two sets of latent factors, i.e. , z D 1 , ..., z D T and z D d that are sampled from the posteriors q ( z D t | x D <t ) and q ( z D d | x D 1: T ) , respectively. The overall architecture consists of five segments including the encoder, LSTM, latent spaces, sampling, and decoder, as shown in Fig. 3. The vanilla structure only gives an arbitrary disentanglement of latent factors. To make the disentanglement facilitate adaptation, we carefully constrain the latent factors as follows.

Video Sequence Reconstruction . The overall architecture of TranSVAE follows a VAE-based structure [22, 50, 1] with two sets of latent factors z D 1 , ..., z D T and z D d . The generative and inference graphical models are presented in Fig. 2. Similar to the conventional VAE, we use a standard Gaussian distribution for static latent factors. For dynamic ones, we use a sequential prior z D t | z D <t ∼ N ( µ t , diag ( σ 2 t )) , that is, the prior distribution of the current dynamic factor is conditioned on the historical dynamic factors. The distribution parameters can be re-parameterized as a recurrent network, e.g. , LSTM, with all previous dynamic latent factors as the input. Denoting Z D = { z D 1 , ..., z D T } for simplification, we then get the prior as follows:

Following Fig. 2 (the 1st subfigure), x D t is generated from z D d and z D t , and we thus model p ( x D t | z D d , z D t ) = N ( µ ′ t , diag ( σ ′ t 2 )) . The distribution parameters are re-parameterized by the decoder which can be flexible networks like the deconvolutional neural network. Using V D = { x D 1 , ..., x D T } , the generation can be formulated as follows:

Following Fig. 2 (the 2nd subfigure), we model the posterior distributions of the latent factors as another two Gaussian distributions, i.e. , q ( z D d | V D ) = N ( µ d , diag ( σ 2 d )) and q ( z D t | x D <t ) = N ( µ ′′ t , diag ( σ ′′ t 2 )) . The parameters of these two distributions are re-parameterized by the encoder, which can be a convolutional or LSTM module. However, the network of the static latent factors uses the whole sequence as the input while that of the dynamic latent factors only uses previous frames. Then the inference can be factorized as:

Combining the above generation and inference, we obtain the VAE-related objective function as:

which is a frame-wise negative variational lower bound. Only using the above vanilla VAE-based loss cannot guarantee that the disentanglement serves for adaptation, and thus we propose additional constraints on the two sets of latent factors.

Mutual Dependence Minimization (Fig. 3, □ ■ ). We first consider explicitly enforcing the two sets of latent factors to be mutually independent. To do so, we introduce the mutual information [2] loss L mi to regulate the two sets of latent factors. Thus, we obtain:

To calculate Eq. (5), we need to estimate the densities of z D d , z D t and ( z D d , z D t ) . Following the non-parametric way in [9], we use the mini-batch weighted sampling as follows:

where z is z D d , z D t or ( z D d , z D t ) , N denotes the data size and M is the mini-batch size.

Domain Specificity & Static Consistency (Fig. 3, □ ■ ). Acharacteristic of the domain specificity, e.g., the video style or the objective appearance, is its static consistency over dynamic frames. With this observation, we enable the static and domain-specific latent factors so that we can remove the spatial divergence by disentangling them out. Mathematically, we hope that z D d does not change a lot when z D t varies over time. To achieve this, given a sequence, we randomly shuffle the temporal order of frames to form a new sequence. The static latent factors disentangled from the original sequence and the shuffled sequence should be ideally equal or be very close at least. This motivates us to minimize the distance between these two static factors. Meanwhile, to further enhance the domain specificity, we enforce the dynamic latent factors from different domains to have a large distance. To this end, we propose the following contrastive triplet loss:

where D ( · , · ) is Euclidean distance, m is a margin set to 1 in the experiments, z D + d , ˜ z D + d , and z D -d are static latent factors of the anchor sequence from domain D + , the shuffled sequence, and a randomly selected sequence from domain D -, respectively. D + and D -represent two different domains.

Domain Temporal Alignment (Fig. 3, □ ■ ). We now consider to reduce the temporal divergence of the dynamic latent factors. There are several ways to achieve this, and in this paper, we take advantage of the most popular adversarial-based idea [14]. Specifically, we build a domain classifier to discriminate whether the data is from S or T . When back-propagating the gradients, a gradient reversal layer (GRL) is adopted to invert the gradients. Like existing video-based UDA methods, we also conduct both frame-level and video-level alignments. Moreover, as TA 3 N [7] does, we exploit the temporal relation network (TRN) [49] to discover the temporal relations among different frames, and then aggregate all the temporal relation features into the final video-level features. This enables another level of alignment on the temporal relation features. Thus, we have:

where d i is the domain label, CE denotes the cross-entropy loss function, Z D i = { z D i _ 1 , ..., z D i _ T } , TrN i is the n -frame temporal relation network, G f , G r , and G v are the frame feature level, the temporal relation feature level, and the video feature level domain classifiers, respectively. To this end, we obtain the domain adversarial loss by summing up Eqs. (8-10):

Weassign equal importance to these three levels of losses to reduce the overhead of the hyperparameter search. To this end, with L mi, L ctc, and L adv, the learned dynamic latent factors are expected to be domain-invariant (the three constraints are interactive and complementary to each other for obtaining the domain-invariant dynamic latent factor), and then can be used for downstream UDA tasks. In this paper, we specifically focus on action recognition as the downstream task.

Task Specific Supervision (Fig. 3, □ ■ ). We further encourage the dynamic latent factors to carry the semantic information. Considering that the source domain has sufficient labels, we accordingly design the task supervision as the regularization imposed on z S t . This gives us:

where F ( · ) is a feature transformer mapping the frame-level features to video-level features, specifically a TRN in this paper, and L ( · , · ) is either cross-entropy or mean squared error loss according to the targeted task.

Although the dynamic latent factors are constrained to be domain-invariant, we do not completely rely on source semantics to learn features discriminative for the target domain. We propose to incorporate target pseudo-labels in task-specific supervision. During the training, we use the prediction network obtained in the previous epoch to generate the target pseudo-labels of the unlabelled target training data for the current epoch. However, to increase the reliability of target pseudo-labels, we let the prediction network be trained only on the source supervision for several epochs and then integrate the target pseudo-labels in the following training epochs. Meanwhile, a confidence threshold is set to determine whether to use the target pseudo-labels or not. Thus, we have the final task-specific supervision as follows:

where ˜ y T i is the pseudo-label of Z T i .

Summary . To this end, we reach the final objective function of our TranSVAE framework as follows:

where λ i with i = 1 , 2 , 3 , 4 denotes the loss balancing weight.

4 Experiments

In this section, we conduct extensive experimental studies on popular video-based UDA benchmarks to verify the effectiveness of the proposed TranSVAE framework.

4.1 Datasets

UCF-HMDB is constructed by collecting the relevant and overlapping action classes from UCF 101 [35] and HMDB 51 [20]. It contains 3,209 videos in total with 1,438 training videos and 571 validation videos from UCF 101 , and 840 training videos and 360 validation videos from HMDB 51 . This in turn establishes two video-based UDA tasks: U → H and H → U .

Jester [27] consists of 148,092 videos of humans performing hand gestures. Pan et al. [29] constructed a large-scale cross-domain benchmark with seven gesture classes, and form a single transfer task J S → J T , where J S and J T contain 51,498 and 51,415 video clips, respectively.

Epic-Kitchens [11] is a challenging egocentric dataset consisting of videos capturing daily activities in kitchens. [28] constructs three domains across the eight largest actions. They are D 1 , D 2 , and D 3 corresponding to P08, P01, and P22 kitchens of the full dataset, resulting in six cross-domain tasks.

Sprites [22] contains sequences of animated cartoon characters with 15 action categories. The appearances of characters are fully controlled by four attributes, i.e. , body, top wear, bottom wear, and hair. We construct two domains, P 1 and P 2 . P 1 uses the human body with attributes randomly selected from 3 top wears, 4 bottom wears, and 5 hairs, while P 2 uses the alien body with attributes randomly selected from 4 top wears, 3 bottom wears, and 5 hairs. The attribute pools are nonoverlapping across domains, resulting in completely heterogeneous P 1 and P 2 . Each domain has 900 video sequences, and each sequence is with 8 frames.

4.2 Implementation Details

Architecture . Following the latest works [31, 37], we use I3D [6] as the backbone 3 . However, different from CoMix which jointly trains the backbone, we simply use the pretrained I3D model on Kinetics [17], provided by [6], to extract RGB features. For the first three benchmarks, RGB features are used as the input of TranSVAE . For Sprites, we use the original image as the input, for the purpose of visualizing the reconstruction and disentanglement results. We use the shared encoder and decoder structures across the source and target domains. For RGB feature inputs, the encoder and decoder are fully connected layers. For original image inputs, the encoder and decoder are the convolution and deconvolution layers (from DCGAN [46]), respectively. For the TRN model, we directly use the one provided by [7]. Other details on this aspect are placed in Appendix.

Configurations . Our TranSVAE is implemented with PyTorch [30]. We use Adam with a weight decay of 1 e -4 as the optimizer. The learning rate is initially set to be 1 e -3 and follows a commonly used decreasing strategy in [14]. The batch size and the learning epoch are uniformly set to be 128 and 1,000, respectively, for all the experiments. We uniformly set 100 epochs of training under only source supervision and involved the target pseudo-labels afterward. Following the common protocol in video-based UDA [37], we perform hyperparameter selection on the validation set. The specific hyperparameters used for each task can be found in the Appendix. NVIDIA A100 GPUs are

used for all experiments. Kindly refer to our Appendix for all other details.

Competitors . We compared methods from three lines. We first consider the source-only ( S only) and supervised-target ( T sup) which uses only labeled source data and only labeled target data, respectively. These two baselines serve as the lower and upper bounds for our tasks. Secondly, we consider five popular image-based UDA methods by simply ignoring temporal information, namely DANN [14], JAN [25], ADDA [38], AdaBN [21], and MCD [32]. Lastly and most importantly, we compare recent SoTA video-based UDA methods, including TA 3 N [7], SAVA [10], TCoN [29], ABG [26], CoMix [31], CO 2 A [37], and MA 2 L-TD [8]. All these methods use single modality features. We directly quote numbers reported in published papers whenever possible. There exist recent works conducting video-based UDA using multi-modal data, e.g. RGB + Flow. Although TranSVAE solely uses RGB features, we still take this set of methods into account. Specifically, we consider MM-SADA [28], STCDA [34], CMCD [18], A3R [48], CleanAdapt [12], CycDA [23], MixDANN [44] and CIA [43].

4.3 Comparative Study

Results on UCF-HMDB . Tab. 1 shows comparisons of TranSVAE with baselines and SoTA methods on UCF-HMDB. The best result among all the baselines is highlighted using bold. Overall, methods using the I3D backbone [6] achieve better results than those using ResNet-101. Our TranSVAE consistently outperforms all previous methods. In particular, TranSVAE achieves 93 . 37% average accuracy, improving the best competitor CO 2 A [37], with the same I3D backbone [6], by 1 . 38% . Surprisingly, we observe that TranSVAE even yields better results (by a 2 . 1% improvement) than the supervised-target ( T sup) baseline. This is because the H → U task already has a good performance without adaptation, i.e. , 88 . 79% for the source-only ( S only) baseline, and thus the target pseudo-labels used in TranSVAE are almost correct. By further aligning domains and equivalently augmenting training data, TranSVAE outperforms T sup which is only trained with target data.

Results on Jester & Epic-Kitchens . Tab. 3 shows the comparison results on the Jester and Epic-Kitchens benchmarks. We can see that our TranSVAE is the clear winner among all the methods on all the tasks. Specifically, TranSVAE achieves a 1 . 4% improvement and a 9 . 4% average improvement over the runner-up baseline CoMix [31] on Jester and Epic-Kitchens, re-

spectively. This verifies the superiority of TranSVAE over others in handling video-based UDA. However, we also notice that the accuracy gap between CoMix and T sup is still significant on Jester. This is because the large-scale Jester dataset contains highly heterogeneous data across domains, e.g. , the source domain contains videos of the rolling hand forward, while the target domain only consists of videos of the rolling hand backward. This leaves much room for improvement in the future.

Compare to Multi-Modal Methods . Wefurther compare with four recent video-based UDA methods that use multi-modalities, e.g. RGB features, and optical flows, although TranSVAE only uses RGB features. Surprisingly, TranSVAE achieves better average results than seven out of eight multi-modal methods and is only worse than CleanAdapt [12] on UCF-HMDB and A3R [48] on Epic-Kitchens. Considering TranSVAE only uses single-modality data, we are confident that there exists great potential for further improvements of TranSVAE with multi-modal data taken into account.

L

svae

L

cls

L

adv

L

mi

L

ctc

PL

U

→

H

H

→

U

J

S

→

J

T

D

1

→

D

2

D

1

→

D

3

D

2

→

D

1

D

2

→

D

3

D

3

→

D

1

D

3

→

D

2

✓

✓

✓

✓

✓

18

.

61

26

.

62

22

.

92

34

.

00

30

.

29

33

.

79

30

.

49

28

.

51

34

.

27

28

.

83

✓

✓

✓

✓

✓

83

.

06

93

.

52

48

.

07

40

.

93

43

.

33

43

.

91

51

.

13

41

.

84

52

.

67

55

.

38

✓

✓

✓

✓

✓

85

.

83

93

.

52

65

.

12

46

.

67

48

.

56

49

.

43

55

.

34

45

.

52

54

.

53

60

.

60

✓

✓

✓

✓

✓

83

.

89

95

.

80

64

.

89

48

.

53

48

.

25

48

.

96

54

.

21

45

.

52

55

.

73

60

.

64

✓

✓

✓

✓

✓

87

.

22

94

.

40

64

.

64

49

.

87

48

.

25

49

.

66

56

.

47

47

.

59

55

.

07

61

.

46

✓

✓

✓

✓

✓

✓

87

.

78

98

.

95

66

.

10

50

.

53

50

.

31

50

.

34

58

.

62

48

.

04

58

.

00

63

.

19

Figure 4: Domain disentanglement and transfer examples. Left: Video sequence inputs for D = P 1 ('Human", □ ■ ) and D = P 2 ('Alien", □ ■ ). Middle: Reconstructed sequences ( □ ■ ) with z D 1 , ..., z D T . Right: Domain transferred sequences with exchanged z D d .

4.4 Property Analysis

Disentanglement Analysis . We analyze the disentanglement effect of TranSVAE on Sprites [22] and show results in Fig. 4. The left subfigure shows the original sequences of the two domains. The 'Human" and 'Alien" are completely different appearances and the former is casting spells while the latter is slashing . The middle subfigure shows the sequences reconstructed only using { z D 1 , ..., z D T } . It can be clearly seen that the two sequences keep the same action as the corresponding original ones. However, if we only focus on the appearance characteristics, it is difficult to distinguish the domain to which the sequences belong. This indicates that { z D 1 , ..., z D T } are indeed domain-invariant and well encode the semantic information. The right subfigure shows the sequences reconstructed by exchanging z D d , which results in two sequences with the same actions but exchanged appearance. This verifies that z D d is representing the appearance information, which is actually the domain-related information in this example. This property study sufficiently supports that TranSVAE can successfully disentangle the domain information from other information, with the former embedded in z D d and the latter embedded in { z D 1 , ..., z D T } .

Complexity Analysis . We further conduct complexity analysis on our TranSVAE . Specifically, we compare the number of trainable parameters, multiply-accumulate operations (MACs), floating-point operations (FLOPs), and inference frame-persecond (FPS) with existing baselines including TA 3 N [7], CO 2 A [37], and CoMix

[31]. All the comparison results are shown in Tab. 4. From the table, we observe that TranSVAE requires less trainable parameters than CO 2 A and CoMix. Although more trainable parameters are used than TA 3 N, TranSVAE achieves significant adaptation performance improvement than TA 3 N (see Tab. 1 and Tab. 3). Moreover, the MACs, FLOPs, and FPS are competitive among different methods. This is reasonable since all these approaches adopt the same I3D backbone.

Ablation Study . We now analyze the effectiveness of each loss term in Eq. (14). We compare with four variants of TranSVAE , each removing one loss term by equivalently setting the weight λ to 0 . The ablation results on UCF-HMDB, Jester, and Epic-Kitchens are shown in Tab. 5. As can be seen, removing L cls significantly reduces the transfer performance in all the tasks. This is reasonable as L cls is used to discover the discriminative features. Removing any of L adv , L mi, and L ctc leads to an inferior result than the full TranSVAE setup, and removing L adv is the most influential. This is because L adv is used to explicitly reduce the temporal domain gaps. All these results demonstrate that every proposed loss matters in our framework.

We further conduct another ablation study by sequentially integrating L cls , L adv , L mi, and L ctc into our sequential VAE structure using UCF-HMDB. We use this integration order based on the average positive improvement that a loss brings to TranSVAE as shown in Tab. 5. We also take advantage of t-SNE [39] to visualize the features learned by these different variants. We plot two sets of t-SNE

𝟗𝟓. 𝟎𝟎

(+𝟐. 𝟐𝟑)

𝟖𝟎. 𝟐𝟕

𝟖𝟒. 𝟒𝟒

(+𝟒. 𝟏𝟕)

𝟖𝟓. 𝟓𝟔

(+𝟓. 𝟐𝟗)

𝟖𝟕. 𝟐𝟐

(+𝟔. 𝟗𝟓)

𝟖𝟕. 𝟕𝟖

(+𝟕. 𝟔𝟏)

(a)

(b)

(c)

(d)

(e)

𝒮

+,&-

(a)

✓

✓

(b)

✓

✓

✓

(c)

✓

✓

✓

✓

(d)

✓

✓

✓

✓

✓

(e)

✓

✓

✓

✓

✓

✓

𝒯

!./

#

ℒ

!"#$

ℒ

%&!

ℒ

#'"

ℒ

()

ℒ

%*%

PL

Accuracy (%)

福

𝟖𝟖. 𝟕𝟗

9

𝟐. 𝟐𝟗

(+𝟑. 𝟓𝟎)

93

. 𝟓𝟐

(+𝟒. 𝟕𝟑)

93

. 𝟕𝟎

(+𝟒. 𝟗𝟏)

94

. 𝟒𝟎

(+𝟓. 𝟔𝟏)

98

. 𝟗𝟓

(+𝟏𝟎.𝟏𝟔)

(a)

(b)

(c)

(d)

(e)

#

ℒ

!"#$

ℒ

%&!

ℒ

#'"

ℒ

()

ℒ

%*%

PL

Accuracy (%)

𝟗𝟔. 𝟖𝟓

𝒮

+,&-

(a)

✓

✓

(b)

✓

✓

✓

(c)

✓

✓

✓

✓

(d)

✓

✓

✓

✓

✓

(e)

✓

✓

✓

✓

✓

✓

𝒯

!./

Si

藏

输

-

figures, one using the class-wise label and another using the domain label. Fig. 5 and Fig. 6 show the visualization and the quantitative results. As can be seen from the t-SNE feature visualizations, adding a new component improves both the domain and semantic alignments, and the best alignment is achieved when all the components are considered. The quantitative results further show that the transfer performance gradually increases with the sequential integration of the four components, which again verifies the effectiveness of each component in TranSVAE . More ablation study results can be found in the Appendix.

5 Conclusion and Limitation

In this paper, we proposed a TranSVAE framework for video-based UDA tasks. Our key idea is to explicitly disentangle the domain information from other information during the adaptation. We developed a novel sequential VAE structure with two sets of latent factors and proposed four constraints to regulate these factors for adaptation purposes. Note that disentanglement and adaptation are interactive and complementary. All the constraints serve to achieve a good disentanglement effect with the two-level domain divergence minimization. Extensive empirical studies clearly verify that TranSVAE consistently offers performance improvements compared with existing SoTA video-based UDA methods. We also find that TranSVAE outperforms those multi-modal UDA methods, although it only uses single-modality data. Comprehensive property analysis further shows that TranSVAE is an effective and promising method for video-based UDA.

We further discuss the limitations of the TranSVAE framework. These are also promising future directions. Firstly, the empirical evaluation is mainly on the action recognition task. The performance of TranSVAE on other video-related tasks, e.g. video segmentation, is not tested. Secondly, TranSVAE is only evaluated on the typical two-domain transfer scenario. The multi-source transfer case is not considered but is worthy of further study. Thirdly, although TranSVAE exhibits better performance than multi-modal transfer methods, its current version does not consider multi-modal data functionally and structurally. An improved TranSVAE with the capacity of using multi-modal data is expected to further boost the adaptation performance. Fourthly, current empirical evaluations are mainly based on the I3D backbone, more advanced backbones, e.g. , [36, 40], are expected to be explored for further improvement. Lastly, TranSVAE handles the spatial divergence by disentangling the static domain-specific latent factors out. However, it may happen that spatial divergence is not completely captured by the static latent factors due to insufficient disentanglement.

88.79

92.29

93.52

93.7

94.4

98.95

96.85

References

Junwen Bai, Weiran Wang, and Carla P Gomes. Contrastively disentangled sequential variational autoencoder. In Advances in Neural Information Processing Systems , volume 34, pages 10105-10118, 2021. 3, 4

Lejla Batina, Benedikt Gierlichs, Emmanuel Prouff, Matthieu Rivain, François-Xavier Standaert, and Nicolas Veyrat-Charvillon. Mutual information analysis: a comprehensive study. Journal of Cryptology , 24(2):269-291, 2011. 5

Qi Cai, Yingwei Pan, Chong-Wah Ngo, Xinmei Tian, Lingyu Duan, and Ting Yao. Exploring object relation in mean teacher for cross-domain detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11457-11466, 2019. 1

Ruichu Cai, Zijian Li, Pengfei Wei, Jie Qiao, Kun Zhang, and Zhifeng Hao. Learning disentangled semantic representation for domain adaptation. In International Joint Conferences on Artificial Intelligence , volume 2019, page 2060, 2019. 3

Ruichu Cai, Fengzhu Wu, Zijian Li, Pengfei Wei, Lingling Yi, and Kun Zhang. Graph domain adaptation: A generative view. arXiv preprint arXiv:2106.07482 , 2021. 3

Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6299-6308, 2017. 7, 8

Min-Hung Chen, Zsolt Kira, Ghassan AlRegib, Jaekwon Yoo, Ruxin Chen, and Jian Zheng. Temporal attentive alignment for large-scale video domain adaptation. In IEEE/CVF International Conference on Computer Vision , pages 6321-6330, 2019. 2, 3, 6, 7, 8, 9, 17

Peipeng Chen, Yuan Gao, and Andy J Ma. Multi-level attentive adversarial learning with temporal dilation for unsupervised video domain adaptation. In IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1259-1268, 2022. 2, 3, 8

Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems , volume 31, pages 1-11, 2018. 5

Jinwoo Choi, Gaurav Sharma, Samuel Schulter, and Jia-Bin Huang. Shuffle and attend: Video domain adaptation. In European Conference on Computer Vision , pages 678-695. Springer, 2020. 2, 3, 8, 14

Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In European Conference on Computer Vision , pages 720-736. Springer, 2018. 7, 14

Avijit Dasgupta, CV Jawahar, and Karteek Alahari. Overcoming label noise for source-free unsupervised video domain adaptation. In Proceedings of the Thirteenth Indian Conference on Computer Vision, Graphics and Image Processing , pages 1-9, 2022. 3, 8

Wanxia Deng, Lingjun Zhao, Qing Liao, Deke Guo, Gangyao Kuang, Dewen Hu, Matti Pietikäinen, and Li Liu. Informative feature disentanglement for unsupervised domain adaptation. IEEE Transactions on Multimedia , 24:2407-2421, 2021. 3

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research , 17(1):2096-2030, 2016. 6, 7, 8

Dayan Guan, Jiaxing Huang, Aoran Xiao, Shijian Lu, and Yanpeng Cao. Uncertainty-aware unsupervised domain adaptation in object detection. IEEE Transactions on Multimedia , 24:2502-2514, 2021. 1

Maximilian Ilse, Jakub M Tomczak, Christos Louizos, and Max Welling. Diva: Domain invariant variational autoencoders. In Medical Imaging with Deep Learning , pages 322-348. PMLR, 2020. 3

Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 , 2017. 7

Donghyun Kim, Yi-Hsuan Tsai, Bingbing Zhuang, Xiang Yu, Stan Sclaroff, Kate Saenko, and Manmohan Chandraker. Learning cross-modal contrastive features for video domain adaptation. In IEEE/CVF International Conference on Computer Vision , pages 13618-13627, 2021. 3, 8

Lingdong Kong, Niamul Quader, and Venice Erin Liong. Conda: Unsupervised domain adaptation for lidar segmentation via regularized domain concatenation. In IEEE International Conference on Robotics and Automation , pages 9338-9345, 2023. 1

Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In IEEE/CVF International Conference on Computer Vision , pages 2556-2563. IEEE, 2011. 7, 14

Yanghao Li, Naiyan Wang, Jianping Shi, Xiaodi Hou, and Jiaying Liu. Adaptive batch normalization for practical domain adaptation. Pattern Recognition , 80:109-117, 2018. 8

Yingzhen Li and Stephan Mandt. Disentangled sequential autoencoder. In International Conference on Machine Learning , pages 5670-5679. PMLR, 2018. 3, 4, 7, 9, 14, 19

Wei Lin, Anna Kukleva, Kunyang Sun, Horst Possegger, Hilde Kuehne, and Horst Bischof. Cycda: Unsupervised cycle domain adaptation to learn from image to video. In European Conference on Computer Vision , pages 698-715. Springer, 2022. 3, 8

Youquan Liu, Lingdong Kong, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, and Ziwei Liu. Segment any point cloud sequences by distilling vision foundation models. arXiv preprint arXiv:2306.09347 , 2023. 1

Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adaptation networks. In International Conference on Machine Learning , pages 2208-2217. PMLR, 2017. 8

Yadan Luo, Zi Huang, Zijian Wang, Zheng Zhang, and Mahsa Baktashmotlagh. Adversarial bipartite graph learning for video domain adaptation. In ACM International Conference on Multimedia , pages 19-27, 2020. 2, 3, 8

Joanna Materzynska, Guillaume Berger, Ingo Bax, and Roland Memisevic. The jester dataset: A large-scale video dataset of human gestures. In IEEE/CVF International Conference on Computer Vision Workshops , pages 1-12, 2019. 7, 14

Jonathan Munro and Dima Damen. Multi-modal domain adaptation for fine-grained action recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 122-132, 2020. 3, 7, 8

Boxiao Pan, Zhangjie Cao, Ehsan Adeli, and Juan Carlos Niebles. Adversarial cross-domain action recognition with co-attention. In AAAI Conference on Artificial Intelligence , pages 11815-11822, 2020. 2, 3, 7, 8

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems , 2019. 7

Aadarsh Sahoo, Rutav Shah, Rameswar Panda, Kate Saenko, and Abir Das. Contrast and mix: Temporal contrastive video domain adaptation with background mixing. In Advances in Neural Information Processing Systems , volume 34, pages 23386-23400, 2021. 2, 3, 7, 8, 9

Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3723-3732, 2018. 8

Antoine Saporta, Arthur Douillard, Tuan-Hung Vu, Patrick Pérez, and Matthieu Cord. Multi-head distillation for continual unsupervised domain adaptation in semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3751-3760, 2022. 1

Xiaolin Song, Sicheng Zhao, Jingyu Yang, Huanjing Yue, Pengfei Xu, Runbo Hu, and Hua Chai. Spatiotemporal contrastive domain adaptation for action recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9787-9795, 2021. 3, 8

Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 , 2012. 7, 14

Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in Neural Information Processing Systems , 35:10078-10093, 2022. 7, 10

Victor G Turrisi, Giacomo Zara, Paolo Rota, Thiago Oliveira-Santos, Nicu Sebe, Vittorio Murino, and Elisa Ricci. Dual-head contrastive domain adaptation for video action recognition. In IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2234-2243, 2022. 2, 3, 7, 8, 9

Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7167-7176, 2017. 1, 8

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research , 9(11), 2008. 9

Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14549-14560, 2023. 7, 10

Garrett Wilson and Diane J Cook. A survey of unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems and Technology , 11(5):1-46, 2020. 1

Ni Xiao and Lei Zhang. Dynamic weighted learning for unsupervised domain adaptation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15242-15251, 2021. 1

Lijin Yang, Yifei Huang, Yusuke Sugano, and Yoichi Sato. Interact before align: Leveraging cross-modal knowledge for domain adaptive action recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14722-14732, 2022. 3, 8

Yuehao Yin, Bin Zhu, Jingjing Chen, Lechao Cheng, and Yu-Gang Jiang. Mix-dann and dynamicmodal-distillation for video domain adaptation. In ACM International Conference on Multimedia , pages 3224-3233, 2022. 3, 8

Fuxun Yu, Di Wang, Yinpeng Chen, Nikolaos Karianakis, Tong Shen, Pei Yu, Dimitrios Lymberopoulos, Sidi Lu, Weisong Shi, and Xiang Chen. Sc-uda: Style and content gaps aware unsupervised domain adaptation for object detection. In IEEE/CVF Winter Conference on Applications of Computer Vision , pages 382-391, 2022. 1

Yang Yu, Zhiqiang Gong, Ping Zhong, and Jiaxin Shan. Unsupervised representation learning with deep convolutional neural network for remote sensing images. In International Conference on Image and Graphics , pages 97-108. Springer, 2017. 7

Jingyi Zhang, Jiaxing Huang, Zichen Tian, and Shijian Lu. Spectral unsupervised domain adaptation for visual recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9829-9840, 2022. 1

Yunhua Zhang, Hazel Doughty, Ling Shao, and Cees GM Snoek. Audio-adaptive activity recognition across video domains. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13791-13800, 2022. 3, 8

Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In European Conference on Computer Vision , pages 803-818. Springer, 2018. 6

Yizhe Zhu, Martin Renqiang Min, Asim Kadav, and Hans Peter Graf. S3vae: Self-supervised sequential vae for representation disentanglement and data generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6538-6547, 2020. 3, 4

Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In European Conference on Computer Vision , pages 289-305. Springer, 2018. 1

Appendix

In this appendix, we supplement the following materials to support the findings and conclusions drawn in the main body of this paper:

Sec. A provides additional implementation details to assist reproductions.

Sec. B contains additional comparative and ablation results of the TranSVAE framework.

Sec. C elaborates on the broader impact of this work.

Sec. D acknowledges the public resources used during the course of this work.

A Additional Implementation Details

In this section, we provide more implementation details including the I3D feature extraction procedure, the concrete model architecture, and the hyperparameter selection in our proposed TranSVAE framework. We also provide detailed instructions for our live demo.

A.1 Notations

We present Tab. A to summarize the notations used in this paper.

A.2 I3D Feature Extractions

We extract the I3D RGB features following the routine described in SAVA [10]. Given a video sequence, 16 frames along clips are sampled by sliding a temporal window with a temporal stride of 1 . Specifically, for each frame in the video, the temporal window consists of its previous seven frames and the following eight frames. Zero padding is used for the beginning and the end of the video. We then feed the sliding windows to the I3D backbone to extract features, which results in a 1024 -dimensional feature vector for each frame of the video.

A.3 Model Architecture

We now provide the detailed model architecture of our TranSVAE . In Fig. A, we show the model with the image as the input, where the encoder and decoder are more complex convolutional and deconvolutional layers. For the model with the RGB features as the input, we can simply replace the encoder and decoder with fully connected linear layers. Note that the dimensionality of all the modules shown above is uniformly applied in all the experiments.

A.4 Hyperparameter Selection

There are several hyperparameters used in TranSVAE , including the balancing weights λ 1 , λ 2 , λ 3 , λ 4 , the number of the video frames T , and the confidence threshold η for generating target pseudo-labels. For λ 1 to λ 4 , we select from the value set { 1 e -3 , 1 e -2 , 1 e -1 , 0 . 5 , 1 , 5 , 10 , 50 , 100 , 1000 } . For T , we select from { 5 , 6 , 7 , 8 , 9 , 10 } . For η , we set its value range from 0 . 9 to 1 . 0 with a step of 0 . 01 .

We set a high-value range of η to ensure a high confidence score on the correctness of the target pseudo-labels. Following the common protocol used in video-based UDA, we conduct an extensive grid search regarding these hyperparameters on the validation set of each transfer task. Tab. B summarizes the exact used values of these hyperparameters for the UCF-HMDB [35, 20], Jester [27], and Epic-Kitchens [11] UDA benchmarks. For the Sprites [22] dataset, we do not do a hyperparameter search as the data is quite simple. We simply set T as 8 , which is the original length of the video sequence. The confidence threshold is set to be 0 . 99 , and λ 1 to λ 4 are all set to be 1 .

A.5 Demo Instruction

As mentioned in the main body, we include a live demo for our TranSVAE framework. This demo can be accessed at: https://huggingface.co/spaces/ldkong/TranSVAE . Here we include the detailed instructions for playing with this demo.

This demo is built upon Hugging Face Spaces 4 , which provides concise and easy-to-use live demo interfaces. Our demo consists of one input interface and one output interface as shown in Fig. B. Specifically, the appearances of the Sprites avatars are fully controlled by four attributes, i.e. , body, hair color, top wear, and bottom wear. We construct two domains, P 1 and P 2 . P 1 uses the 'Human' body while P 1 uses the 'Alien" body. The attribute pools of 'Human" and 'Alien" are non-overlapping across domains, resulting in completely heterogeneous P 1 and P 2 . Each video sequence contains eight frames in total.

For conducting domain disentanglement and transfer with TranSVAE , users are free to choose the action and the appearance of the avatars on the left-hand side of the interface. Next, simply click the 'Submit" button and the adaptation results will display on the right-hand side of the interface in a few seconds. The outputs include:

The 1st column: The original input of the 'Human" and 'Alien" avatars, i.e. , { x P 1 1 , ..., x P 1 8 } and { x P 2 1 , ..., x P 2 8 } ;

The 2nd column: The reconstructed 'Human" and 'Alien" avatars { ˜ x P 1 1 , ..., ˜ x P 1 8 } and { ˜ x P 2 1 , ..., ˜ x P 2 8 } ;

The 3rd column: The reconstructed 'Human" and 'Alien" avatars using only { z D 1 , ..., z D 8 } , D ∈ { P 1 , P 2 } , which are domain-invariant;

TranSVAE(

(encoder): encoder(

(c1): dcgan_conv(

(0): Conv2d(3, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))

(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

(2): LeakyReLU(negative_slope=0.2, inplace=True) ) )

(c2): dcgan_conv( (main): Sequential(

(0): Conv2d(64, 128, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))

(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

(2): LeakyReLU(negative_slope=0.2, inplace=True) ) )

(c3): dcgan_conv( (main): Sequential(

(0): Conv2d(128, 256, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))

(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

(2): LeakyReLU(negative_slope=0.2, inplace=True) ) )

(c4): dcgan_conv( (main): Sequential(

(0): Conv2d(256, 512, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))

(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

(2): LeakyReLU(negative_slope=0.2, inplace=True) ) )

(c5): Sequential(

(0): Conv2d(512, 1024, kernel_size=(4, 4), stride=(1, 1))

(1): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

)

(decoder): decoder_woSkip

(0): ConvTranspose2d(1024, 512, kernel_size=(4, 4), stride=(1, 1))

(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

(2): LeakyReLU(negative_slope=0.2, inplace=True) )

(upc2): dcgan_upconv( (main): Sequential(

(0): ConvTranspose2d(512, 256, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))

(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

(2): LeakyReLU(negative_slope=0.2, inplace=True) ) )

(upc3): dcgan_upconv( (main): Sequential(

(0): ConvTranspose2d(256, 128, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))

(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

(2): LeakyReLU(negative_slope=0.2, inplace=True) ) )

(upc4): dcgan_upconv( (main): Sequential(

(0): ConvTranspose2d(128, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))

(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

(2): LeakyReLU(negative_slope=0.2, inplace=True) ) )

(0): ConvTranspose2d(64, 3, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))

)

(relu): LeakyReLU(negative_slope=0.1)

(dropout_f): Dropout(p=0.5, inplace=False)

(dropout_v): Dropout(p=0.5, inplace=False)

(z_prior_lstm_ly1): LSTMCell(512, 512)

(z_prior_mean): Linear(in_features=512, out_features=512, bias=True)

(z_prior_logvar): Linear(in_features=512, out_features=512, bias=True)

(z_lstm): LSTM(1024, 512, batch_first=True, bidirectional=True)

(f_mean): Linear(in_features=1024, out_features=512, bias=True)

(f_logvar): Linear(in_features=1024, out_features=512, bias=True)

(z_rnn): RNN(1024, 512, batch_first=True)

(z_mean): Linear(in_features=512, out_features=512, bias=True)

(z_logvar): Linear(in_features=512, out_features=512, bias=True)

(fc_feature_domain_frame): Linear(in_features=512, out_features=512, bias=True)

(fc_classifier_domain_frame): Linear(in_features=512, out_features=2, bias=True)

(TRN)

(bn_trn_S): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

(bn_trn_T): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

(fc_feature_domain_video): Linear(in_features=256, out_features=256, bias=True)

(fc_classifier_domain_video): Linear(in_features=256, out_features=2, bias=True)

(relation_domain_classifier_all): ModuleList(

(0): Linear(in_features=256, out_features=256, bias=True)

(1): ReLU()

(2): Linear(in_features=256, out_features=2, bias=True) )

(0): Linear(in_features=256, out_features=256, bias=True)

(1): ReLU()

(2): Linear(in_features=256, out_features=2, bias=True) )

(0): Linear(in_features=256, out_features=256, bias=True)

(1): ReLU()

(2): Linear(in_features=256, out_features=2, bias=True) )

(0): Linear(in_features=256, out_features=256, bias=True)

(1): ReLU()

(2): Linear(in_features=256, out_features=2, bias=True) )

(0): Linear(in_features=256, out_features=256, bias=True)

(1): ReLU()

(2): Linear(in_features=256, out_features=2, bias=True) )

(0): Linear(in_features=256, out_features=256, bias=True)

(1): ReLU()

(2): Linear(in_features=256, out_features=2, bias=True) )

(0): Linear(in_features=256, out_features=256, bias=True)

(1): ReLU()

(2): Linear(in_features=256, out_features=2, bias=True) )

)

(pred_classifier_video): Linear(in_features=256, out_features=15, bias=True)

(fc_feature_domain_latent): Linear(in_features=512, out_features=512, bias=True)

(fc_classifier_doamin_latent): Linear(in_features=512, out_features=2, bias=True)

Reshape

Conv

LeakyRelu

Conv

LeakyRelu

Conv

LeakyRelu

Conv

LeakyRelu

Conv

Reshape

Reshape

Transpose

Shape

Shape

Gather

Gather

Unsqueeze

Unsqueeze

Concat

Concat

Expand

Expand

LSTM

Transpose

Reshape

Shape

Gather

Unsqueeze

Concat

Transpose

ConstantOfShape

Expand

Gather

Gather

RNN

Slice

Slice

Mul

Squeeze

Concat

Equal

Transpose

Gemm

2x612

2x512

Where

Unsqueeze

MatMul

onnx:Unsqueeze_278

Expand

Add

Concat

onnx:Concat_300

Reshape

ConvTranspose

BatchNormalization

LeakyRelu

ConvTranspose

BatchNormalization

LeakyRelu

ConvTranspose

BatchNormalization

LeakyRelu

ConvTranspose

BatchNormalization

LeakyRelu

ConvTranspose

Sigmoid

Shape

Shape

Shape

Gather

Gather

Gather

Unsqueeze

Unsqueeze

Unsqueeze

Concat

Reshape

)

Human - Source Avatar

action_source

Original

Reconstructed

Output

Exchange

Input

w/Zt

pue

slash

spellard

walk

hair_source

O green

yellow

rose

red

wine

top_source

blue

white

bottom_source

。

white

golden

red

silver

Alien - Target Avatar

action_target

slash

spellcard

walk

hair_target

violet

silver

purple

grey

。

golden

top_target

grey

khaki

linen

ocre

bottom_target

olive

。

Clear

Submit

The 4th column: The reconstructed 'Human" and 'Alien" avatars by exchanging z D d , which results in two sequences with the same actions but exchanged appearance, i.e. , domain disentanglement and transfer.

B Additional Experimental Results

In this section, we provide additional quantitative and qualitative results for our TranSVAE framework.

B.1 Additional Ablation Studies

Component Integration Study . We further analyze the effect of each loss term without disentanglement, i.e. , not adding L svae, on the UCF-HMDB dataset, as shown in Tab. C. Without L svae, the framework is to learn representations with corresponding constraints as many existing video-based UDA methods do. Precisely, we have the following remarks for the top-half table:

Only with L adv, the framework is equivalent to TA 3 N [7], which learns the domain-invariant temporal representation. We can see the adaptation results are similar to that of [7].

L mi is a term specifically designed for disentanglement purposes. Thus, the effect of using L mi alone for adaptation is random as verified in the above table, achieving worse results

in the U → H task and better results in the H → U tasks compared with the source-only baseline.

L ctc enforces the learned static representation to be domain-specific and brings indirect benefit to adaptation, thus obtaining slightly better results than the source-only baseline.

In sum, without L svae, the adaptation performance of each loss term alone is far from optimal. Moreover, combining all the terms can slightly improve the adaptation performance over the individual ones, however, is still obviously worse than TranSVAE results. The above ablation study shows the necessity of the disentanglement in the TranSVAE framework and proves that L svae is an indispensable loss term.

Regarding the bottom half table showing the results with L adv, we have the following remarks:

Each individual term with L adv achieves better adaptation results than the source-only baseline.

Each individual term with L adv consistently outperforms the corresponding ones without L adv .

The final TranSVAE framework combined all the loss terms yields much better results than each individual case.

This set of ablation studies indicates that 1) each term in TranSVAE brings positive effects to adaptation, and 2) disentanglement and adaptation are interactive and complementary in our framework to obtain the best possible UDA results.

Number of Frames T . Tab. D shows the transfer performance with the variation of the number of frames T on the UCF-HMDB dataset. As can be seen, the two tasks achieve the optimal performance with different T , specifically T = 8 for U → H and T = 9 for H → U . Based on this observation, we apply a grid search on the validation set to obtain the optimal T for each task in our experiments.

Target Pseudo-Label Threshold η . We show the sensitivity analyses of the transfer performance with respect to the target pseudo label threshold η on the UCF-HMDB dataset in Tab. E. The results show that different tasks yield the best transfer result with different η , specifically η = 0 . 93 for U → H and η = 0 . 96 for H → U . Thus, we also apply the grid search on the validation set to obtain the optimal η for each task.

B.2 Additional Qualitative Results

For those who cannot access our live demo, we have included more qualitative examples for domain disentanglement and transfer in Fig. C. We also provide a link 5 for GIFs demonstrating various disentanglement and reconstruction results.

In the GIFs link, we show two cases, the first two columns for the first one and the last two columns for the second one. Each case contains a source and a target cartoon character performing an action. For each row, we have the following remark:

•

Left:

'Human', slash,

𝐱

𝐏

!

•

Right:

'Alien', walk,

𝐱

𝐏

"

Reconstruction

•

Left:

'Human', slash,

" 𝐱

𝐏

!

•

Right:

'Alien', walk,

" 𝐱

𝐏

"

Domain Disentanglement

•

Left:

'Null', slash

•

Right:

'Null', walk

•

Left:

'Alien', slash

•

Right:

'Human', walk

•

Left:

'Human', spellcard,

𝐱

𝐏

!

•

Right:

'Alien', slash,

𝐱

𝐏

"

Reconstruction

•

Left:

'Human', spellcard,

" 𝐱

𝐏

!

•

Right:

'Alien', slash,

" 𝐱

𝐏

"

Domain Disentanglement

•

Left:

'Null', spellcard

•

Right:

'Null', slash

•

Left:

'Alien', spellcard

•

Right:

'Human', slash

The second row shows the reconstructed results of TranSVAE , i.e. z D d and z D t . As can be seen, TranSVAE reconstructs the image with a high quality.

The third row shows the reconstructed results only using the static latent factors, i.e. , z D d and 0 D t , where we replace z D t with zero vectors. As can be seen, the reconstructed results are basically static containing the appearance of the character which is the main domain difference in the Sprites dataset [22]. Specifically, we find they generally lack arms. This is reasonable as the target action is slashing or spelling cards with arms moving, and such dynamic information on the arm is captured by z D t .

The fourth row shows the reconstructed results only using the dynamic latent factors, i.e. , 0 D d and z D t , where we replace z D d with zero vectors. As can be seen, the reconstructed results are performing the right action but the appearance is mixed up. This shows that z D t are indeed domain-invariant and contain the semantic information.

The last row shows the reconstructed results of exchanging the dynamic latent factors between domains, i.e. , ( 0 S d , z T t ) and ( 0 T d , z S t ). As can be seen, the reconstructed results are with the original appearance but perform the transferred action. This indicates the potential of TranSVAE for some style-transfer tasks.

C Broader Impact

This paper provides a novel transfer method to use cross-domain video data, which effectively helps reduce the annotation efforts in related video applications. Although the main empirical evaluation is on the video action recognition task, the model structure proposed in this paper is also applicable to other video-related tasks, such as action segmentation, video semantic segmentation, etc . More generally, the idea of disentangling domain information sheds light on other data modality style transfer tasks, e.g. , voice conversion. The negative impacts of this work are difficult to predict. However, as a deep model, our method shares some common pitfalls of the standard deep learning models, e.g. , demand for powerful computing resources, and vulnerability to adversarial attacks.

D Public Resources Used

We acknowledge the use of the following public resources, during the course of this work:

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe proposed method in this paper handles the spatial and temporal domain divergence separately via disentanglement.\n\nTwo sets of latent factors are considered (i) static information encoding and (ii) dynamic information encoding.\n\nThe transfer sequential VAE framework is proposed to model the generation of cross-domain videos and better serve for the adaptation process.\n\nThe proposed method facilitates the spatial divergence removal and the temporal divergence reduction through adversarial learning.\n\nThey show good performance on several action recognition benchmarks.\n\nSTRENGTHS:\n+ Unsupervised video domain adaptation presented in this paper is an interesting topic.\n\n+ In general, the paper is well-written, well-organized.\n\n+ Extensive experiments show the effectiveness of the proposed model.\n\n+ There are also some nice visualizations and comparisons presented in the paper.\n\nWEAKNESSES:\nMajor:\n\n- The proposed method seems more like a combination of modules/blocks from existing works, the reviewer is not very clear to the following aspects (i) the complexity analysis of the proposed model w.r.t. existing unsupervised domain adaptation methods in action recognition (ii) the major difference / contribution compared to existing works (iii) why bother to design such a complicated model? The novelty is below the acceptance bar for this venue although the reviewer does like the experiments, comparisons and visualizations presented in the paper.\n\n- Is the proposed model trained in an end-to-end way? Fig. 3 shows that there are many different components and different losses / regularizations, it is still not very clear to reviewer even though the final loss presented in Eq. (14) is a sum of different loss components.\n\n- How do you choose the hyper-parameters for e.g., T, $\\eta$, and a series of $\\lambda$s for different loss components. The supplementary material only provides the parameters used in the experiments on different evaluations of different datasets, but (i) how to choose them is not mentioned (ii) analysis of the parameters is not presented and discussed, e.g., different magnitudes of the $\\lambda$ for different sets of experiments presented in Table A of supplementary material (why for $\\lambda_4$ the values are generally large).\n\n- The choose of backbone is limited to I3D, which is published in CVPR'17, ~6 years ago, why not choosing the newest models for analysis? Many popular pipelines such as masked modeling (MAE for videos) and transformer-based video models are much more popular, are there any specific reasons of choosing I3D as backbone?\n\n- Table 4 is not referenced in the paper? Table 4 is not being thoroughly analyzed and discussed. The reasons why existing methods achieve better performance on some experiments are not discussed in the paper. The reasons why other methods outperform the proposed model should be analyzed and discussed in the paper, e.g., with multi-modal signals, will the proposed model outperform the method CIA? Also it is suggested to conduct a complexity analysis among the proposed model versus the existing SOTA methods, e.g., the number of model parameters, processing speeds, FLOPs, etc.\n\nMinor:\n\n- It is suggested to have a notation section for the maths symbols used in the paper to make them clearer to readers. For example, regular fonts are scalars; vectors are denoted by lowercase boldface letters, matrices by the uppercase boldface, etc. \n\n- Are there specific reasons to show both Fig. 5 and 6? They look almost the same and the only difference is: Fig. 5 for $\\textbf{U} \\rightarrow \\textbf{H}$ and Fig. 6 is $\\textbf{H} \\rightarrow \\textbf{U}$\n\nQUESTIONS:\nPlease see my comments above.\n\nI would consider raising scores if the authors are able to provide satisfactory responses to the raised concerns. .\n\nLIMITATIONS:\nDiscussed in either the main submission or supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an unsupervised domain adaptive approach for video de-entangling based on unraveling spatial and temporal domain discrepancies. The proposed TranSVAE framework models the problem and optimizes the latent factors through multiple objective constraints. Experimental results demonstrate the effectiveness and superiority of this method in behavior recognition tasks.\n\nSTRENGTHS:\n1. This paper focuses on the unsupervised domain adaptation of videos, which is more challenging compared to images, as it considers not only spatial structures but also temporal relationships.\n\n2. The motivation and writing of this paper are well done.\n\n3. The experimental results show improvement compared to the baseline method.\n\nWEAKNESSES:\n1. It is mentioned that the approach of disentanglement for video tasks is relatively mature. Although the authors applied it to handle the domain adaptation problem by separating spatial and temporal domain differences, this approach may not be considered as novel.\n2. It is noted that there is a curiosity about why the authors did not use the same backbone as CoMix for comparison experiments. It is suggested that using the same backbone for comparison would provide a fairer evaluation.\n3. The baselines compared by the authors are considered outdated, as they include methods that are even two years old. It is recommended to include a comparison with more advanced state-of-the-art methods as following and analyze the differences.\n\n[a] Yin, Yuehao, et al. \"Mix-DANN and Dynamic-Modal-Distillation for Video Domain Adaptation.\" Proceedings of the 30th ACM International Conference on Multimedia. 2022.\n\n[b] Lin, Wei, et al. \"CycDA: Unsupervised Cycle Domain Adaptation to Learn from Image to Video.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n4. Some minor:\n(1) The clarity of Figure 3 is mentioned as not being sufficient to intuitively understand the overview of the authors' method.\n\n(2) It would be beneficial to provide information about the inference time or FLOPs (floating-point operations) of the model for a more comprehensive understanding.\n\nQUESTIONS:\n1 Although the novelty of the proposed method in this paper is average, the experiments and ablation studies are comprehensive, and the work is solid.\n\n2 I am particularly interested in the author's response to the second and third points raised in the weaknesses. If I am satisfied with the explanations, I may consider raising  score.\n\nLIMITATIONS:\nNo needed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a Transfer Sequential VAE (TranSVAE) to model temporal and spatial domain divergence through feature disentanglement for unsupervised domain adaptation for action recognition from videos. The architecture consists of 5 components - encoder, LSTM, latent spaces, sampling and decoder. Video is generated using a VAE based structure from two sets of latent factors: Dynamic Random Variable to encode the semantic information for downstream tasks and Static Random Variable for domain-related spatial information. To make disentanglement facilitate adaptation, the latent factors are constrained following specific criteria. The approach is tested on UCF-HMDB, Jester and Epic Kitchens dataset and outperforms the concurrent related work by a significant margin.\n\nSTRENGTHS:\n●\tThe approach of using static and dynamic random variable to encode semantic and domain-related spatial information using the TranSVAE architecture is interesting\n\n●\tResults are comprehensive, and well elaborated on multiple datasets, consistent performance improvements over previous methods of single and multi-modal data. \n\n●\tComprehensive ablation study shows the utility of the loss functions deployed.\n\nWEAKNESSES:\n●\tSummarizing the main contributions point by point would make the paper a better read\n\n●\t[Section 3] A more detailed description of the architecture and input, output of each component would have made it easier to understand the architecture\n\n●\tMethodology needs to rewritten for better explainability. (See Questions: 1 LINE 115)\n\n●\t[Section 4.3] Providing quantitative results with ResNet-101 as backbone would have proved the effectiveness of using I3D over ResNet-101 on the TranSVAE architecture\n\n●\tAblation experiment without disentanglement would justify the necessity of disentanglement.\n \n●\tRecent results are missing A3R [1] for Epic-Kitchens. CleanAdapt [2], CycDA [3] for U-> H H->U.\n\n[1]: Audio-adaptive activity recognition across video domains. CVPR’22. \n[2]: Overcoming Label Noise for Source-free Unsupervised Video Domain Adaptation. ICVGIP’22, \n[3]: CycDA: Unsupervised Cycle Domain Adaptation to Learn from Image to Video, ECCV’22\n\nMinor Weakness: static variable z_d & dynamic z_t should be written in Video Sequence Reconstruction Line 124-125.\nEquation 1,2,3 can be explained much better if the formulas were derived.\n\n**Post rebuttal**\nThank you authors, the response satisfactory answers the questions I had.\nThe rebuttal shows more results (lot of comparisons missing in original submission, Recent SOTA comparison, and FLOPs comparison). Hopefully these will be added to revised paper as well. Better expiation of methodology in order to understand the pipeline (more annotations in the figure, table explaining symbols, etc.), has been provided.\n\nQUESTIONS:\n●\tLine 115 Framework Overview is not clear. Z_1 , ..., z_T and z_d. What do they mean. Whats ”d” here? What does x_{<t} and x_{1:T} mean at LINE 118. Line 123 again uses it without explaining what do they mean/represent/ are obtained?\n\n●\tLSTM module is unclear. From Figure 3, LSTM (yellow) seems to indicate all the frames (from both videos) are aware of one another and module looks like Fully connected network, while from description 127-129 It seems like the same LSTM applies separately to the source and target videos.\n\n●\t[Section 3] Since there is no mention of reconstruction loss, how is the decoder getting trained?\n\n●\tIn most of the cases (Tab 1,2) the accuracy of supervised targets (T_sup) is higher than the TranSVAE, so does that mean the target pseudo-labels used in TranSVAE for all the datasets are almost correct?\n\n●\tWhy some of the recent results missing? TRN [32],  CIA [21]. for H->U, and U->H. CIA [21] for epic kitchen.\n\nLIMITATIONS:\nYes, provided.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on video domain adaptation. The authors' primary focus is to eliminate the temporal invariant component of the features initially, followed by the implementation of an adversarial-based domain adaptation technique. To achieve disentanglement, they employ a method of temporally random shuffling of the video, thus identifying domain-specific and static latent factors that remain unaffected by temporal shifts. These domain static factors, along with the Domain-Invariant Sequential Factors, are subsequently utilized separately for domain adaptation. The authors provide an extensive evaluation of their proposed TranSVAE method by conducting rigorous experiments on widely recognized datasets such as UCF-HMDB, Jester, and Epic-Kitchens. The results demonstrate that the TranSVAE model surpasses several state-of-the-art techniques, including those employing multi-modal approaches.\n\nSTRENGTHS:\n1. The proposed idea of handling spatial and temporal domain divergence separately through disentanglement for domain adaptation is interesting. \n2. The idea of using shuffling to disentangle the domain static and dynamic latent factors is interesting.\n2. The method can outperform sota with only RGB modality on multiple datasets.\n\nWEAKNESSES:\n1. The proposed model contains many components but no proper ablation study is conducted to evaluate the effect of each component. In my opinion, the core part should be whether to add $L_{mi}$ and/or $L_{ctc}$ into the model. The ablation study in Figures 5 and 6 is not sufficient. The authors should add ablation study to specifically evaluate this core part, without the influence of other components of the model, especially $L_{svae}$.\n2. While this method outperforms previous works, it apparently uses much more parameters because of the use of a decoder. An analysis of training and inference time and model parameters is needed. \n3. The EPIC-Kithens dataset has a new official split for domain adaptation, which is a larger and better split with an online evaluation server for the test set. This split is used in a lot of previous works. I wonder why the authors do not perform experiments on this set.\n\nQUESTIONS:\n1. I hope the authors can provide better ablation study for better highlighting the core technical part of this work. In my opinion, the use of $L_{svae}$, $L_{cls}$, $L_{adv}$, and PL are not the main technical contribution of this work. \n2. I hope the authors can provide a comparative analysis on training time, inference time, and model parameters. \n3. The EPIC-Kitchens100 dataset should be a better playground for evaluating this method.\n\nLIMITATIONS:\nLimitations are discussed by the authors in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the unsupervised domain adaptation (UDA) problem by viewing it from a disentanglement perspective. It suggests separating two sets of latent factors, one of which is responsible for domain-related spatial information and another encodes dynamic temporal information. As a result, a new model for UDA called Transfer Sequential VAE (TranSVAE) is introduced. To achieve better disentanglement and better learning the paper considers additional constraints: minimizing mutual dependence, task-specific supervision, static consistency, and temporal domain alignment through adversarial-based idea [31]. The experiment set includes 4 dataset settings: UCF-HMDB, Jester, Epic-Kitchens, and Sprites. The proposed method outperforms related works on all these settings visually showing strong disentanglement properties on the Sprites dataset.\n\nSTRENGTHS:\n1) The paper is well-written with a good structure and useful visualizations. The motivation of the method is clear and easy to understand and follow.\n2) The method achieves good performance on several datasets including real-world domains (UCF-HMDB, Jester, Epic-Kithens) and synthetic domains (Sprites). Additionally, it is visually shown a good disentanglement property on the Sprites dataset and analysis of different loss parts to study the domain and semantic alignment of the features. \n3) The ablation study extensively analysis the impact of different proposed constraints showing that incorporating all of the losses achieves the best results. \n4) The proposed method is also performing on par with multi-modal methods that are using other modalities such as optical flow. Sometimes it even outperforms most of them.\n\nWEAKNESSES:\nI would like to see more discussions about related works in the disentanglement area and more outlined explanations of differences of the proposed TranSVAE from them, especially with the papers for video generation [26, 27, 28]. Basically, they also consider the separation of static and dynamic latent factors and considers the minimization of mutual information between them. So it seems that the main difference lies in UDA setting for video action recognition itself (task-specific supervision) and other constraints in consideration (static consistency and temporal domain alignment). The temporal domain alignment is based on the existing ideas of [31], so the paper looks to me mostly as a smart combination of the existing approaches not as a fully novel model.\n\nQUESTIONS:\nI have no additional questions except the Weakness section discussion.\n\nLIMITATIONS:\nThe paper adequately addressed the limitations and potential negative social impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}