Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixing
Authors: Aadarsh Sahoo
Rutav Shah
Rameswar Panda
Kate Saenko
Abir Das
Venue: NeurIPS 2021 Poster (2021)
Keywords: Video Domain Adaptation
Contrastive Learning
Action Recognition

Abstract: Unsupervised domain adaptation which aims to adapt models trained on a labeled source domain to a completely unlabeled target domain has attracted much attention in recent years. While many domain adaptation techniques have been proposed for images, the problem of unsupervised domain adaptation in videos remains largely underexplored. In this paper, we introduce Contrast and Mix (CoMix), a new contrastive learning framework that aims to learn discriminative invariant feature representations for unsupervised video domain adaptation. First, unlike existing methods that rely on adversarial learning for feature alignment, we utilize temporal contrastive learning to bridge the domain gap by maximizing the similarity between encoded representations of an unlabeled video at two different speeds as well as minimizing the similarity between different videos played at different speeds. Second, we propose a novel extension to the temporal contrastive loss by using background mixing that allows additional positives per anchor, thus adapting contrastive learning to leverage action semantics shared across both domains. Moreover, we also integrate a supervised contrastive learning objective using target pseudo-labels to enhance discriminability of the latent space for video domain adaptation. Extensive experiments on several benchmark datasets demonstrate the superiority of our proposed approach over state-of-the-art methods. Project page: https://cvir.github.io/projects/comix.

Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixing
Aadarsh Sahoo'
Rutav Shah' Rameswar Panda²
Kate Saenko Abir Das'
IIT Kharagpur, 2 MIT-IBM Watson Al Lab, $ Boston University
Esahoo_aadareae rutavmse, abir@cse. .iitkpp....................... ac, in, rpanda0ibm. com, saenko@bu. edu
Abstract
Unsupervised domain adaptation which aims to adapt models trained on a labeled source domain a completely unlabeled target domain has attracted much attention in recent years. While many domain adaptation techniques have been proposed for images, the problem of unsupervised domain adaptation in videos remains largely underexplored. In this paper, we introduce Contrast and Mix (CoMix), new contrastive learning framework that aims learn discriminative invariant feature representations for unsupervised video domain adaptation. First, unlike existing methods that rely on adversarial learning for feature alignment, we utilize temporal contrastive learning to bridge the domain gap by maximizing the similarity between encoded representations of an unlabeled video at two different speeds well minimizing the similarity between different videos played at different speeds. Second, we propose a novel extension to the temporal contrastive loss by using background mixing that allows additional positives per anchor, thus adapting contrastive learning to leverage action semantics shared across both domains. Moreover, we also integrate supervised contrastive learning objective using target pscudo-labels to enhance discriminability of the latent space for video domain adaptation. Extensive experiments several benchmark datasets demonstrate the superiority of our proposed approach over sate-of-thhe-a methods. Project page: https://cvirr gi httpttuwb.../ooojtcccccoooiiioio
1 Introduction
Unsupervised domain adaptation (UDA), which alleviates the requirement of large amounts of annotated data by adapting a model learned on a labelled source domain to an unlabelled target domain, has drawn great deal of attention in the last few years [2 80 Much progress has been
made in developing deep UDA methods by minimizing the cross-domain divergence [39] 70]. adding
adversarial domain discriminators [20] 74]. and image-to maage-to-image translation techniques [26] 51).
However, despite impressive results on commonly used benchmark datasets (e.g., [61] 75 (57), most
of the methods have been developed only for images and not for videos, where the annotation task is often more complicated requiring tedious human labor in comparison to images.
More recently, very few works have attempted deep UDA for video action recognition by directly
matching segment-level features 88 50
with attention weights [ 53). However, (1)
trivially matching segment-level feature distributions by extending the image-specific approaches, without considering the rich temporal information may not alone be sufficient for video domain adaptation; (2) prior methods often focus on aligning target features with source, rather than exploiting any action semantics shared across both domains (e.g., difference in background with the same action: videos in the top of Figure [ are from the source and target domain respectively, but both capture the same action walking); (3) existing methods often rely on complex adversarial learning which is unwieldy train, resulting in very fragile convergence.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Meanwhile, self-supervised pretext tasks like predicting rotation and translation have recently emerged as an alternative adversarial learning for unsupervised domain adaptation in images [8] 7T. While these works show the promising potential of self-supervised learning in aligning source and target domains, the more recent very successful contrastive representation learning 9 23 52 has never been used to adapt video action recognition models to target domains. Motivated by this, in this paper, we explore the following natural, yet important question: whether and how contrastive learning could be exploited for the challenging and practically important task of unsupervised video domain adaptation for human action recognition?
To this end, we introduce Contrast and Mix (CoMix), simple yet effective approach based on contrastive learning to adapt video action recognition models trained on a labeled source domain to unlabelled target domains. First, we propose to represent video as a graph and then utilize temporal contrastive self-supervised learning over the graph representations as nexus between source and target domains to align features, without requiring any additional adversarial learning, most prior works do in
Source
Target
video domain adaptation [8] II 531. Specifi- cally, maximize the similarity between coded representations of the same video at two different speeds as well minimize the sim- ilarity between different videos played at dif- ferent speeds, leveraging the fact that changing video speed does not change an action on both domains. While minimization of contrastive self-supervised losses in both domains simulta- neously helps in domain alignment, it ignores action semantics shared across them as the loss treats each domain individually. To alleviate this, we incorporate new synthetic videos into the temporal contrastive objective, which are obtained by mixing background of video from one domain to video from another domain, as shown in Figure [ (bottom). Importantly, since mixing background doesn't 't change the temporal dynamies, we introduce pseudo-labels for the mixed videos to be same as the label of the original videos and consider additional positives per anchor (see Figure which encourages the model to generalize to new samples that may not be covered by temporal contrastive learning in hand. In other words, mixed background video of an input sample the embedding space act as small semantic perturbations that are not imaginary, i.e., they are representative of the action semantics shared across source and target domains. Finally, rather than relying only on the supervision of source categories to learn a discriminative representation, we generate pseudo-labels for the target samples in every batch and then harness the label information using a temporal supervised contrastive term, that pushes the examples from the same class close and the examples from different classes further apart (Figure 2 right). While our modified contrastive losses are motivated by the supervised contrastive learning B0J, pseudo labels for exploiting shared action semantics and discriminative information from target domain, instead of using true labels as an alternative to supervised cross-entropy loss (which is not present for target samples). To the best of our knowledge, ours the first work that successfully leverages contrastive learning in an unified framework to align cross-domain features while enhancing discriminabilty of the latent space for unsupervised video domain adaptation.
Mix (Source, Target BG)
Mix (Target, Source BG)
Figure 1: Background Mixing. Top row shows two representative videos from the source and target domain respectively. Both videos capture the same action "walk- ing" with different backgrounds. Bottom row shows videos obtained after mixing target background with source video and vice versa.
To summarize, the main contributions of our work are as follows:
We introduce Contrast and Mix (CoMix), new contrastive learning framework to learn discrimina- tive invariant feature representations for unsupervised video domain adaptation. Overall, CoMix is simple and easy to implement which perfectly fits into modern mini-br batch end-to-end training.
We propose a novel extension to temporal contrastive loss by using background mixing that allows additional positives per anchor, thus adapting contrastive learning to leverage action semantics shared across both domains. We also integrate supervised contrastive learning objective using pscudo label information from the target domain to enhance discriminabilty of the latent space.

---

Anchor
Anchor
Anchor
Negalives
Negatives
16-zlips
Negalives
Positive
BGM Positive
TPL. Positive
12/84-c1pps
Temporal Contrastive Learning (TCL) )
TCL with Background Mixing (TCL w/ BGM)
Incorporating Target Pseudo-labels
(TCL w/ TPL)
Figure 2: Temporal Contrastive Learning with Background Mixing and Target Pseudo-labels. Temporal contrastive loss (left) contrasts a single temporally augmented positive (same video, different speed) per anchor against rest of the videos in a mini-batch as negatives. Incorporating background mixing (middle) provides addi- tional positives per anchor possessing same action semantics with different background alleviating background shift across domains. Incorporating target pseudo-labels (right) additionally enhances the discriminabilty by contrasting the target videos with the same pseudo-label as positives against of the videos as negatives.
We conduct extensive experiments on several challenging benchmarks (UCF-HMDB S Jester [53] and EEpic-Kitchent (50) for video domain adaptation demonstrate the superiority of our approach over state-of-the-ard methods. Our experiments show that CoMix delivers a significant performance increase over the compared methods, e.g., CoMix outperforms SAVA [ (ECCV'20) by 3.6% on UCF-HMDB [8] and T 8 (ICCV" 19) by 9.2% on Jester 455 benchmark respectively).
2 Related Work
Action Recognition. Much progress has been made in developing variety of ways to recognize
video actions, by either applying 2D-CNNs 6 37 47 79] or 3D-CNN: 4 17 22 73 Many successful architectures are usually based on the two-stream model 667. processing RGB frames and optical-flow in two separate CNNs with a late fusion in the upper layers 22 SlowFast network [8]] employs two pathways for recognizing actions by processing video at different frame rates. Mitigat- ing background bias in action recognition has also been presented in [10 36). Despite remarkable progress, these models critically depend on large labeled datasets which impose challenges for cross-domain action recognition. In contrast, our work focuses on unsupervised domain adaptation for action recognition, with labeled data in source domain, but only unlabeled data in target domain.
Unsupervised Domain Adaptation. Unsupervised domain adaptation has been studied from mul- Representative works minimize some measurement of
tiple perspectives (see reviews 12
distributional discrepancy [2 39
65 70
or adopt adversarial learning S 20
40
56
74] to generate
comainn-nvarrinn features. Leveraging image translation (25 26
style transfer [15] 91] is
also another popular trend in domain adaptation. Deep self-training that focus on iteratively training the model using both labeled source data and generated target pseudo labels have been proposed in 46 90]. Semi-supervised domain adaptation leveraging few labeled samples from the target domain has also been proposed for many applications [4
A very few methods have recently attempted video domain adaptation, using adversarial learning combined with temporal attention B 42 53, multi-modal cues [50], and clip order prediction [.]. While existing video DA methods mainly rely on adversarial learning (which is often complicated and hard to train) in some form or other, they do not take any action semantics shared across domains into considera- tion. Our approach on the other hand, successfully leverages temporal contrastive learning to learn domain- comain-invariant features while exploiting shared action semanties through background mixing for video domain adaptation. Recently, self-supervised tasks like predicting rotation and translation have been used for unsupervised domain adaptation and generalization, mainly for images 3 38 71].
33 63
By contrast, we focus on the more challenging problem of domain adaptation for human action recognition, where our goal is to align domains by learning consistent features representing different speeds of unlabeled videos. We further propose temporal supervised contrastive loss to ensure discriminabilty by considering pscudo-labeling in an unified framework for video domain adaptation.
Contrastive Learning. Contrastive representation learning is becoming increasingly attractive due to its great potential to leverage large amount of unlabeled images [9] 16.
23
48
52
and
videos 19 31 54
60 59 78). Speed of a video is investigated for self-supervised
28
86
and
3

---

unlike the problem we consider in this paper. Recent works [4 87 utilize contrastive learning with different augmentations for learning unsupervised representations of graph data. Contrastive learning has also been recently used in supervised settings, where labels are used to guide the choice of positive and negative pairs [30]. While our approach is inspired by these, we propose novel temporal contrastive learning framework with background mixing for video domain adaptation, which to our best knowledge has not been explored in the literature.
semi-supervised learning 688 94]
Image Mixtures. Mixup regularization [89] and its variants 2 76 88) that train models on virtual examples constructed as convex combinations of pairs of images and labels have been used to improve
the generalization of neural networks. Very few methods apply Mixup in domain adaptation, but
mainly to stabilize the domain discriminator 62 85] or to smoothen the predictions 44 Several works have recently leveraged the idea of different image mixtures [34 66] for improving contrastive representation learning. Our proposed background mixing can be regarded as an extension this line of research by adding background of a video from one domain to video from another domain, to explore shared semantics while learning -nvvainnnnaaiint features for action recognition.
3 Proposed Method
Unsupervised video domain adaptation aims to improve the model generalization performance by transferring knowledge from labeled source domain to an unlabeled target domain. Formally, we have a set of labelled source videos DD Dsource= = (vv((+) y² )
No and a set of unlabelled target videos M==
Diarget= = ( | with a common label space L. Given these data sets, goal is to learn single model for action recognition that performs well on previously unseen target domain videos.
Approach Overview. Figure 3 illustrates an overview of CoMix. Our action recognition model consists of a feature encoder FF with temporal graph encoder G. Given a video, the feature encoder FF first extracts clip-level fea- tures, and then a graph encoder G utilizes those features to model intrinsic temporal relations for providing a robust encoded representation for action recognition. CoMix adopts supervised learning on the source vidcos, as the labels are available, jointly with two novel temporal con- trastive learning loss terms to align the features for domain adaptation. Specifically, we maxi- mize the similarity of the encoded representation of the fast version of video (represented by s clips) with that of the slow version of the same f) as
Sianco Doeuis
Target Dendt
Supersioo Lousnee
Fouture Alignment
Seara
Torge
Contrast and Mis
Figure 3: An Overview of our Approach. Given la- beled videos in source domain and only unlabeled vidcos target domain, CoMix adopts supervised learning on source videos. jointly with temporal contrastive learning on both domains to align features. Additional cross- domain contrastive supervision is obtained using back- ground mixing across domains and using target pscudo- labels for enhancing discriminability of the latent space. CoMix provides a more simpler yet effective approach than adversarial learning for aligning both domains.
video (represented by clips, where well as minimize the similarity of the representa- tions of different videos within each of the two domains. However, as temporal contrastive loss treats each domain individually, we further add two new sets of synthetic videos that contain source videos mixed with target background and vice versa, respectively for introducing the background variations among the videos while keeping the action semantics intact. Finally, we generate pseudo-labels for the target videos in every mini-batch and utilize them using another temporal supervised contrastive term. This term contrasts target videos with the same pseudo-label as positives to learn features discriminative for the target domain. We now describe each of our proposed components individually in detail in the following subsections.
Video Representation. Capturing long-range temporal structure in videos is crucial for action recog- nition, which in turn affects the overall generalization performance of a model when adapting across domains. Thus, we adopt graph convolutional neural network (G) on top of a 3D convolutional neural network (F) as our video feature encoder. Specifically, for a video V with n clips, the feature extractor F maps the clips into the corresponding sequence of features, which alone do not incorporate the rich temporal structure of the video. Therefore, we use the temporal graph encoder which constructs a fully connected graph on top of the clip-level features, with learnable edge weights through a parameterized adjacency matrix, as in [8I) With these graph representations, we apply a graph convolutional neural network with three layers and finally perform average pooling over all

---

the node features to output the encoded representation of the video V. In summary, the end-to-en -nn network ((F()))
Ie takes sequence of clips from a video as input and outputs confidence scores (logits) over the number of classes e for recognizing actions.
Temporal Contrastive Learning. Given video representations, our goal is to leverage contrastive self-supervised learning in both domains for unsupervised domain adaptation. To this end, we use temporal speed invariance in videos as a proxy task and enforce this with pairwise contrastive loss. Specifically, our key idea is to represent videos in two different temporal speeds (fast and slow) to obtain their encoded representations and then consider the fast and slow version representations of the same video to constitute positive pairs, while versions from different videos constitute negative pairs. Formally, let consider a mini-batch of B videos {V), n: V pt V with corresponding feature representations {z,, 2 .... z "1 J. where each of the videos Va is represented using n number of sampled clips. Let f be the number of clips used to represent the fast version of the videos (forwarded through the base branch), and s be that used for the slow version (forwarded through the auxiliary branch), with s < f. as shown in Figure 4 Given positive and negative pairs, the model trained such that it learns to maximize agreement between positive pairs, while minimizing agreement between negative pairs. This is achieved by employing temporal contrastive loss (Ltot) as
h(zz,,z.) 21,)
Cia V',, V$) = log
(1)
B Σ h(zz,,z) zb))
h(zz,, 21,)
j=1jei oE(s.f)
where, v) = exp( u /T) is the exponential of cosine similarity measure and T is |
the temperature hyperparameter [ We use f 16, and choose from {12, 8, 4] follow- ing a random uniform distribution in every training iteration where randomness encourages the model learn from a variety of temporal speed variations learn robust representations.
Background Mixing, As temporal contrastive loss treats each domain individually, it ignores shared action seman- ties which is vital for domain alignment. Thus, we propose new perspective of temporal contrastive loss through background mixing, specifically to alleviate the cross- domain background shift, as seen in Figure I The basic idea is to obtain the background frames for the videos in one domain and mix it with the frames of the videos from the other domain. More details on how we extract the backgrounds are provided in supplementary material. This introduces variation in each the domains by adding new synthetic videos with the same action semantics earlier, but possessing background from the other domain. Given two videos vi(s) E Dsource and Vi(t) E Dtarget with corresponding background frames (single image per video) as (G* and ( both domains by a convex combination of the background with each of the frames in the videos as follows.
Legtu
Lagks
Logis
Feature treider
Festere Enceder
Fratere Uncnder
obtain the synthetic videos in
Vi(+) = (1- Vi(.) + A BG
12/5/4 Clips
Clips boadh (Sadbec)
12/4 Clipu Aucilayy branch Deggattee
Vo = (1 (1 + X (G
(2)
Auslitory brunch (Poellioo)
where, À is sampled from the uniform distribution [0, y]. Gri(t and vi(t) correspond to the video from source do- main with target background and vice versa, respectively. The main operation in our proposed background mixing to generate a synthetic video with background from the other domain while retaining the temporal action seman- ties intact. Since mixing background doesn't change the motion pattern of video which actually defines an action, we assume both the original and mixed video to be of the same action class and go beyond single instance positives in Eq. | by adding additional positives per anchor, as supervised contrastive learning [30] (sce Figure for an illustrative example). The modified temporal contrastive
Figure 4: Temporal Contrastive Loss. Given unlabeled videos, we maximize simi- larity between encoded representations of the same video at two different speeds (fast and slow) as well as minimize similarity between different videos played at different speeds.
5

---

loss with background mixing (Cogm) is defined as below:
h(zz.. p)
Lbgm (V), V2) [P(2)]] pEP(2))
Σ
log
B
Σ DEP(
h(zz,, p) +
Σ
{h(z), z:) + hlzzz, 201}
j=1.vii rE(x.f)
(3)
where, P(z)) = {zl, 22, } is the set of positives for the anchor zj.. and / represent the feature representation of the corresponding background-min -ixedd video depending the domain to which V belongs. Note that for anchor zjy, there are 3 positive pairs: (a) slow version of the mixed video (2)), (b) fast version of the mixed video (2)), and (c) slow version of the original video (z)). Also, the loss is computed for all positive pairs in the mini-batch, i.e., (V), VT). (V., 5? VJ), (V), V.).
and (V., Vz). Simultaneous minimization of figm in both source and target domains not only learns temporal dynamies but also helps to better align the features for video domain adaptation by leveraging action semantics shared across both domains. Our background mixing is especially effective in video domain adaptation it enforces the model be robust to domain changes (i.e., difference in background as shown in Figure D while leaving the action semantics intact. Further, it can also be adopted as a data augmentation strategy for improved generalization in standard video action recognition: we leave this as an interesting future work.
Incorporating Target Pseudo Labels. While temporal contrastive loss with background mixing helps in aligning the learned representations across the two domains, we cannot fully rely on source categories to learn features discriminative for target domain. Therefore, we propose contrastive loss (30) over pseudo-labeled target samples, an extended version of temporal contrastive loss in Eqn. to enhance discriminabilty by allowing many samples per anchor to be positive, so that videos of the same pseudo-label can be attracted to each other in the embedding space. Let A be the subset of videos assigned pseudo-labels using a confidence threshold, from a mini-batch of B videos, the supervised temporal contrastive loss for incorporating target pseudo-labels (Cipt) is defined as
supervised
Cipt (Vi, V.)
1
Σ log Σ pEP(.))
h(z),.) P) h(z h(z).p) Σ h(zz, z5)
(4)
[P(z))
PEP(2T)
a.d.a "E(s.f)
is the set of all positives for video Vi and j represent the pseudo-label for target video V Note that the set of positives (P(.) includes all the target domain samples (fast and slow) classified as the same action class as that of the anchor (zz,) through the pseudo labels. Following (95), leverage temporal ensemble prediction for given video V from the target domain to produce robust and better-calibrated version of pseudo-labed Specifically,, we obtain the encoded (logits) representations z and z from the base and auxiliary branch respectively and then compute the pseudo-label as ÿ = arg max, softmax(z) weed)) where
where, P(zz) = 27 P
o (2/)
z "uedd represents the mean of both logits. We consider the class index & on which the model is most confident among e claasess provided it is higher than a confidence threshold.
Optimization. Besides the losses Lbgm and Lipt, we minimize the standard supervised cross-entropy loss (Loe) on the labelled source videos as follows.
Lee (VilV)) 3- E(o') (gg(((((V)())))
(5)
Overall, the loss function for training our model involving both source and target domain data is,
LCoMie C(e) + Abgm (clgt. + c(gg)) + Aept Clpd
((t)
where Abgm and Aept weights to balance the impact of individual loss terms. To reduce the g(s) number of hyper-parametenn we use the same weight Abym for both 48mm and c(t) bim. Notably, for the semi-supervised domain adaptation setting, we also use supervised cross-entropy loss for the few labeled target domain videos in addition to the source domain videos.

---

4 Experiments
Datasets. We evaluate the performance of our approach using several publicly available benchmark datasets for video domain adaptation, namely UCF-HMDB 77 Jester [53], and Epic- Kitchens (50)
UCF-HMDB (assembled by authors in
an overlapped subset of the original UCF
[69 and
HMDB datasets [32], containing 3, 209 videos across 12 classes. Jester (assembled by authors in 53
is a large-scale cross-domain dataset that contains videos of humans performing hand gestures [45 from two domains, namely Source and Target that contain 51, 498 and 51, 415 video clips respectively across 7 classes. Sépic-Kitcheee
(assembled by authors in [50]) a challenging egocentric dataset that consists of videos across & largest action classes from three domains, namely D1, D2 and D3,
corresponding to POS, P01 and P22 kitchens on the full Epic- Kitchens dataset [13]. We use the standard training and testing splits provided by the authors in 7 53 55 to conduct our experiments on each dataset. More details about the datasets can be found in the supplementary material.
Baselines. We compare our approach with the following baselines. (1) source only (a lower bound) and supervised target only (an upper bound) baselines that trains the network using labeled source data and labeled target data respectively, (2) popular UDA methods based on adversarial learning (e.g., DANN [20), and ADDA [4),, (3) existing video domain adaptation methods, including SAVA [.]. N 8. ABG 42 and TCoN [53]. We also compare with Source + Target (which simply uses all labelled data available to to train the network) and ENT 633 in semi-supervised domain adaptation experiments. We directly quote the numbers reported in published papers when possible and use source code made publicly available by the authors of TA N 8 on both Jester and Epic- Kitchens.
Implementation Details. Following [1] we use 13D 4 as the backbone feature encoder network, initialized with Kinetics pre-trained weights. For the temporal graph encoder, we use 3-layer GCN similar to [8]. We follow the standard 'pre-train then adapt' procedure used in prior works [74 11 and train the model with only source data to provide a warmstart before the proposed approach is employed. The dimension of the features extracted from the I3D encoder is 1024 which is the same as the node-feature dimension of the initial layer of the GCN. The final layer of the GCN has its node-feature dimension same the number of action classes in dataset and uses mean aggregation strategy to output the logits. We use a clip-length of 8-frames and train all the models end-to-ent using SGD with a momentum of 0.9 and a weight decay of 1c-7. We use initial learning rate of 0.001 for the I3D and 0.01 for the GCN in all our experiments. We use a batch size of 40 equally split over the two domains, where each batch consists of clips from the same video, where is 16 for the fast version (f) and 12, 8, or 4 for the slow version (s). For inference, we use 16 uniformly sampled clips per video and use the base branch of the model to recognize the action. The temperature parameter is set to T = 0.5. We extract backgrounds from videos using temporal median filtering (58) and empirically set 7 0.5 for background mixing. We use a pseudo-label threshold of 0.7 in all our experiments and smooth the cross-entropy loss with e = 0.1, following [72 49). We set Abgm and Aepl from {0.01, 0.1} depending on the dataset. We report the average action recognition accuracy over 3 random trials. We use 6 NVIDIA Tesla V100 GPUs for training all our models.
Results on UCF-HMDB. Table [ shows results of our method and other competing approaches on UCF. HMDB dataset. Our CoMix frame- work achieves the best average performance of 90.3%, which is about 2.2% more than the previ- ous state-of-the-art performance on this dataset. While comparing with the recent method, SAVA [II] using the same I3D backbone, CoMix ob- tains 4.5% and 2.7 % improvement
Table 1: Results on UCF-HMDB Dataset. CoKix establishes new statc-of-thc-art for unsupervised vidco domain adaptation on UCF-HMDB, by significantly outperforming existing methods.
Method
Backbone
HMDBB
HMDB -UCF
Average
DANN
20
KesNet- I0I
75.3
76.4
10.8
JAN
ResNet- 101
79.3
77.0
AdaBN
35
ResNet-1U1
75.5
77.4
76.4
MCD
RosNet-101
79.3
76.8
TA'N 8
ResNet-1U1
78.3
81.8
$0.1
42
ResNet- 101
85.1
TCoN
53
ResNet-101
57.2
89.1
88. 1
Source Only
85.8
54.5
DANN
20
BD
80.7
88.0
$4.3
ADDA
79.1
88.4
TA"N
8
BDD
81.4
90.5
$5.9
on UCF->HMBB and HMDB task respectively, without relying on frame attention or adversarial learning.
SAVA
91.2
CoRix
13D
N6.7
93.9
90..3
Supervised Target
L3D
95.0
96.8
95.9
These improvements clearly show that our temporal graph contrastive learning with background mixing is not only able to better leverage the temporal information but also shared action semanties, essential for effective video domain adaptation. In summary, CoMix outperforms all the existing video
7

---

Table 2: Results Jester and Epic-Kitchens Datasets. CoMix outperforms TA'N 8 by 9.2% the challenging Jester dataset. On
CoMix achieves the best performance on 5 out of transfer tasks including the best average performance among all compared methods.
Jester
Spic-Kitchem.
Method
Backbone
Average
Source
TTargtt
D2
>D1
D3
>D1
D1
:D2
D3
>D2
D1
>D3
D2
>D3
Source Only
B3D
51.5
35.4
34.6
32.8
35.8
34.1
39.1
35.3
DANN 20
13D
55.4
38.3
38.8
37.7
42.1
36.6
41.9
39.2
ADDA
74
I3D
52.3
36.3
36.1
35.4
41.4
34.9
40.8
37.4
TA 8
I3D
55.5
40.9
39.9
34.2
44.2
37
42.8
39.9
CoMix
D3D
64.7
38.6
42.3
42.9
49.2
40.9
45.1 2
43.7 2
Supervised Target
B3D
95.6
57.0
57.0
64.0
64.0
63.7
63.7
61.5
DA methods on UCF-HMDB, showing the efficacy of our approach in learning more transferable features for cross-domain action recognition without using any target labels.
Results on Jester and Epic-Kitchens, On the large-scale Jester dataset, our proposed approach, CoMix also outperforms other DA approaches by increasing the Source Only (no adaptation) accuracy from 51.5% to 64. 7 %, as shown in Table 2 (left). In particular, our approach achieves absolute improvement of 9.2 % over T (8). which corroborates the fact that CoMix can well handle not only the appearance gap but also the action gap present on this dataset (e.g., for the action class "rolling hand", source domain contains videos of "rolling hand forward"' while the target domain only consists of videos of "rolling hand backward"). Table - (right) summarizes the results on Epic- Kitchens which is another challenging dataset consisting of total 6 transfer tasks with large imbalance across different action classes. Overall, CoMix obtains the best on 5 tasks including the best average performance of 43. %, compared to only 35.3% % and 39.9% 9% achieved by the source only and TA 'N g respectively. While the improvements achieved by our approach are encouraging on both Jester and Epic-Kitchens the accuracy gap between CoMix and supervised target is still significant (30.9% % on Jester and 18.3% % on Epic-KKtchens)) which highlights the great potential for improvement in future for unsupervised video domain adaptation.
Comparison with MM-SADA (50). MM-SADA 50) is another state-of-the-art approach for video domain adaptation that leverages the idea of using multi- (RGB and Optical flow) data to learn better domain invariant representations. The approach has two main components: adversarial learning and multi-modal supervision. While CoMix does not use optical flow features anywhere, the RGB-only version of MM-SADA still uses optical flow features for the multi-modal self-supervision. Interestingly, CoMix (43.2%) shows very competitive performance using only RGB features when compared the above (43.9%) on the Epic-Kitchent dataset. Additionally, we train MM-SADA (RGB-only) (but perform multimodal supervision using both RGB and flow following the original paper (50) on UCF-HMDB dataset and notice that CoMix outperforms it by a margin of 3% on an average (UCF - HMDB: 82.2% vs 86,7% HMDB + UCF: 91.2% vs 93.9%, Avg: 86.7% vs 90.3%), showing its effectiveness in unsupervised video domain adaptation.
Semi-supervised Domain Adapta- tion. To further study the robustness of our proposed approach, we extend the unsupervised domain adaptation to a semi-supervised setting, where one (1-shot) and three target labels (3- shot) per class are available for train- ing. Table shows that our simple ap- proach consistently outperforms the adversarial DA methods (DANN (20).
Table 3: Semi-Supervised Domain Adaptation on UCF-HMDB and Jester Datasets. CoMix significantly outperforms the com- pared methods in both one-shot and three-shot settings.
UCF
HMFB
HMDB-UCF Jester(S)
Jester(T)
Method
T-shot
3-shot
I-shol
3-shot
I-hoot
3-shot
Source
Target
83.2
85.8
90.3
93.7
53.8
55.0
DANN 20
85.4
86.9
92.1
93.1
55.1
59.9
ADDA 74
83.6
86.3
91.2
93.0
59.5
61.3
ENT 63
85.6
88.6
92.8
95.8
58.6
61.5
CoMix
88.4
93.1
95.4
96.6
65.3
69.6
and ADDA (74) including the semi-supervised method, ENT [63]. on both UCF-HMDB and Jester datasets. Remarkably, CoMix with three target labels per class improves the performance of Source + Target baseline from 93.77 7 7% to 96.6% %, which is only 0.2% lower than the supervised target upper bound (in Table D on HMDB UCF task (96.6% vs 96.8%) These results well demonstrate the utility of our proposed approach in many practical applications where annotating a few videos per class is typically possible and therefore worth doing given the boost it provides.
Effectiveness of Individual Components. As seen from Table 4 the vanilla temporal contrastive learning (TCL) achieves an average accuracy of 85.8% on UCF-HMDB while 57.5% on Jester (Ist row), which is already better than DANN 20), and ADDA 77 (ref. Table 1 showing its
8

---

effectiveness over adversarial learning in aligning features. While both background mixing (BGM) and incorporation of target pseudo-labels (TPL) individually improves the performance over TCL (+2.9% %. +5.6 % using BGM and +1.9% +5.4 % using TPI., respectively), addition of both of them leads to the best average performance of 90.3% on UCF-HMDB dataset and 64.7% on the Jester dataset. This corroborates the fact that both cross-domain action semantics (through BGM) and discriminabilty (through TPL) of the latent space play crucial roles in video domain adaptation in addition to the vanilla contrastive learning for aligning features.
Table 4: Ablation Study on UCF-HMDB and Jester. TCL: Temporal Contrastive Learning, BGM:
Table 5: Comparison with MixUp Strategies.
Background mixing outperforms other alternatives in leveraging shared action semantics on UCF-HMDB
Background Mixing. TPL: Target Escudd-LLabche
TCL BGM UPL
Avernge JesteriS) JesteriKi
Method
U-H
H-U
Average Jestersjj-Jssterii
x
$3.3
55.1
85.8
57.5
Gaussian Noise
81.7
90.6
87.6
313
56.2
81.2
88.7
63.1
Video MixUp
85.1
91.7
88.4
62.2
62.9
Video CutMix
92.1
88.5
50.7
93.9
90.3
64.7
Background Mixing
86.7
93.9
90.3
61.77
Comparison with Different MixUp Strategies. We explore the effectiveness of background mixing by comparing with different MixUp strategies (Table 5]: (a) Gaussian Noise: adding White Gaussian
Noise to videos in both domains; (b) Video MixUp 89 : directly mixing one video with another
from different domain, as in images; (c) Video CutMix S8 : randomly replacing a region of a
video with another region from the other domain. The proposed way of generating synthetic videos by mixing background of video from one domain to a video from another domain, outperforms all three alternatives on UCF-HMDB as well as on the more challenging Jester dataset. Note that while both MixUp and CutMix destroy motion pattern of original video, background mixing keeps semantic consistency without changing the temporal dynamics.
Effect of Background Pseudo-labels. We investigate the effect of pseudo-labels on background mixed videos (i.c., both videos considered to be of same action class while creating positives) by simply adding them as unlabeled videos without any modification to the contrastive objective Eq M CoMix without background pseudo-labell decreases the performance from 90.3% to 89.0% (-1.3% %: Table (6) showing its effectiveness in leveraging action semantics shared across both domains.
Effect of Source Contrastive Learning, CoMix adopts contrastive learning on both source and target domains, although we already have supervised cross-entropy loss source videos. We observe that applying contrastive learning on target domain only, by removing source contrastive objective boom from Eq. 6 lowers down the performance from 90.3% to 88.4% (-1.9% UCF-HMDB (Table O This shows the importance of training the model using the same temporal invariance objective on both domains simultancously to achieve effective alignment across domains.
Effect of Random Speed Invariance. We re-
Table 6: Ablation Study on Contrastive Learning.
move randomness in video speed from the aux-
Method
U-H
H-SU
Average
iliary branch of our temporal contrastive learn
CoMix
86.7
93.7
90.3
ing framework and observe that CoMix (with 16 clips in the base branch and only 8 clips in the auxiliary branch) leads to an average top- 1 ac- curacy of 89.6% compared to 90.3% (-0.7%: %
wo Background Picudo-labels
85.8
92.2
89.0
w/o Source Contrastive Learning
85.1
91.8
88.4
w/o Random Speed Invariance
86.4
92.8
89.6
Table (6), showing the importance of random speed invariance in learning robust features.
Self- Ke--Triinng vs Supervised Contrastive Learning, We directly use self- self-training that uses cross-
entropy loss on target pseudo labels instead of c(f and find that the average performance drops to 88.7% on UCF-HMDB. indicating the advantage of supervised contrastive objective in enhancing discriminability of the latent space by successfully leveraging label information from target domain.
Effect of Graph Representation. (a) Removal of Graph representation CoMix: We exam- ine the effect of graph representation for videos and find that by removing GCN from our frame- work lowers down the performance from 90.3% to 88.1% on UCF-HMDB dataset, which shows that graph contrastive learning is more useful in capturing the temporal dependencies, essential for video domain adaptation. (b) Effect of Graph
Table 7: Baseline Comparisons w/ GCN Represen-
tations on UCF-HMDB and Jester Datasets.
Method (w/ GCN)
[ -H
H H--U
Average JesteriS)-Jesteriie JesteriRi
Source Only
82.5
57.7
85.1
54.0
DANN 20
80.0
86.3
83.2
62.9
TA B
52.5
72.4
62.3
51.7
Calix
86.7
93.9
90.3
64.7
9

---

Source Only
TCL
TCL w/ BGM
TCL BGM & TPL (CaMis)
Figure Feature Visualizations using t-SNE. Plots show visualization our approach with different components on UCF-+IMMB task. Blue and red dots represent source and target data respectively. Features for both target and source domain become progressively discriminative and improve from left to right by adoption of our novel components within temporal contrastive learning framework. Best viewed in color.
Additionally, in Table 7 we compare with domain adversarial adaptation methods DANN 20] and TA3N [8] including the Source only baseline with GCN feature representation both UCF-HMDB and Jester datasets. CoMix improves the Source only accuracy by 5.2% and 10.7% respectively on UCF-HMDB and Jester datasets. Furthermore, CoMix outperforms DANN 20 with the same GCN equipped as ours, both datasets (+7.1% 111 +1.8%, respectively) showing its effectiveness over adversarial learning in aligning features for video domain adaptation. TA3N 8 performs very poorly (62.3% and 51.7%) when equipped additionally with graph repre- sentations. We believe this is because TA3N already utilizes Temporal Relational Network [92] for modeling temporal relations, which probably hinders in learning GCN features for successful domain adaptation in videos. (c) Alternatives for Graph Representation: We replace GCN using MLP/LSTM of similar complexity and notice that both alternatives are inferior GCN on UCF-HMDB (MLP: 88.1%, LSTM: 84.3%, GCN: 90.3%), which shows the effectiveness of GCN in our contrastive learning framework for capturing the temporal dependencies, essential for video domain adaptation.
Representation Baseline Methods:
We experiment with a different background extraction strategy 993 that uses Gaussian Mixture Models (GMM) to extract the backgrounds and observe that the very simple and fast strategy based on temporal median filtering (58) outperforms GMM by 2.3% on average on UCF-HMDB (UCF-HMDBB: 85.3% 86.7%, HMDB UCC: 90.7% vs 93.9%, Avg: 88.0% vs 90.3%). Note that our CoMix framework is agnostic to the method used for background extraction and can be incorporated with any other background extraction techniques for videos, e.g., learnable background segmentation strategies such as 182 55
Effect of Background Extraction Method.
Feature Visualizations. We use t-SNE 43 to visualize the features learned using different com- ponents of our CoMix framework. As seen from Figure alignment of domains including discrim- inability improves
adopt "TCL" and "BGM" to the vanilla Source only model. The best results are obtained when all the three components "TCL", "BGM" and "TPL" i.e., CoMix are added and trained using an unified framework (Eq. (6) for unsupervised video domain adaptation. Additional results and analysis including more qualitative examples are included in the supplementary material.
5
Conclusions
In this paper, we introduce a new end-to-end temporal contrastive learning framework to bridge the domain gap by learning consistent features representing two different speeds of the unlabeled videos. We also propose two novel extension to temporal contrastive loss by using background mixing and target pseudo-labels, that allows additional positive(s) per anchor, thus adapting contrastive learning to leverage cross- coss-domain action semantics and label information from the target domain respectively in an unified framework, for learning discriminative invariant features. We demonstrate the effectiveness of our approach on three standard datasets, outperforming several competing methods.
Broader Impact. Our research can help reduce burden of collecting large-scale supervised data in many real-world applications of human action recognition by transferring knowledge from auxiliary datasets. The positive impact that our work could have on society is in making technology more accessible for institutions and individuals that do not have rich resources for collecting and annotating large-scale video datasets. Negative impacts of our research are difficult to predict, however, shares many of the pitfalls associated with standard deep learning models such as susceptibility to adversarial attacks and lack of interpretablity. Other adverse effects could be potential attrition in jobs in certain sectors of economy where fewer employees (security guards, nurses, etc.) are needed to monitor human activities as a result of wider adoption automated video recognition systems. A Acknowledgements. This work was partially supported by the ISIRD Grant EEE,
10

---

References
[1] Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T Freeman, Michael Rubinstein, Michal Irani, and Tali Dekel. Speednet: Learning the Speediness in Videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9922-9933 2020.
[2] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: Holistic Approach to Semi-Supervised Learning. In Advances in Neural Information Processing Systems, pages 5049-5059, 2019.
[3] Fabio Carlucci, Antonio D' Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain Generalization by Solving Jigsaw Puzzles. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pages
2019.
[4] Joao Carreira and Andrew Zisserman. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017.
[5] Chao Chen, Zhihong Chen, Boyuan Jiang, and Xinyu Jin. Joint Domain Alignment and Discriminative Feature Learning for Unsupervised Deep Domain Adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3296-3303, 2019.
[6] Chun-Fu Richard Chen, Rameswar Panda, Kandan Ramakrishnan, Rogerio Feris, John Cohn, Aude Oliva, and Quanfu Fan. Deep Analysis of CNN-Based Spatio-Tempon Temporal Representations for Action Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6165-6175, June 2021.
[7] Jin Chen, Xinxiao Wu, Lixin Duan, and Shenghua Gao. Domain Adversarial Reinforcement Learning for Partial Domain Adaptation. IEEE Transactions on Neural Networks and Learning Systems, pages 1-15, 2020.
[8] Min-Hung Chen, Zsolt Kira, Ghassan AlRegib, Jaekwon Yoo, Ruxin Chen, and Jian Zheng, Tempo- ral Attentive Alignment for Large-Scale Video Domain Adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6321-632323 2019.
[9] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In International conference on machine learning, pages 1597-1607, 2020.
[10] Jinwoo Choi, Chen Gao, Joseph CE Messou, and Jia-Bin Huang. Why Can't Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition. Advances in Neural Information Processing Systems, 32:853 865, 2019.
[11] Jinwoo Choi, Gaurav Sharma, Samuel Schulter, and Jia-Bin Huang, Shuffle and Attend: Video Domain Adaptation. In European Conference on Computer Vision, pages 678 695, 2020.
[12] Gabriela Csurka. A Comprehensive Survey on Domain Adaptation for Visual Applications, pages 1-35. Springer International Publishing, Cham, 2017.
[13] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling Egocentric Vision: The Fpie-Kitchens Dataset. In Proceedings of the European Conference on Computer Vision (ECCV), pages 720-736. 2018
[14] Hal Daumé III, Abhishek Kumar, and Avishek Saha. Frustratingly Easy Semi-Supervised Domain Adaptation. In Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing, pages 53 33-59, 2010.
[15] Aysegul Dundar, Ming- Yu Liu, Ting-Chun Wang, John Zedlewski, and Jan Kautz. Domain Stylization: Strong, Simple Bascline for Synthetic to Real Image Domain Adaptation. arXiv preprint arXiv: 1807.00384, 2018.
[16] Dehidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a Little Help From My Friends: Nearest-Neighborr Contrastive Learning of Visual Representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9588-9597, October 2021.
[17] Christoph Feichtenhofer. X3d: Expanding Architectures for Efficient Video Recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2020.
11

---

[18] Christoph Feichtenhofer. Haoqi Fan. Jitendra Malik, and Kaiming He. Slowfast Networks for Video Recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6202-6211, 2019.
[19] Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Ginshick, and Kaiming He. A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3299-3309, 2021.
[20] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-Adwe Adversarial Training of Neural Networks. The Journal of Machine Learning Research, 17(1):2096 2030, 2016.
[21] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bemhard Schölkopf, and Alexander Smola. A Kernel Two-Sample Test. The Journal Machine Learning Research, 13(1):723-73, 2012.
[22] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Learning Pentiol Features with 3D Residual Networks for Action Recognition, In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 3154-3160, 2017.
[23] Kaiming He. Haoqi Fan. Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast Unsupervised Visual Representation Learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 9729-9738, 2020.
[24] R Devon Hjelm, Alex Fedorov, Samuel I. Lavnic- Mariii-arrchilonn, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning Deep Representations by Mutual Information Estimation and Maximization. In International Conference Learning Representations, 2018.
[25] Judy Hoffman, Eric Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle- Consistent Adversarial Domain Adaptation. In International conference on machine learning, pages 1989-1998, PMLR, 2018.
[26] Lanqing Hu, Meina Kan, Shiguang Shan, and Xilin Chen. Duplex Generative Adversarial Network for Unsupervised Domain Adaptation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 1498-1507, 2018
[27] Arshad Jamal, Vinay P Namboodiri, Dipti Deodhare, and KS Venkatesh. Deep Domain Adaptation in Action Space. In BMVC, volume 2. page 4, 2018.
[28] Simon Jenni, Givi Meishvili, and Paolo Favaro. Video Representation Learning by Recognizing Temporal Transformations. In Proceedings of the European Conference on Computer Vision (ECCV), pages 425-442.
Springer, 2020.
[29] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-Scale Video Classification with Convolutional Neural Networks. In Proceedings of the IEFE conference on Computer Vision and Pattern Recognition, pages 1725-1732, 2014.
[30] Pramnay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised Contrastive Learning, Advances in Neural Information Processing Systems, 33, 2020.
[31] Bruno Korbar, Du Tran, and I Lorenzo Torresani. Cooperative Learning of Audio and Video Models From Self Self-Supervised Synchronization. arXiv preprint arXiv:1807.00230, 2018
[32] Hildegard Kuehne, Hucihan Jhuang, Estibaliz Garrote, Tomaso Poggio, and Thomas Serre. HMDB: A Large Video Database for Human Motion Recognition. In 2011 International conference on computer vision, pages
JEEE, 2011.
[33] Abhishek Kumar, Avishek Saha, and Hal Daume. Co-Regularization Based Supervupee Domain Adaptation. Advances in neural information processing systems, 23:478-486. 2010.
[34] Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun BhannLiang Li, Jinwoo Shin, and Honglak Lee. I-Mix: A Domain- Agnostic Strategy for Contrastive Representation Learning. arXiv preprint arXiv:2010. 08888, 2021.
[35] Yanghao Li. Naiyan Wang, Jianping Shi, Xiaodi Hou, and Jiaying Liu. Adaptive Batch Normalization for Practical Domain Adaptation. Pattern Recognition, 80: 109-117, 2018.
[36] Yingwei Li, Li, and Nuno Vasconcelos. Resound: Towards Action Recognition Without Representation Bias. In Proceedings of the European Conference Computer Vision (ECCV), pages 2018.
12

---

[37] Ji Lin, Chuang Gan, and Song Han. Temporal Shift Module for Efficient Video Understanding, In CVF International Conference on Computer Vision (ICCV), pages 7082-7092, 2019.
[38] Weizhe Liu, David Ferstl, Samuel Schulter. Lukas Zebedin. Pascal Fua, and Christian Leistner. Do- main Adaptation for Semantic Segmentation via Patch-Wiss Contrastive Learning. arXiv preprint arXiv:2104.11056 2021.
[39] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning Transferable Features with Deep Adaptation Networks. International conference on machine learning, pages 97-105, 2015.
[40] Mingsheng Long, Zhangjie Cao, Jianmin Wang. and Michael Jordan. Conditional Adversarial Domain Adaptation. In Advances Neural Information Processing Systems, pages 1640-1650, 2018.
[41] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep Transfer Learning With Joint Adaptation Networks. International conference on machine learning, pages 2208 2217. PMLR, 2017.
[42] Yadan Luo, Zi Huang, Zijian Wang, Zheng Zhang, and Mahsa Baktashmotlagh. Adversarial Bipartite Graph Learning for Video Domain Adaptation. In Proceedings of the 28th ACM International Conference on Multimedia, pages 19-27, 2020.
[43] Laurens van der Maaten and Geoffrey Hinton. Visualizing Data Using t-SNE. Journal of machine learning research, 9NNov) 2008.
[44] Xudong Mao, Yun Ma, Zhenguo Yang. Yangbin Chen, and Qing Li. Virtual Mixup Training for Unsuper- vised Domain Adaptation. arXiv preprint arXiv: 1405,0 2019.
[45] Joanna Materzynska, Guillaume Berger, Ingo Bax, and Roland Memisevic. The Jester Dataset: Large- Scale Video Dataset Human Gestures. In Proceedings of the IEEEJCVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
[46] Mei, Chuang Zhu, Jiaqi Zou, and Shanghang Zhang. Instance Adaptive Self-Training for Unsupervised Domain Adaptation. In European Conference on Computer Vision. Springer, 2020.
[47] Yue Meng, Chung-Ching Lin, Rameswar Panda, Prasanna Sattigeri, Leonid Karlinsky, Aude Oliva, Kate Saenko, and Rogerio Feris. AR-Net: Adaptive Frame Resolution for Efficient Action Recognition. In European Conference on Computer Vision, pages 86-104, 2020.
[48] Ishan Misra and Laurens van der Maaten, Self-Supervised Learning of Pretext-Invariant Representations, In Proceedings of the /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages -777, 2020.
[49] Rafael Müller Simon Kornblith, and Geoffrey E Hinton. When Does Label Smoothing Help? In Advances in Neural Information Processing Systems, pages 4694-4703, 2019.
[50] Jonathan Munro and Dima Damen. Modal Domain Adaptation for Fine-Grained Action Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 122-132, 2020.
[51] Zak Murez, Soheil Kolouri, David Kriegman, Ravi Ramamoorthi, and Kyungnam Kim. Image to Image Translation for Domain Adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4500-4509, 2018
[52] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning With Contrastive Predictive Coding. arXiv preprint 1807 007448, 2018.
[53] Boxiao Pan, Zhangjie Cao, Ehsan Adeli, and Juan Carlos Niebles. Adversarial Cross-Domain Action Recog- nition With Co-Attention, Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11815-1 11822, 2020.
[54] Tian Pan, Yibing Song, Tianyu Yang. Wenhao Jiang, and Wei Liu. Videomoco: Contrastive Video Repre- sentation Learning With Temporally Adversarial Examples. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 1 11205-177 11214, 2021.
[55] Prashant W Patil, Akshay Dudhane, and Subrahmanyam Murala. Multi-frame Recurrent Adversarial Network for Moving Object Segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2302-2311, 2021.
[56] Zhongyi Pei, Zhangjie Cao. Mingsheng Long, and Jianmin Wang. Multi-Adversarial Domain Adaptation. Thirty-second AAAI conference artificial intelligence, 2018
13

---

[57] Xingehao Peng, Ben Usman. Neela Kaushik, Judy Hoffmman, Dequan Wang, and Kate Saenko. VisDA: The Visual Domain Adaptation Challenge. arXiv preprint crXiv:1710.06924, 2017.
[58]
Massimo Piccardi. Background Subtraction Techniques: A Review. In 2004 IEEE International Conference on Systems, Man and Cybernetics, volume 4, pages 3099-3104, IEEE, 2004.
[59] Rui Qian, Tianjian Meng, Boqing Gong, Ming- Huugn Yang. Huisheng Wang, Serge Belongic, and Yin Cui. Spatiotemporal Contrastive Video Representation Learning. Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 066646674,250 2021.
[60] Adrià Recasens, Pauline Luc, Jean Saan-Bapisste Alayrac, Luyu Wang. Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Patraucean, Florent Altché, Michal Valko, et al. Broaden Your Views for Self- Supervised Video Learning. arXiv preprint 103. 16559, 2021.
[61] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting Visual Category Models to New Domains. In European conference on computer vision, pages 213 Springer, 2010.
[62] Aadarsh Sahoo, Rameswar Panda, Rogerio Feris, Kate Saenko, and Abir Das. Select, Label, and Mix: Learning Discriminative Invariant Feature Representations for Partial Domain Adaptation. arXiv preprint arXiv:2012.03355 03333588 2020.
[63] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-SSuervised Domain Adaptation via Minimax Entropy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8050-8000 8058, 2019.
[64] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum Classifier Discrepancy for Unsupervised Domain Adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3723 3732, 2018
[65] Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein Distance Guided Representation Learning for Domain Adaptation. In Shirty-Second AAAI Conference on Artificial Intelligence, 2018.
[66] Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, Trevor Darrell, and Eric Xing. Un-Mix: Rethink- ing Image Mixtures for Unsupervised Visual Representation Learning. arXiv preprint arXiv:2003.05433 3(7), 2020.
[67] Karen Simonyan and Andrew Zisserman. Two-Stream Convolutional Networks for Action Recognition in Vidcos. In Advances in Neural Information Processing Systems, pages 568-576, 2014.
[68] Ankit Singh, Omprakash Chakraborty, Ashutosh Varshney, Rameswar Panda, Rogerio Feris, Kate Saenko, and Abir Das. Semi- Supervsuee Action Recognition with Temporal Contrastive Learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10389-10399, 2021.
[69] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A Dataset of 101 Human Actions Classes From Videos in the Wild. arXiv preprint arXiv:1212.0400; 2012.
[70] Baochen Sun and Kate Saenko. Deep Coral: Correlation Alignment for Deep Domain Adaptation. In European conference on computer vision, pages 443-450. Springer, 2016.
[71] Yu Sun, Eric Tzeng, Trevor Darrell, and Alexei A Efros. Unsupervised Domain Adaptation Through arXiv preprint arXiv: 1909. 1825, 2019.
[72] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
[73] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning Spatiotemporal Features With 3D Convolutional Networks. In Proceedings of the IEFE international conference on computer vision, pages 4489-4497, 2015.
[74] Eric Tzeng, Judy Hoffman, Kate Sacnko, and Trevor Darrell. Adversarial Discriminative Domain Adapta- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167-7176, 2017.
[75] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep Hashing Network for Unsupervised Domain Adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2017.
14

---

[76] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, loannis Milliagkas, David Lopez-Paz. and Yoshua Bengio. Manifold Mixup: Better Representations by Interpolating Hidden States. In International Conference on Machine Learning, pages 1138-6447. PMLR, 2019.
[77] Jiangliu Wang. Jianbo Jiao, and Yun-Hui Liu. Self-Supervised Video Representation Learning by Pace Prediction. In European Conference on Computer Vision, pages 504-521, 2020.
[78] Jinpeng Wang. Yuting Gao, Ke Li, Yiqi Lin, Andy J Ma, Hao Cheng, Pai Peng. Feiyue Huang, Rongrong
Ji, and Xing Sun. Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition. pages 11804-18113 2021.
[79] Limin Wang. Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang. and Lue Van Gool. Temporal Segment Networks: Towards Good Practices for Deep Action Recognition. In Europeon conference on computer vision, pages 20-36, 2016.
[80] Mci Wang and Weihong Deng. Deep Visual Domain Adaptation: A Survey, Neurocomputiing, 312:135-155: 2018.
[81] Xiaolong Wang and Abhinav Gupta. Videos as Space-Time Region Graphs. In Proceedings of the European conference on compuder vision (ECCV), pages 399-417, 2018.
[82] Xueying Wang, Lei Liu, Guangli Li, Xiao Dong, Peng Zhao, and Xiaobing Feng. Background Subtraction on Depth Videos With Convolutional Neural Networks. In 2018 International Joint Conference on Neural Networks (ICCN), pages 1-7. IEEE, 2018.
[83] Yuan Wu, Diana Inkpen, and Ahmed El-Roby. Dual Mixup Regularized Learning for Adversarial Domain Adaptation. In European Conference on Computer Vision, pages 540-555. Springer, 2020.
[84] Yaochen Xie, Zhao Xu, Zhengyang Wang, and Shuiwang Ji. Self- Supervuee Learning of Graph Neural Networks: A Unified Review, arXiv preprint 1XXx:22002.105:: 2021.
[85] Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang. Qi Tian, and Wenjun Zhang. Adversarial Domain Adaptation with Domain Mixup. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6502-6509, 2020.
[86] Yuan Yao. Chang Liu, Dezhao Luo,, Yu Zhou, and Qixiang Ye. Video Playback Rate Perception for Self Supervieee Spatio-Temporal Representation Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6548-6557 2020.
[87] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang. and Yang Shen. Graph Contrastive Learning With Augmentations. Advances in Neural Information Processing Systems, 33, 2020.
[88] Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization Strategy To Train Strong Classifiers With Localizable Features. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 6023-6032, 2019.
[89] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Mixup: Beyond Empirical Risk Minimization. arXiv preprint 1rXiv:17 170.09942, 2017.
[90] Yabin Zhang, Bin Deng, Kui Jia, and Lei Zhang. Label Propagation with Augmented Anchors: Simple SuperSupe Learning baseline for Unsupervised Domain Adaptation. In European Conference on Computer Vision, pages 781-797. Springer, 2020.
[91] Yiheng Zhang, Zhaofan Qiu, Ting Yao, Dong Liu, and Tao Mei. Fully Convolutional Adaptation Networks for Semantic Segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8810-6818, 2018.
[92] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal Relational Reasoning in Videos. In Proceedings of the European Conference on Computer Vision (ECCV), pages 803 818, 2018
[93] Zoran Zivkovic. Improved Adaptive Gaussian Mixture Model for Background Subtraction. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., volume 2, pages 28-31. IEEE, 2004.
[94] Yuliang Zou, Jinwoo Choi, Qitong Wang, and Jia-Bin Huang. Leaming Representational Invariances for Action Recognition. arXiv preprint ar3iv:2103. 2021.
[95] Yuliang Zou, Zizhao Zhang, Han Zhang, Chun- LiaunLLing La Li, Xiao Bian, Jia- -in Huang, and Tomas Pfister. PseudoSeg: Designing Pseudo Labels for Semantic Segmentation. arXiv preprint ar 2020.
15


Reviews:
[{"review_id": "yxlBmo_y_q1", "reviewer": "Reviewer_UMhT", "summary": "This work proposes a video-based contrastive learning framework by maximizing the similarity between encoded representations of the same video at two different speeds as well as minimizing the similarity between different videos played at different speeds. Results on UCF-HMDB, Jester, and Epic-Kitchens validate its effectiveness. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Merits]\n\n1. No adversarial loss is needed, which helps to stabilize the training.  \n\n2. Formulating background mixing into the CL framework is novel. \n\n3. Results on UCF-HMDB, Jester, and Epic-Kitchens as shown in Table 1-5 are good.\n\n4. Code is provided. \n\n\n[Improvement Suggestions]\n\n1. How the background information is extracted?  Why In Line 65 \"Importantly, since mixing background doesn’t change the temporal dynamics\". Why? e.g. when the synthetic videos are generated via background mixing, what if it becomes false positive when the background information is added to the original frame?\n\n2. This is an application of contrastive learning (Formula 3) with mixup (Formula 2) and pseudo labeling (Formula 4) in the scope of UDA for action recognition. Except for the second point (BG part in Formula 3) mentioned above, all components have been proposed in the previous work. There is no specific new method in terms of hard negative selection for the background. Could the author[s] justify the novelty of the proposed method precisely? \n\n3. How long does it take for the SSL convergence? If it is stable across multiple seeds? Why only a single number is reported for all experiments? Table numbers look good, but the current form seems to lack ablation illustrations. \n\n4. Could author[s] add discussions or results with the baseline mixup UDA method https://arxiv.org/pdf/2001.00677.pdf?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This work has no potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "YShH0x3mNT", "reviewer": "Reviewer_bY6e", "summary": "This paper proposes a new contrastive method for unsupervised video domain adaptation. This method employs temporal contrastive learning based on graph convolution and back ground mixing approach for enhancing action similarity, so called CoMix. \nAdditionally, for tagert domain, CoMix uses a pseudo label-based target domain contrastive learning. The authors evaluate CoMix on UFC-HMDB, Epic-Kitchen, and Jester datasets with intensive comparison and ablation studies. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength\n- Unsupervised video domain adaption is a challenging but significant research topic. \n- Leveraging contrastive learning, graph representation for temporal relations, and background mixing in video domain adaption is an interesting idea and seems novel (considering NeurIPS deadline). \n- It is well-written, well-organized, and easy to read.\n- Including ablation study, experiments seem thorough except for the results on Epic-Kitchen dataset.\n- Source code was submitted.\n\n### Weakness\n- Some important previous references were missed in related work such as \n  - For domain adaption: MCD [Saito et al. 2018]\n  - For Image Mixture: PuzzleMix [Kim et al. 2020] and CoMixup [Kim et al. 2021]\n- Also, the comparison with MM-SADA [Munro and Damen 2020] was missed even if it was referred to in the paper as [51]. In particular, the comparison is important because MM-SADA proposes Epic-Kitchen dataset and MM-SADA with RGB only not using optical flow shows comparable or better performances than the proposed CoMix. This should be clarified.\n- The authors employs graph convolution for temporal contrastive learning. In this study, because the number of clips are not large (=16), the fully connected graph is not a problem. However, when the number of clips increases (for longer clip sequence), the connections also exponentially increases, which might restrict scale-up of this method. \n-  For background mixing, the authors uses a very simple image processing method. How about using more advanced pretrained foreground-background decomposition method? How large can the background mixing method influence on the performance?\n- How sensitive are the results by target pseudo-label confidence threshold? Any results?\n- For GCN ablation, did the author try to use MLP, LSTM, or Transformer instead of GCN? Because removing GCN makes the total parameter reduced, other NN can be helpful to validate the effect of GCN.  \n- Minor comments\n  - L276 and L308 in P7, SAVA was wrongly cited. [10] --> [12]\n  - Overall, there is some redundancy in Introduction, Proposed Method and captions in Figures 2 and 3.\n  - Figure 3 can be more detailed. If the page limitation is an issue, More detailed figure can be located at supplementary material. \n  - Instead of naive video cutmix, VideoMix [Yun et al. 2020] can be used. \n  - [Song et al. 2021] can be added in related work although it is not published until the NeurIPS deadline. \n\n#### References\n- [Saito et al. 2018] Maximum classifier discrepancy for unsupervised domain adaptation. CVPR 2018.\n- [Kim et al. 2020] Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup. ICML 2020.\n- [Kim et al. 2021] Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversity. ICLR 2021.\n- [Yun et al. 2020] VideoMix: Rethinking Data Augmentation for Video Classification. arXiv 2020. \n- [Song et al. 2021] Spatio-temporal Contrastive Domain Adaptation for Action Recognition. CVPR 2021.\n\n\n=========================== After rebuttal =======================\nI read the other reviewers' comments and author response. Beacuse the authors addressed most of my concerns, I decided to raise my score. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "#### No negative societal impact\n", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concern", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "Oakc9sQEE8", "reviewer": "Reviewer_LMAg", "summary": "This paper aggregates (a) contrastive learning, (b) background mixing and (c) target pseudo-labels on video domain adaptation and achieves excellent performance (outperforming SOTA). The paper is written well and easy to follow.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has solid results and ablations on several key datasets in video domain adaptation. The paper reads well.\n\nHowever, the novelty is limited. Using contrastive learning to leverage labeled and unlabeled data (UDA is a special case) is not new. Sampling at different speeds and background mixing can be considered as data augmentation (SimCLR already showed that data augmentation is crucial for contrastive learning). In addition, using pseudo-labels in the target domain is not new either.\n\nThe idea of background mix is interesting, but it is not clear WHY it is more effective than Mixup or CutMix? Is it more effective in video domain adaptation? or it is a general approach for video recognition tasks.\n\nGood performance is crucial for a paper, but I would like to see more insightful understanding in a NeurIPS paper. e.g. why sampling at different speeds is optimal for video domain adaptation, or it is just one of many working sampling strategies? is it dataset specific? why background mix is crucial for video domain adaptation? does background encode shortcut in the source domain?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see limitations in the main review\n\nMinor comments:\nFigure 1 is too small to see how background is mixed\n\nFigure 2 and 3 are not easy to follow.\n\nFigure 5: please add more explanation on \"become progressively discriminative\".\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "2C84NVUnvMr", "reviewer": "Reviewer_CMgF", "summary": "In this paper, the authors tackle the problem of video domain adaptation for the task of action recognition. To align source and target domain features, they propose to use contrastive learning instead of domain adversarial learning. For contrastive learning, two instances of different speeds from the same video constitute a positive pair. Two instances of different speed from the different video constitute a negative pair. In addition to the contrastive learning, they propose the background mixup in order to bridge the domain gap between source background and target background. They also incorporate target pseudo-labels to learn more discriminative target classifier. They validate the efficacy of the model on the publicly available benchmarks: UCF-HMDB, Jester and Epic-Kitchens.\n\nThe claimed contribution of the paper are as follows.\n 1) Introducing a new UDA framework, Contrast and Mix (CoMix): contrastive learning with speed, and background mixup for bridging the domain gap \n2) Integrate the pseudo labeling for UDA \n3) Extensive experiments on three datasets, and achieving state-of-the-art performance on UCF-HMDB and Jester.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper has the following strengths.\n\n+ The problem addressed, unsupervised domain adaptation for video recognition, is an interesting, but unsolved problem.\n+ The proposed method has a good motivation and sensible design choices. Speed is an almost invariant feature for recognizing actions in many cases (except a few examples such as walk vs. jog/run, slow dance vs. fast dance). Background shift between source videos and target videos is a known problem. Pseudo-labeling is shown to be helpful for the UDA in image domain [1].\n+ The paper is structured well. It is easy for readers to follow except for a few confusing sentences.\n\n[1] Unsupervised Domain Adaptation for Distance Metric Learning, Sohn et al., ICLR 2019.\n\n\nHowever, this paper has weaknesses as well.\n- The comparison with adversarial domain adaptation methods (ADDA, DANN, TA$^3$N, SAVA or even the source only baseline) is unfair in Table 1. CoMix is based on the video-level representations from GCN, while the compared methods are not using GCN representations. The authors claim the proposed method is more effective than domain adversarial methods based on this result. However, we do not know where the improvement comes from. Does it come from GCN or CoMix (contrastive learning with background mixup)? The ablation on the GCN shows CoMix w/o GCN gives 88.1% on the UCF-HMDB which is close to SAVA (86.7%, note that SAVA does not have pseudo-labeling) than CoMix with GCN. Therefore, it is not straightforward to draw conclusion that CoMix is better than adversarial DA methods. It would be better to see the results from the source only, DANN and ADDA baselines with GCN equipped. \n- There is no comparison with MM-SADA [51] on the Epic-Kitchens dataset. MM-SADA is a multi-modal method. Therefore, it is unfair to compare CoMix (43.2% on avg) with RGB+Flow version of MM-SADA (50.3% on avg). However, comparison with RGB only MM-SADA (43.9% on avg) should be there. RGB only MM-SADA (43.9%) shows slightly higher performance than CoMix (43.2%). This result might suggest adversarial learning is still valuable for the domain adaptation problem.\n- All ablation experiments are conducted on UCF-HMDB which is a quite small (only 3K videos with 12 classes), and performance-saturated dataset. It is hard to draw concrete conclusions from UCF-HMDB experiments. It is better to conduct ablation experiments on the Epic-Kitchens or Jester dataset which are larger and less performance-saturated.\n\nIf the majority of my concerns are resolved by the rebuttal, I will consider increasing my rating.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The initial reviews raised several concerns about novelty, missing experiments, and analyses that provide insights. The rebuttal addressed most of the concerns; some reviewers raised their ratings after the rebuttal. Overall, the paper shows how to combine existing ideas to tackle a challenging problem in a simple and effective manner. The reviews are generally positive, mainly due to the good empirical performance, but they are not overwhelmingly enthusiastic about the technical novelty. I carefully read all the reviews, rebuttal, and discussion, and also read the paper in detail. I tend to agree that the paper lacks novelty but must say the proposed approach is simple, effective, and performs well. I am recommending an acceptance, but wouldn't mind if it is rejected.
Decision: Accept (Poster)