Discovering Informative and Robust Positives for Video Domain Adaptation
Authors: Chang Liu
Kunpeng Li
Michael Stopa
Jun Amano
Yun Fu
Venue: ICLR 2023 poster (2023)
Keywords: Domain Adaptation
Video Recognition

Abstract: Unsupervised domain adaptation for video recognition is challenging where the domain shift includes both spatial variations and temporal dynamics. Previous works have focused on exploring contrastive learning for cross-domain alignment. However, limited variations in intra-domain positives, false cross-domain positives, and false negatives hinder contrastive learning from fulfilling intra-domain discrimination and cross-domain closeness. This paper presents a non-contrastive learning framework without relying on negative samples for unsupervised video domain adaptation. To address the limited variations in intra-domain positives, we set unlabeled target videos as anchors and explored to mine "informative intra-domain positives" in the form of spatial/temporal augmentations and target nearest neighbors (NNs).
To tackle the false cross-domain positives led by noisy pseudo-labels, we reversely set source videos as anchors and sample the synthesized target videos as "robust cross-domain positives" from an estimated target distribution, which are naturally more robust to the pseudo-label noise. Our approach is demonstrated to be superior to state-of-the-art methods through extensive experiments on several cross-domain action recognition benchmarks.


Published as a conference paper at ICLR 2023
DISCOVERING INFORMATIVE AND ROBUST POSITIVES FOR VIDEO DOMAIN ADAPTATION
Chang Liu', Kumpeng Li', Michael Stopa', Jun Amano & Yun Fu' Northeasternn University, Boston, MA, USA
Konica Minolta, San Mateo, CA, USA
1iu. chang 6enortheastern. edu, kinpeng. 11.gg....1111 .11.9999gmaaii com Imstopa, j amano Y @ kmbs. koni www.icaaa.nauuss..mmm us, yunfuêece neu, edu
ABSTRACT
Unsupervised domain adaptation for video recognition is challenging where the domain shift includes both spatial variations and temporal dynamics. Previous works have focused on exploring contrastive learning for cross-domain alignment. However, limited variations in intra-domain positives, false cross- rross-domain positives, and false negatives hinder contrastive learning from fulfilling intra-domain discrim- ination and cross-domain closeness. This paper presents a non-contrastive learning framework without relying on negative samples for unsupervised video domain adaptation. To address the limited variations in intra-domain positives, we set unlabeled target videos anchors and explored to mine "informative intra-domain positives" in the form of spatial/temporal augmentations and target nearest neigh- bors (NNs). To tackle the false cross-domail positives led by noisy pseudo-labed we reversely set source videos as anchors and sample the synthesized target videos "robust cross-domain positives" from an estimated target distribution, which are naturally more robust to the pseudo-label noise. Our approach is demonstrated to be superior to state-of- state-of-the-are methods through extensive experiments on several cross-domain action recognition benchmarks.
INTRODUCTION
Recent breakthroughs in deep neural networks have transformed numerous computer vision tasks,
including tasks such as image and video recognition (He al. 2016 Carreira & Zisserman 2017a
Mittal et al.] (2020_. Nevertheless, achieving such remarkable results typically necessitates time- consuming human annotations. To address this issue, semi-supervised learning (Miyato al.] (018)) and self-supervised learning (SSL) (He et al.) 2020) have been studied to utilize the knowledge available dataset with abundant labeled samples to improve the performance of models trained on datasets with scarce labeled data. However, the domain shift problem between the source and target datasets usually exists in roal-world scenarios, leading to performance degradation. Unsupervised domain adaption (UDA) has been exploited transfer knowledge across datasets with domaín diserepancies to mitigate this problem. Although many methods have been created specifically for images, there still a significant lack of exploration in the field of UDA for videos.
Recently, some studies have endeavored to perform UDA for video action recognition through the direct alignment of frame/clip-level features (Chen et al., 2019a Pan et al. [2000a] Choi et al. 2020). However, these methods usually extend the image-based UDA methods without considering long- term temporal information or action semantics. [Song al. (202I) and Kim al. (2021b) alleviate these issues with contrastive learning to learn such long-term apatial-tempora representations by instance discrimination. To further understand how contrastive learning helps UDA, we firstly recall that domain- wise discrimination and class wise closeness are the two main criteria to solve UDA problems (Shi Sha 2012 Tang et al. 2020). Considering an unlabeled video from the target domain anchor, we explain that Song et al. (2027) introduces intra-domain positives (in the form of cross-modde moos-mooaa correspondence, e.g. optical flow) to help UDA by learning discriminative representation in the target domain. Additionally, Kim et al. (2021b) utilizes cross(sources domain positives with the help of pseudo-labee labels and benefits the class-wise closeness by pushing the samples of the same class but different domains closer.

---

Published as conference paper at ICLR 2023
Intra-donne draa-dmmainnno Pas from Self caff-correspandence
100
Poe from Ground-ruth
Top 1 Classification Accuracy
Cross-domain Pos from Pseudo-labels
96
Cress domain Pas from Ground Iruth
80
Purity (%)
60
94
NN k
trg
trg NN k=10
20
NN k=20
SrC
Pseudo labels
90
0
2
10
20
30
40
w/ False Negatives w/ False Negatives
Epochs
(a) Study on positives and negatives selection
(b) Purity comparison
Figure 1: On HMDB UCC:: (a) Effeet of false negatives and intra/cross-dmmann positives. We study two contrastive methods in video DA: STCDA (Song al. (02T) with intra-domain positives and LCMC (Kim al. 2021b) with cross domain positives. We observe that limited variations intra-domain positives, pseudo- noise in tross-domal positives and false negatives are the three issues that largely hinder the performance. (b) Purity comparison among nearest neighbors (NNs) and pseudo label. We found the NNs in target domain clean and fit informative and robust intra-domain positives. Note that purity is the cleanness of pseudo-supervision compared to the ground truth.
By thoroughly studying the effect the positives and negatives selection from the existing contrastive- based methods in Figure (a), we empirically find that limited variations in intra-domain positives, pseudo-label noise in cross-domain positives and false negatives are the three issues that largely hinder the performance. Explicitly, when selecting the negatives based on the ground-trutte intra- domain positives from ceffccorrespondecc (Song et al. 2021 is 3.5% below the ones from ground truth. Further, selecting the dross-domain positives based on pseudo-labels induces 2% drop compared to the ground-trun truth. Importantly, when selecting the negatives from either different instance (Song al.] (02T) or pseudo-labels (Kim et al. 2021b), the performance of the four baselines in Figure (a) drops by 2 ~ 4%. Besides, the purity analysis in Figure (b) also indicates that the pseudo-labels of target video are noisy and thus not reliable for selecting cross-domain positives and negatives. Based on the observations, several straightforward questions might be raised:
Are there any unexplored intra-domain positives to enrich intra-domain variations?
2. How alleviate the pseudo-label noise issue when selecting cross- positives?
3. How address the adverse effect from false negatives?
To answer the first question, we propose to leverage the temporal and spatial augmentations of the unlabeled target video intra-domain positives. The rationale is that partially modifying the spatial/temporal information of videos could alter the samples without changing the whole action semantics. Incorporating those samples positives could help the model be invariant to spatial/temporal shift. Take step further, we also explore the anchor's 's nearest neighbors (NNs) in target feature space, which capture rich target-domain variations. By analyzing the purity of the NNs in both source and target domain in Figure (b), we observe that target-domain NNs are clean and thus fit as intra-done mnrra-domain positives.
To address the second question, we are motivated from (ie et al. 2018) that though the pseudo-labee labels assigned to target samples are noisy, the class-conditional centroids pt, of target features would weaken this noise by averaging all the target features of same pseudo class. To incorporate it into our contrastive learning framework, we re-formulate the optimization reversely by setting source videos anchors and then mining target vidcos cross-domain positives. We then estimate the gaussian target distribution (( >x) of target features based on pscudo-labels. Given a source video as anchor, synthesized target features can be drawn from this distribution that shares the same class as the source anchor. Consequently, we could leverage those synthesized features as cross-doman positives the estimated target distribution is robust pseudo-label noise.
To tackle the last question, we present an effective non-contrastive learning framework without relying negative samples for video domain adaptation. Specifically, motivated by the recent
2

---

Published as a conference paper at ICLR 2023
Feuture space Deep features Pulling
Spatial Aug.
2nd Neurest Neighhur
Estimated Target distriution Aill
Pancing
Climbing
Target nchor
Target A nchor
Source Anchor
Temporal Aug
Ird Nesrest Neighhor
(a) SpatialTempont Aug. as Positives
(b) Target NN' Positives
(c) Synthesized Target features as Positives
Figure 2: Overview of our intra/cross-domain positives. (a) spatial/temporal augmentations as intra-domain positives. (b) target nearest neighbors as robust intra-domain positives. Note that we consider self- soff-augmentation as the 1st NN. (c) synthesizing target features from an estimated target distribution as cross-domain positives. In short, intra-domain positives help target-domain discrimination while cross tross-domain positives support class closeness across domains.
advances in SSL Grill et al. 2020 Chen & He 202T) to discard the negatives and maximize the similarity between the anchor positives in feature space with the help of an extra MLP head and stop- noopggradeent operation. Beyond adopting this non-contrastive optimization into our setting, we highlight our compelling finding that the MLP head crucial for intra-domain positives optimization but largely humpers the convergence for cross-domain positives. Consequently, we remove the MLP head for the latter case. To summarize our main contributions:
By studying the effects of the positives and the negatives from previous contrastive- based video DA methods, make the first attempt to emphasize that limited variations in intra- domain positives, pscudo-label noise in inter-domain positives and false negatives are the three under-explored bottlenecks
We contribute a unified solution to address the above issues by introducing more informative and robust intra-domain and cross-domain positives without relying on negative samples for video DA problem.
We conduct comprehensive experiments and analysis on challenging cross-domain action recognition benchmarks (UCF-HMDB and Appic-Kicchens, and thoroughly verify our superi- ority over state-of-the-an methods.
RELATED WORK
2.1 ACTION RECOGNITION
Action recognition is challenging task with significant potential for practical applications. Many previous approaches are deployed in both 2D (Karpathy al. 2014a), and 3D CNN-based framework (such as I3D (Carreira & Zisserman 2017a). ResNet3D (Hara et al.) (017)), which achieves significant progress on RGB modality. To incorporate multiple modalities or temporal dynamics, the two-stream architecture (Simonyan & Zisserman 2014) is commonly employed technique in which RGB and optical-flor (Karpathy et al. 20014b) are processed independently using two CNNs, followed by late fusion. The SlowFast (Feichtenhofer al.) 2019) utilizes dual branches recognize actions through video with different frame rates. Although supervised benchmarks have shown encouraging results, such models heavily rely on large datasets that demand considerable human annotation effort. Conversely, our research is centered on UDA scenarios for action recognition, where the data in the target domain lacks labeling.
2.2 SELF-SUPERVISED LEARNING
Self-supervised representation has made significant strides in improving performance images and videos through the use various pretext tasks, such as rotations (Gidaris et al. 2018 8) and jigsaw puzzles (Noroozi & Favaro 2016), as well contrastive learning (Chen al. 2020 He et al. 2020 Han et al. 2020). Recently, non con-raatre learning methods (i.e., B BYO Grill et al. (020) and
SimSiam CChn & He 202T) dominate the SSL performance by learning meaningful representation
3

---

Published as a conference paper at ICLR 2023
by maximizing the similarity between two positive samples without considering negative pairs. While our method is motivated by the non-contrastive learning above, we highlight our difference in mining "informative" and "robust" intra-domain and cross cross-domain positives beyond instance-level variations for better target-domain discrimination and class-wise closeness. Importantly, we observe that the predictor in SimSiam (Chen & He| (02T), which is designed for preventing model collapse, would lead to convergence issue for cross-domain positives. In contrast, we remove the predictor for cross-domain positives in optimization while maintain for intra-domain positives. We will explain the details in the experiment section.
2.3 UNSUPERVISED DOMAIN ADAPTATION
There are several directions make UDA effective. Specifically, mainstream methods seek for
learning domain-invariant feature representations by minimizing distribution discrepancy (Tzeng
ct al.
2014:
Chung et al. 2022 Qiu et al.
2021 Liu et al.
202T). utilizing adversarial learning (Ganin
al.
2016
Long al. [2018] Saito et al. 2018 Pan et al. 2020b), cross dooss-domain augmentation (Wang
ct al.
2021:
Li ct al. 2021b) and contrastive alignment (Saito et al. 2020; Kim et al.] [202aa] Harary
et al. 2022
Li al. 2021a). Self-training (Xie et al.) 2018]] Zou et al. 2018 2019) also thrives
on UDA by training the model with labeled source data and pseudo-labeled target data. However,
less attention has been paid video domain adaptation until recently. Existing video UDA methods
can be summarized into two folds: temporal attention-based adversarial learning (Chen et al.
2019b; Luo et al.
Pan al. 2020a) and SSL-based cross-domain alignment(Munro & Damen 2020 Song et a 2021 Kim et al. (2221b). Explicitly, self suff-superissoons such as correspondence (Munro& Damen 2020) and contrastive learning (Song et al. [222]] Kim et al. [2021b] Sahoo et al. (021). have been exploited. Compared to the image-based DA methods, our method provides new perspective for domain discrimination and class-wise closeness via exploring positive samples in contrastive learning. Compared to the contrastive-base video DA methods, (1) we provide empirical insights on the limitations of existing contrastive methods by analyzing the effect of positives ans negatives; (2) we address the issues by proposing more informative and robust intra-domain positives and cross-domain positives without replaying on negative samples.
2020
2020; Choi et al.
predicting video clips order (Choi al.) 2020), mulli-mmddai
Discussion on Xie et al. (2018). It is worth noting some fundamental differences between our algorithm and Xie et al. (018). From the optimization view, they propose centroids alignment across domains where target centroids are updated with the current target batch based on pseudo- labels and thus will receive gradients. If the pseudo labels are noisy, wrong gradients will hurt the training. In comparison, we align the cross-domain samples within constrastive framework where our synthesized target features are detached from gradients; thus, only the source batch will receive the robust gradients. Algorithmically. besides the estimation of target centroids, we further integrate target covariance to estimate the whole gaussian distribution of target domain. Table / shows that we have a significant improvement over Xie et al. 2018
3 PROPOSED METHOD
The problem of UDA involves a labeled source domain D. = "e"), 1,, comprising N, source videos, and an unlabeled target domain TD; = (at) N. of N; target videos. Both domains share common label space L. The joint distributions of source and target domain are not identically and independently distributed, specifically defined as P(**, y²) + "(x'yy") The objective is to develop deep neural network for action recognition on both the labeled source videos D. and unlabeled target videos D,. The model should be able to generalize effectively on new target domain videos. To specify, the model is comprised of feature extractor F followed by a classifier G.
3.1 INTRA-DOMAIN POSITIVES
The objective of intra-domain positives is to achieve domain-level discrimination, where the data in the source and target domains segregated into separate semantic clusters.
4

---

Published as a conference paper at ICLR 2023
Intra-source Discrimination. As source domain videos have labels, simply minimize the supervised cross-entropy loss to achieve source-domain discrimination:
N
Lee
(I)
where Pr,y, = G(F()))) is the y/-th element of a K-dimensional prediction and K indicates the number of class.
Self-supervision as Intra-domain positives. Without proper regularization on unlabeled target data, target-domain discrimination cannot be realized for free. Motivated by the self-supervieed learning methods (Chen & He 2021; Bai et al. 202T). our first proposal is to train a model to learn a discriminative representation of the target domain by maximizing the similarity between different augmented views of the unlabeled target data. Specifically, we consider 1) spatial augmentation, including standard image-based augmentation such as random flip, crop and color distortion. 2) temporal augmentation, including video play speed (Benaim et al. 2020p, and shuffled clips (Fernando) ct al. 2017). Precisely, we randomly double the sampling rate (x2) for video play speed or randomly shuffle clip (1/3 consecutive frames) of a video.
The rationale behind is that the effectiveness of the intra-domain positives are highly coupled with how "informative" and "clean" " the positives given anchor video. "Informative" positives usually capture rich spatial variations (e.g., background/veepprn (e.g., video speed/action length). Our choices of spatial and temporal augmentations above could partially change the anchor video along these variations without changing the whole action semantics. Consequently, incorporating those samples as intra-domain positives could help the model be invariant to the variations in spatial and temporal axis.
changes) or temporal dynamics
Explicitly, our model takes two randomly augmented views of and x from a target video xf input: one view is strongly augmented by one of the augmentations above while the other view is weakly augmented. The feature extractor F operates on both views, converting them into feature space. two-layer prediction MLP head, represented by H, then takes the feature of one view, transforms it, and matches with the feature the other view. shown Figure (a). our intra-target domain alignment loss can be formulated as maximizing the cosine similarity between the two views:
Lintra xb2) ) = Dhh 25),
(2)
h1
D(M, z1) = | 2 2
(3)
where M4
H(F(x))) is the predictor output, =/
((r)) is the spatial-tempone feature, ||-||
12-norm and D)...) applied to o/ and we implement it by modifying Equ. 3 as D stopgrad(2))
)) is cosine distance function. Specifically, the operation of stop-gradient is
Explanation on optimization without negatives. Unlike traditional contrastive learning objec- tive (Song et al. 2021 Kim et al. 2021b), we only involve positive pairs (e.g, (af x+)) into the optimization without negative samples. The insight (Chen & He 2021) behind is that the negative samples in contrastive- learning play role in preventing models from collapsing solutions (e.g, constant outputs). Aaapuutt.Altetenteettt Alternatively, it is found that applying the predictor H-)) on one view of and the stop-gradient operation slopgrad(.) the other view o/j could also avoid model collapse.
As the existing contrastive video DA methods are largely constrained by the false negatives issue (Figure T). we leverage this optimization cut off the reliance on the negatives and thus avoid the егrоr accumulation led by false negatives.
Target nearest neighbors as Intra-domain positives. The empirical finding in Figure [bb shows that intra-domain NNs (target NNs) have higher purity than cross cross-domant domain NNs (source NNs) and pseudo-labels. It motivates us further explore the intra-domain NNs intra-domain positives which capture rich spatial and temporal variations in target domain.
Following the notation of Eqn. we add the detached feature 2 of the veakly-augmente view and its class pseudo label y² = arg max G(z)) to a memory bank, denoted as M. = {(2f. yf )1 [ } in an online fashion where the size equals to the number of target dataset N,. M, is updated with the
5

---

Published as a conference paper at ICLR 2023
newest features and pseudo labels given current batch. Then retrieve the nearest neighbors )-1) Finally, we extend the Equ. 3 to
of a5 in the memory bank to obtain target features {2, f1k incorporate target NNs as intra-domain positives:
k 1
Li-intra (x')
M 25), k
The overall loss is calculated by summing up the aforementioned loss function across all target videos ,D By pulling unlabeled target videos close their NNs, the model could achieve the better larget-domain discrimination. Importantly, the Eqn. i equivalent to the Eqn. 3 when k = 1. As k increases, the intra-domain positives could cover richer target-domain variations while under the risk of bringing false positives. The trade-off off between the "informativeness" and "robustness" of intra-domain positives is discussed in experiment section in Figure
3.2 ROBUST CROSS-DOMAIN POSITIVES
While the intra-domain positive focuses on domain-wise discrimination, ignores the cross-domain alignment in the shared action semantic space. A natural solution is to extend Eqn. 4 by incorporating source domain positives which cover cross-domain spatial-tempors variations. As target videos do not have labels, we resort to use the pseudo-labels for choosing source samples or directly using source NNs as positives. However, Figure (b) suggests, both source NNs and pscudo labels have low purity. Incorporating cross-domain positives based on such noisy supervisions would result in false domain alignment.
To tackle this issue, we formulate it reversely to utilize source videos as anchors and mine for more informative target domain positives. Previous work (Xie et al. 2018) motivates us that, though the pseudo-labels assigned to larget samples are noisy, the cluss-conditional centroids nº of target features featurrs would weaken this noise by averaging all the target features of same pseudo class. Consequently, the target centroid of each class 1t is more robust to pscudo-label noise. To further explore target domain variations, we also estimate the corresponding covariance matrix Σ from the target features of the same pseudo-class.
Instead of directly mining real target videos as positives based on noisy NNs or PL, we randomly sample synthesized target feature from a normal distribution N(r,,, Σd) of the same action class to the source anchor video in the form of
N (ul a2
(5)
where the coefficient a controls the target domain's variation. In the implementation, the target class centroids and the covariance matrix are computed based on the target features of the same pseudo-class from the memory bank Mr in Eqn. 4 As the estimation in the first few epochs is not so accurate, we
(t/T") X ao where and T are the current epoch and maximum epochs. As result, the negative impact of the estimated covariance in the early training stage will be reduced.
Similarly to Eqn. our model takes source video z as input (anchor) and processes it with the feature extractor Then we match it to the synthesized positives sampled from the class-conditional target distribution of class y" from Eqn. 5 in feature space. As shown in Figure 2 (c), when the number of sampled positives is M, our cross-domain alignment loss can be formulated as:
L. Lx-eross
Σ
where 2" F(x*)) is the feature of the source anchor and 5, is the target positive sampling from N(u, a2,;))) We further derive a close-form upper-bound for the expectation Eqn. under all possible target features L. in appendix B.I
Explanation on cross-domain optimization. Unlike Eqn. we do not transform the 2 by the predictor H which is originally designed for avoiding model collapse. Interestingly, find that discarding the predictor helps better convergence for cross- cross-domain positives in optimization. Our insight is that, as source feature z and the synthesized target feature 2! are from different distribution,

---

Published as a conference paper at ICLR 2023
pulling them close requires non-trivial optimization efforts and thus models are less prone to collapse. In contrast, matching different views of a target video zf from the same distribution is relatively "easy". Without regularization from the predictor H, models will easily match all the views (e.g, augmentations or intra-domain NNs) of the target video into a constant output.
3.3 OVERALL OPTIMIZATION
In summary, the overall optimization includes the intra-domain positives in section 3.1 and the cross-domain positives in section 3.2 as follows:
C L. + Lz-eross + Xit-intra-
(7)
where À a hyperparameter to control the strength of intra-domain alignment. We empirically set the À to 15 for the convergence of intro due to the predictor. Our framework can be easily extended to two-stream network by adding Eqn. for optical flow modality L Also, the optical flow correspondence of target videos can be integrated as intra-domain positives in Le-intra (Eqn T5). Our algorithm with pseudo code will be found in appendix [
EXPERIMENT
4.1 DATASETS
We conducted experiments assess our method two prevalent benchmark datasets for video domain adaptation, specifically UCF+HMDB (Chen et al.,
2019a) and Epic-Kitchens (Munro & Damen 2020). To ensure consistency with prior research, we utilized the training and testing
partitions provided by the respective authors in (Chen et al.
2019a Munro & Damen 2020
UCF HMDB is first assembled by Chen al. (Chen et al. 2019a for studying video domain adap- lation problem. This dataset is a subset of the UCF (Soomro el al. 20122) and HMDB datasets (Kuchne)
al. 2. comprising 3, 209 videos distributed among 12 classes that overlap for human activity recognition. We present our results on two different scenarios: UCF-> HMDB and UCF HMDB.
Epic-Kitchens a fine-grained egocentric action recognition dataset with videos from three different domains (D1, D2, and D3) representing PO8, PO1, and P22 kitchens, respectively, from the full Epic-Kitchens dataset (Damen al. 2018). The dataset is challenging and includes videos from the eight largest action classes. It is released by the authors in (Munro & Damen, 2020) for exploring video domain adaptation on the fined-grained setting.
4.2 COMPARISON WITH STATE-OF-THE-AN METHODS
Baselines. We conducted a comparative study with various baseline methods, including: (1) source only and supervised target only approaches, which utilize labeled source data and labeled target data, respectively, to train the neural network; (2) existing UDA methods that employ adversarial learning, such as DANN (Ganin et al. 2016) and ADDA (Tzeng et al., 2017). (3) existing UDA based on pseudo-labeling, including CRST (Zou et al. 2019) and MSTN (Xie et al. 2018). (4) existing video domain adaptation methods, including TA (Chen et al. 2019a), ABG (Luo et al. 2020), TCoN (Pan et al. 2020a). MM-SADA (Munro & Damen 2020), SAVA (Choi et al. 2020). STCDA (Song al. 2021 and LCMC
(Kim al. (022bb)
Results on UCF-HMDB. Table 2 illustrates the performance of our method and other competing methods the UCF-HMDB dataset. Our framework outperforms the previous of-te-of-the-ar methods on various backbones and achieves the highest performance. Compared to the recent methods incorporating self-supervision such clip-order prediction (SAVA (Choi et al. 2020)), )
uppearance-motion correspondence (MM-SADA (Munro & Damen 2020)) and contrastive learning (STCDA (Song al. (021) and LCMC (Kim et al.) 2021b)). our method makes a significant improvement using 13D backbone with 1.8 % on
and 3.5 % on HIMD->UCH task
respectively. Though not specifically designed for motion space, our method can extend two-stream backbone with additional gains to 86.1 1 % on UCF-HHMDB and 95.4 15.4% % on HMDB- BEDB--UCE UCC, achieving the state-of-the-art performance.
7

---

Published as a conference paper at ICLR 2023
Table 1: Results on Epic-Kitchens Datasets. All methods use two-stream backbone including both RGB and flow modality if not specity.
Method
Backhone
D2 -
D1
D3
-
D1
D1
-
D2
D3
D2
D1
+ D3
D2
D3
Mean
Source-oaly
12.5
46.5
AdaBN
eL al. 2018
ED
44.6
47.8
47.0
54.1
40.3
48.8
47.2
MMD
Lome
ED
43.1
48.3
46.6
55.2
39.2
48.5
46.8
MCD
Stalo 3 s0
2018
BD
42.1
47.9
46.5
52.7
43.5
51.0
47.3
MM-SADA Muitro Damen 2020 (GB)
BD
41.7
42.1
45.0
48.
39.7
46.1
43.9
Flow)
48.2
LCMC ni 20216
BD
49.5
51.3
50.3
56.3
46.3
$2.0
51.0
STCDA
(RGB)
44.4
41.1
47.7
45.5
41.2
47.6
44.6
STCDA (Song etal.
2021 (RGB + Flow)
IID
49.0
52.6
52.0
53.6
45.5
52.5
51.2
Ours (RGBS)
EDD
44.8
41.8
48.1
46.8
41.9
48.2
45.2
Ours (RGB- Flow)
49,9
53.5
52.7
$7.5
46.9
53.4
52.3
Supervised-taree
13D
62.8
62.8
71.7
71.7
74.0
74,0
69.5
Results on Epie-Kitchens. Ta- ble [ shows the results on Epic- Kitchens, which is another challeng- ing dataset comprising six transfer tasks with imbalanced data distribution across different tion classes. Our approach demon- strates superior performance on all six transfer tasks with an average accuracy of 52.3%. Compared to adversarial alignment (AdaBN (Li
Table 2: Results on UCF-HMDB dataset. Two-stream denotes the methods using I3D two-stream network for both RGB and flow stream respectively. Supervised Target denotes the baseline training with labeled target data only. stands the self-implemennttionn
total of
Method
Backbone
UCF - HMDB
HMDB - UCF
Source-nnly (Chen et al. 2019h
ResNet-101
71.6
73.9
DANN Cianin et al 2016
ResNet-101
75.3
76.4
JAN (Lang et
74.7
AdaRN da
201N
ResNei-101
75.5
77.4
MCD
Sailo
2018
RexNet-101
74.4
79.3
TAPN
Chen ctal
2019g
ReaNct-101
78.3
81.8
ABG
Luo al.
1020
ResNet-101
79.1
85.1
al. 2018) and MCD (Saito al.
TCoN
2020a
ResNet-101
87.2
89.1
2018) coeescorreeeoneecce learn-
Supervised Target
CChen
2019a
ReaNct-101
82.8
Source-only (Chri
2020
BDD
80.3
ing (MM-SADA (Munro & Damen 2020h) and contrastive learning meth- ods (STCDA (Song et al. al.| (20TT) and LCMC (Kim et al. [b)) similar observations can be found that our method outperforms all the compet- ing methods in the challenging Epic- Kitchens dataset by 1.1 %.
DANN
Ganih ct nl. 2010
[30
80.7
88.0
ADDA
Zcng ct
201
B3D
79.1
88.4
MSTN
Xie al,
2018
13D
80.2
89.9
CRST
Lang
80.9
'
2019a
81.4
SAVA
C hoi ct
2020
I3D
82.2
91.2
STCDA Song et al. 2
B3D
61.9
91.9
Ours
BD
84.0
94.7
Suparvised Target
Choi ot
2020
95.0
96.8
Source-nnly (ong
Two-stremm
MM-SADA Munro
84.2
STCDA Sung
2021
Two-streum
83.1
4.3 ABLATION
LCMC
Kim al
102
Two-stremm
84.7
92.8
STUDY AND ANALYSIS
Ours
Two-stream
86.1
95.4
Supervised Target Song al. 2021
Twa-streum
95.8
97.7
Effectiveness of Proposed Positives.
As presented from Table 3 intra-
domain positives bring a significant improvement UCF-*HMDB by 2.6% and HMDB UCF by 5.0%. Cross-domain positives also improve the performance by large margin. Surprisingly, each proposed component individually outperforms the previous state-of-the-are methods (e.g.. SAVA (Choi et al. (020)) and STCDA (Song et al., 202T)) on the I3D backbone. Incorporating these two components into the same framework brings further gains on both UCF-HHDBB by 7% and HMDB-+UCF by 5.9%. These results suggest that intra/cross-domat positives could benefit each other. Precisely, the better target domain discrimination brought by intra-domain positives helps the better estimation of target distribution in Eqn In return, cross- doos--domai positives help cross- cross-domain alignment and result more accurate target NNs for intra-domain positives Eqn. Last, the performance could be further boosted with our optimization without negatives.
Effeet of Optimization w/o Negatives. As shown in Table optimization without negatives boosts the performance of both intra-domain positive, cross-domain positives and jointly training by significant margin. Explicitly, when jointly training, it improves 1.5% on UCF-HHMB and 1.6 % on HMDB UCF compared to the baseline of conventional contrastive optimization with negatives.
Effect of Augmentation Strategies. When = 1, our Intra-Pos could generalize to self-supervised learning where the target anchor videos pushed to be close to their augmentations. Explicitly, we incorporated spatial augmentations such as random cropping and color distortion, as well as temporal augmentations like video playrate change and clips shuffling. Table shows that all types
8

---

Published as a conference paper at ICLR 2023
of augmentations benefit the performance, and integrating all of them gives the best performance. Interestingly, UCF-HMDB favors the temporal augmentations more than the spatial augmentations while opposite observation is found in HMDB UC...
Table 3: Ablation Study on UCF-HMDB for each Intra-domain Positives in Cross-domain Positives in Eqn 6
Table Ablation Study on augmentation strate- gies and cross-domain positive. The gray rows are the Intra-Pos with all augmentations and the Cross- Pos with covariance estimiation and w/o predictor.
component. Intra-Pos:
Equ[4 Cross-Pos:
w/o Negatives: Optimization negatives See 5.1
Method
UCF
HMDB
HMDB - UCF
Intra Crow-Pos
Neg
Intra-Pos (default, k=1)
81.8
92.3
x
80.3
$3.8
Spatial Aug.
x
81.8
92.6
w/ Speed Aug.
81.1
91.2
/
82.9
93.8
w/ Shuffle Aug.
81.3
91.4
x
80.9
90.9
Cross-Pos Pos
92.3
82.5
82.5
93.1
92.3
w/o E
82.2
91.8
84.0
w/ predictor
80.2
89.1
Effect of Class- conditional Covariance. Our cross-domain align- ment pulls source video the synthesized target video representations from the es- timated target distribution. We claim that the estimated class mean feature vector provides the shared action semantics from the target domain. Its corresponding covariance brings target spatial-temporal variations to the source samples. Ta- ble shows performance drop on Cross-domain without the covariance 2 on both tasks of UCF-HMDB, demonstrating the importance of integrating target variations.
Lambda (i)
alph
93.6
92.2
Accurzy (N)
93.4
$2.0
90.2
92.8
81.8
03.0
92.6
82.8
91.6
0.1
0.25
0.75
Number of neighbors (k)
Alpha (e)
Figure 3: Hyperparameters sensitivity the number of neighbors & (bottom x-axis, left y-axis), intra-target coefficient À (top x-axis, right y-axis) and class covariance coefficient og on HMDB
Effect of Predictor in Cross-domain postives. Unlike previous con-cotrrattive learning method CChen & He [202]), we find that applying the predictor results in poor performance in Tab. 4 We further tried increasing the optimization strength with larger loss ratio but still failed. We hypothesize that, due to the domuin shift, pulling the source features closer to the target requires non-trivial optimization effort and thus the model less prone to the "collapsing solutions". Therefore, adding predictor would make the optimization even harder.
Hyperparameters Sensitivity. Figure S presents the effect of choosing different neighbors k with A = 15. We observe that adding neighbors in Eqn generally boosts the performance and & 5 gives the best performance. For A, we set it À from (5, 10, 15, 20} with = 5. Empirically, we observe that relatively large À helps the convergence the predictor in Figure and À 15 achieves the best performance. For On, it can be implicitly considered as the radius target class distribution. It is shown that Σ becomes most effective when dg 0.25 in Figure 3 In this case, the sampled target variations from Σ are more likely to be clean. In summary, our method is robust to different hyperparameters.
CONCLUSION AND FUTURE WORK
In this work, we introduce the bottlenecks of in existing contrastive based video DA methods and propose a unified solution to address them without relying on negatives by mining informative and robust intra-domain positives and cross-domain positives. Our approach has been thoroughly evaluated on various benchmark datasets, and the experimental results demonstrate its superiority over the current sfate-of-hhe-a methods. Regarding the future work, multimodal learning would be an interesting direction to be integrated with our algorithm by mining cross-mit cross-modal nearest neighbors and estimating target distribution from other modalities.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
In our paper, we strictly follow the ICLR ethical research standards and laws. All datasets we employed are publicly available, and all related publications and source codes are cited appropriately,
REPRODUCIBILITY STATEMENT
We highlight several components within this submission that effectively support the reproducibility of this work. 1) Implementation details are provided in Appendix C backhone architecture, training schemes and hyperparameters are carefully stated. 2) A more detailed pseudo code for our algorithm be found in Alg.
where the training dataset,
ACKNOWLEDGEMENT
This work was supported by Konica Minolta research funding.
REFERENCES
Yutong Bai, Haoqi Fan, Ishan Misra, Ganesh Venkatesh, Yongyi Lu, Yuyin Zhou, Qihang Yu, Vikas Chandra, and Alan Yuille. Can temporal information help with contrastive self-supervised learning? 2021.
Sagie Benaim, Ariel Ephrat, Oran Lang, Inbar Mosseri, William T Freeman, Michael Rubinstein, Michal Irani, and Tali Dekel. Speednet: Learning the speediness videos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9922-9933 2020.
Joao Carreira and Andrew Zisserman. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. IEEE Conference on Computer Vision and Puttern Recognition, pp. (299-6308, 2017a.
Joao Carreira and Andrew Zisserman. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 6299-6308, 2017b.
Min-Hung Chen, Zsolt Kira, Ghassan AlRegib, Jaekwon Yoo, Ruxin Chen, and Jian Zheng. Temporal attentive alignment for large-scale video domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 6321-6330, 2019a.
Minghao Chen, Hongyang Xue, and Deng Cai. Domain adaptation for semantic segmentation with maximum squares loss. In IEEE International Conference Computer Vision, pp. 2090-2099, 2019b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, pp. 1597-1607, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Jinwoo Choi, Gaurav Sharma, Samuel Schulter, and Jia-Bin Huang. Shuffle and Attend: Video Domain Adaptation. In Proceedings of the European Conference Computer Vision, pp. 678-695, 2020.
Inseop Chung, Daesik Kim, and Nojun Kwak. Maximizing cosine similarity between spatial features for unsupervised domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1351-1360, 2022.
Dima Damen, Hazel Doughty. Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro. Toby Perrett, Will Price, al. Scaling Egocentric Vision: The Epic-Kitchens Dataset. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 720-736, 2018.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast Networks for Video Recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6202-6211, 2019.
10

---

Published as a conference paper at ICLR 2023
Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Sulf-super video representation learning with odd-one-out networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domainnaaveessaiil training of neural networks. The Journal of Machine Learning Research, 17(1):2096-2030) 2016.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach to self-supervised learning. In Advances in Neural Information Processing Systems, 2020.
Tengda Han, Weidi Xie, and Andrew Zisserman. Self-supervised co-training for video representation learning. In NeurlPS, 2020.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Learning Residual Networks for Action recognition. In IEEE International Conference Computer Vision Workshops, pp. 3154-3160, 2017.
Features with 3D
Sivan Harary, Eli Schwartz, Assaf Arbelle, Peter Staar, Shady Abu-Hussein, Elad Amrani, Roei Herzig, Amit Alfassy, Raja Giryes, Hilde Kuehne, et al. Unsupervised domain generalization by learning bridge across domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5280-5290, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 9729-99299999 97389, 2020.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014a.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-Scale Video Classification with Convolutional Neural Networks. In Proceedings of the IEEE conference on Computer Vision and Patter Recognition, pp. 7725-1732 2014b.
Donghyun Kim, Kuniaki Saito, Tae- Hyun Oh, Bryan A Plummer, Stan Sclaroff, and Kate Saenko. Cds: Cross-domain self-supervised traininnn In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 0123-91333 2021a.
Donghyun Kim, Yi-Hsuan Tsai, Bingbing Zhuang, Xiang Yu, Stan Sclaroff, Kate Saenko, and Manmohan Chandraker. Learning cross- modal contrastive features for video domain adaptation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 13618-13627, 2021b.
H. Kuehne, H. Jhuang. E. Garrote, T. Poggio, and T. Serre. Hmdb: A large video database for human motion recognition. In Proceedings of the International conference Computer Vision, 201
Kai Li, Chang Liu, Handong Zhao, Yulun Zhang, and Yun Fu. Ecacl: A holistic framework for semi-supervised domain adaptation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8578-8587, 2021a.
Shuang Li, Mixue Xie, Kaixiong Gong, Chi Harold Liu, Yulin Wang, and Wei Li. Transferable semantic augmentation for domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11516-11551 1525, 2021b.
11

---

Published as a conference paper at ICLR 2023
Yanghao Li, Naiyan Wang, Jianping Shi, Xiaodi Hou, and Jiaying Liu. Adaptive Batch Normalization for Practical Domain Adaptation. Pattern Recognition, 80:109-117, 2018.
Chang Liu, Lichen Wang, Kai Li, and Yun Fu. Domain generalization via feature variation decorrela- tion. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 1683-1691, 2021.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International Conference on Machine Learning, pp. 97-105. PMLR, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International Conference on Machine Learning, pp. 2208-2217. JML.R. org, 2017.
Mingsheng I Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pp. 1640-1650, 2018.
Yadan Luo, Zi Huang, Zijian Wang, Zheng Zhang, and Mahsa Baktashmotlagh. Adversarial Bipartite Graph Learning for Video Domain Adaptation. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 19-27, 2020.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, (Nov):2577-2665, 2008.
Gaurav Mittal, Chang Liu, Nikolaos Karianakis, Victor Fragoso. Mei Chen, and Yun Fu. Hyperstar: Task-aware hyperparameters for deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8736-8745, 2020.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and cun-ssupee learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, (18):1979-1992 2018.
Jonathan Munro and Dima Damen. Multi-Modal Domain Adaptation for Fine-Grained Action Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 122 132, 2020.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. ECCV, pp. 69 84, 2016.
Boxiao Pan, Zhangjie Cao, Ehsan Adeli, and Juan Carlos Niebles. Adversarial Recognition With Co-Attention. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 11815-11822 2020a.
Action
Fei Pan, Inkyu Shin, Francois Rameau, Seokju Lee, and In So Kweon. Unsupervised intra-domain adaptation for semantic segmentation through self-supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3764-3773, 2020b.
Zhen Qiu, Yifan Zhang, Hongbin Lin, Shuaicheng Niu, Yanxia Liu, Qing Du, and Mingkui Tan. Source-free domain adaptation via avatar prototype generation and adaptation. arXiv preprint arXiv:2106.1533 153226 2021.
Aadarsh Sahoo, Rutav Shah, Rameswar Panda, Kate Saenko, and Abir Das. Contrast and mix: Temporal contrastive video domain adaptation with background mixing. Advances in Neural Information Processing Systems, 34, 2021.
Kumiaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 3723-3732, 2018.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. Advances in neural information processing systems, 33:16282-16223 2020.
12

---

Published as a conference paper at ICLR 2023
Yuan Shi and Fei Sha. ceformation-theoreeiiie learning of discriminative clusters for unsupervised domain adaptation. In International Conference on Machine Learning, pp.
2012.
Karen Simonyan and Andrew Zisserman. Two-Stream Convolutional Networks for Action Recogni- tion in Videos. In Advances Neural Information Processing Systems, pp. 568-576, 2014.
Xiaolin Song, Sicheng Zhao, Jingyu Yang, Huanjing Yue, Pengfei Xu, Runbo Hu, and Hua Chai Spatio-temporal contrastive domain adaptation for action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp- 9787-9795, 2021.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset 101 human actions classes from videos in the wild. /2X1::222..442, 2012.
Hui Tang, Ke Chen, and Kui Jia. Unsupervised domain adaptation via structurally regularized deep clustering. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 8725-8735, 2020.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: 142.34444 2014.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 7167-7176, 2017.
Yulin Wang, Gao Huang, Shiji Song, Xuran Pan, Yitong Xia, and Cheng Wu. Regularizing deep networks with semantic data augmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3733-3777 -7748 2021.
Xiaofu Wu, Suofei Zhang, Quan Zhou, Zhen Yang, Chunming Zhao, and Longin Jan Latecki. Entropy minimization versus diversity maximization for domain adaptation. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Shaoan Xie, Zibin Zheng, Liang Chen, and Chuan Chen. Learning semantic representations for unsupervised domain adaptation. In Proceedings of the International conference machine learning, pp. 5423-5432. PMLR, 2018.
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Domain adaptation for semantic segmenta- tion via balsnebalanced self self-training. arXiv preprint arXiv: (010.01110079/11 2018
Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In IEEE International Conference Computer Vision, pp. 5982-5991, 2019.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
In this material, we show the additional optimization detail, comparison to imagge-based Domain adaptation methods and algorithm pseudo code to main submission.
B
OPTIMIZATION DETAIL
In summary, our framework can be readily implemented and optimized within an end-to-end deep learning framework using the overall objective function:
Call
ggb s-00
Crgb -
grgb -intraa
B.1 CLOSE-FORM UPPER-BOUSE FOR CROSS- OPTIMIZATION
The naive implementation not computational efficient when M is large. To tackle this issue, we consider the case that the number of M goes to infinity, and then derive close-form upper-bound for the expectation of Eqn. 6 under all possible target features as:
C g-roes
Egge Dlog P(=*,
(9)
- log Egg je 2""
(10)
log et "ho +=(2))
(11)
D(2", M())) 2 (2 (2
(12)
For simplicity, all the z", 2, 14. and Ey, above are 12 normaazle if not coupled with D(: The Eqn emerges from the Jensen inequality E Eog X) log]E[X] on the concave function log(.).
Based on the moment generation function E[eaX] = cart Ea where X ~ N(,,, E). Equ can be obtained from Eqn[[Ojnnde [nder Gaussian assumption. The overall cross-domain alignment loss above is averaged for all source videos N.. Notably, the computational cost of Eqn. 12 is different from backpropagating losses over M x N₂ source-target pairs. As the N(H,,, a. v. is estimated offline based on the memory bank, Eqn. 12 can be computed efficiently without explicitly sampling target features.
B.2 EXTENSION TO TWO-STREAM NETWORK
For fair comparison to the methods that utilizing optical flow such MM-SADA Munro & Damen (202)), STCDA Song et al. (202T) and LCMC Kim et al. (2021b), our method could be extended to two-stream network using both RGB and optical flow as inputs well. Following the setting above, we have:
fooe all
=
fflom L-scce
+ flow no
flow X intra
(13)
Besides Crgb and flow we also incorporate the mross-modality self-supervision in target domain as:
(2 Coross (xrob, _ [loww) = Dz
, ffoo D( flow
22
rgb
(14)
where 21th F² (1770), of flow F Fx (I1( are the features of RGB and flow images.
To sum up, the overall objective function for two-stream network is shown as follows:
Lau = Lrgh + Flow + XCrross A
(15)
C IMPLEMENTATION DETAILS
Following Choi et al. 22020; Song et al. 2021): (2017)) as the backbone feature encoder network, initialized with Kinetics pre-trained weights.
: Kim et al. (2021b), we use I3D Carreiaa & Zisserman
14

---

Published as a conference paper at ICLR 2023
Algorithm 1 Robust Cross-domain Positives for video DA
Input: Labeled source domain D. = ((rf, yf [ unlabeled target domain Dr = {(x)))
warm-start iteration To, maximum iteration 7 and batch size B; hyper-parameters: À and on-
Output: Parameters model 2 and predictor ② H---
1: for = 1 to To do
2: Update è with labeled source videos Lee.
3: end for
4: for = Ty to T do
5: (1/T) x oo
6: Sample [( vf) and {< from D. and D., respectively.
7: Obtain deep features {2; ) {z) and softmax outputs (pt {0}} for source and
target samples, respectively.
8: Generate target pseudo labels based on {
9: Update memory bank M, {(1 y (/')// with target features and pseudo labels in current
batch I.
10:
Randomly apply strong spatial or temporal augmentation on the target samples the second {
view
11:
Obtain their deep features and transform them as {h}} by the predictor θ
12:
Given the current target features {2/} mine the top-K nearest neighbors from M,, as
{ ' ) and compute the intra-domain positives loss Li-intra-
13:
for each class do
14:
Estimate target features means μe according to memory module Mr.
15:
Estimate the target intra-class covariance matrix Σ; according to memory module Ml. end for
16:
17:
Based (N ).), compute the cross-domain alignment loss C Ls-cross
Update @p, OH by minimizing the loss Lrgb in Eqn. 8 with stochastic gradient descent (SGD).
18:
19:
end for
The dimension of the features extracted from the 13D encoder is 1024. We randomly sample 16 consecutive frames out of a video clip with a size of 224 x 224. For inference, we use 16 uniformly sampled frames per video to recognize the action. We follow the standard 'pre-train then adapt' procedure used in prior works Tzeng et al. (0117); Choi et al. (2020) to train the model with only source data for epochs as a warm start before the proposed approach is employed. Then we train models in Eqn. 8 for 40 epochs in total. For the hyperparameters, we select À from (5. 10, 15, 20}, ag from 10.1, 0.25, 0.5, 0.75) and the number of neighbors & from {1, 2, 5, 10, 20). We conduct hyperparameter sensitivity experiments in sec. 4.3 We set A: = 5, À 15 and og 0.25 for all datasets. We train the models end-to-end using SGD with a momentum of 0.9 and a weight decay of 1e-7. We use initial learning rate of 0.01 for the I3D with cosine Icarning rate scheduler for our experiments. We use a batch size 32 equally split over the two domains. We report the average action recognition accuracy over 3 random trials. We use four 12G NVIDIA GPUs for training.
D COMPARISON TO IMAGE-BASED DOMAIN ADAPTATION BASELINE
We compare approach with the following imsge d DA basclines. (1) existing UDA based on advanced pseudo-labeling and entropy minimization methods which can considered as cross(inter)- domain alignment approaches, including CRST Zou al. (2019) and MSTN Xie et al.) (018) and
MEDM Wu et al. (2021). (2) existing DA method based on intra-domain discrimination IDA Pan
al. 2020b).
We shows results of our method with other competing approaches on UCF-HMDD dataset in Table 5 Our framework outperforms both inter-domain and intra-domain methods by significant margin.
15

---

Published as a conference paper at ICLR 2023
Table 5: Results on UCF-HMDB dataset for Image-based baselines. Supervised Target denotes baseline training with labeled target data only. stands for the self-implementation.
Method
Backbone UCF + HMDB
HMDB - UCF
Source-only Choi et al.
(020
I3D
80.3
88.8
MSTN
Ganin et (0016
IBD
80.2
89.9
CRST
Long et al.
(2017
I3D
80.9
90.2
MEDM
Wu et al.
22021
I3D
80.6
89.8
IDA Pan al. 2020b
I3D
81.6
91.4
Ours
I3D
84.0
94.7
Supervised Target Choi et al. 2020
I3D
95.0
96.8
Tigler
(a) Source only
(b) Intra-domain
(c) Cross-domain
(d) (all)
Figure 4: Feature Visualizations using t-SNE on BMMDB>UCH
80
Purity (%)
60
40
20
trg k=5
k=5
PL
0
0
2
10.
20
30
40
Epochs
Figure 5: On UCF-HMDB: Purity comparison among nearest neighbors (NNs) and pseudo label.
E ADDITIONAL ANALYSIS AND ABLATION STUDY
Purity comparison on ICF--MDDB We conduct purity comparison on UCF-HHMBB to fur- ther verify the observation in Figure [b). Given UCF-HMDB is more challenging task than HMDB-+UFF, we found similar observation that the target NNs are generally more robust and accurate than the source NNs and pseudo-labels.
Importantly, we first highlight our contribution in exploring the power intra-domain positives which are missing in previous contrastive-basee methods, such as (Song et al. (202I) and Kim et al. (2021b). Further, we consider the target (intra-domain NNs are generally more robust than PL and source NNs and suitable for intra-domain positives based on Figure [b) and Figure
Feature Visualizations. We visualize the feature embedding by t-SNE MMatten & Hinton 2008 among each component of proposed methods, including the source only model, intra-domain, cross- domain and ours. As seen from Figure 4 intra-domain shows better class discrimination while cross-domain aligns two domains more tightly. Ours incorporates the merits of both two methods.
16


Reviews:
[{"review_id": "YxNydntlai4", "reviewer": "Reviewer_4bRo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper points out the existing limitations of current contrastive-based VDA methods, including limited variations of positives within the same domain, inaccurate cross-domain positive pairs, and inevitable false negatives. To address the above limitations, the authors introduce target-domain nearest neighbors and synthetic features from the target domain based on class centroids to perform VUDA in a non-contrastive manner. The proposed method is well-motivated, and its experimental results demonstrate its performance.", "summary_of_the_review": "This paper presents a simple-yet-effective method for VDA, which leverage some machine-learning techniques to address the limitations of previous methods. The paper is presented clearly with sufficient details and analysis, while part of the motivation requires further justifications. Based on the quality of this paper, I would recommend a “marginally above the acceptance threshold”. ", "strength_and_weaknesses": "Strength:\nThis paper begins with an easy-to-follow analysis of what hinders existing contrastive-based methods, which motivates the following proposed methods. Overall, the proposed modifications are simple-yet-effective and well-motivated. The experimental results can further justify the effectiveness of the proposed methods.\n\nWeakness:\nSome details need to be further discussed. \na)\tAuthors claim that “intra-domain positives provide the least performance gains” indicates \"their limitation in variations\", while this causation is not convincing purely based on the comparison in Figure 1 (a). One could argue that such inferior performance is simply due to other reasons, e.g., the lack of minimizing domain discrepancy compared to cross-domain methods. It would be better for authors to provide more justification for this statement. \nb)\tWhile Figure 1 presents an intuitive analysis of the limitations of existing contrastive-based methods based on HMDB->UCF, I wonder whether these empirical observations also exist in other adaptation scenarios. More specifically, for Figure 1 (b), the higher purity of vanilla NN can be due to the fact that UCF is a relatively less challenging dataset (i.e., source only can already achieve 88.8). It is also known that multiple samples from UCF101 are clipped from the same raw video, which could also be the main reason why NN here is more accurate. Therefore, such observation on UCF101 may not work for other more challenging datasets. \nc)\tIn Table 2, authors present the performance of their method utilizing RGB and RGB+Flow for comparison, while only introducing RGB+Flow results from previous methods. To make a fair comparison, authors should also, from my perspective, include the RGB results from previous methods in Table 2. \nd)\tSome minor formatting issues can be observed (e.g., Table 2 appears earlier than Table 1, and a citation error in Table 1 where MSTN^* Ganin et al. (2016) should be MSTN^* Xie et al. (2018)).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KnsZpZQrddD", "reviewer": "Reviewer_9Kq9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles unsupervised domain adaptation for the video recognition task. The paper raises concerns with the use of contrastive methods for aligning source and target domain features. Those concerns involve intra-domain positives, false positives in a cross-domain matches that hinder the contrastive learning process and negatively affect the adaptation. The proposed solution is to utilize intra-domain positives and mine robust cross-domain positives that reduces likelihood of pseudo-label noise. The proposed solution is shown to be superior than the state-of-the-art methods, by experiments on several cross-domain action recognition benchmarks.", "summary_of_the_review": "Overall, the paper provides good contribution to the domain adaptation for the task of video action recognition. There are some concerns raised in the review, addressing them should provide more clarity on the final decision.", "strength_and_weaknesses": "**Strengths:**\n- The paper raises some interesting points on the way state-of-the art models are trained for domain adaptive action recognition in a contrastive learning. The paper suggests that impurity of the pseudo-label and cross-domain FP/FN hinder contrastive learning, that results in a sub-optimal domain alignment.\n- The proposed solution moves away from the contrastive learning and modifies it to include intra-domain positives and reduce reliance on negative samples. Furthermore, rather than following existing strategy of creating cross-domain positives, a new strategy is proposed that synthesizes positives for a source-domain anchor that likely reduces the pseudo-label noise.\n- There are more minor observations for improvements such as value of MLP in contrastive learning for intra-domain and cross-domain optimization.\n- Experimental analysis is exhaustive, includes state-of-the art methods for comparison and shows reasonable improvements with proposed strategy.\n- Ablation studies also provide interesting insights into the individual component of the method and effects of hyper-parameters used on the final performance.\n\n\n**Concerns:**\n- One of the aspect of the method includes the use of fitting a Gaussian distribution on the target domain features, which is more or less similar to the strategy used here [1]. I would like to know more about how proposed method is different/better than [1] both intuitively and empirically. This would provide more information on the overall contribution of the work.\n- The proposed solution revolves around reducing the FN/FP influence of the cross-domain alignment, but this intuition is never verified directly empirically. It would be beneficial to have FN/FP numbers before and after the use of proposed approach to further solidify the points raised in the introduction.\n- Similarly, there should be an empirical analysis to show how the proposed synthesis strategy is reducing pseudo-label noise. The paper claims that such synthesis will weakens the noise, but is never empirically verified.\n\n**Typos:**\n- \"robust intra-domain and cross-domain positives without *replying* on negative samples\nfor video DA problem.\" --> relying\n\n**References:**\n[1] Ding, Ning, et al. \"Source-Free Domain Adaptation via Distribution Estimation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2tItRCDyoO", "reviewer": "Reviewer_cfxm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work is the first to strengthen that limited variations in intra- domain positives, pseudo-label noise in inter-domain positives, and false negatives are the three under-explored key problems. Motivated by these concerns, this paper proposes a unified solution by introducing more informative and robust intra-domain and cross-domain positives without relying on negative samples for video DA. This model achieves state-of-the-art performance on several challenging cross-domain benchmarks for video DA. ", "summary_of_the_review": "This paper has proposed a novel contrastive-based unified video DA method without relying on negatives by mining informative and robust intra-domain positives and cross-domain positives. ", "strength_and_weaknesses": "Strength:\n- This paper has several interesting findings like \"MLP head is crucial for intra-domain positives optimization but largely hampers the convergence for cross-domain positives\". The proposed method is with adequate insights from these findings. \n\n- This work introduces the bottlenecks of existing contrastive-based video DA methods.\n\nWeakness:\n- It seems that this work can be applied to existing video da models. I suggest the author should do further ablation study to testify whether this unified model can be applied to other video da models like TA3N. \n- This work needs more ablation studies to support the proposed findings. Are these findings similar in different video DA benchmarks? \n- This work only conducts experiments on two small cross-domain datasets UCF-HMDB and Kitchens. These two benchmarks only have a few classes. Validating the method in large-scale cross-domain datasets with more classes will make your conclusions more convincing. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0d65dmJi_n", "reviewer": "Reviewer_srUr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper handles noisy pseudo-labeling in domain adaptation for action recognition. The paper shows that the previous contrastive learning-based methods can be noisy and improves the model performance with robust cross-domain positives. Experiments are conducted on two benchmarks.\n", "summary_of_the_review": "While the method outperforms existing SOTA methods, the novelty is somewhat limited since the proposed components are from previous works. And more extensive experiments are needed.\n\n-- POST REBUTTAL --\nThank the authors for the response. My concerns are partially addressed. I still believe that the idea of neighborhood clustering is not new to domain adaptation. Even though there is a difference from NC[2], the proposed clustering does not seem very novel. Other implementation details seem to be engineering optimization with limited novelty. However, this work is meaningful as it provides extensive experimental results and analysis, which can contribute to the research community. Therefore, I increase my score to 6", "strength_and_weaknesses": "\nStrength\n---\n- The paper analyzes the noisy pseudo labeling for domain adaptation in action recognition and proposes a way to solve this with Gaussian data augmentations and target nearest neighbors.\n- From a simple gaussian distribution, the method synthesizes the target features and uses these for cross-domain alignment.\n\nWeakness\n--- \n- The novelty is somewhat limited as the proposed components are borrowed from previous works (e.g., SimSiam). And the neighborhood clustering concept is also presented in [1], which is not referenced in this paper.\n- Extensive comparison is needed. For example,  the benefit of synthesizing features is not very clear. How are the proposed cross-domain components compared to the previous cross-domain alignment methods? In addition, Instead of using the cross-domain alignment in this paper, how does the classical pseudo-labeling performs? How does the pseudo labeling in Sahoo et al. 2021 work compared to the cross-domain component in this paper? It is not clear that the proposed components are better than previous alignment methods.\n-  As mentioned, the paper is missing some references for relevant works (e.g., feature generation [1], contrastive alignments for DA [2, 3, 4]).\n\n[1] Li, Shuang, et al. \"Transferable semantic augmentation for domain adaptation.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*\n. 2021.\n[2] Saito, Kuniaki, et al. \"Universal domain adaptation through self supervision.\" *Advances in neural information processing systems*\n 33 (2020): 16282-16292.\n[3] Kim, Donghyun, et al. \"Cds: Cross-domain self-supervised pre-training.\" *Proceedings of the IEEE/CVF International Conference on Computer Vision*\n. 2021.\n[4] Harary, Sivan, et al. \"Unsupervised Domain Generalization by Learning a Bridge Across Domains.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*\n. 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster