Leveraging Endo- and Exo-Temporal Regularization for Black-box Video Domain Adaptation
Authors: Yuecong Xu
Jianfei Yang
Haozhi Cao
Min Wu
Xiaoli Li
Lihua Xie
Zhenghua Chen
Venue: Accepted by TMLR ()

Abstract: To enable video models to be applied seamlessly across video tasks in different environments, various Video Unsupervised Domain Adaptation (VUDA) methods have been proposed to improve the robustness and transferability of video models. Despite improvements made in model robustness, these VUDA methods require access to both source data and source model parameters for adaptation, raising serious data privacy and model portability issues. To cope with the above concerns, this paper firstly formulates Black-box Video Domain Adaptation (BVDA) as a more realistic yet challenging scenario where the source video model is provided only as a black-box predictor. While a few methods for Black-box Domain Adaptation (BDA) are proposed in the image domain, these methods cannot apply to the video domain since video modality has more complicated temporal features that are harder to align. To address BVDA, we propose a novel Endo and eXo-TEmporal Regularized Network (EXTERN) by applying mask-to-mix strategies and video-tailored regularizations. They are the endo-temporal regularization and exo-temporal regularization, which are performed across both clip and temporal features, while distilling knowledge from the predictions obtained from the black-box predictor. Empirical results demonstrate the state-of-the-art performance of EXTERN across various cross-domain closed-set and partial-set action recognition benchmarks, which even surpasses most existing video domain adaptation methods with source data accessibility. Code will be available at https://xuyu0010.github.io/b2vda.html.

Published in Transactions on Machine Learning Research (02/2024)
Leveraging Endo- and Exo- Temporal Regularization for Black-box Video Domain Adaptation
Yuecong Xu*
auya0014$e. Be.ntu.ddu.s...
Deportment of Electrical and Computer Engineering National University of Singapore
Jianfei Yang"
yang04 78 (en.0078@e.ntiu. 0tt../..........eeeeee..
School Mechanical and Acrospace Engineering Nanyong Technological University, Singopore
Haozhi Cao
haozhi002@c.ntte www.:/eeeenn.......... edu.sg
School of Electrical and Electronic Engineering Nanyang Technological University, Singapore
Min Wu, Xiaoli Li
fwumin, ælli @22r. a atstar edu.
Institute for Infocomm Research
Agency for Science, Technology and Research (A *STAR), Singapore
Lihua Xie
elharie@ntun edd...d.......
School of Electrical and Electromic Engineering Nanyang Technological University, Singapore
IEEE Fellow
Zhenghua Chent
chen0832@c.nnu.ede wtw.:e................
Institute for Infocomm Research and Centre for Frontier Research Agency for Science, Technology and Research (A *STAR), Singapore
Reviewed on OpenReview: https: // o/pnreevee.. net/ forum? id- ccoP08mrQJ
Abstract
To enable video models to be applied scamlessly across vidco tasks in different environments, varions Video Unsupervised Domain Adaptation (VUDA) methods have been proposed to improve the robustness and transferability of video models. Despite improvements made in model robustness, these VUDA methods require access to both source data and source model parameters for adaptation, raising serious data privacy and model portability issues. To cope with the above concerns, this paper firstly formulates Black-box Video Domain Adaptation (BVDA) as more realistic yet challenging scenario where the source video model is pro- vided only as a black-box predictor. While few methods for Black-box Domain Adaptation (BDA) are proposed in the image domain, these methods cannot apply to the video domain since video modality has more complicated temporal features that are harder to align. To ad- dress BVDA, we propose novel Endo and eXo-TEmppral Regularized Network (EXTERN) by applying mask-to-mix strategies and video-tailored regularizations. They are the endo- temporal regularization and exo-temporal regularization, which are performed across both clip and temporal features, while distilling knowledge from the predictions obtained from the black-box predictor. Empirical results demonstrate the stale-of-the-all performance of EXTERN across various cross-domain closed-set and partial-set action recognition bench- marks, which even surpasses most existing video domain adaptation methods with source data accessibility. Code will be available at https://xuyu0010 github. io/b2vda. html
"Equal contribution. Cooresponndigg author.

---

Published in Transactions on Machine Learning Research (02/2024)
Temporal Features nn Sample Class
Viruel Temporal Features One Sample Class 8
Ciip Features
Classter
Figure 1: Clip features of target videos may be scattered, where clips from the same video may be separated by the decision boundary, resulting in differed label predictions. Such clip features violate both the cluster assumption and the masked temporal hypothesis. We augment the target vidco domain with virtual tempo- ral features through novel mask-to-mix strategy and apply endo-temporal regularization. The resulting temporal features are more discriminative and comply with both the cluster assumption and the masked temporal hypothesis.
1 Introduction
Video Unsupervised Domain Adaptation (VUDA) (Chen et al. 2019a Xu et al. 2021a 2022a) airms lo transfer knowledge from a labeled source video domain to an unlabeled target video domain. It has wide applications in scenarios where massive labeled vidco data may not be available. Despite their effectiveness in improving the robustness of video models (Xu et al.) 2022b). current VUDA methods require access the source video data which contains personal and private information, raising serious concerns about data privacy and model portability (Liang et al.) 2020a 2022). The Source-Free Video Domain Adaptation (Xul ct al. 2022c Tian et al. 2021) is subsequently formulated to learn a target model without access to source dala, yet it still relies on the well-trained source model parameters which allow generative models to recover
source videos (Goodfellow et al. 2014 Creswell & Bharath 2018
Luan et al. 2021 Wang et al. 2021
Privacy-preservingg is vital in applying action recognition models to real-world applications such as smart hospitals and security surveillance where action recognition models are leveraged for anomaly behavior recognition (Sultani et al. 2018: [Said et al.) 2021 Zhou et al., 2019 Zhong et al. 2022). In these cases, current VUDA methods are totally inapplicable when sharing models across organizations due to their violation of privacy-related regulations, such as the European GDPR (Goddard) 2017) and Singaporean PDPA (Chik, 2013). To further cope with the video privacy issue, we formulate and study a more realistic and challonging vidco domain adaptation sconario termed the Black-boz Video Domain Adaptation (BVDA) where the source video model provided for adaptation only as black-box predictor (e-g., API service). In privacy-concernee scenarios, BVDA helps to derive an accurate model in the target domain without access to both the paramcters and data in the source domain.
Without access to source data and model, existing VUDA methods that aim at enhancing transferability through statistical alignment (e.g.. TAMAN (Xu et al. 2023) or adversarial alignment (e.g., TAAN (Chen 2020 are not applicable. There have been a few recent research
et al. 2019a) and SAVA CChoi et al.
efforts (Lingg et al. 2021 Yang et al.
2022 aiming at Black-box Domain Adaptation for images. One rep-
where target features are extracted by obtaining pseudo-labels from the black-box predictors while applying structural regularizations (Viola & Wells III 1997 Verma et al. 2019) that encourage better model discriminability and generalization ability. However, structural regularizations of DINE are tailored for images that only contain spatial features. In comparison, charac- terized by the multi-modality nature, vidcos consist of spatial features and additional temporal information. This result. in additional challenges in aligning temporal feat ures. As a result, solutions for images such as DINE cannot show significant improvements for the BVDA task. Previous studies (Chen et al. 2019b Yang ct al. 2020a Kundu et al., 2022 Huang al.) 2022) prove that improving discriminability would benefit the effectiveness of domain adaptation. Inspired by these studies, we propose improve the discriminability of temporal features to tackle BVDA effectively when neither source data source model is accessible.
resentative work DINE Liang et al.
2022)
2

---

Published in Transactions on Machine Learning Research (02/2024)
One common strategy to extract video temporal features is to split longer videos into series of shorter clips. Therefore, temporal features can be constructed explicitly with the series of clip features (Wang et al. 2018 Zhou et al.) 2018). Meanwhile, humans are capable of recognizing actions correctly even with only representative clips from videos (Isik et al. 2018). Intuitively, if the target model is able to perform similarly to human perception and obtain discriminative features with consistent predictions given only partial clip information, the representational capacity of the target model and the discriminability of the extracted target temporal features could be improved significantly even without knowledge from the source domain. We term the above hypothesis the masked-tempont hypothesis as this hypothesis depicts the ideal characteristios of features obtained after certain clips are masked out. Our proposed method is built upon this hypothesis.
To this end, we propose the Endo and eXo-TEmporal Regularized Network (EXTERN) to address the BVDA task. EXTERN extracts robust temporal features a self-supervised manner by applying both the endo-temporal regularization and the exo-temporal regularization while distilling knowledge from the predictions obtained from the source predictor. Specifically, the endo-temporal regularization is designed to improve the discriminability of clip features by regularizing the behavior of clip features obtained within the same target video. This drives the clip features towards complying with the cluster assumption (Rigollet. 2007 Xiao et al., 2023) and the masked-temponey hypothesis as presented in Fig. | The endo-temporal regularization is achieved by augmenting the target video domain with virtual temporal features through a novel mask-to-mix strategy over clip features corresponding to the same video.
Meanwhile, the exo-temporal regularization designed to drive the proposed model to extract more stable and discriminable temporal features, characterized by being linearly smooth in-between features obtained from different target videos. The exo-temporal regularization is achieved by auginenting the target video domain with interpolated temporal features. It is remarkable that our EXTERN achieves outstanding results, outperforming most existing VUDA methods that require source data and models, This demonstrates that training the target model from seratch may help overcome the negative effect of domain shift,, paving a new way for tackling VUDA.
In summary, our contributions are threefold. (i) We formulated a realistic and more challenging task, Black- bor Video Domain Adaptation (BVDA). To the best of our knowledge, this is the first work to address black- box domain adaptation for privacy-preserving and portable video model transfer. (ii) We propose EXTERN to address BVDA, which enhances discriminative temporal feature extraction through an endo-temporal regularization using a mask-to-mix strategy along with an exo-temporal regularization, driving clip features towards complying with the masked- teakkell hypothesis. (fii) Extensive experiments show the efficacy of EXTERN, achieving state-of-the-art performances across cross-domain action recognition benchmarks under closed-set and partial-set video domain adaptation settings. EXTERN achieves remarkable 7.7% and 12.4% average improvements under the closed-set and the partial-set settings respectively, even outperforming most existing adaptation methods with access to source data.
2 Related Work
(Video) Unsupervised Domain Adaptation ((V)UDA). UDA and VUDA aim to distill shared knowl- edge across a labeled source domain and an unlabeled target domain, which improve the robustness and transferability of deep learning models (Zhao et al. 2020). (V)UDA methods could be broadly catego-
rized into four categories: i) reconstruction-baedd methods Ghifary et al. 2016 Yang et al.] 2020b Wei
et al. 2022), where domain-invariiat features are extracted by encoders trained with data-reconstruuttion
objectives; ii) discrepancy- based methods (Saito et al.) 2018 Xu et al.) 2019 YUAN et al.) 2022 Zhang
et al. 2023). where domain alignment is achieved by applying metric learning approaches, optimized with 2016), and MMD [Long Levi et al.
metric- based objectives such as MDD (Zhang et al. 2019). CORAL (Sun ct al.
el al. 2015); iii) adversarial-base methods (Tzeng et al.] 2017 Xu el al. 2022b
2022), ; where
methods leverage additional domain discriminators along with feature generators, trained jointly in an ad-
versarial manner (Huang et al., 2011 by minimizing adversarial losses (Ganin & Lempitsky 2015;; and iv)
semantic-basod methods (Choi et al. 2020 Kim et al.) 2021 Sahoo et al.) 2021 Song et al.
2021 Du et al.
2023), where domain-invariant features are obtained subject certain semantic ic constraints by leveraging
3

---

Published in Transactions on Machine Learning Research (02/2024)
Mask
Mix
Video Sample
Target Data
Frame Features
Clip Features
Endo-Temporal
Clip
Regularization
Gsp
Gr
Virtual Temporal Features
Virtual Temporal Predictions
Mask-to-M ix
HT
Mixed Unmasked
Clip Predictions
Target redictions
Temporal Features
Source Data
Mean Aggregation
Hr
PrrgetPrediction
Train
Source model
Exo-Temporal Regularization
Source API
Source API
Figure 2: An overview of the proposed EXTERN. EXTERN extracts knowledge from the black-box source predictor (i.c., Source APT) through a distillation process. EXTERN further extracts temporal fcatures in a self-supervised manner by applying both the endo-temporal regularization and eazottemporal regularization. To apply the endo-temporal regularization, the virtual temporal features constructed with mask-to-me strategy.
auxiliary tasks such as contrastive learning (Chen et al. 2020 and clip order prediction (Choi et al. 2020 Compared to UDA research which is primarily focused on image-based tasks, VUDA research lags behind owing to the challenges brought by aligning temporal features. Despite the challenges, there has been a substantial increase in research for VUDA, backed by the introduction of various cross-domain closed-set or partial-set video datasets (Chen et al. 2019a Xu et al. 2021a 2023). Regardless the improvements in video model robustness and transfcrability, VUDA approaches all require access both source data and source model parameters during adaptation, which could raise serious privacy concerns given the amount of private information of the subjects and scenes contained in videos.
Black-box Domain Adaptation (BDA). With the increased importance of data privacy with concerns of possible data leakage through white-box attack given model parameters (Zhang et al. 2020), there have been a few research that explores BDA with images. BDA enables image models to be adapted to the unlabeled target domain without either the source data or the source model paramoters, with the source model provided only as a black-box predictor (Fang et al. 2022 Yu et al. 2023). Under such settings, BBSE (Lipton et al. 2018) focused on black-box label shift, and requires hold-out source set for class confusion matrix estimation, which is commonly unavailable. More recently, LNL. (Zhang et al. 2021 proposed to tackle BDA by an iterative noisy lcarning approach via pscudo labels that are refined with KL divergence, while DINE (Liang et al. 2022) leveraged knowledge distillation with information maximization and structural regularizations. Despite the above recent advances, BVDA has never been explored, which is more challenging as temporal features must also be aligned. We propose to tackle BVDA by applying a temporal feature tailored endo-temporal regularization leveraging a mask-to-mix strategy, along with exo- temporal regularization.
3 Methodology
For Black-box Video Domain Adaptation (BVDA), we only have access to a black-bor video pre- dictor Hg (i.c., the constraint Source APT) which trained from the labeled source vidco domain Ds = ((Vis, Ye, containing ns vidcos, with Vis E Vs. Vis Vs, where Vs the source input dis- tribution and Ys the source label space. We also given unlabeled target video domain Dr (Vrr pr
4

---

Published in Transactions on Machine Learning Research (02/2024)
with ny i..dd videos Vir Vr where Vr is the target input distribution. Additionally, assumee that source and target video domains share the same label space with C classes, i.e., Y/s Y'r with Yr denoting the target label space. Meanwhile the source and target videos follow different data distributions, Therefore, there cxists a domain shift (Ben-David et al. 2006) between Ds and Dz. The objective BVDA is to learn a mapping model Vr + Vr to perform the action recognition task on the unlabeled Dr while both Dg and the parameters of Hs are not accessible.
Constrained by the absence of both the source data and the parameters of the source model, neither VUDA methods nor SFVDA methods could be directly applied for BVDA. To tackle BVDA, we resort to alternative strategy where we adapt target models to the embedded semantic information of the source data by resorting to the hard or soft predictions of the target domain from the black-box source predictor Dr = (Vr). Essentially, for BVDA, such a strategy aims to extract effective temporal features with high discriminability and comply with the cluster assumption, where the input distribution contains separated data clusters and that data samples in the same cluster share the same class label (Rigollct) 2007). We therefore propose EXTERN which drives temporal feature towards high discriminability in a sell-supervised manner relying both the endo- rrdo-temporal regularization and the ezo-temporal regularization. We first introduce the backbonc structure of the target model, followed by a thorough illustration over EXTERN and its key components.
3.1 Backbone Network
Videos differ from images greatly due to the existence of temporal features. A key prerequisite for the target network to be adapted to the embedded source semantic information is that its backbone could extract temporal features explicitly. An efficient and popular approach constructs the temporal features explicitly with a series of clip features, obtained through clips sampled from the corresponding videos (Wang el al. 2018 Zhou al. 2018). One notable example the Temporal Relation Network (TRN) (Zhou et al. [2018) TRN has been widely adopted in previous video domain adaptation tasks, such VUDA (Chen et al.
Xu et al. 2022b), PVDA (Xu et al. 2021a) and SFVDA (Xu et al., 2022cb bringing stale-of-the-art results. This is thanks to its ability in extracting accurate temporal features by reasoning over correlations
2019a;
between spatial representations which coincides with the process of humans recognizing actions.
with n,()) being the spatial feature of the j-th frame in the i-th source video. The subscript for the target domain T is omitted for clarily. The spatial features are extracted from the source spatial feature generator Gsp which is formulated as a 2D-CNN (e.g., ResNet-50 or ResNet-101 (He et al., 2016 Li et al., 2021 Subsequently, the temporal feature of V, is constructed by a combination of multiple clip features, obtained from the temporal feature generator G.. More specifically, G, builds each clip feature with temporal-ordered sampled frames where r E [2,k]. Formally, a clip feature for V,, denoted as ed is defined by:
Formally, we define an input target video with h frames as V, = ",""",,!""......
a!
= -E m ((V.")
(1)
Here (V:(*) )m (1,1), (0))) ,..m. m is defined as the m-th clip with r temporal-ordered frames, where a and are the frame indices. a and may not be consecutive as the clip could be extracted with nonconsecutive frames, but should be both constrained within the range of [1, k] with b > a. Eventually, the clip feature (!(r) is obtained by fusing all the time ordered frame-level spatial features through an integration function g()), which is usually formulated as a Multi-Layer Perceptron (MLP). The above computation would result in a total of (A- 16-11 1) clips. Finally, the temporal feature of V,, denoted as tG, is computed through the mean aggregation applied across all clip features, defined as:
k Σ, I,, (r)
t
(2)
3.2 Endo and Exo- Temporal Regularizations
Endo-temporal Regularization Current methods for BDA (Liang al. 2022) attempt to obtain fea- tures from the unlabeled target data with high discriminability via. self-supervised learning by applying
5

---

Published in Transactions on Machine Learning Research (02/2024)
structural regularizations. However, such regularizations are only tailored for spatial features since relevant studies are only conducted over image-based BDA tasks. Comparatively, videos contain temporal features that are constructed from a series of clip features. As depicted in Fig, [ clip features of discriminative temporal features may still be scattered across the decision boundary thus predicted with different label pre- dictions. This results in indistinct semantic information contained within the temporal features and causes inferior target domain performance. The key to tackling BVDA is to improve the discriminability of clip features by regularizing the behavior of clip features from the same target video. This is achieved by our proposed endo-temporal regularization performed over clip features.
More specifically, clip features that are discriminative should meet both the cluster assumption and the muusked-lemppore hypothesis. Notably, the masked-lemporal hypolhesis matches the human intuition that a combination of partial clip features from the same video would still result in consistent prediction as the overall temporal features that combines all clip features if both the temporal features and their clip features are of high discriminability. In other words, the clip features of high discriminability should be clustered toward the corresponding temporal feature.
To attain discriminable clip features, first augment the target domain with virtual temporal features obtained via a novel yet simple mask-to-mix strategy as depicted in Fig. 2 We first define the virtual temporal feature of Vi as i, It is constructed by mixing a set of clip features el!"), E [2, selected by random masking process. If the temporal feature &; is built upon set of (k - 1) clips, there will be exactly (k- 3) clips masked randomly, leaving two randomly unmasked clips: (((1)) and cg."²), where r1, E [2, k]. For each mini-batch, the selection of masked clips is random across all input vidcos within the mini-batch and across each epoch. This is to ensure that the virtual temporal features of V; are built upon all possible clip pairs from Vú across the whole training process. Different from the temporal feature which is constructed through a simple mean aggregation where all clip features are combined with equal weights, we compute the virtual temporal feature via the MixUp (Zhang et al.) 2018) operation. The virtual temporal feature is computed as a linear combination of the two unmasked clips assigned with random weights, defined as:
f (cl ,d, ca.12))
= -x,ot(")) + (1 Ao)c!!??)
where X E Beta(ap, a.) is the weight assigned to ci!"!) sampled from a Beta distribution with ag as the hy- perparameter. The virtual temporal feature constructed is essentially the linear intermediate representation between the unmasked clip features.
With the constructed virtual temporal feature, the the virtual temporal prediction to be consistent with the prediction of the corresponding target video (or equivalently, the corrosponding target temporal feature). To achieve such prediction consistency, we aim to minimize the divergence between the virtual temporal prediction and the largel prediction. The virtual temporal prediction is obtained from the target predictor Hr directly, i.e., ÿ: = Hr().). The target pre- diction is obtained by applying the target predictor to the target temporal feature, i.c,, y: = (t,).. The minimization of prediction divergence is formulated as:
hypothesis is satisfied by encouraging
Spre = (KL()))))
(4)
where (ll(- denotes the Kullback-Leibler divergence.
To further ensure that the temporal feature contains distinctive semantic information, the clip features should also comply with the cluster assumption, where clip features of the same target video share the same class label. Previous studies (Zhang et all 2018 suggest that the discriminability of extracted features can be improved and thus the cluster assumption is met if the fcature generator behaves lincarly in-between training samples. The cluster assumption of clip feat ures is therefore complied by employing the interpolation consistency training (ICT) technique (Verma et al. 2019). Specifically, such a technique encourages the virtual temporal prediction (ū:) to be consistent with the mixed unmaskod clip prediction. The mixed unmasked clip prediction is computed as the linear combination of the target predictions from both unmasked clips. The mixed unmasked clip prediction is computed as the linear combination of the target predictions
6

---

Published in Transactions on Machine Learning Research (02/2024)
from both unmasked clips, defined as:
Umizzi (M (trl =rcHr(cl, + (1- (o) (ct)
(5)
Subsequently, we aim to optimize the loss function:
Cusr = (Be )m
(6)
where lce denotes the cross-entropy loss.
Overall, the proposed endo- temporal regularization drives the target model to obtain discriminable temporal features by extracting clip features with higher discriminability that comply with the masked- teemped-e hy- pothesis and the cluster assumption. The endo-temporal regularization is applied by jointly optimizing Eq. 4 and Eq. 6
Lendo = Lovi + Cpre-
The implementation of the endo-temporal regularization can be observed to be very simple, yet it brings significant improvements towards tackling BVDA, would be prosented in Sec.
Exo-temporal Regularization To further enhance the discriminability of the temporal feature, we ex- tend the promotion of linear behavior in-between training samples towards the temporal features from different target videos. The temporal features are thus regularized by our proposed exo-temporal regular- ization. Given pair of videos V,, we employ the 1CT (Verma et al. 2019) across their corresponding temporal features t, ty. Such operation is equivalent to augmenting the target video domain with inter- polated temporal features which would drive the model towards better generalization. Formally, similar to how virtual temporal features are constructed, the interpolated temporal features t,, tú are obtained by applying MixUp (Zhang et al. 2018). With y Ht(t;) and y = Hr(t,) denoting the target prodictions of t and ty. the exo-temporal regularization aims to optimize the loss function:
Lewo = Lce (Mii (tg, t2)), MixUPx, (3), M;)),
(8)
where lor is the cross-entropy loss. MixUpA, is defined as shown in Eq. and and d Bela(as, an) is the weight assigned to t, with a the hyperparameter.
Our mask-to-mix strategy utilizes MixUp for the construction of virtual temporal features, which seeks to obtain consistent prediction as the lemporal features such that the corresponding clip features satisfy the maakedd-tmppore hypothesis, which different from existing domain adaptation works based on Mixup that regards it as a data augmentation approach (Xu et al. 2020 2019).
Yan et al., 2020 Wu et al. 2020 Panfilov et al.
3.3 Endo and eXo- TEmpora Regularized Network
With the endo-temporal regularization and ezo-temporal regularization terms defined, we propose the EX- TERN to address BVDA leveraging on both regularizations, depicted in Fig. 2 EXTERN builds upon the TRN backbone structure as specified in Sec. 3.1
Extracting Knowledge via Knowledge Distillation. To extract knowledge from the black-box predic- tor Hs, knowledge distillation (KD) (Hinton et al. 2015) has proven to be effective solution. The target model is seen as the student, and is trained to learn predictions analogous to that produced by the source model, which is seen as the teacher. However, due to domain shift between the source domain D3 and target domain Dr, the output from the source model could be noisy and inaccurate. To cope with such drawbar back, the adaptive label smoothing (AdaIS) technique (Liang et al. (022) with sclf-distillation (Laine & Aila 2017) and exponential moving average (EMA) (Grebenkov Serror (014) update recently proposed. Here, only the predictions of the top-c classes directly utilized while predictions of other classes are
7

---

Published in Transactions on Machine Learning Research (02/2024)
forced to uniform distribution
8 Σ VE
PETI T
σ = AdaLS.(0i)
(9)
=
otherwise;
where di E V is the prediction of the target video V; obtained from the black-box source predictor Hs (i.e.. the teacher prediction), while vf denotes the prediction of the p-th class and T, denotes the class index set of the top-c predictions for input V,. The teacher prediction further dynamically updated per training epoch maintain EMA prediction. We apply AdaLS with EMA to reduce noisy information by focusing only the top-predicted classes. Extracting source knowledge is eventually achieved by optimizing:
Ld = Ev.DD DKL (0.1)))
(10)
Learning Adaptive Clip Weights. Previous research (Nguyen al. 2020: Chen et al. 2019a shows
that features with lower prediction uncertainty would possess higher discriminability. Therefore, to better aggregate the clip fcatures for the temporal feature, we assign clip weight to each clip feature by attending to clip features with lower prediction uncertainly. Specifically, the clip weight is defined as the additive inverse of the target predictions of the corresponding clip, computed as:
w (( = (cH)
(11)
where the constant is added as residual connection for more stable optimization. Subsoquently, the temporal feature is obtained as the weighted aggregation of all clip features, with Eq. 2 modified as:
t. = --
E, w aago d
(12)
Information Maximization. Inspired by prior works in BDA (Liang et al. 2022 Yang et al. 2022) we maximize the mutual information (MI) to encourage diversity among target predictions and to promote their individual certainty:
mm h ((VVE, - h(s)),
(13)
where y4 = Hy (G = (t,) ) is the target prediction for input V; and h(y)) = - - of log yf is the conditional entropy function. Maximizing MI could marginally improve the performances for BVDA, as would be shown later in the ablation studies (Sec. 4.3).
The aforementioned techniques of leveraging AdaLS, adaptive clip weights and information maximization have been proven to be elfective for black-box domain adaptation for images (Liang el al. 2022) and source- free video unsupervised domain adaptation (Xu et al. 2022c They are leveraged to build a strong baseline for BVDA.
Table 1: Results for BVDA on UCF-HMDB,utt and Sports- DA for closod-set video domain adaptation.
3dethads
Publication
Data Modal
Privary
U101--Hi
UCF.HMDDB
Hii-attion
Avx,
X100-11101
Ka00 -S10
SIX--11111
Sports-DA
5ib--E000
U101-K000
U101-81X
TWR Diee a. (ars FODN--8
25.11
3%-97
77.34
90.95
71:16
88.95
12.50
02 73
(B:74
79 70
15.75
75312
K2.11
11.11
HD SEOT
TPAMIRU' TPAMICU")
50.20
71.33
N1.50
72.13
05.03
72.74
SP-SHOT
50:32
50.76
61.05
74.50
15.94
DINK [1nig
81.30
95.54
17.59
10.90
EXTERS
77.70
00.97
Cotuu Lrunteer 2016
JCML-15
25.63
85.75
79.85
MICADTT
JCML-16
77.00
73.58
74.05
SAVA Chuk
75.50
59.28
76.25
SHOT
T2AMI221))
77:44
56.77
15.07
plrsm
84.04
58.78
$8.93
ATOwN
20022
ECCV-12
83.21
$1.07
87.44
77.54
04.36
90.32
60.20
55.00 17
00
Overall Objective. Summarizing all the loss functions as presented in Eqs. optimization objective of EXTERN is expressed
10
13
the overall
L = Lsd + (Lendo + Lezo) - Lmis
(14)
where Preg is the hyperparameter that controls the strength of the regularizations and is empirically set 1. We refer to the settings of DINE. (Liang et al. 2022) by setting or as 0.3 and 3.
8

---

Published in Transactions on Machine Learning Research (02/2024)
Table 2: Results for BVDA on Daily-DA for closed-set video domain adaptation.
Primary
Avg.
33.55
11133
13.11
LNU
30.79
41:94
12.35
13.53
62.32
13.33
16.30
32.75
43.34
21.04
14.23
30.15
$1.30
2311
23.56
43.87
12.00
18.20
50.14
15:32
15.57
32.30
45:12
23:25
15.23
02.50
11.02
27:022
11:20
10.30
41.07
15.10
16.27
12.30
48.93
$2.07
23.25
DANS
2010
35:20
35.34
23.36
23.31
65:30
61.80
16.38
15.35
48:29
28.46
15.00
57.35
11.00
35
07.00
23.36
13.00
53:71
63.07
04.05
38.05
58.34
22.85
17.35
20.40
$1.00
20:03
25.22
32.181
3180
68.30
12:01
31.81
61.34
21.74
13.30
11:11
AGAN
27.08
42.31
31.30
21.17
47.07
13385
22.51
34.75
45.79
01.73
34.31
ATCGN N0.
23.33
$3.27
35.00
$3.30
61.90
05 36
36.30
13.31
27:44
17:00
43.45
38.54
4 Experiments
In this section, we evaluate our proposed EXTERN across a variety of cross-domain action recognition benchmarks, covering wide range of cross-domain scenarios. We demonstrate exceptional performances all benchmarks. Moreover, thorough ablation studies and analysis of EXTERN are performed to further justify the design of EXTERN.
4.1 Experimental Settings
Datasets. We evaluate EXTERN on three benchmarks: UCF-HMDBBaI (Chen et al. 2019a). Sports- DA [uu al. 2023) and Daily-DA (Xu al. 2023). UCF-HMDB ful is one of the most common bench- marks for VODA and is constructed from UCF101 (U101) (Soomro al. 2012) and HMDB51 (H51) (Kuehne et al. 2011), each corresponding to a separate domain. Sports-DA DA is a large-scale benchmark with three domains, built from UCF101, Sports-IM (SIM) (Karpathy et al. 2014). and Kinetics (K600). Daily-DA incorporates both normal and low-lllmmination videos with four domains, built from ARID (A11) (Xu et al. 2021b) (a low-illumination video dataset), HMDB51, Moments-in- Time (MIT) (Monfort et al. 2019) and
Kinetics (Kay et al. 2017). The distant domain shift due to immense illuminatiin difference renders it more challenging.
Implementation. We implement our method with the PyTorch (aazzke al. 2019 library. To obtain video features, we instantiate Temporal Relation Network (TRN) [Zouu al.) 2018) with ResNet-50 (He et al. 2016) the model backbone for both the black-box source model and the target model. The TRN is leveraged thanks to its capability in extracting explicit temporal features via reasoning over correlations between spatial representations which coincides with how humans recognize actions. The TRN has therefore been widely adopted in previous video unsupervised domain adaptation tasks, including closed-set video domain adaptation (Chen et al. 2019a Xu et al. 2022b), multi-sct video domain adaptation (Xu et al. 2023), and partial-set video domain adaplation (Xu el al. 2021a). delivering state-of-the-arl results in the respective tasks. For the source model, an additional fully connected layer inserted before the last fully connected layer which acts the classifier. For the target model, following DINE (Liang et al.) 2022). Batch Normalization [Ioffe & Szegedy 2015) and additional fully connected layer are inserted before the final classifier, which is constructed with weight normalization (Salimans & Kingma 2016) layer and a fully connected layer.
Similar strategies are applied for training the black-box source model and the target model, where the TRN backbones are both initialized from pre-trained weights obtained by pre-training on ImageNet (Deng) et al. 2009). All new layers are trained from scratch, with their learning rates set to be 10 times that of the pretrained-lodded layers. For the black-box source model, the training lasts for 100 epochs for tasks related to the Sports-DA and the MiniKinetics-UCH dataset, and for 50 epochs for all other datasets. For the target model, the training lasts for 20 epochs for tasks related to the UCF-HMDB, fall datasct and the UCF-HMDBport. dataset, 30 epochs for tasks related to the Daily-DA datasel and the HMDB-ARID, pertial dataset, and 50 epochs for the Sports- DA dataset and the MiniKinetics- UCF dataset. The stochastic gradient descent (SGD) algorithm (Bottou 2010) used for optimization, with the weight decay set to 0.0001 and the momentum set to 0.9. The batch size is set to 32 input videos per GPU. Hyper- parameters a 0.3 and Breg 1.0 empirically set and fixed.
9

---

Published in Transactions on Machine Learning Research (02/2024)
Baselines. We compare EXTERN with state-of-the-art BDA approaches as well as competitive UDA/VDA approaches. For fair comparisons, all compared methods are re-implemented with the exact same backbone as EXTERN. Specifically, for BDA approaches, LNL (Zhang et al. 2021] is a noisy label learning method
where pseudo labels are refined with KL divergence and leveraged for iterative network training. HD-SHOT and SD-SHOT obtain the model through self-training and apply SHOT (Liang al.
2021 by employing
a cross-entropy loss and weighted cross-entropy loss respectively. We also compare with
mothods including:
DINE [Liang al., 2022), DANN (Ganin & Lempitsky 2015), MK-MMD
Longg et al.,
2015
TAAN (Chen)
et al.
)019a). SAVA (Choi et al., 2020), ACAN (Xu et al,, 2022b), SHOT Liang et al.
2021
ATCoN (Xu
et al.
2022c), BA (Liang et al. 2020b), PADA (Cao et al. 2018 and PATAN (Xu et al.
2021a). We
report
the average accuracies over five runs with identical settings.
Table 3: Results for BVDA on UCF-HMDB video domain adaptation.
partial, parrtaa HMDB-ARID partial and MiniKinetics-UCF for partial-set
Publication
Privacy
ARTD parchet
Dita
Medll
1-14-1--
AVe.
H-10-AA-5
A-10--11-5
Avy.
1-16-18-88
Avg.
TRN
Zbou
2018
IS
39.08
82.35
V0.09
21.54
25.33
25.44
04.30
57.50
75,03
LNL
Aanung
08.85
222
25.57
77.00
HD-SHTT
Linng vl m
3121
TPAMI(2))
56.41
80.02
68.51
23.50
26.84
25.07
59.35
80.62
74.78
SD-SROT
T
02.
82.42
71.97
28.74
25.62
24.68
61.077
88.79
74,93
[Linng
CVPR-21
EXTELN
7
7
71.43
90.00
61.02
23.05
38.67
30.87
13.89
96.49
86. 19
TAIN
Clum it
0m
ROCV-19
30.98
73.70
02.35
20.05
27.08
24.02
63.21
02 14
77.60
Cinon Lempessk
77.63
22.
Loog
2015
ICML-15
59.10
82,25
70.70
22.51
25.79
24.05
69.26
69
78. 08.08
SAVA
Chuc ct n: X120
BOCV-20
54.74
89.41
199.08
25.27
27.58
25.66
66.49
90.31
78,40
PADA
ECCV
85,96
77.11
21.28
32.60
20.94
72.72
12.62
82.17
BA
Linn
88.41
S10.1
26.81
32.20
29.51
76.41
PATAN
05 21
A21
10VV-21
X
73.00
91.85
82.72
30.54
$5.51
32.98
77.91
06.50
86.00
4.2 Overall Results and Comparisons
We show the results on UCF-HMDBBut and Sports-DA in Tab. and results on Daily-DA Tab. 2 Our proposed EXTERN achieve state-of-the-art results across all the three cross-domain benchmarks. On average, EXTERN outperforms all BDA approaches designed for image-based DA tasks (i.e., LNL, HD/SD-SHOL and DINE), outperforming the best method by relative 7.0%, 6.0% and 10.2% respectively. This justifies the effectiveness of the designed regularizations tailored for temporal features whose discriminability relics on clip features complying with the mashed-temporal hypothesis and the cluster assumption. It could also be observed that the prior BDA approaches may fail to tackle BVDA well, with at least one task of Sports-DA and Daily-DA benchmarks showing inferior performance to that of the source-only model. Prior BDA approaches focused solely on spatial features, and may not obtain clip features that meet the masked- temporrll hypothesis and temporal feature with distinct semantic information. This result in negative impacts compared to the source-only baseline. Notably, EXTERN even pulperfomms various VUDA approaches (Chen al.) 2019a Choi et al., 2020 Xu et al. 2022b) with source data accessibility, Since EXTERN performs adaptation based solely on prediction results, it is found that EXTERN would not be affected by noise contained within the source data or the source model, generating superior adaplation results. This shows that training a target model from scratch with strong regularizations while adapting solely with source predictions can be as effective as data-based domain alignment techniques.
Closed-set Domain Adaptation
Partial-set Domain Adaptation. Apart from closod-set video domain adaptation, we further demon- strate the generalization ability of our proposed EXTERN by evaluating on partial-set video domain adap- tation (PVDA) tasks. To achieve this, we follow (Xu et al. 2021a) and leverage on three other benchmarks: UCF CCF-HMDB partiaa,, HMDB-ARID partial and MiniKinetics- UCF. Specifically, UCF-HMDB Barttaa partial is built from UCF101 and HMDB51 from 14 overlapping categorics and contains two PVDA tasks: U-14-H-7 and H-14->U-7 HMDB-ARID partial is built from HMDB51 and ARID, which is more challenging thanks to the distant domain shift. The dataset collected from 10 and contains two PVDA tasks: H-10~>- and A-10-;H-5. MiniKinetics-UCHH a largo-scale dataset built from MiniKinetics (Xie et al. 2017) and UCF101 containing 45 overlapping categories, also containing two PVDA tasks: M-45-+U-18 and U-45->M-14
10

---

Published in Transactions on Machine Learning Research (02/2024)
Table 4: Ablation studies of learning objectives and clip weights on UCF- (CF-HMDBBjul foll and UCF-HMDE partial
Components
UCP HM
UCF parei
Methods
Avg.
Cba
Ems
Caso
Cene/Lenda Lenda
Eyprl Lendo
Wed
U101-H51
H51 -1101
U-14-HH-
H-14-~U-7
85.64
/
/
$6.34
61.91
71
/
82.66
$9.19
61.52
86 17
80,78
/
/
83.89
$9.93
65.21
88.16
81,80
7
/
86.47
90.63
68.11
88, 23
83.36
/
87 :2
90.89
69.05
89.47
84.16
EXTERN
84.65
90.11
66.62
87.59 57
82.24
85.83
90.37
67.71
83.21
91.16
70.19
91.64
87.65
91.16
70.04
89.84
84.67
87.92
91.33
70.38
90.08
84,93
88.89
91.95
71.43
90.60
85.72
95
95
95
95
93.48
91.59
92.95 92.95 91.71
91.59 $1.95
91.77 91.99
88.06
88.09
88.53
90
88.06
00.71
80.09
88.53
85
85
80
0.2
0.4
0.6
0.8
1.0
1.2
0.1
0.2
0.3
0.5
0.7
0.9
0.2
0.4
0.6
0.8
1.0
1.2
0.1
0.2
0.3
0.5
0.7
0.9
Brog
Bree
a,
(a) S Sensitiity to U101-H51
(b) Sensitivity to on U101-H51
(c) Sensiiiiityt to Prog on H51-U101
(d) Sensitivity to on H51-U101
Figure 3: Sensitivity of hyperparameters Breg and on UCF-HMDB full-
(a) Clip features (Jett) and temporal loatures (right) (b) Clip features (left) and temporal features (right) (c) Clip features (left) and temporal features (right) with DINE
with source-on only model
with EXTR
2008) visualizations of clip features and temporal features extracted by (a) source-only model, (b) DINE [Liang el. al. 2022), and (c) EXTERN, with class information. Different colors denotes different classes.
Figure t-SNE (Van der Maaten Hinton
The results for partial-set cross-domain action recognition are presented in Tab. 3 Partial-set more challenging due to the asymmetric label spaces with the existence of "source-only" classes, causing negative transfer (Cao et al., 2018 Xu et al. (2211a). Negative transfer affects all previous BDA approaches, where all approaches will under-perform against the source-only baseline in al. least. benchmark. Despite such challenge, EXTERN still achieves outstanding results, outperforming the best BDA approach by relative 8.0%, 23.1% and 6.2% on the three benchmarks respectively. EXTERN also surpasses several PDA approaches even though EXTERN is not specifically catered for label shift.
4.3 Ablation Studies and Analysis
To gain a deeper understanding of the offectiveness of EXTERN while justifying design, we perform detailed ablation studies shown in Tab. and Fig. Specifically, the ablations studies explore EXTERN from two perspectives: the effects of individual learning objectives and the effects of assigning clip weights.
11

---

Published in Transactions on Machine Learning Research (02/2024)
Table 5: Ablation studies of regularization weights on UCF- SCF--MMDBJA fui and UCF-HMIDI pertill
pertial-
Weights
UCF-HMDBjau:
UCF-HMDB
Methods
Avg.
Grrey
U101 -551
H15 H11-U101
U-14->H--7
1-14-uU-7
1.0
1.0
88.89 89
91.95
71.43
90.60
85.72
1.0
0.0
80.08
86.34
61.91
82.71
77.76
1.0
0.1
82.75
88.40
64.57
84.34
80.04
83.79
EXTERN
1.0
1.5
88.53
92.12
72.48
89.42
85.64
1.0
2.0
89.
91.42
70.80
90.84
83,39
0.0
1.0
85.83
90.37
67.71
85.91
63.21
68.45
89.33
63.76
0.5
1.0
88.43
91.48
70.98
90.31
85,30
1.5
1.0
88.24
90.84
71.75
90.18
65.25
2.0
1.0
87.92
91.10
70.42
89.88
84.83
The ablation studies are conducted on UCF-HMDBBjar and UCF-HMDB partial with the same TRN backbone as previous experiments.
Endo and Exx-Tempont Regularizations. As demonstrated Tab. there is notable performance drop when either Lemo; Lendo or any of the componentts of Lendo (i.e., feer and Lyre) is removed from the learning objective, thus justifying that the designed learning objectives complement each other. Further, by applying the proposed endo and exo-temporal regularizations alone (optimizing Lendo: Loir, and ppee); EXTERN could outperform all prior BDA approaches and even some UDA/PDA approaches. This further proves the effectiveness of both the endo-temporal regularization and ezo-temporol regularization since these regularizations are tailored to temporal features.
Meanwhile, the weights of both the endo- and exo-temporal regularizations first set equally as Breg- A larger performance drop occurs when the endo-temporal regularization is not applied. Therefore, it is intuitive that the two regularization terms may have distinct contributions to the final performance. A detailed comparison over the contribution of the different regularizations is presented in Tab. 5 where the weights of the exo- and endo-temporal regularizations are denoted as Sre, and 3 respectively. The results suggests that the endo-temporal regularization would contribute more towards EXTERN's performance. This is thanks to its capability in pushing clip features towards the masked- temporral hypothesis. When Gtey 1.0 and Bryy < 1.0, small weight increase in the endo-temporal regularization would result in notable performance improvement of EXTERN. EXTERN achieves the overall best performance when both regularizations are balanced, further increasing 3reg would achieve better results in certain adaptation tasks.
Knowledge Distillation, Information Maximization, and Clip Weight. Tab, also shows that ap- plying knowledge distillation and information maximization brings further improvements towards EXTERN. However, the scale of which brought by these techniques is marginal compared applying the temporal feature-tailored regularizations. Meanwhile, results also justify the need to construct temporal features at- tentively with clip weight, bringing consistent performance gain, though the gain is also relatively marginal.
Hyperparameter Sensitivity. We focus studying the hyperparameter sensitivity of Breg which con- trols the strength of the regularizations and a, which relates to the construction of virtual temporal feature t. Here Breg is in the range of 0.2 to 1.2 and is in the range of 0.1 to 0.9. As shown in Fig. 3 the results of EXTERN falls within a marginal 0.83% which ranges from 88.06% to 88.89% for the U101-H551 task, and a marginal 0.62% which ranges from 91. 31.33% to 91.95% for the H51->U101 task. EXTERN obtains the best results for both tasks at a, = 0.3 and Preg = 1.0. The minimal variations show that the performance of EXTERN is robust to both hyperparameters. Meanwhile, despite the slight variations, EXTERN maintains the best results with all the hyperparameter settings.
Feature Visualization. We further understand the characteristics EXTERN by plotting the t-SNE embeddings VVan der Maaten & Hinton 2008) of both the clip features and temporal features extracted by the source-only model, DINE and EXTERN for H51 H511, as shown in Fig, 4 It is clearly observed that both the clip features and temporal features from EXTERN are more clustered and discriminable, justifying
12

---

Published in Transactions on Machine Learning Research (02/2024)
Table Detailed comparison of EXTERN with related but different VUDA and UDA methods.
Method
Publication
Task
Techniques
Domain Adaptation (BDA); source image data not avail- able, source image model provided as black-box predictor whose parame- ters are not available, target label not available, image-based.
Black-box
DINE extracts target image features by obtain- ing pscudo-labe from the black-box prodictors while applying structural regularizations.
DINE
Liang
CVPR-21
al.
2022
(a) ATCoN is designed such that the source and target models are identical in their structure, where the target model leverages the source clas- sifier directly; (b) ATCoN learns temporal con- sistency which includes feature consistency and source prediction consistency across local tempo- ral features; (c) ATCoN attends local temporal features based on prediction confidence obtained from source classifier.
Source-free Video Domain Adaptation (SFVDA): source source video data not avail- able, source vidoo model parameters available, target label not avail- able, video-based.
ATCoN
Xu
ECCV-22
al.
2022c
(a) TAON (Chen al. 2019a aligns source and target videos by applying adversarial-based
Video Unsupervised Domain Adapta- tion (VUDA): source video data and source video model are available, tar- get label not available, video-basod,
domain adaptation with domain discriminators across both spatial and local temporal fea-
TAON
Chen
ICCV-19
al.
2019a
tures; (b) TAON (Chen et al. 2019a attends to the local temporal features with high domsin discriminsbility.
(a) DM-ADA leverages domssn MixUp which augments the target domain with source domain data. (b) DM-ADA utilizes soft domain labels to improve the generalization ability of the fea- ture extractor and obtain domain discrimina- tor judging samples' difference relative to two mains with refined scores.
Unsupervised Domain Adaptation (UDA): source image data and source image model are available, target label not available, image-based.
DM-ADA
Xu
AAAI-20
al. 2020
(a) EXTERN extracts temporal features self-supervised manner by applying the endo- temporal regularization and the ezo-temporal reg- ulerization; (b) EXTERN distills knowledge from the prodictions obtained from the source predictor; (c) The
Black-box Video Domain Adaptation (BVDA): source video data not avail- able, source vidco model provided as black-box predictor whose parame- ters are not available, target label not available video-based
EXTERN
regularization drives clip features towards satisfying the clus- ter assumption (Rigollet. 2007] and the masked- temporal hypothesis by augmenting the tar- get vidco domain with virtual temporal fea- tures through a mask-to-mix strategy over clip features.
(Ours)
that the applied regularizations can promote higher discriminability and better compliance with the cluster assumption. We can also observe that the distribution of clip features is more similar to the distribution of temporal features with EXTERN. This intuitively proves that EXTERN drives clip fcatures towards satislying the basked-lemporde hypothesis where clip features are aligned towards the temporal features, and ensures that the temporal features contain distinct semantic information with high discriminability.
4.4 Detail Comparison with Related VUDA and UDA Methods
To highlight the novelty of EXTERN, we further compare our EXTERN with previous VUDA and UDA methods detail. Specifically, we compare with DINE [Ling et al., 2022), ATCoN (Xu et al. 2022c TA'N iChen et al. (2019a), and DM-ADA (Xu al., 2020) which an image-based UDA method that leverages MixUp (Zhang al. 2018). The methods are compared from two perspectives: the tasks they tackle and the techniques leveraged, shown in Table 0
13

---

Published in Transactions on Machine Learning Research (02/2024)
5 Conclusion and Future Work
In this work, we pioneer in formulating and exploring the realistic yet more challenging task of Black-bor Video Domain Adaptation (BVDA) for privacy-preserinng and portable video model transfer. We propose EXTERN for BVDA which obtains effective and discriminative temporal features by driving clip features to satisfy the mosked-tempony hypothesis and the cluster assumption. This is achieved by applying a novel endo-temporal regularization following a mask-to-mix strategy, along with an exo-temporal regularization. Results across multiple cross-domain action recognition benchmarks under both closed-set. and partial-set domain adaptation settings justify the efficacy of EXTERN. We believe that such a superior performance of EXTERN could pave new way for tackling video domain adaptation without compromising data privacy.
While the proposed EXTERN has proven to be effective for the BVDA task, there is st.ill room for future improvements. We observe that EXTERN may not perform well in tasks where the domain gap is relatively large, as depicted in Table 2 for tasks involving the ARID domain. This suggest that the current endo- and exo-temporal regularizations which are built on features constructed with the linear MixUp approach may not work under large domain shift scenarios. Virtual temporal features and interpolated temporal features constructed via higher order computation could be explored. Further, the temporal relationship between clip features could also be leveraged more explicitly as an additional modality for adaptation. Additionally, current BVDA tasks have only been explored for the closed-set and partial-set scenarios. More challenging scttings such as the open-set sctting where the target vidco domain may contain target-private classes may also be further explored.
References
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19:137-144, 2006.
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMP- 2011 pp. 177-186. Springer, 2010.
Zhangjie Cao, Lijia Ma, Mingsheng Long, and Jianmin Wang. Partial adversarial domain adaptation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 135 150, 2018.
Min-Hung Chen, Zsolt Kira, Ghassan AlRegib, Jackwon Yoo, Ruxin Chen, and Jian Zheng. Temporal attentive alignment for large-scale video domain adaptation. In Proceedings of the IEEE International Conference Computer Vision, pp. 6321-6330, 2019a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1507-1607. PML.R, 2020.
Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation. In International conference on machine learning, pp. 1081-1090. PMLR, 2019b.
Warren B Chik. The singapore personal data protection act and an assessment of future trends in data privacy reform. Compuler Lawe & Securily Review, 29(5):554-35 575, 2013.
Jinwoo Choi, Gaurav Sharma, Samuel Schulter, and Jia- Bin Huang. Shuffle and attend: Video domain adap- tation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 678-695. Springer, 2020.
Antonia Creswell and Anil Anthony Bharath. Denoising adversarial autoencoders. IEFE transactions neural networks and learning systems, 30(4):968 984, 2018
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largo-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pottern recognition, pp. 248-255, Icce, 2009.
14

---

Published in Transactions on Machine Learning Research (02/2024)
Yuntao Du, Hongtao Luo, Haiyang Yang, MingCai Chen, and Chongjun Wang. Bidirectional view based consistency regularization for semi-supervised domain adaptation. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreviee wtt.://eeee.oooom///d//mmcacc..
Yuqi Fang, Pew-Thian Yap, Weili Lin, Hongtu Zhu, and Mingxia Liu. Source-free unsupervised domain adaplation: A survey. arxio preprint (Xiv:2301.00263 2022.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Interna- tional conference on machine learning, pp. 1180-1189. PMLR, 2015.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li. Deep reconstruction- classification networks for unsupervised domain adaptation. In European conference on computer vision, pp. 597-697777 613. Springer, 2016.
Michelle Goddard. The eu general data protection regulation (gdpr): European regulation that has a global impact. International Journal of Market Research, 59(6))703 705, 2017.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Denis S Grebenkov and Jeremy Serror. Following a trend with an exponential moving average: Analytical results for a gaussian model. Physica A: Statistical Mechanics and its Applications, 394:288 303, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognilion, pp. 770 778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URL httt:://axxi...rrggg///// http:/xwwwwrrrra/a/////eeo abarggabs/15500 02531
Jingke Huang, Ni Xiao, and Lei Zhang. Balancing transferability and discriminability for unsupervised domain adaptation. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Ling Huang, Anthony D Joseph, Blaine Nelson, Benjamin IP Rubinstein, and J Doug Tygar. Adversarial machine learning. In Proceedings of the 4th ACM workshop on Security and artificial intelligence, pp. 43 58, 2011.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456. PMLR, 2015.
Leyla Isik, Andrea Tacchelti, and Tomaso Poggio. A fast, invariant representation for human action in the visual system. Journal of neurophysiology, 1 119(2):631-64) 2018.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large- scale video classification with convolutional ncural networks. In Proceedings of the IEEE conference on Compuler Vision and Pallern Recognilion, pp. 1725 1732, 2014.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset, 2017.
Donghyun Kim, Yi-Hsuan Tsai, Bingbing Zhuang, Xiang Yu, Stan Sclaroff, Kate Sacnko, and Manmohan Chandraker. Learning cross-modal contrastive features for video domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13618-13627, 2021.
Hildegard Kuchne, Hucihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In 2011 International Conference on Computer Vision, pp. 2556 2563. IEFF, 2011.
15

---

Published in Transactions on Machine Learning Research (02/2024)
Jogendra Nath Kundu, Akshay R Kulkarmi, Suvaansh Bhambri, Deepesh Mchta, Shreyas Anand Kulka- rni, Varun Jampani, and Venkatesh Babu Radhakrishnan. Balancing discriminability and transferability for source-free domain adaptation. In International Conference on Machine Learning, pp. 11710-11111 11728. PMLR, 2022.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In 5th International Conference on Learning Represenlations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL. httl.https://pppo
Matan Levi, Idan Atlias, and Aryeh Kontorovich. Domain invariant adversarial learning. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. U8uJAUMzJ9
URL https://openreviie htt.:/ppwwreiiieeeeeeeooooooaoee net net/forumfdd
Zewen Li, Fan Liu, Wenjie Yang, Shouheng Peng, and Jun Zhou. A survey of convolutional neural networks: analysis, applications, and prospects. IEEE transactions on neural networks and learning systems, 2021.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pp. 6028 6039. PMLR, 2020a.
Jian Liang, Yunbo Wang, Dapeng Hu, Ran He, and Jiashi Feng. A balanced and uncertainty-aware approach for partial domain adaptation. In European Conference on Computer Vision, pp. 123- 140. Springer, 2020b.
Jian Liang, Dapeng Hu, Yunbo Wang, Ran He, and Jiashi Feng. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Jian Liang, Dapeng Hu, Jiashi Feng, and Ran He. Dine: Domain adaptation from single and multiple black- box predictors. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black PMLR, 2018.
box predictors. In International conference machine learning, pp.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97 105. PMLR, 2015.
Xiao Luan, Jiezhong Zheng, and Weisheng Li. Learning unsupervised face normalization through frontal view reconstruction. IEEE Transactions on Circuits and Systems for Video Technology, (2(8):5201-5211 2021.
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. IEEE transactions on pattern analysis and machine intelligence, 42(2):502 508. 2019.
Dang- -hha Nguyen, Wei-Lun Tseng, and Hong- Shuai. Domain-adaptive object detection via uncertainty-aware distribution alignment. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 2499 200-2507,22 2020.
Egor Panfilov, Aleksei Tiulpin, Stefan Klein, Mika T Nieminen, and Simo Saarakkala. Improving robustness of deep learning based knee mri segmentation: Mixup and adversarial domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0 0, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep 2019.
learning library. In Advances in neural information processing systems, pp.
Philippe Rigollct. Generalization error bounds in semi-supervised classification under the cluster assumption. Journal of Machine Learning Research, 8(7), 2007.
16

---

Published in Transactions on Machine Learning Research (02/2024)
Aadarsh Sahoo, Rutav Shah, Rameswar Panda, Kate Saenko, and Abir Das. Contrast and mix: Temporal contrastive video domain adaptation with background mixing. Advances in Nearal Information Processing Systems, 34:23386 23400, 2021.
Abdel Mlak Said, Aymen Yahyaoui, and Takoua Abdellatif. Elficient anomaly detection for smart hospital iot systems. Sensors, 21(4)10026 2021.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3723-3732 2018.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016.
Xiaolin Song, Sicheng Zhao, Jingyu Yang, Huanjing Yuc, Pengfei Xu, Runbo Hu, and Hua Chai. Spatio- temporal contrastive domain adaptation for action recognition. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 9787-9795, 2021.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint
2012.
Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4479-6488, 2018.
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 30, 2016.
Jiayi Tian, Jing Zhang, Wen Li, and Dong Xu. Vdm-da: Virtual domain modeling for source data-free domain adaptation. IEFE Transactions on Circuits and Systems for Video Technology, (2()):374 2021.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167-7176, 2017.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. In International Joint Conference on Artificial Intelligence, pp. 3635-3641, 2019.
Paul Viola and William M Wells III. Alignment by maximization of mutual information. International journal of computer vision, (22):33771 1997.
Jianyi Wang, Mai Xu, Xin Deng, Liquan Shen, and Yuhang Song. Mw-gan+ for perceptual quality en- hancement on compressed video. IEEE Transactions on Circuits and Systems for Video Technology, 32 (7):4224-4237, 2021.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Lue Van Gool. Temporal segment networks for action recognition in videos. IEFE transactions pattern analysis and machine intelligence, 11(11):2740-2755 2018.
Pengfei Wei, Lingdong Kong, Xinghua Qu, Xiang Yin, Zhiqiang Xu, Jing Jiang, and Zejun Ma. Unsupervised video domain adaptation: A disentanglement perspective. arXie preprint arXiv:2208.07s 07365. 2022.
Yuan Wu, Diana Inkpen, and Ahmod El-Roby. Dual mixup regularized learning for adversarial domain adaptation. In European Conference on Computer Vision, pp. 540 555. Springer, 2020.
An Xiao, Hanting Chen, Tianyu Guo, QINGHUA ZHANG, and Yunhe Wang. Deep plug-and-play clustering with unknown number of clusters. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https:://pparrevvie... http:///ww.......oooo//o/ooooaaa
17

---

Published in Transactions on Machine Learning Research (02/2024)
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning for video understanding. arXiv preprint arXio:171 712.04888.. 1(2):5, 2017.
Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 6502 6509, 2020.
Xuemiao Xu, Hai He, Huaidong Zhang, Yangyang Xu, and Shengfeng He. Unsupervised domain adaptation via importance sampling, IEEE Transactions on Circuits and Systems for Video Technology, 30(12): 4688-4699, 2019.
Yuecong Xu, Jianfei Yang, Haozhi Cao, Zhenghua Chen, Qi Li, and Kezhi Mao. Partial video domain adap- tation with partial adversarial temporal attentive network. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp.
2021a.
Yuecong Xu, Jianfei Yang, Haozhi Cao, Kezhi Mao, Jianxiong Yin, and Simon See. Arid: A new dataset for recognizing action in the dark. In International Workshop Deep Learning for Human Activity Recognition, pp. 70 84. Springer, 2021b.
Yuecong Xu, Haozhi Cao, Zhenghua Chen, Xiaoli Li, Lihua Xie, and Jianfei Yan. Video unsupervised domain adaptation with deep learning: A comprehensive survey. arXiv preprint 2r1iv:2211. 104012, 2022a.
Yuecong Xu, Haozhi Cao, Kezhi Mao, Zhenghua Chen, Lihua Xie, and Jianfei Yang. Aligning correlation information for domain adaptation in action recognition. IEEE Transactions on Newral Networks and Learning Systems, 2022b.
Yuecong Xu, Jianfei Yang, Haozhi Cao, Keyu Wu, Min Wu, and Zhenghua Chen. Source-free video domain adaptation by learning temporal consistency for action recognition. In European Conference on Computer Vision, pp. 147-164. Springer, 2022c.
Yuecong Xu, Jianfei Yang, Haozhi Cao, Keyu Wu, Min Wu, Zhengguo Li, and Zhenghua Chen. Multi-source video domain adaptation with temporal attentive moment alignment network. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.000 006777, 2020.
Jianfci Yang, Han Zou, Yuxun Zhou, Zhaoyang Zeng, and Lihua Xic. Mind the discriminability: Asymmetric adversarial domain adaptation. European Conference on Compuler Vision, pp. 589 606. Springer, 2020a.
Jianfei Yang, Xiangyu Peng, Kai Wang, Zheng Zhu, Jiashi Feng, Lihua Xie, and Yang You. Divide adapt: Mitigating confirmation bias for domain adaptation of black-box predictors. arXiv preprint arXiv:220.114677, 2022.
Jinyu Yang, Weizhi An, Sheng Wang, Xinliang Zhu, Chaochao Yan, and Junzhou Huang. Label-driven reconotruction for domain adaptation in semantic segmentation. In European Conference on Computer Vision, pp. 480 498. Springer, 2020b.
Zhiqi Yu, Jingjing Li, Zhekai Du, Lei Zhu, and Heng Tao Shen. A comprehensive survey on source-free domain adaptation. arXiv preprint 2rXi::2302. 2023.
Zhiri YUAN, Xixu HU, Qi WU, Shumin MA, Cheuk Hang LEUNG, Xin Shen, and Yiyan HUANG. A unified domain adaptation framework with distinctive divergence analysis. Transcttoon on Machine Learning Research, 2022. ISSN 2835-8856. URL htttss// mett:/forrumm/dxyyryssggge
Dexuan Zhang, Thomas Westfechtel, and Tatsuya Harada. Unsupervised domain adaptation via mini- mized joint error. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https:
18

---

Published in Transactions on Machine Learning Research (02/2024)
Haojian Zhang, Yabin Zhang, Kui Jia, and Lei Zhang. Unsupervised domain adaptation of black-box source models. arxiv preprint arXiv:200 (rXim:211.02800 0010098999. 2021.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations, 2018. URL https::/openrreviev.. net/foruum'idrr1dppt-1
Yinghua Zhang, Yangqiu Song, Jian Liang, Kun Bai, and Qiang Yang. Two sides of the same coin: White- box and black-box attacks for transfer learning. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Dala Mining, pp. 2989 2997, 2020.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning, pp. 7404-7413 7413. PMLR, 2019.
Sicheng Zhao, Xiangyu Yue, Shanghang Zhang, Bo Li, Han Zhao, Bichen Wu, Ravi Krishna, Joseph E inngioovanniVincnnelllii
Gonzalez, Alberto L Sangiovanni- Vincentelli, Sanjit A Seshia, et al. A review of single-source deep unsu- pervised visual domain adaptation. IEEE Transactions on Neural Networks and Learning Systems, 33(2): 473 473-493,10 2020.
Yuanhong Zhong, Xia Chen, Yongting Hu, Panliang Tang, and Fan Ren. Bidirectional spatio-temporal feature learning with multiscale evaluation for video anomaly detection. IEEE Transactions on Circuits and Systems for Video Technology, 32(12):8285 8296, 2022.
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 803 818, 2018.
Joey Tianyi Zhou, Le Zhang, Zhiwen Fang, Jiawei Du, Xi Peng, and Yang Xiao. Attention-driven loss for anomaly detection in video surveillance. IEEE transactions on circuits and systems for video technology. 30(12):4639 4644, 2019.
19


Reviews:
[{"review_id": "GHsVhwxAdl", "reviewer": "Reviewer_Qj8M", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: One of the main contributions, the two proposed regularization terms are built with random masking and MixUp operations, making it easy to comprehend and reproduce. Meanwhile, the proposed method with these regularization terms obtains superior performance against VUDA and SFVDA approaches in most experiments. This also proves the correctness of the proposed masked-temporal hypothesis. This hypothesis will not only promote future work about BVDA but also be a substantial inspiration for VUDA and SFVDA tasks. The quality, clarity, and significance of this work are outstanding.\n\nWeaknesses: As an important concept in the paper, cluster assumption does not get any definition elaboration. I suggest you give a brief definition and briefly explain why the initial features violate the cluster assumption in Figure 1. Furthermore, you assign the same weight to the two regularization terms. I think you can conduct more experiments exploring the relative importance of the two regularization terms individually if possible, as they may have distinct contributions to the final performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes and addresses a novel Black-box Video Domain Adaptation (BVDA) task with a proposed EXTERN model. Based on the masked-temporal hypothesis and cluster assumption, the authors propose two forms of regularizations. The endo-temporal regularization term is based on the target temporal feature, virtual temporal feature generated with mask-to-mix strategy and MixUp predictions. The exo-temporal regularization term is calculated by ICT across corresponding temporal features. The whole network is based on the two regularization terms with knowledge distillation operations, clip feature weights and information maximization loss. Experiments comprehensively evaluate the effectiveness of the proposed EXTERN method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the Weaknesses above. \nFurther, some domain adaptation reviews and surveys are suggested, such as “Source-free unsupervised domain adaptation: A survey”, “A Comprehensive Survey on Source-free Domain Adaptation”, “A Review of Single-Source Deep Unsupervised Visual Domain Adaptation”. Please also add a caption for Figure 2.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oP3DH9OF8l", "reviewer": "Reviewer_SMeM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- This paper is one of the first studies to address black box video domain adaptation (BVDA).\n- This paper proposes a novel pipeline, EXTERN, to deal with BVDA.\n- This paper reveals that EXTERN exhibits high adaptation performance compared to the baseline methods on various cross-domain closed-set and partial-set action recognition benchmarks.\n- The proposed method is simple, and the paper is easy to read.\n\nWeaknesses\n- The paper mentions complex temporal features as a difficulty of Video Domain Adaptation (VDA), but only the temporal generator G_t explicitly deals with temporal information. After G_t, clip features are mainly used, but the temporal relationship between clip features is rarely considered.\n- Motivation to use MixUp could be stronger. At the very least, the motivation or necessity should be stated more clearly in the paper than it is now. For example, the masked temporal hypothesis states that mixup features are learned to approach temporal features. However, it is equally effective if the clip features are learned to approach the temporal features instead of the mixup features directly.\n- The core technology of the proposed method relies heavily on interpolation consistent training (ICT) and MixUp, so the technical novelty is not that high.\n- As a paper, it needs more elaboration. For example, the figure on page 4 has no caption. Symbols such as \\mathcal{V}, \\mathcal{Y}, and \\mathcal{H} are used without definitions. Also, in Equation 8, what does it mean to compute the cross entropy loss between temporal feature t and the prediction result?\n- Theoretical background on the proposed method is lacking.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper discusses black box video domain adaptation (BVDA) for privacy protection and portable video model transfer. Although several black-box region adaptation methods have been proposed for the image domain, these methods are not applicable to the video domain because the video modality has more complex temporal features and is more difficult to align. To address BVDA, this paper proposes a new Endo and eXo-TEmporal Regularized Network (EXTERN) by applying mask-to-mix strategies and video-adapted regularization. The proposed method drives clip features to satisfy the masked-temporal hypothesis and the cluster assumption to obtain practical and discriminative temporal features. Empirical results show that EXTERN exhibits high adaptive performance compared to the baseline methods on various cross-domain closed-set and partial-set action recognition benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper needs more elaboration, and the reviewer thinks it is necessary to clarify some of the concerns mentioned in the Weaknesses section.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VwgdOhqsxw", "reviewer": "Reviewer_pFxC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\na) The results are very good across several benchmarks and settings\nb) It appears that the method can be applied in any other similar settings while preserving some privacy\n\nCons \\\na) The first half of the paper that describes the methods is rather confusing; too much information that is hard to follow through - figure 2 does not have a caption and there are a couple of references to figures in the text that are not correct. \\\nb) Even after reading the sections a few times I struggle to find how endo- and exo-temporal regularisations are defined; yes some equations do exist etc. but the clarity is poor. Better structured information is required to comprehend the significance and contextualise the results with respect to the methods.\nCan you please elaborate on this? \"Overall, the proposed endo-temporal regularization drives the target model to obtain discriminable temporal features by extracting clip features with higher discriminability that complies with the masked-temporal hypothesis and the cluster assumption.\" How does it push it to obtain discriminable features?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper concerns black box domain adaptation for video where source data are not available and the model is provided only as a black box predictor (no parameters/weights available). Also the target labels are not available. \\\n\nWhat this paper proposes is to create two regularisation methods termed endotemporal and exotemporal regularisation. \\\n\nThe contributions are:\na) The two new regularisation terms that are used to extract temporal features in a self-supervised manner \\\nb) The method is using a distillation approach to extract knowledge from the predictions obtained from the source predictor alone \\\nc) results and experiments are extensive and show very good performance across several benchmark datasets and ablations", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "a) Clarify needs to be improved - the whole paper seems to be contextualised around the two regularisation terms so this needs to be fleshed out more clearly \\\nb) figure 2 needs fixing; maybe another figure can be added to show how the two regularisation terms work \\\nc) there are several typos and long sentences. \\\nd) Conclusion is short; please expand and provide a short discussion on importance, generalisation and future perspectives", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three expert reviewers raised some concerns about the paper that were resolved by the authors' response. Post-response, all reviewers were in favor of accepting the paper, indicating that the paper was novel and clearly demonstrated its claims. The AE is in agreement and recommends the paper for acceptance.
Decision: Accept (poster)