Diversifying Spatial-Temporal Perception for Video Domain Generalization
Authors: Kun-Yu Lin
Jia-Run Du
Yipeng Gao
Jiaming Zhou
Wei-Shi Zheng
Venue: NeurIPS 2023 poster (2023)
Keywords: video understanding and analysis
video domain generalization

Abstract: Video domain generalization aims to learn generalizable video classification models for unseen target domains by training in a source domain.
A critical challenge of video domain generalization is to defend against the heavy reliance on domain-specific cues extracted from the source domain when recognizing target videos. To this end, we propose to perceive diverse spatial-temporal cues in videos, aiming to discover potential domain-invariant cues in addition to domain-specific cues. We contribute a novel model named Spatial-Temporal Diversification Network (STDN), which improves the diversity from both space and time dimensions of video data. First, our STDN proposes to discover various types of spatial cues within individual frames by spatial grouping. Then, our STDN proposes to explicitly model spatial-temporal dependencies between video contents at multiple space-time scales by spatial-temporal relation modeling. Extensive experiments on three benchmarks of different types demonstrate the effectiveness and versatility of our approach.

Diversifying Spatial-Temporal Perception for Video Domain Generalization
Kun-Ye Yu Lin
Jia-Run Du
Yipeng Gao Wei-Shi Zheng
Jiaming Zhou'
School of Computer Science and Engineering, Sun Yat-sen University, China
2 Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China
[linky5, gww.ppayy........ ,000p23,,houj555 2houjnh50omaaaaa sysu. edu.
wszhengdieee.orz
Abstract
Video domain generalization aims to learn generalizable video classification models for unseen target domains by training video domain generalization is to defend against the heavy reliance on domain- specific cues extracted from the source domain when recognizing target videos. To this end, we propose to perceive diverse -rappal-teppora cues in vidcos, aiming to discover potential contribute a novel model named cues addition which improves the diversity from both space and time dimensions of video data. First, our STDN proposes to discover various types of spatial cues within individual frames by spatial grouping. Then, our STDN proposes to explicitly model spatial- temporal dependencies between video contents at multiple space-time scales by spatial-temporal relation modeling. Extensive experiments on three benchmarks different types demonstrate the effectiveness and versatility of our approach.
source domain. A critical challenge of
comain--pecifie cues. We Diversification Network (STDN),
Introduction
Recently, advanced deep network architectures have achieved competitive results for video classifica- tion 2 4
6 7 leading to wide applications in surveillance systems, sport analysis, health However, existing video classification models rely on the i.i.d. assumption, i.e., training and test videos independently and identically distributed. This assumption would be casily violated, since models often face unfamiliar scenarios in real-world applications. For example, housework robot will work in house, and surveillance system will encounter illumination change caused by camera viewpoint or weather [12 13 14]. Holding such an assumption, the performance of video classification models would drop significantly in unfamiliar test scenarios.
monitoring, etc [9
10
To alleviate the above problem, our work studies the video domain generalization task, which aims to learn video classification model that is generalizable in unseen target domains by training in a source domain [5] [6]. this task, videos from the source and target domains follow different distributions though with identical label space. For example, shown in Figure I humans in the source domain play basketball shooting on indoor basketball courts while those in the target domain play outdoors. Different from the video domain adaptation task with available unlabeled target videos for training [17 [8] 19 20], video domain generalization can only access the source domain during training, which is much more challenging but more practical.
'Equal contributions Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Source Domain (Training)
Target Domain (Test)
Fending
Shoot Ball
Biking
Kick Ball
Fanding
Shoot Ball
Biliny
Kick
Generalize
Figure 1: Video classification models suffer from the misguidance of domain specifi cues when generalizing to unseen domains. As shown in the figure, in the source domain, the static backboard provides clearer cue compared with the blurred basketball in motion, thus prevailing video classifica- tion models are prone to recognize the class "Shoot Ball" by the backboard. However, the backboard is invisible in the target domain due to viewpoint change, and thus previous models learned in the source domain would make mistakes in recognition. Videos the figure are from the UCF-HMDB benchmark, Best viewed in color.
A critical challenge of video domain generalization is to defend against the reliance domain- specific cues in the source domain that are correlated with class labels. For example, as shown in Figure video classification models prefer to leverage the backboard for recognizing the class "shoot ball" the source domain, since the static backboard provides a clearer cue compared with the blurred basketball in motion (static patterns are usually easy-to-fit [21 22 23 24)). However, in the target domain, the backboard is occluded due to the viewpoint, thus recognizing the class by the backboard would cause recognition errors. It is challenging of the target domain. Typically, existing works explore invariance across domains for learning generalizable video features [5] [6]. For example, Yao et al. propose to learn generalizable temporal features by encoding information of local features into global features, assuming that local temporal features are more invariant across domains compared with global ones [15].
address this problem in lack of any knowledge
In this work, we propose a novel approach for video domain generalization, which explore spatial- temporal diversity in videos. Our approach aims perceive diverse correlaedd cues from abundant video contents, and thus we would leverage not only easy-to-fit domain-specific cues but also other potential domain-invariant that our model can capture not only static backboards but also dynamic basketballs in the source domain). As a result, our approach can alleviate the overfitting domain-specific cues in the source domain and generalize better in target domains by leveraging those potential -nvariatn cues. Specifically, we propose explore the diversity from both space and time dimensions of video data, leading to a novel architecture named Spatial-Temports Diversification Network (STDN). Our contributions are summarized as follows:
for recognizing videos in target domains (e.g., expect
We propose Spatial Grouping Module to discover various groups of spatial cues within individual frames by embedding a clustering- process, enriching the diversity from a spatial modeling perspective.
We propose Temppoall Relation Module to explicitly model dencies between video contents at multiple space-time scales, enriching the diversity from a spatial-temporal relation modeling perspective.
depen-
Extensive experiments are conducted on three benchmarks of different types, including two newly designed benchmarks, and the results demonstrate the effectiveness and versatility of our proposed method.
2
Related Works
Video Classification aims to recognize actions events in videos. deep learning architectures have been proposed for video classification. 3D CNNs extend the
Recently, many advanced
convolution to 3D convolution for video feature learning [ 2
3
26
27
28
29
30].
Another

---

type of models first applies 2D convolution for frame-level spatial modeling and then conducts temporal modeling based on frame features [
6
31
32
33 Some works propose to couple explicit
shifts along the time dimension for efficient temporal modeling
34 35 More recently, pioneer
works have made efforts in adapting Vision Transformer (36] for video classification [37 38 [39
40 42 usually assume an identical test distribution to the training one, which is not practical in real-world applications.
43
44 45). Although these advanced architectures achieve appealing performance, they
Video Domain Generalization aims to train video classification models in a source domain for generalizing to unseen target domains. With target videos inaccessible during training, existing works usually assume different types of invariance across domains to defend against the reliance on domain- domain-specific cues (25] IS [6]. For example, Yao et al. propose to learn generalizable temporal features according to an assumption from empirical findings, i.e., local temporal features are more invariant across domains compared with the global ones [5]; Planamente et al. propose to constrain consistency across visual and audio modalities by relative norm alignment for addressing domain generalization of egocentric action recognition [16]. In this work, we propose to perceive diverse class-correlated spatial-temporal cues in source videos, which alleviates the misguidance of domain-specific cues in way that is orthogonal to previous works.
Video Domain Adaptation aims to learn transferable video classification models for a label-free target domain by transferring knowledge from label-sufficient source domain [17 [8]. Different from video domain generalization, video domain adaptation is oriented a specific seen unlabeled target domain. Typically, existing works learn invariance between labeled source videos and unlabeled target videos to tackle video domain adaptation. A class of representative works propose to learn Nomain-invariant temporal features by designing temporal modeling modules [18 19. 46 47). In addition, Choi et al. (20 48) propose self-supervised methods adaptive to video data. Furthermore, multi-modal works explore information interaction between different modalities (e.g. RGB, Flow,
Audio) for domain- -amarntinvariiatt feature learning 44 50 51 52
General Domain Generalization, also known as out-of of-distribution generalization, studies learning models generalizable to -fdistiibbbttiinnn data for the image classification task In recent years,
plethora of methods have been proposed to address domain generalization [25] [54 [35] [6]. Prevailing
methods are mainly based on feature alignment
(7
58
59 domain adversarial learning 660 61
invariant risk minimization [62 63
64 655, meta learning 66
67
68
69 77), data augmentation
72,
73
74
75,
etc. In addition, ensemble learning is an effective way to learn generalizable
models 76. 77 78]. And recently, Zhu et al. develop a theory showing that ensemble learning
can provably improve test accuracy by discovering the "multi-view" structure of data (79), which partially inspires our approach. Among cebitecture-basel methods 88 8I] Meng et al. propose to redesign attention modules for learning diverse task-related features 88). Different from existing general domain generalization methods, video classification, which explores diverse class-correlated information in intrinsic space and time dimensions of video data. There are some works that study the identification of out-of-distribution data of different categories from training data 82 the scope of our work.
propose domain generalization method specific to
86 87 88 but this topic not within
3 Spatial-Temporall Diversification Network
In this section, illustrate our proposed Spatial-Temporal Diversification Network (STDN) in detail, which perceives diverse class-correlan carselorrel spatial-temporal cues from video contents for generalization in unseen target domains.
3.1 Problem Formulation
In video domain generalization, a set of labeled videos D = {(zz)) y)} from a source domain are given for training, where x E X' and y E y denote a source video and its corresponding class label. Given only the source video set, the goal of video domain generalization is to learn a video classification model that is generalizable unseen target domains. The source and target domains follow different but related distributions with the same label space y = {0, 1, ..... C' 1}. where C denotes the number of video classes. Following the standard video domain generalization setting [55, each video is evenly divided into N segments, and one frame is sampled from each segment as the model input
3

---

Video
Backbone
Spatial Grouping Module
Spatial-Tempont Relation Module
Frame
Frame
Frame S
K Group
Classficr
:
Spatial Features
Frame Group
Grouping
L Learnable A Anchors
10 Spatial Spatial-Relation Modeling
Spatial-Relation Feature
Temporal-Relation Modeling
Temporal-Relation Feature
Figure An overview of our proposed Spatial-Temports Diversification Network (STDN). We use video of N 5 segments with K 4 spatial groups for example. After backbone feature extraction, our STDN extracts spatial cues of K types for each frame by the Spatial Grouping Module, enriching the diversity in spatial modeling. Then, our STDN explicitly models spatial-temporal dependencies at multiple space-time scales, enriching the diversity spatial-temporal relation modeling. Best viewed in color.
during training and testing, i.e., {21, (21,222 .2...... FNN and Fn denotes the n-th sampled frame from the n-th segment.
3.2 Model Overview
Aiming generalization in unseen target domains, our idea is perceive rich and diverse class- correlated cues in the source domain. In this way, our model would leverage not only easy-to-fi domain sppecifcc cues but also other potential domain-invariant cues for recognizing videos in the target domain, alleviating the misguidance of domain-specinn cues. Considering the intrinsic space and time dimensions of video data, we propose to explore the diversity both spatial and temporal modeling. An overview of our proposed STDN is shown in Figure 2 Firstly, given the video H, our STDN takes N sampled frames as input and separately extracts N spatial feature maps {21, 22,,.., vN by the backbone (e.g., ResNet (89),, where 2n E #xx x denotes the feature map of the n-th frame, D denotes the feature dimension and H W denotes the size of feature maps. Then, we extract spatial cues of K types (groups) from each spatial feature map by our proposed Spatial Grouping Module, aiming to enrich the spatial diversity. In the Spatial Grouping Module, two entropy- based losses are introduced to enhance the distinction between different spatial cues. On top of the Spatial Grouping Module, we propose to explicitly model between video contents multiple space-time scales by proposed Temmpoall Relation Module. The learning of the Pratial-Temporal Relation Module guided by relation discrimination loss, which ensures the diversity of the extracted spatial-temporal relation features. Finally, diverse spatial-temporal features are aggregated for video domain generalization.
dependencies
3.3 Spatial Grouping Module
Our proposed Spatial Grouping Module aims to discover diverse class-correlated spatial cues from abundant contents of individual frames, which enriches the diversity from spatial modeling per- spective for video domain generalization. Our Spatial Grouping Module extracts various spatial cues of different types by partitioning features from different spatial positions into several groups within individual frames. In this way, our Spatial Grouping Module discovers more diverse spatial cues, compared with prevailing approaches that extract an integrated feature for each frame (e.g., by average pooling).
As shown in Figure (a), given the spatial feature map E H of the n-th frame, our proposed Spatial Grouping Module learns to extract K spatial cues by aggregating the HW spatial features. Specifically, the proposed spatial grouping process is conducted based on K learnable anchor features {an,1 On.2...., an.k) ). where an,k E R denotes the anchor feature of the k-th spatial group for the n--hh frame. Then, calculate the probability of assigning spatial feature to each spatial group, which is formulated as follows:
exp (-dist (Zn,ir kn,k) /T)
Pn.i,k
(1)
Σ/=1 exp (-dist (2n,s, On.s) (imms)/r /T)

---

Spatial Grouping Module
(b) Spatial-Temae Temporal Relation Module
Spatial Relation Modeling
Frame
Temporal Modeling
1 time scale
Frame
Spatial Features
spaico
12: .3...
time scale
Sgice
P
123
3
H
Lernaable Anthors
Entropy Minimization Lenii
Grouping
Entropy
3 space scale =
time scale
Maximization Legruz
2 4
Lernable Anchor Spatial Feature
Figure 3:
Overviews of our proposed (a) Spatial Grouping Module and (b) Dpatial-Tempores Relation Module. Best viewed in color.
where 2 RD denotes the i-th spatial feature in the feature map 2n (i [1, ... HW) W. dist(..-) denotes the Euclidean distance metric and т is the temperature factor. According to Eq. ( if the spatial feature Σ is closer the anchor feature an then the Zn,á will be assigned the k-th spatial group with higher probability. After group partition, our Spatial Grouping Module produces K integrated features representing K different spatial cues by aggregating spatial features in each group. The integration process is formulated as follows:
HW
Σ
Priik * Zn,i
(2)
11
Pr.i,
where 2i, denotes the spatial cues integrated from the k-th group within the n-th frame.
In order extract spatial cues of diverse types, we introduce two entropy- losses enhance the distinction between different spatial groups. The first one is an entropy minimization loss to enhance the confidence of group assignment for each spatial feature. The loss is formulated as follows:
N HW K ΣΣ>
L Lemin NHW
Pn.i, log (Pm,i.k)
For the assignment probability vector Pa, [P., Pr.4...... Pr.i. Kx if the entropy is minimized, then the feature 2n,5 will be confidently assigned a specific spatial group. The second loss is an entropy maximization loss for the mean assignment probability vector, which guarantees that those HW spatial features are assigned to different spatial groups. Specifically, the loss formulated as follows:
N
1
Teemax
P k log (Pn,k),
(4)
N
where Pr.* NW Pr,i, denotes the mean probability of assigning features to the k-th group within the n-th frame. For the mean assignment probability vector pr, [P... ... Pn. x RXDD if the entropy is maximized, then the spatial features (=n,)) will be uniformly assigned K spatial groups. By using the two entropy-based losses, we guarantee that spatial features are different from each other across different spatial groups, enriching the diversity of extracted spatial cues.
HW
the Spatial Grouping Module, the learnable anchor feature for each group is extracted by weighted combining those HW spatial features, and the weights are calculated conditioned on the feature map on by using a lightweight two-layer convolutional network. this way, the spatial grouping process can be regarded as conducting clustering over spatial features within individual frames. All involved parameters in the module are end-to-end trained together with the main network, i.e., we contribute a parametric clustering module to group spatial features for improving the spatial diversity.
3.4 Spatial-Tempores Relation Module
Our proposed Spatial-Tempone Relation Module aims to discover diverse class-correlated spatial- temporal cues from abundant video contents, which enriches the diversity from spatial-temporal relation modeling perspective for video domain generalization. As demonstrated by previous works 44 90), there rich dependencies between entities over space and time dimensions in videos, which
5

---

is crucial for video classification. Accordingly, we propose to explicitly model dependencies between video cues at multiple space-time scales. Our proposed Opatial-Tempell Relation Module conducts dependency modeling at space and time dimensions separately, and an overview of the module is shown in (
First, based on the spatial cues extracted by our Spatial Grouping Module, we conduct spatial dependency modeling between these spatial cues at multiple space scales. Specifically, given the representations of spatial cues * = I ",,2....,,,, " E R2*D x for the n-thh frame, we extract the spatial relation feature at the 1-th space scale by the spatial dependency modeling function R7 as follows:
Rf (= = Ag [H] 2i k kn) E ERR:
where [E[] denotes the expectation calculation and (........ denotes a linear projection function after feature concatenation. The index set {ki, ,2....,k) ..) denotes the index of spatial features uniformly sampled from the K spatial features, where the index / E (2.3..... ..., }} indicates the space scale, ki # l2 # ... + l and & 21. K}. For each frame, we extract K 1 spatial relation features by dependency modeling at K space scales separately. And, we concatenate these spatial relation features and produce an integrated feature for each frame, which is given "(,,) E R.D. In the integrated feature ên. by 2, = ["))) Rī(=;)² "[.,,, R (=a)²,
G(2n) E RD denotes the global feature extracted from the feature map 2n by a convolution layer.
Second, based on the frame-level integrated features, we conduct temporal dependency modeling between frames at multiple time scales. Specifically, given N frame-level features denoted by 2 = [ , ,2v]. we extract the temporal relation feature at the m-th time scale by the temporal dependency modeling function R (.) as follows:
R (2) = En <m2c--emm (2n1: 80z.
mm.) E E
(6)
where Hü, ((5'))... denotes a linear projection function after feature concatenation. The index set {πi, R2.. .e..... R denotes the index of frame features randomly sampled from the N frame features, Note that
where the index m E 3.. ... we keep the relative order of sampled frames for temporal modeling. By using N - 1 temporal dependency modeling functions, we extract N temporal relation features at N time scales for each video.
indicates the time scale and n. E ...
To ensure the diversity of temporal relation features, we propose relation diserimination loss to constrain that different temporal dependency modeling functions (i.e., different time scales) capture different temporal cues. This loss constrains that a relation classifier can distinguish one relation feature from not only relation features of other classes but also relation features of the same class but of other time scales. Thus, avoids the feature collapse of learned temporal relation features. Specifically, the loss is formulated as follows:
Lyel N
CE(FFa((2m)
where En RR, denotes the temporal relation feature at the m-th time scale, (-ei(-) denotes a relation classifier that classifying (N - 1))= C' classes, and CE(., denotes the cross-entropy loss. The Vm denotes the relation label of the video æ with label y, i.e., Vm = y * (N - 1) + (m - 2) E {0, 1, ... (N - 1) + C' 1}. In this way, the loss forces different temporal dependency modeling functions to capture different class-correlated temporal cues in the video since the captured temporal cues are discriminative across scales. By incorporating the the relation discrimination loss Liel, we extract rich and diverse spatial- temiiialltmprall cues.
Relation Module with
Feature Aggregation: After exploring spatial-temporal diversity by our proposed Spatial Group- ing Module and Tempoalll Relation Module, our model discovers diverse class-correlated spatial-temporal cues from abundant video contents. Then, we aggregate these diverse spatial- temporal features for video classification. Specifically, the feature aggregation is formulated as N 1 = >>>>>>>> 2 H (2m), where H.,(.) denotes a small SE-based block [91 for modulating the m-th temporal relation features.
Overall Training and Test: We adopt video classification loss on top of the aggregated feature x given by Leis CE(F(2), v). where F(-) is video classifier. Overall, the training loss of our

---

SGD with a batch size of 32, momentum of 0.9, learning rate of Table I: Comparison with state-of-the-art methods on the UCF-HMDB benchmark. Red and blue denotes the best and second best. Results of all compared methods are from VideoDG [15].
Arch
DG Method
UCF >HMDB
HMMB >UCF]
Arch
DG Method
UCF III-HIMBB HMDB +UCF
FRM
51.410.
68.610.3
ERM
52.2.0.3
69.22...
ADA.em
$1.140.
68.220.2
ADA.cm
51.320.3
68.6 68.620.3
TSN
ADAgiie!!
71
49.640.
67.4
TSM 7
pDApisee
52.7=0.3
68.3
M-ADA
92
52.4+0.2
69.2+0.2
M-ADA
92
52.5+0.2
69. 10.3
Jigsaw
93
51.510.3
68.510.3
Jigsaw
93
52.5
68.9
ERM
54.310.3
71.410.3
ERM
52.4.0.3
69.82..3
ADA
71
55.2+0.3
71.9+0.3
ADA
71
52.8+0.2
69 69.6g0.5
APN
ADAgire!
56.910.2
72.2
TRN 66
ADAgizel
52.1.0.3
70.6
M-ADA
55.6+0.3
71.5+0.3
M-ADA
53,4
69, =0.33
Jigsaw 93
55.2
72.4.0.3
Jigsaw
53.3.00.
70. 10.1
VideoDG [15
59.1
74.940.3
STDN (Ours)
60.2
77.1=0.4
proposed STDN is given follows:
L = Lels + Aent Lemin + Aeut Lemax
Avel
where Aent and Avel hyperparameters for trade-off. Following the standard protocol source videos for training and test the model on target videos for evaluation.
15]
use
4 Experiments
4.1 Benchmarks and Experimental Setups
To demonstrate the effectiveness and versatility of our proposed Spatial-Temporal Diversification Network (STDN), we adopt three benchmarks of different types for experiments, including two newly designed benchmarks, namely EPIC-Kitchens-DC and Jester-DG. For these new benchmarks, we select video categories and construct domains following previous video domain adaptation We split the source video set into training and validation sets following previous source validation protocols (25 15], i.e., reasonable in-domain model selection scheme for better generalization ability in unseen target domains. We reproduce general domain generalization methods (cooperated with video classification architectures) and state-of-the-eee video domain generalization methods for comparison. We report mean and standard deviation of accuracy over three random trials for all methods.
works 49 [9]
UCF-HMDB is the most widely used benchmark for cross-domain video classification [5 [8]. which contains 3,809 videos of 12 overlapping sport categories shared by UCF101 (94) and HMD- B51 [95]. The videos in UCF101 and the videos in HMDB51 are captured from unconstrained environments and different camera viewpoints. This benchmark includes two subtasks, i.c., UCF-HHMB and HMDB-UCE
mostly captured from certain scenarios similar environments,
EPIC-Kitchens-DCC is a cross-scene egocentric action recognition benchmark, which consists of 10,094 videos across 8 egocentric action classes from three domains (scenes), following Munro et al. [49]. The three domains
SPiC-Kiichens-Dee (i.e., D1, D2, D3) correspond to three largest kitchens (i.e., PO8, PO1, P22) from the large-scale egocentric action recognition dataset EPIC- Kitchens-55 196.. This benchmark includes six subtasks constructed from three domains.
Jester-DG is cross-domain hand gesture recognition benchmark. We select videos from the Jester dataset 977 and construct two domains following Pan et al. [19. The source (S) and target (T) domains contain 51,498 and 51,415 video clips across 7 categories, respectively. The vidcos in EPIC-Kitchens-DG and Jester-Do DG benchmarks are both hand-centric, but they are captured from different views, namely first-person and third-person views.
Implementation details: We use ResNet50 88 pretrained on ImageNet 98 as the backbone for frame-level feature extraction following the standard video domain generalization protocol [15]. The backbone takes frames of size 224 x 224 as input and outputs feature maps of size 7 x 7 x 2048. We take N = 5 frames for each video for temporal modeling, We set K 4, 0.5, D, 192 and D. = 256. F(.) linear classifier and (.a(() an MI.P classifier. All parameters are optimized using
7

---

Table 2: Comparison with state-of-the-art methods on the EPIC-Kitchens-DG and Jester-DG benchmarks. Red and blue denotes the best and second best. Results of all compared methods are reproduced following their official implementations.
Epic- Seie-Kitchens-GG
Jester-DG
Arch
DG Method
D1-12 D1 D1-+D3 D2 DD D2-1D3 D3->D1
D3 D3-+D2 Average
S-T
FRM
33.610.6
31.3
22.5
36.1
31.7_a.
40.210.7
34.2.0.4
47.5
Mixup 72
33.620.2 29.1
31.2-2.7
36.520.4
33.0_o.5
39.7.0.1
33.920.3
47.8 47.8.
TSN S
IRM
62
34.610.1 30.311.
31.2-12
36.3102
32.3_0. 40.010.
34.1
47.610.5
ADA
71
33.8+0.2 30.3+0.
31.1-14
36.4+0.0
33.3-04
40.1+0.8
34.2+0.6
47. 47.6-0.3
COP
100
34.720.1 29.7a1.
31.520.8
36.710s 31.5_01
40.3
34.1
47.3.0.9
ERM
34,610.5 32.311.
30.21..9
34,810.
31.2_1.9
39.911.
33.8
47.1
Mixup
72
34.1 29.310.6
28.1
32.410a
31.7_a.7
39.1
32.4
47.310.4
TSM 7
IRM 62
34.320.2 29.3+15
31.128
36.340.7
31.6-1.
38.5
33.5-0.7
46.820.7
ADA
34.3+0.8 30.641.
30.0-1.
34.6+1.
31.8_a.
38.8
28.9
33.4
47.120.5
COP 100
34.910.3 32.111.1
30.5
32.210.6
31.8_18
1.5
17.5
1..2
16.4
ERM
35.2+0.8 30.242.1
35.3210
43.221.2
36.4_o.3
4
37.6
46.920.5
Mixup
72
34.710.2 30.910.7
33.8.0.7
44.7105
36.4_03
14.4
37.5
47.8
APN
IRM
62
33.6.0.5 28.5+1.
34.3-0.1
42.8+08
34.6_0.
44.
36.3
47.0-0.3
ADA
36.010.3 29.5
35.0.a.4
43.020.3
37.6_0.4
45.6
47.8 27.8
27.4
COP 100
36.52a.8 32.1
37.6-1.8
41.820.8
37.9-0.2
49.9
47.2 39.3
FRM
36.811.4 32.111.
34.211.0
44.710.
37.6_0.9
18.910.1
19.7
16.210.4
Mixup
72
37.5+1.0 31.2+1.
35.3-1.5
43.2+1.4
39.0_0.
48.
39.0=0.6
46.7
TRN 6
IRM
62
37.8+2.3 30.5
37.0_26
42.6+1.
40.3_a.
47.9
39.3
46.5
ADA
38.4+1.0 30,4+12
35.9-1.3
41.2+0.6
38.8-a5
47.5+1.1
38.7
46.1
COP 100
36.8 34.2127
35.8_1.
39.6124
18.1_0.5
38.9 49.0
47.810.5
VideoDG 15
36.2
31.920.2
36.5
40.5
39.5
49.1
39.0
47.5
STDN (Ours)
40.5 38.6.0.1 38.5g26
44.0
40.4_1.. 47.2
41.6
49.8
le-3 and weight decay of 5e-4. By default, the trade-off hyperparameters are set as Aemt 0.1 and Apel = 0.5. We adopt an efficient feature augmentation technique namely MixStyle [74] for simulating novel target domains during training. We modify the original MixStyle to adapt the video data, i.e., we calculate the mean and standard deviation across both space and time dimensions within each channel of each instance (instead of only space dimension for image data). All experiments
are conducted by PyTorch [99] with four NVIDIA GTX 1080Ti GPUs. The code is released at http://ggttubb.coouuuullllni
4.2 Results
Comparison with State-of-the-arts We compare our proposed STDN with two types of state-of- the-arts: 1) general domain generalization (DG) methods cooperated with different video classification architectures; 2) statc-of-the-a video domain generalization methods. For the two newly designed benchmarks (i.e., Spicc-Kichhnns-GG and Jester-DG), we adopt five different types of general domain generalization methods for comparison, including Empirical Risk Minimization (ERM), Mixup (72), Invariant Risk Minimization (IRM) 62). Adversarial Data Augmentation (ADA) 77.. Clip Order Prediction (COP) [000. All results are summarized in Table [nd On all the three benchmarks, our STDN outperforms all the -tate-of-thh--ar methods. Specifically, our STDN obtains performance improvement by 2.2% and 2.3% on HMDB HMDB-+UCF and Kitic-Kithhnns-GG respectively, which significant compared with previous state-of-the -att---r-----sss.... In addition, VideoDG [15] obtains lower performance than their proposed architecture APN [55 on Sppic-Kitchens-DG and Jester-DG. By contrast, our superiority on three benchmarks of different types verifies the effectiveness and versatility of our proposed STDN, demonstrating the effectiveness of perceiving diverse spatial-temporal cues. In the supplemental material, we make an attempt to compare with two variants of Planamente et al. [66 to show our effectiveness, following some works in the domain adaptation field [Io [02 103
Ablation Study: We analyze the effects of each component in our proposed STDN, as shown
in Table 3. Following the training scheme of TSN [5], we apply a classifier on top of the backbone as our baseline. By stacking our proposed Spatial Grouping Module (SGM) on top of the backbone, we obtain significant improvement over the baseline (i.e., 2.2% on UCF-HMDB and 2.0% on HMDB-UCFF) demonstrating the effectiveness of extracting dif- ferent types of spatial cues within individual frames. Then, by introducing the temporal de-
8

---

Baseline
Spetial Grouping Module
Normalized MSE
SGM+STRM
0.5441
SGM+TRM
0.5124
TRM
0.3329
Crans
TRM w/o Lrel
0.0434
Dindes Bodddin Triles 142
Dawles Bondes- Indos: (163
Figure 4:
Diversity analysis on UCF-HHMBB
Figure 5: T-SNE visualization of spatial features. For both the baseline and SGM, we cluster the set of spatial features into K = 4 clusters by K- means before visualization. In the figure, dots stand for spatial feature vectors and stars stand for cluster centers, and different colors denote different clusters.
We use the normalized Mean Square Error (MSE) evaluate the feature diversity of variants of TR-
M. i.e. measuring difference between temporal relation features of different time scales. A higher value of normalized MSE indicates higher diver- sity.
pendency modeling of our Spatial-Temporal Relation Module (denoted by TRM), we obtain 1.8% and 1.4% improvement on UCF-HHMBB and HMDB-+UCF, respectively. It should be noted that the relation discrimination loss Liel is an important part in temporal dependency modeling, since we obtain very minor perfor- mance improvement without the loss in tempo- ral dependency modeling. By introducing the
Table 3: Ablation study on UCF-HMDB
Method
UCF-HMDB
HMDB-UCC
full Spatial- Temporll Relation Module (STR- Backbone
52.7
71.9
+SGM
54.920.3
73.9
M), we obtain 3.4% and 2.3% improvemen- t over "Backbone+SGM" on UCF-HMDB and HMDB-UCC respectively. These results demonstrate the effectiveness of modeling de- pendencies between various video cues both space and time dimensions, which enriches the diversity in spatial-temporia relation modeling, More- over, by introducing the feature augmentation technique MixStyle, we obtain further improvement. Finally, our full model aggregates diverse spatial-temporal features, leading to better generalization performance on both HMMB and HMDB UCF.
+TRM
56.720.2
75.3+0.4
+STRM
58,3
76.2+0.3
+MixStyle
59.3
76.640.2
Full STDN
60.2a0.5
77.1
Diversity Analysis:
We make quantitative analysis to the diversity of learned video features for our model. Specifically, we evaluate the difference between temporal relation features of different time scales, measured by the normalized Mean Square Error (MSE) between feature vectors. A higher value of normalized MSE indicates a large difference. As shown in Figure 4 without our relation discrimination loss Leel, learned temporal relation features at different time scales hold very small difference (implying feature collapse). By introducing Lyel, our TRM improves the diversity, indicated by the higher MSE value. By introducing our Spatial Grouping Module, the diversity is further improved as various spatial cues are extracted from each frame. Moreover, by modeling spatial dependencies, our model further enlarges the difference between features across scales.
We make qualitative analysis to the grouping process of our proposed Spatial Grouping Module (SGM). Specifically, we use t-SNE [104] for visualizing feature distributions of spatial features, and we adopt the model trained without our SGM as the baseline (adopts average pooling to extract an integrated feature for each frame) for comparison. Also, we use the Davies-Bouldin Index! as a quantitative metric to measure the clustering performance, i.e., lower value of the Davies-Bouldin Index indicates better separation between clusters. As shown by the qualitative and quantitative results in Figure our SGM extracts spatial features with better cluster separation than the baseline, which is attributed to that SGM enhances the distinction between features in different spatial groups These results indicate that our proposed spatial grouping process forces the model to learn features encoding more different information. In the supplemental material, we also show Grad-CAM examples to qualitatively compare our SGM with the baseline.
Analysis of Spatial Grouping:
'The coiis-Bouldii Index distance.
105 measures a ratio between the intra-cluster distance and inter-cluster

---

Baseltne
SIDN (Ours)
Baseline
SIDN (Ours)
Source Domain (Training)
Target Domain (Test)
Figure 6: Grad-CAM visualization on UCF-HMDB. As shown in the figure, compared with the baseline, our proposed STDN captures more diverse blass-correlated cues in the source domain, iee, including domain-spenn specific backboards and domain-invariant basketballs. a result, our proposed STDN generalizes better in the target domain, where backboards are invisible and thus our STDN uses the basketball for recognition instead. Best viewed in color.
Grad-CAM Visualization: We compare our proposed STDN with TRN model (the baseline) by Grad-CAM [106]. As shown in Figure the baseline prefers to use the domain-specifit backboard for recognition, which causes recognition errors the target domains as backboards are invisible. In contrast to the baseline, our proposed STDN perceives more diverse source domain, including some Nomain-invariann cues such as basketballs. As a result, our STDN can predict the correct video class by recognizing the basketball in the target video. These results demonstrate that, our proposed diversity- -based approach can discover some potential domain-invariant invariant cues, which alleviates the overfitting to domain-specte cues and leads to better generalization in the target domain.
cues from the
Conclusion
In this work, we propose to explore spatial-temporal diversity to address the video domain generaliza- tion task. Our proposed Spatial-Temporal Diversification Network learns diverse spatial-temporal features in videos, which discovers potential domain-invarial cues and thus alleviates the heavy reliance on domain-specins cues. We conduct extensive quantitative and qualitative experiments on three benchmarks (including two newly designed benchmarks), and the results demonstrate the effectiveness and versatility of our approach.
Acknowledgements. This work was supported partially by the NSFC 1124112071 ,U191 1401), Guangdong NSF Project (No. 2023B151504002 thank Zhilin Zhao, Yi-Xing Peng, and Yu-Ming Tang for their valuable suggestions on model design or writing.
The authors would like to
References
[1] Du Tran, Lubomir D. Bourdey, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotem- poral features with 3D convolutional networks. IEEE International Conference on Computer Vision, 2015.
[2] João Carreira and Andrew Zisserman. Quo Vadis, Action Recognition? A new model and the kinetics dataset. In IEEE Conference Computer Vision and Pattern Recognition, 2017.
[3] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look spatiotemporal convolutions for action recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2018
[4] Xiaolong Wang, Ross B. Girshick. Abhinav Gupta. and Kaiming He. Non-local neural networks. IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[5] Limin Wang. Yuanjun Xiong, 7he Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool, Temporal Segment Networks: Towards good practices for deep action recognition. In European Conference on Computer Vision, 2016.
[6] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos.
In European Conference Computer Vision, 2018
10

---

[7] Ji Lin. Chuang Gan, and Song Han. TSM: Temporal shift module for efficient video understanding. In IEEE/CVF International Conferencee on Computer Vision, 2019.
[8] Haoxin Li, Wei-Shi Zheng, Yu Tao, Haifeng Hu, and Jian-Huang Lai. Adaptive interaction modeling via graph operations search. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[9] Yu Kong and Yun Fu. Human action recognition and prediction: A survey. International Journal of Computer Vision, I (005):1366-1400)) 2022.
[10] Zehua Sun, Qiuhong Kc, Hossein Rahmani, Mohammed Bennamoun, Gang Wang, and Jun Liu. Human action recognition from various data modalities: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3200 3225, 2023.
[11] Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip H.S. Torr, and Song Bai. MOSE: A new dataset for video object segmentation in complex scenes. In IEEE/CVF International Conference on Computer Vision, 2023.
[12] Riccardo Volpi, Pau de Jorge, Diane Larlus, and Gabriela Csurka. On the road to online adaptation for semantic image segmentation. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
[13] Attila Lengyel, Sourav Garg, Michael Milford, and Jan van Gemert. Zero-shot day-night domain adaptation with physics prior. In JEEE/CVF International Conference on Computer Vision, 2021.
[14] Yuecong Xu, Jianfei Yang, Haozhi Cao, Kezhi Mao, Jianxiong Yin, and Simon See. ARID: A compre- hensive study on recognizing actions in the dark and new benchmark dataset. CoRR, abs/2006.03876. 2020.
[15] Zhiyu Yao, Yunbo Wang, Jianmin Wang, Philiip S. Yu, and Mingsheng Long. VideoDG: Generalizing temporal relations in videos to novel domains. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):9989-800 8004, 2022.
[16] Mirco Planamente, Chiara Plizzari, Emanuele Alberti, and Barbara Caputo. Domain generalization through audio-visual relative norm alignment in first person action recognition. In IEEL/CVF Winter Conference Applications of Computer Vision, 2022.
[17] Arshad Jamal, Vinay Namboodiri, Dipti Deodhare, and K. S. Venkatesh. Deep domain adaptation in action space. British Machine Vision Conference, 2018.
[18] Min- MiinHHug Chen, Zsolt Kira, Ghassan Alregib, Jackwon Yoo, Ruxin Chen, and Jian Zheng. Temporal attentive aligmment for large-scale video domain adaptation. In JEEE/CVF International Conference on Computer Vision, 2019.
[19] Boxiao Pan, Zhangjie Cao. Ehsan Adeli, and Juan Carlos Niebles. Adversarial cross-domain action recognition with co-attention. In AAAI Conference on Artificial Intelligence, 2020.
[20] Jinwoo Choi, Gaurav Sharma, Samuel Schulter, and Jia-Bin Huang. Shuffle and attend: Video domain adaptation. In European Conference on Computer Vision, 2020.
[21] Yingwei Li, Yi Li, and Nuno Vasconcelos. RESOUND: Towards action recognition without representation bias. In European Conference on Computer Vision, 2018.
[22] Jinwoo Choi, Chen Gao, Joseph C. E. Messou, and Jia-Bin Huang, Why Can't I Dance in the Mall? Learning to mitigate scene bias action recognition. In Advances Neural Information Processing Systems, 2019.
[23] Jinpeng Wang, Yuting Gao, Ke Li, Yiqi Lin, Andy J. Ma, Hao Cheng, Pai Peng. Feiyue Huang. Rongrong Ji, and Xing Sun. Removing the background by adding the background: Towards background robust self-supervised video representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[24] Haoxin Li, Yuan Liu, Hanwang Zhang, and Boyang Li. Mitigating and evaluating static bias of action representations the background and the foreground. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[25] Ishaan Guirajani and David Lapez-Paz. In search of lost domain generalization. In International Conference Learning Representations, 2021.

---

[26] Du Tran, Heng Wang. Matt Feiszli. and Lorenzo Torresani. Video classification with channel-separated convolutional networks. In IEEE/CVF International Conference on Computer Vision, 2019.
[27] Christoph Feichtenhofer. X3D: Expanding architectures for efficient video recognition. In IEEE/CVF Conference Computer Vision and Puttern Recognition, 2020.
[28] Shiwen Zhang, Sheng Guo, Weilin Huang, Matthew R. Scott, and Limin Wang. V4D: 4D convolutional neural networks for video-level representation learning. In International Conference on Learning Representations, 2020.
[29] Kunchang Li, Xianhang Li, Yali Wang, Jun Wang, and Yu Qiao. CT-Net: Channel tensorization network for video classification. International Conference on Learning Representations, 2021.
[30] Junyan Wang. Zhenhong Sun, Yichen Qian, Dong Gong, Xiuyu Sun, Ming Lin, Maurice Pagnucco, and Yang Song. Maximizing spatio-temporal entropy of deep 3d cnns for efficient video recognition. CoRR, 0os/2303.02693,, 2023.
[31] Noureldien Hussein, Efstratios Gavves, and Arnold W. M. Smeulders. Timeception for complex action recognition. In JEEE Conference on Computer Vision and Puttern Recognition, pages 254-263, 2019.
[32] Jiaming Zhou, Kun- Lin, Haoxin Li, and Wei-Shi Zheng. Graph-based high-order relation modeling for long-term action recognition. In IEEE Conference on Computer Vision and Puttern Recognition, 2021.
[33] Jiaming Zhou, Kun-Yo Yu Lin, Yu-Kun Qiu, and Wei-Shi Zheng. Twinformer: Fine-to-coarse temporal modeling for long-term action recognition. IEEE Transactions on Multimedia, 2023.
[34] Hao Shao, Shengju Qian, and Yu Liu, Temporal interlacing network. In AAAI Conference on Artificial Intelligence, 2020.
[35] Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz. Gate-shift networks for video action recogmition. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, In Advances in Neural Information Processing Systems, 2017.
[37] Rohit Girdhar, João Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer network. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[38] Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network. CoRR, abs/2102.00719, 2021.
[39] Hao Zhang, Yanbin Hao, and Chong- Wah Ngo. Token shift transformer for video classification. In ACM International Conference Multimedia, 2021.
[40] Yanyi Zhang, Xinyu Li, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and Joseph Tighe. VidTr: Video transformer without convolutions. In IEEE/CVF International Conference on Computer Vision, 2021.
[41] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for vidco understanding? International Conference on Machine Learning, 2021.
[42] Anurag Arnb, Mostafa Dehghani, Georg Heigold. Chen Sun, Mario Lucic, and Cordelia Schmid. VIVIT: vidco vision transformer. In IEEF/CVF International Conference on Computer Vision, 2021.
[43] Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. VLT: lision-language transformer and query generation for referring segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, (5(6):7900-7111 2023.
[44] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang. Stephen Lin, and Han Hu. Vidco swin transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[45] Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang. Chen Chen, and Mu Li. AIM: Adapting image models for efficient video action recognition. In International Conference on Learning Representations. 2023.
[46] Yuecong Xu, Haozhi Cao, Kezhi Mao, Zhenghua Chen, Lihua Xie, and Jianfei Yang. Aligning correlation information for domain adaptation in action recognition. IEEE Transactions on Neural Networks and Learning Systems, 2023.
12

---

[47] Yadan Luo, Zi Huang, Zijian Wang. Zheng Zhang, and Mahsa Baktashmotlagh. Adversarial bipartite graph learning for video domain adaptation. In ACM International Conference Multimedia, 2020.
[48] Aadarsh Sahoo, Rutav Shah, Rameswar Panda, Kate Saenko, and Abir Das. Contrast and mix: Temporal contrastive video domain adaptation with background mixing. In Advances in Neural Information Processing Systems, 2021.
[49] Jonathan Munro and Dima Damen. Multi-modal domain adaptation for fine- ane-ggained action recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[50] Xiaolin Song, Sicheng Zhao, Jingyu Yang. Huanjing Yue, Pengfei Xu, Runbo Hu, and Hua Chai. Spatio- temporal contrastive domain adaptation for action recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[51] Donghyun Kim, Yi-Hsuan Tsai, Bingbing Zhuang, Xiang Yu, Stan Sclaroff, Kate Saenko, and Manmohan Chandraker. Learning cross-modal contrastive features for video domain adaptation. In IEEE/CVF International Conference on Computer Vision, 2021.
[52] Lijin Yang, Yifei Huang. Yusuke Sugano, and Yoichi Sato. Interact before align: Leveraging cross- moss- knowledge for domain adaptive action recognition. In JEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
[53] Yunhua Zhang, Hazel Doughty, Ling Shao, and Cees G. M. Snoek. Audio-adaptive activity recognition across video domains. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[54] Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre- Aeevisee-Alvise Rebuffi, Ira Ktena, Krishnamurthy Dvi- jotham, and Ali Taylan Cemgil. A fine- fine-grained analysis on distribution shift. In International Conference on Learning Representations, 2022.
[55] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE Transactions Puttern Analysis and Machine Intelligence, 45(4):4396-4415, 2023.
[56] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, and Tao Qin. Generalizing to unseen domains: 2021.
A survey on domain generalization. CoRR,
[57] Baochen Sun and Kate Saenko. Deep CORAL: correlation alignment for deep domain adaptation. In European Conference on Computer Vision Workshops, 2016.
[58] Mingsheng Long. Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In International Conference on Machine Learning, 2015.
[59] Divyat Mahajan. Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In International Conference on Machine Learning, 2021.
[60] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarie training of neural networks. Journal of Machine Learning Research,
159:35, 2016.
[61] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In European Conference on Computer Vision, 2018.
[62] Martín Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. CoRR, abs/1907 02893, 2019.
[63] David Krueger, Fthan Caballero, Jöm--enrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Rémi Le Priol, and Aaron C. Courville. Out-of-distibutiion generalization via risk extrapolation (rex). In International Conference on Machine Learning, 2021.
[64] Shiyu Chang, Yang Zhang, Mo Yu, and Tommi S. Jaakkola. Invariant rationalization. In International Conference on Machine Learning, 2020.
[65] Xiao Zhou, Yong Lin, Weizhong Zhang, and Tong Zhang. Sparse invariant risk minimization. In International Conference on Machine Learning, 2022.
[66] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Leamning to generalize: Meta-learning for domain generalization. In AAAI Conference Artificial Intelligence, 2018.
[67] Yogesh Balaji. Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain generalization using metaaregulaarization In Advances in Neural Information Processing Syxtems, 2018
13

---

[68] Yiying Li. Yongxin Yang, Wei Zhou, and Timothy M. Hospedales. Feature-critic networks for heteroge- neous domain generalization. In International Conference on Machine Learning, 2019.
[69] Qi Dou, Daniel Cnelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via model-agnostic leamnngg of semantic features. In Advances in Neural Information Processing Systems, 2019.
[70] Jin Chen, Zhi Gao, Xinxiao Wu, and Jieho Luo. Meta-causal learning for single domain generalization. CoRR, abs/2304.03709 2023.
[71] Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C. Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In Advances in Neural Information Processing Systems, 2018.
[72] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lapez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
[73] Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto I.. Sangiovanni- Vincentelli, Kurt Keutzer, and Boqing Gong. Domain randomization and pyramid consistency: Simulation-to-ret generalization without accessing target domain data. In IEEE/CVF International Conference on Computer Vision, 2019.
[74] Kaiyang Zhou, Yongxin Yang. Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference on Learning Representations, 2021.
[75] Sangrok Lee, Jongscong Bac, and Ha Young Kim. Decompose, adjust, compose: Effecttive normalization by playing with frequency for domain generalization. CoRR, abs//2333.02322 2023.
[76] Pavel Izmailov. Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads wider optima and better generalization. In Conference on Uncertainty in Artificial Intellignnee, 2018.
[77] Junbum Cha. Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. SWAD: Domain generalization by seeking flat minima. In Advances Neural Information Processing Systems, 2021.
[78] Xu Chu, Yujie Jin, Wenwu Zhu, Yasha Wang, Xin Wang, Shanghang Zhang, and Hong Mei. DNA: Domain generalization with diversified neural averaging. International Conference on Machine Learning, 2022.
[79] Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self- distillation deep learning. In International Conference on Learning Representations, 2023.
[80] Rang Meng, Xianfeng Li, Weijie Chen, Shicai Yang. Jie Song, Xinchao Wang, Lei Zhang, Mingli Song, Di Xie, and Shiliang Pu. Attention diversification for domain generalization. In European Conference on Computer Vision, 2022.
[81] Scokeon Choi, Debasmit Das, Sungha Choi, Seunghan Yang, Hyunsin Park, and Sungrack Yun. Progres- sive random convolutions for single domain generalization. CoRR, abs/2304.00422 2023.
[82] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of of-distribution examples in neural networks. In International Conference on Learning Representations, 2017.
[83] Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations, 2018
[84] Kimin Lee, Honglak Lee, Kibok Lec, and Jinwoo Shin. Training confidence-calibraaee classifiers for detecting out-of-distribution samples. In International Conference on Learning Representations, 2018.
[85] Jingkang Yang, Pengyun Wang. Dejian Zou. Zitang Zhou, Kunyuan Ding, Wenxuan Peng. Haoqi Wang. Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwci Liu. Openood: Benchmarking generalized out-of-distributiin detection. In Advances in Neural Information Processing Systems, 2022.
[86] Zhilin Zhao, Longbing Cao, and Kun- Yu Lin. Revealing the distributional vulnerability of discriminators by implicit generators. IEFE Transactions on Pattern Analysis and Machine Intelligence, 45(7):8888- 8901, 2023.
[87] Zhilin Zhao, Longbing Cao, and Kun- Yu Lin. Supervision adaptation balancing in didstriibbutonn generaliza- tion and -fissiiibbutii detection. IEEE Transactions on Pattern Analysis and Machine Intelligence. pages 1-16, 2023.
14

---

[88] Zhilin Zhao and Longbing Cao. tions on Machine Learning Research, 2023.
Dual representation learning for out-of-distribution detection. Transac-
[89] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[90] Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. In European Conference on Computer Vision, 2018.
[91] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitatio networks. In JEEE Conference on Computer Vision and Puttern Recognition, 2018.
[92] Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[93] Fabio Maria Carlucci, Antonio D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[94] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCFI01: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/ 2012.
[95] Hildegard Kuchne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso A. Poggio, and Thomas Serre. HMDB: A large video database for human motion recognition. In IEEF International Conference on Computer Vision, 2.
[96] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The EPIC-KITCHENS dataset. In European Conference on Computer Vision, 2018.
[97] Joanna Materzynska, Guillaume Berger, Ingo Bax, and Roland Memisevic. The jester dataset: A large- scale video datasct of human gestures. In IEEE/CVF International Conference on Computer Vision Workshops, 2019.
[98] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2009.
[99] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward 7. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjic Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, 2019.
[100] Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[101] Jianfci Yang. Xiangyu Peng, Kai Wang, Zheng Zhu, Jiashi Feng, Lihua Xie, and Yang You. Divide to Adapt: Mitigating confirmation bias for domain adaptation of black-box predictors. In International Conference Learning Representations, 2023.
[102] Ziyi Zhang, Weikai Chen, Hui Cheng, Zhen Li. Siyuan Li, Liang Lin, and Guanbin Li. Divide and Contrast: Source-free domain adaptation via adaptive contrastive learning. In Advances in Neural Information Processing Systems, 2022.
[103] Yipeng Gao, Kun-Yu Yu Lin, Junkai Yan, Yaowei Wang, and Wei-Shi Zheng. AsyFOD: An asymmetric adaptation paradigm for few-shot domain adaptive object detection. JEEF/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2023.
[104] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(1)), 2008
[105] David L. Davies and Donald W. Bouldin. A cluster separation measure. IEEE Transactions Pattern Analysis and Machine Intelligence, 1(2):224 227, 1979.
[106] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual explanations from deep networks via gradicnt-based localization. International Journal of Computer Vision, 128(2)336-359, 2020.
15


Reviews:
[{"review_id": "QI2P59SeUy", "reviewer": "Reviewer_bwaF", "summary": "The paper presents STDN, a spatio-temporal diversification network designed for domain generalization. It introduces a spatial grouping module that effectively groups features from individual frames across different spatial frames. Additionally, a spatio-temporal relation module is proposed to model spatial-temporal correlations at multiple scales. The experiments demonstrate the network's good performance on three benchmarks.", "questions": "Please see my comments above.", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to follow\n- The results on four setup is comprehensive (I do have concerns on the results, pleas also read the next section)\n", "weaknesses": "\nRegarding the methodology:\n- The author asserts that domain-specific cues are crucial for achieving good generalization (L43-45). Subsequently, the paper states that spatial grouping is employed to enhance the diversity of spatial modeling. It is necessary to provide further justification as to why this diversity aids in learning domain-specific features instead of introducing noise.\n- The term \"domain-specific feature\" is frequently used in this paper; however, it is never explicitly defined. Moreover, the results fail to substantiate that the proposed method effectively learns these domain-specific features. Taking the basketball example into consideration, the backboard can be regarded as the domain-specific feature within the training set due to its construction, and the basketball itself might not be the case as you can kick the basketball, so what is the domain specific feature? should it be data driven or manually defined? When the paper claims that the proposed method can learn more representative features, supporting evidence must be provided. Currently, aside from visualization (which will be discussed later), there is a lack of evidence to support the theory that the proposed method effectively learns domain-specific features. \n- It is important to note that improved results do not necessarily demonstrate that the proposed method resolves the domain generalization problem. For instance, if a stronger backbone were employed, significantly better performance could be achieved under the same experimental conditions; however, this would not imply that the stronger backbone more effectively addresses the domain generalization issue. To this end, the most straight forward experiments would be comparing against the baselines use I3D backbone, as I3D learns spatio-teamporal feature without grouping and multi-scale.\n\nConcerning the results:\n- Firstly, it should be noted that the works listed for comparison in Table 1 are incomplete. The authors could easily find numerous works on UCF-HMDB that exhibit considerably better performance.\n- Given the substantial emphasis placed on spatial modeling and spatio-temporal modeling, it is crucial to compare the proposed method against previous works that utilize 3D backbones, such as CoMiX (refer to https://arxiv.org/pdf/2110.15128.pdf), which demonstrates superior performance. Additionally, it is important to include the set of baselines cited by this paper and establish a fair comparison, such as using ResNet101 as the backbone.\n- It is worth pointing out that there may be instances where Grad-CAM highlights the \"domain-specific\" features, yet the network makes incorrect classifications. Thus, Grad-CAM alone cannot serve as conclusive evidence for improved domain generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Overall clarification of our idea**\n\nWe would like to clarify that, our key idea is to discover diverse class-correlated cues in videos, aiming to *alleviate the overfitting of domain-specific cues* in the source domain, NOT focus on learning domain-specific cues, as stated in our introduction (Line 31-32, 43-48). As diverse class-correlated cues are discovered from the source domain, our STDN can leverage various types of cues for recognition in the target domain. Compared with previous models, the set of diverse class-correlated cues is more likely to include recognition cues that are invariant (shared) between the source and target domains, leading to better generalization in the unseen target domain.\n\n#### **Q1. Concern about noise**\n\nRE: First, as illustrated in the overall clarification, our key idea is to perceive diverse class-correlated cues (NOT focus on learning domain-specific cues) for video domain generalization. \n\nTo further justify the effectiveness of diverse features learned by our Spatial Grouping Module, we evaluate a trained STDN (4 spatial groups) by dropping features from a specific spatial group (STDN-$i$ denotes that the $i$-th group is dropped for each video). As shown in the table below (UCF->HMDB), dropping any one of the 4 spatial groups will cause performance degradation compared with the full STDN, which demonstrates that our learned diverse spatial features encode effective class-correlated information more than noise. \n\n| STDN-1 | STDN-2 | STDN-3 | STDN-4 | Full STDN |\n| :-: | :-: | :-: | :-: | :-: |\n| 59.7 | 59.5 | 59.7 | 59.1 | 60.2 |\n\n#### **Q2. About the domain-specific cues**\n\nRE: First, as illustrated in the overall clarification, our proposed STDN aims to perceive diverse class-correlated cues (NOT focus on learning domain-specific cues) for video domain generalization (VDG). \n\nSecond, we clarify the definition of domain-specific cues. In our work, domain-specific cues are learned from the data rather than manually defined, similar to [48, 19, 20, 21]. In principle, domain-specific cues are recognition cues that are associated with video categories in one domain (e.g., source) but do NOT have a correlation with the (identical) categories in another domain (e.g., target). Taking the EPIC-Kitchens-DG (EPIC) benchmark as an example, videos from different domains are recorded in different environments (native kitchens), and thus the domain-specific cues learned from an EPIC domain should be some cues from the environment that are statistically correlated with the video categories (e.g., specific decorations). \n\nOur work is motivated by that, previous video classification models are prone to overfit some domain-specific cues in the source domain (e.g., class-correlated contexts, as demonstrated by [19, 20, 21]), and thus this impairs the generalization performance in unseen target domains. Accordingly, we propose a diversity-based approach to tackle VDG. In Figure 5, we quantitatively demonstrate that our proposed STDN can improve the feature diversity, i.e., diverse information is encoded.  \n\n#### **Q3. Experiments based on I3D backbone**\n\nRE: The results based on I3D backbone on UCF->HMDB are shown in the below table, and all methods are implemented under the same augmentation setting. As shown in the table, our proposed STDN outperforms VideoDG (previous SOTA), which demonstrates the effectiveness of our proposed design. \n\n| Baseline (TRN [6]) | VideoDG [13] (SOTA) | STDN (Ours) |\n| :-: | :-: | :-: |\n| 68.0 | 68.7 | 72.1 |\n\n#### **Q4&Q5. Incomplete comparison on UCF-HMDB (Table 1)**\n\nRE: The works listed for comparison on UCF-HMDB are complete. The works you mentioned that have better performance (e.g., CoMix [41]) are designed for video domain **adaptation** (VDA), which is a different task from video domain **generalization** (VDG). In VDA, unlabeled videos from the target domain are accessible for training, while VDG (our setting) cannot access to any target videos. A comparison between VDA and VDG methods is not fair, since VDA methods can leverage extra target videos for training and naturally lead to better performance. \n\nAs for the comparison results using 3D backbone, please refer to Q3. \n\n#### **Q6. About the Grad-CAM visualization**\n\nRE: First, as illustrated in the overall clarification, our key idea is to perceive diverse class-correlated cues (NOT focus on learning domain-specific cues) for video domain generalization. \n\nThe Grad-CAM visualization in Figure 4 is a qualitative analysis to intuitively illustrate that our model can discover diverse class-correlated cues. We also conduct a quantitative analysis in Figure 5, and the results show that our proposed STDN can improve the feature diversity. Furthermore, the comparison experiments on three different benchmarks (Table 1&2) and ablation study (Table 3) quantitatively demonstrate the effectiveness of our proposed diversity-based modeling. In our reply to Q1, we quantitatively demonstrate that our model learns effective class-correlated cues more than noise. \n"}, {"review_id": "cfKLMbv62d", "reviewer": "Reviewer_JaYX", "summary": "The paper proposes Spatial-Temporal Diversification Network (STDN) for Video Domain Generalization (VDN). VDN is a new problem which is similar to video domain adaptation, but more challenging due to no unlabeled videos from target domain is provided. STDN is mainly designed into two modules: Spatial Grouping (soft-clustering) and Spatial-Temporal Relation (similar idea as TRN [6]). Experiments are done on 3 different benchmarks: UCF-HMDB, EPIC-Kitchens-DG, Jester-DG with good improvements over baselines. Written presentation is clear and mostly easy to read. ", "questions": "- Given the proposed method works well on VDG, it is convinced that STDN can learn diverse set of spatial and temporal features. It is natural to ask if STDN also work on generic video classification problems e.g, on Kinetics, Something-Something-v2? It will be great if it work, if not, where it falls short, it may give further insights for video understanding?\n\n- The Spatial-Temporal Relation Module shares some similarity with TRN [6], it would be nice to have a few sentences to compare and contrast.\n\n* minor comments:\n- citation format needs further correction, e.g., [77]", "limitations": "The reviewer does not foresee any potential negative social impact of this work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The motivation of the proposed method is clearly presented and experimental results are solid, i.e., good improvements over baselines.\n* Various ablations, qualitative analysis provide better understanding about the proposed method.\n* Written presentation is clear and easy to read and understand.", "weaknesses": "* Missing a direct comparison with [14], even though [14] may use additional audio modality, an attempt to compare with [14] may make the paper more solid.\n\n* Although the problem of general domain generalization has been studied recently, the video domain generalization is less explored, which can be either good (this paper and a few other [13,14] are among the first ones) or bad (the problem is too small with limited impact).\n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Q1. Comparison with AVRNA [14]**\n\nRE: We conduct experiments to compare our STDN with AVRNA on UCF->HMDB. Since AVRNA involves the audio modality but our work focuses on the RGB modality, we implement two variants of AVRNA for comparison as follows:\n\n(1) Hard Norm Alignment loss (HNA): apply the HNA loss (Eq. (4) of [14]) for normalizing video features. \n\n(2) Relative Norm Alignment loss (RNA): first divide the source domain into two subdomains following [R2, R3] (as the target domain is not accessible), and then apply the RNA loss (Eq. (3) of [14]) for aligning feature distributions of the two subdomains. \n\nWe implement these two variants based on TRN, and all of these experiments are conducted under the same augmentation setting. As shown by the table below, our STDN significantly outperforms the two variants of AVRNA on UCF->HMDB, which demonstrates the effectiveness of our model. \n\n| Baseline (TRN) | HNA | RNA | STDN (Ours) |\n| :-: | :-: | :-: | :-: |\n| 53.1 | 54.3 | 55.8 | 58.3 |\n\n[R2] Zhang et al.: Divide and Contrast: Source-free Domain Adaptation via Adaptive Contrastive Learning. NeurIPS 2022\n\n[R3] Yang et al.: Divide to Adapt: Mitigating Confirmation Bias for Domain Adaptation of Black-Box Predictors. ICLR 2023\n\n#### **Q2. Significance of studying video domain generalization**\n\nRE: In our opinion, video domain generalization (VDG) is a critical research area to develop robust video classification models capable of effectively generalizing to unseen test domains. The setting of VDG aligns closely with real-world applications, since models would face unfamiliar scenarios in practice. \n\nCompared with the widely studied general domain generalization that focuses on image data, VDG focuses on more complex video data with an extra time dimension. Therefore, VDG would suffer from large and complex domain shifts (e.g., variations of motion, unexpected absence or misalignment of short-term snippets), which cannot be addressed by general domain generalization methods. Therefore, advanced spatial-temporal modeling methods should be developed to address VDG. \n\nHoping to further the development of this field, our work designs two new benchmarks for evaluating VDG methods, with numerous reproduced baselines. As shown in Table 1&2, general domain generalization methods perform poorly in VDG. By contrast, our work proposes a diversity-based spatial-temporal modeling approach tailored to challenges of VDG, which achieves superior performance on all the three benchmarks. \n\n#### **Q3. Performance of generic video classification**\n\nRE: The results of our proposed STDN vs. TRN on the validation set of Something-Something-v2 (SSv2) are given in the table below. We use the same training recipe for both methods (e.g., resnet50 backbone, 5 segments, 80 epoch). As shown in the table, our STDN can obtain improvement over TRN in generic video classification. Together with our superior performance on video domain generalization (Table 1&2), we demonstrate that our STDN can effectively generalize to videos from unseen test domains and perform well in generic video classification. \n\n| Baseline (TRN [6]) | STDN (Ours) |\n| :-: | :-: |\n| 42.6 | 43.2 |\n\n#### **Q4. Discussion about TRN [6]**\n\nRE: Thank you, we will add discussion in our manuscript. \n\nOverall, our proposed Spatial-Temporal Relation Module (STRM) is different from TRN, although STRM is designed based on TRN. Our STRM is designed for video domain generalization (VDG), while TRN is for generic video classification. Specifically, our STRM proposes to perceive diverse class-correlated cues in videos by modeling multi-scale spatial-temporal dependencies, aiming to alleviate the overfitting of domain-specific cues in the source domain and make the model generalize better in unseen test domains. By contrast, TRN is a multi-scale temporal modeling module that does not address the unique challenges of VDG. \n\nTechnically, our STRM differs with TRN in two aspects: 1) our STRM models multi-scale spatial dependencies within individual frames, which enriches the spatial diversity; 2) we propose a relation discrimination loss to constrain that different temporal dependency modeling functions capture different temporal cues, which enriches the temporal diversity. By enriching the diversity in both space and time dimensions, our STRM discovers diverse class-correlated cues from source videos for effective generalization to unseen test domains. \n\n#### **Q5. Citation format**\nRE: Thank you, we will improve this. "}, {"review_id": "VbyvfRD9sh", "reviewer": "Reviewer_m6Ue", "summary": "The paper addresses the problem of video domain generalization for classification task. The core idea of the paper is to enhance the diversity in class-correlated cues both in spatial and temporal dimensions with the assumption that in this diverse pool, it is more likely to capture the domain-generalizable features. To capture diversity in spatial dimensions, a spatial grouping module is proposed which forms K integrated features representing K groups of features by aggregating spatial features in each group. Two different entropy-based losses are used to ensure diversity in spatial cues. Next, the paper learns spatial relation features by sampling the integrated feature from different space scales. These spatial relation features, limited to space only, are then leveraged to learn temporal relation features. To improve the effectiveness of temporal relation features, a relation discrimination loss is also used to avoid collapse of learned temporal relation feature. The overall loss for optimisation is composed of task-specific loss, two entropy losses and a relation discrimination loss. Experiments have been conducted on three different datasets and the results claim to achieve better performance than the existing method and other image-based DG baselines.", "questions": "The paper tackles the problem of generalizing video classification task to novel (unseen)domains. Overall, the paper presents interesting technical contributions aimed at capturing diverse spatial-temporal features across multiple time-scales and the results show that it the method is effective against the existing method and image-based baselines. However, there are some concerns/questions (as listed in the weaknesses 1-4) due to which my initial rating for the paper is ‘weak accept’. \n\nAlso:\n\nWhat 'model selection criterion' is used to report the results?\n\nWhat is the performance of the method in different types of domains shifts and how the performance varies as the domain shift increases?\n\nWhat would be the performance gain compared to baseline with a different backbone, such as vision transformer-based?", "limitations": "The supplementary material mentions that the work doesn't consider the multi-modal nature of video data as it contains different modalities such as, RGB data, optical flow, and audio.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Generalizing to novel domains for video modality is an important and challenging task and it carries several applications in real-world. Also, not much work has been done for video domain generalization.\n\n2) The idea of spatial-temporal relation feature is interesting to capture the diverse class-correlated cues in search of domain-invariant features in the video data.\n\n3) Results claim to demonstrate better performance against competing methods in all three datasets, including EPIC-kitchens-DG, UCF-HMDB, and Jester-DG.\n\n4) Ablation studies show the clear performance contribution of different components in the proposed method.", "weaknesses": "1) The analyses of spatial grouping using the tSNE in Fig. 6 is not very convincing. It is not very clear how the claim that the spatial grouping does the feature group of spatial features is justified in this diagram. It is important to better understand either by visualizing or some other quantification measures that what is the clustering ability of spatial grouping mechanism.\n\n2) It is not very clear how the diverse class-correlated cues in space and time, for which the spatial-temporal diversity module is developed,  are domain-invariant information that the paper claims to extract (L:6). Fig. 6 uses Grad-cam visualization to show the attention heatmaps but it is very general and can be applicable to any classification task.\n\n3) In table 1, the improvement from UCF to HMDB is not very encouraging over VideoDG [13]? The paper doesn’t discuss any potential reasons for this. In fact, the the performance improvement from UCF to HMDB is less than than VideoDG [13] if Mixstyle [67], which is an off-the-shelf component,  is not used in the overall framework.\n\n4) What is the performance of the method when only using MixStyle [67] for the datasets used in Table 3?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Q1. About the t-SNE visualization**\n\nRE: In Figure R1 (please see the PDF in the global response), we show an improved version of Figure 6, which includes a quantitative analysis of cluster separation. According to Figure R1, we qualitatively (t-SNE) and quantitatively (Davies-Bouldin Index) verify that our proposed Spatial Grouping Module (SGM) extracts spatial features with better clustering than the baseline (we use $K$-means for both methods to cluster spatial features for visualization). \nThe better clustering of our SGM indicates that, the features from different spatial groups encode different spatial information. \n\nIn addition, we use Grad-CAM visualization to qualitatively compare our SGM with the baseline. As shown in Figure R2, our SGM focuses on different class-correlated cues by using features from different spatial groups, while the baseline focuses on very similar regions across different groups. This demonstrates that our SGM can extract diverse class-correlated spatial cues. \n\n#### **Q2. How the proposed model extracts domain-invariant cues?** \nRE: We would like to clarify that, our proposed STDN is not an approach that explicitly extracts domain-invariant cues between the source and target domains, since target data are not accessible during training in video domain generalization (VDG). Accordingly, we propose an alternative approach for VDG. Our STDN perceives diverse class-correlated cues in the source domain, such that the model can leverage various types of cues for recognition in the target domain. The set of rich and diverse class-correlated cues is more likely to include recognition cues that are invariant (shared) across the source and target domains, compared with previous models. \n\nOur quantitative analysis in Figure 5 demonstrates that our model can improve the feature diversity, namely diverse information is encoded in our learned features. \nAnd, the state-of-the-art performance on three benchmarks (Table 1&2) and the ablation study (Table 3) demonstrates the better generalization in target domains, which indicates that our model effectively discovers domain-invariant cues across the source and target domains. \n\n#### **Q3. Comparison with VideoDG [13]**\n\nRE: The VideoDG method actually involves a strong data augmentation technique, i.e., Robust Adversarial Domain Augmentation (RADA). By dropping RADA from VideoDG and dropping MixStyle from our STDN, we conduct a fair comparison between our STDN and VideoDG. As shown in the table below, our STDN outperforms VideoDG on both UCF->HMDB and HMDB->UCF under the same augmentation setting. \n\n| | UCF->HMDB | HMDB->UCF |\n| :-: | :-: | :-: |\n| VideoDG w/o RADA [13] | 54.3 | 71.4 |\n| STDN w/o MixStyle (Ours) | 58.3 | 76.2 |\n\nIt is an interesting question that why VideoDG obtains higher improvement over the baseline on UCF->HMDB than other benchmarks. We conjecture that, it is because the adopted RADA augmentation well simulates the distribution shift from UCF to HMDB (especially with Adversarial Pyramid Network for temporal modeling, as shown by the ablation study in Table 1 of [13]). Even though VideoDG performs well on UCF->HMDB, our STDN outperforms it. Besides, our STDN obtains superior performance on other benchmarks, which verifies the effectiveness and versatility of our design.\n\n#### **Q4. Performance with only MixStyle [37]**\n\nRE: The performance of using only MixStyle is given in the following table. As shown in the table, our proposed STDN obtains significant improvement over MixStyle, which demonstrates the effectiveness of our proposed design.\n\n| | UCF->HMDB | HMDB->UCF |\n| :-: | :-: | :-: |\n| MixStyle | 55.7 | 73.5 |\n| Full STDN (Ours) | 60.2 | 77.1 |\n\n#### **Q5. Model selection criterion**\n\nRE: Following VideoDG [13], we conduct model selection according to the validation set of the source domain.\n\n#### **Q6. Analysis of different types of domain shifts**\n\nRE: We are delighted to talk about this interesting question. As demonstrated by the experiment results, our proposed STDN obtains substantial improvement over previous SOTAs under different domain shifts, e.g., environment change (EPIC-Kitchens-DG), subclass change (Jester-DG) and large illumination shift (HMDB->ARID, as shown in the Q2 of Reviewer SFFk). These results demonstrate that our method is a promising solution for video domain generalization. \n\nRegarding the performance variation at different levels of domain shifts, it is very challenging to analyze this by using existing video domain generalization benchmarks. We thank for your valuable idea for our future works, and we will attempt to quantitatively analyze this problem in the future (e.g., by constructing new benchmarks).\n\n#### **Q7. Results based on other backbones**\n\nRE: We conduct a comparison with TRN [6] and VideoDG [13] (previous SOTA) on UCF->HMDB using ViT-B/32 and I3D as backbones. As shown in the table below, our proposed STDN outperforms VideoDG based on both backbones, which demonstrates the effectiveness of our proposed STDN. \n\n| Backbone | TRN [6] | VideoDG [13] (SOTA) | STDN (Ours) | \n| :-: | :-: | :-: | :-: | \n|ViT-B/32 | 61.3 | 61.8 | 64.8 | \n| I3D | 68.0 | 68.7 | 72.1 |"}, {"review_id": "qV1BXMoDTi", "reviewer": "Reviewer_YkHU", "summary": "In this manuscript, the authors proposed a novel Spatial-Temporal Diversification Network (STDN) for video domain generalization.  More precisely, the proposed method introduces the Spatial Grouping Module and Spatial=Temporal Relation Module to discover various groups of spatial cues within individual frames and to model spatial-temporal dependencies. Experimental results on three benchmarks show the effectiveness of the proposed method.", "questions": "N.A.", "limitations": "N.A.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written and well-organized. \n\nThe proposed method achieves state-of-the-art results across three benchmarks.\n\nThe proposed method is straightforward and interesting. ", "weaknesses": "The Temporal Relation Module is derived from the work cited as [6]. It would be beneficial for the authors to acknowledge this in their manuscript.\n\nIn my opinion, the proposed spatial grounding module is similar to spatial attention and the KNN model. The authors are suggested to conduct an ablation study to compare these methods for a more thorough analysis.\n\nThe proposed method is founded on Temporal Segment Networks TSN). However, the comparative methods use ensembles with various backbones. It would be advisable for the authors to replicate this approach for a fairer comparison.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Q1. Discussion about TRN [6]**\n\nRE: Thank you, we will add discussion about this. \n\nAlthough our proposed Spatial-Temporal Relation Module (STRM) is founded on TRN, they are designed for different tasks with different motivations. Our STRM is designed for video domain generalization, while TRN is for generic video classification. Specifically, our key idea is to explicitly perceive diverse class-correlated cues in videos by modeling multi-scale spatial-temporal dependencies, aiming to alleviate the overfitting of domain-specific cues in the source domain and make the model generalize better in unseen test domains. By contrast, TRN is a multi-scale temporal modeling module that does not address the unique challenges of VDG. \n\nIn our STRM, we contribute two novel technical designs for perceiving diverse cues: 1) we model multi-scale spatial dependencies within individual frames, which enriches the spatial diversity; 2) we propose a relation discrimination loss to constrain that different temporal dependency modeling functions capture different temporal cues, which enriches the temporal diversity. By enriching the diversity in both space and time dimensions, our STRM discovers diverse class-correlated cues from source videos for effective generalization to unseen target domains. \n\n#### **Q2. Analysis of our proposed Spatial Grouping Module (SGM)**\n\nRE: We conduct further comparison experiments to demonstrate the effectiveness of our SGM. We compare our SGM with two methods as follows: \n\n(1)\tSpatial Attention (SA): following CBAM [R4], we use $K$ spatial attention modules on top of the backbone to extract $K$ features for subsequent feature modeling. \n\n(2)\t$K$-means Grouping (KG): we use $K$-means algorithm to partition spatial features into $K$ groups, and then the $K$ cluster centers are used as features for subsequent feature modeling. \n\nWe implement the two methods with our proposed Spatial-Temporal Relation Module (STRM). The results on UCF->HMDB are given in the table below, which shows that our SGM outperforms the two comparative methods. Our superiority is attributed to that we force the network to learn different types of features according to the guidance of our entropy-based losses, which constrain the distinction between different spatial groups. \n\n| STRM+SA | STRM+KG | STRM+SGM (Ours) |\n| :---: | :---: | :---: |\n| 56.0 | 55.5 | 58.3 |\n\n[R4] Woo et al.: CBAM: Convolutional Block Attention Module. ECCV 2018.\n\n#### **Q3. Comparison with methods using various backbones**\n\nRE: We would like to clarify that our comparison is fair, since we use ResNet50 as the backbone for both comparative methods and our STDN. In order to compare with general domain generalization methods in video domain generalization (VDG), we combine these methods with classical temporal modeling modules (i.e., TSN [5], TRN [6], TSM [7] and APN [13]) to adapt video data. Our proposed STDN has temporal modeling capabilities, and thus extra temporal modeling modules are needless. All comparative methods and our STDN is based on TSN, i.e., using the sparse temporal sampling strategy. \n\nIn addition, we conduct experiments based on other backbones. The following table shows a comparison between our STDN and two existing methods (TRN and VideoDG) on UCF->HMDB. Our superior performance demonstrates the effectiveness of our design. \n\n| Backbone | TRN | VideoDG [13] (SOTA) | STDN (Ours) | \n| :---: | :---: | :---: | :---: |\n| ViT-B/32 | 61.3 | 61.8 | 64.8 |\n| I3D | 68.0 | 68.7 | 72.1"}, {"review_id": "nSNsvpd4A0", "reviewer": "Reviewer_SFFk", "summary": "In this work, the authors propose a Spatial-Temporal Diversification Network (STDN) for video domain generalization. First, they intrdouce a spatial grouping method to summarize the spatial clues in each frame. Then,  they further build up spatial-temporal relations in a multi-scale manner. Finally, they show the effectiveness of the method via different experiments of video domain generalization.", "questions": "Please see the weakness section.", "limitations": "Please see the weakness section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The domain generalization problem is important for video understanding in practice.\n2. The paper is written in a good structure, basically.\n3. The experiments somehow show the effectiveness of the design.", "weaknesses": "1 Design.\n\nI am not quite convincing by the proposed design in the paper. Bascially, the spatial grouping (or clustering) or spatial-temporal relation modeling are not particularly designed for domain generalization. It could be used for the traditional video classification problem without any difficulty. Why are these designs important for domain generalization?  \n\n2 Experiment.\n\n2.1 The setting is not quite challenging, actually. The data sets bascially belong to the same domain. It would be interesting to see the cross domain setting, like action recognition from dark videos in UG2+ Challenge.\n\n2.2 It would be interesting to show the results of traditional video classification setting on the popular benchmarks, like Kinetics400 or Something-Something V1 or V2. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Q1. Why the proposed diversity-based modeling is important for video domain generalization (VDG)?**\n\nRE: Our model is designed for VDG, although it is applicable to traditional video classification. The core idea of our proposed designs, including our proposed Spatial Grouping Module and Spatial-Temporal Relation Module, is to perceive diverse class-correlated cues in videos. A detailed illustration to answer the question is as follows: \n\n(1) As illustrated in our manuscript (Line 31-32, Figure 1), previous video classification models usually suffer from the overfitting of some domain-specific cues in the source domain (these domain-specific cues are easy-to-fit, e.g., class-correlated contexts, as demonstrated by [19, 20, 21]). In VDG, videos from the source and target domains follow different distributions, which means that some class-correlated cues in the source domain may be unseen or not correlated with categories in the target domain. As a result, a model that overfits to the cues specific in the source domain would fail to generalize in the target domain. \n\n(2) Since the target domain is unseen, it is impossible to explicitly learn the invariant cues across the source and target domains from the data. Thus, we propose an alternative approach. \n\n(3) We propose to discover diverse class-correlated cues in the source domain, such that our model can leverage different types of cues for recognition in the target domain. The set of rich and diverse class-correlated cues is more likely to include recognition cues that are invariant (shared) across the source and target domains, compared with previous models that overfit domain-specific cues (as discuss in (1)). Thus, our proposed model can generalize better in the target domain. \n\nRegarding traditional video classification, it is a very different task from VDG. In traditional video classification, the training and test videos follow the same distribution (namely an identical domain), thus the class-correlated cues specific in the training domain would be effective in the test domain (this is very different from VDG). In our reply to Q3, we quantitatively demonstrate that our STDN can effectively generalize to videos from unseen test domains and perform well in traditional video classification. \n\n#### **Q2. Experiments on dark videos**\n\nRE: Our proposed STDN effectively generalizes to dark videos by learning from normal videos. Specifically, we conduct experiments on the challenging HMDB->ARID benchmark following ACAN [R5] (ARID is the dataset from UG2+ challenge which consists of dark videos). On HMDB->ARID, methods are implemented using the I3D backbone, and these experiments are conducted using the same augmentation setting. As shown by the table below, our proposed STDN outperforms VideoDG [13] (previous SOTA) on HMDB->ARID, which demonstrates the effectiveness of our model. \n\n| Baseline (TRN [6]) | VideoDG [13] (SOTA) | STDN (Ours) |\n| :-: | :-: | :-: |\n| 41.1 | 41.3 | 44.4 |\n\n[R5] Xu et al.: Aligning Correlation Information for Domain Adaptation in Action Recognition. TNNLS 2023. \n\n#### **Q3. Results on traditional video classification**\n\nRE: The results of our proposed STDN vs. TRN (the network that our model is founded on) on the validation set of Something-Something-v2 (SSv2) are given in the table below. We use the same training recipe for both methods (e.g., resnet50 backbone, 5 segments, 80 epoch). As shown in the table, our STDN can obtain improvement over TRN in traditional video classification. Together with our superior performance on video domain generalization (Table 1&2 in our main manuscript), we demonstrate that our STDN can effectively generalize to videos from unseen test domains and perform well in traditional video classification. \n\n| Baseline (TRN [6]) | STDN (Ours) |\n| :-: | :-: |\n| 42.6 | 43.2 |"}]

Author rebuttal:
Thanks to all reviewers for your constructive comments. We are encouraged that the reviewers found that, our work studies an important and practical problem (Reviewer m6Ue, SFFk) with a clear motivation (Reviewer JaYX), proposes a straightforward and interesting idea (Reviewer m6Ue, YkHU), presents comprehensive and solid experimental analysis or obtains state-of-the-art performance (Reviewer bwaF, JaYX, m6Ue, YkHU) and has good writing (Reviewer bwaF, JaYX, YkHU, SFFk). 

We have carefully addressed your concerns and provided detailed responses for each review.


Meta-review: The topic of video domain generalization is considered of interest, the approach is sufficiently novel, and the experiments are comprehensive and convincing. The author-responses to the remaining reviewer concerns have been well received and post-rebuttal all reviewers are supportive of acceptance. The AC agrees and encourages the authors to include all the promised additions and comparative experiments in the camera-ready paper.
Decision: Accept (poster)