SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Overall Training and Test: We adopt a video classification loss on top of the aggregated feature  $\tilde{z}$  given by  $L_{\mathrm{cls}} = \operatorname{CE}(F(\tilde{z}), y)$ , where  $F(\cdot)$  is a video classifier. Overall, the training loss of our
Table 1: Comparison with state-of-the-art methods on the UCF-HMDB benchmark. Red and blue denotes the best and second best. Results of all compared methods are from VideoDG [15].
Arch	DG Method	UCF→HMDB	HMDB→UCF	Arch	DG Method	UCF→HMDB	HMDB→UCF
TSN [5]	ERM	51.4±0.2	68.6±0.3	TSM [7]	ERM	52.2±0.3	69.2±0.3
\( ADA_{sem} \) [71]	51.1±0.3	68.2±0.2	\( ADA_{sem} \) [71]	51.3±0.3	68.6±0.3
\( ADA_{pixel} \) [71]	49.6±0.3	67.4±0.2	\( ADA_{pixel} \) [71]	52.7±0.3	68.3±0.2
M-ADA [92]	52.4±0.2	69.2±0.2	M-ADA [92]	52.5±0.2	69.1±0.3
Jigsaw [93]	51.5±0.3	68.5±0.3	Jigsaw [93]	52.5±0.3	68.9±0.3
APN [15]	ERM	54.3±0.3	71.4±0.3	TRN [6]	ERM	52.4±0.3	69.8±0.3
\( ADA_{sem} \) [71]	55.2±0.3	71.9±0.3	\( ADA_{sem} \) [71]	52.8±0.2	69.6±0.5
\( ADA_{pixel} \) [71]	56.9±0.2	72.2±0.3	\( ADA_{pixel} \) [71]	52.1±0.3	70.6±0.2
M-ADA [92]	55.6±0.3	71.5±0.3	M-ADA [92]	53.4±0.3	69.9±0.3
Jigsaw [93]	55.2±0.4	72.4±0.3	Jigsaw [93]	53.3±0.3	70.1±0.3
VideoDG [15]	59.1±0.3	74.9±0.3	STDN (Ours)	60.2±0.5	77.1±0.4
proposed STDN is given as follows:
$$
L = L _ {\mathrm {c l s}} + \lambda_ {\mathrm {e n t}} L _ {\mathrm {e m i n}} + \lambda_ {\mathrm {e n t}} L _ {\mathrm {e m a x}} + \lambda_ {\mathrm {r e l}} L _ {\mathrm {r e l}}, \tag {8}
$$
where  $\lambda_{\mathrm{ent}}$  and  $\lambda_{\mathrm{rel}}$  are hyperparameters for trade-off. Following the standard protocol [15], we use source videos for training and test the model on target videos for evaluation.
4.1 Benchmarks and Experimental Setups
To demonstrate the effectiveness and versatility of our proposed Spatial-Temporal Diversification Network (STDN), we adopt three benchmarks of different types for experiments, including two newly designed benchmarks, namely EPIC-Kitchens-DG and Jester-DG. For these two new benchmarks, we select video categories and construct domains following previous video domain adaptation works [49, 19]. We split the source video set into training and validation sets following previous source validation protocols [25, 15], i.e., a reasonable in-domain model selection scheme for better generalization ability in unseen target domains. We reproduce general domain generalization methods (cooperated with video classification architectures) and state-of-the-art video domain generalization methods for comparison. We report mean and standard deviation of accuracy over three random trials for all methods.
UCF-HMDB is the most widely used benchmark for cross-domain video classification [15, 18], which contains 3,809 videos of 12 overlapping sport categories shared by UCF101 [94] and HMD-B51 [95]. The videos in UCF101 are mostly captured from certain scenarios or similar environments, and the videos in HMDB51 are captured from unconstrained environments and different camera viewpoints. This benchmark includes two subtasks, i.e., UCF  $\rightarrow$  HMDB and HMDB  $\rightarrow$  UCF.
EPIC-Kitchens-DG is a cross-scene egocentric action recognition benchmark, which consists of 10,094 videos across 8 egocentric action classes from three domains (scenes), following Munro et al. [49]. The three domains of EPIC-Kitchens-DG (i.e., D1, D2, D3) correspond to three largest kitchens (i.e., P08, P01, P22) from the large-scale egocentric action recognition dataset EPIC-Kitchens-55 [96]. This benchmark includes six subtasks constructed from three domains.
Jester-DG is a cross-domain hand gesture recognition benchmark. We select videos from the Jester dataset [97] and construct two domains following Pan et al. [19]. The source (S) and target (T) domains contain 51,498 and 51,415 video clips across 7 categories, respectively. The videos in EPIC-Kitchens-DG and Jester-DG benchmarks are both hand-centric, but they are captured from different views, namely first-person and third-person views.
Implementation details: We use ResNet50 [89] pretrained on ImageNet [98] as the backbone for frame-level feature extraction following the standard video domain generalization protocol [15]. The backbone takes frames of size  $224 \times 224$  as input and outputs feature maps of size  $7 \times 7 \times 2048$ . We take  $N = 5$  frames for each video for temporal modeling. We set  $K = 4$ ,  $\tau = 0.5$ ,  $D_s = 192$  and  $D_t = 256$ .  $F(\cdot)$  is a linear classifier and  $F_{\mathrm{rel}}(\cdot)$  is an MLP classifier. All parameters are optimized using mini-batch SGD with a batch size of 32, a momentum of 0.9, a learning rate of
Table 2: Comparison with state-of-the-art methods on the EPIC-Kitchens-DG and Jester-DG benchmarks. Red and blue denotes the best and second best. Results of all compared methods are reproduced following their official implementations.
Arch	DG Method	Epic-Kitchens-DG	Jester-DG
D1→D2	D1→D3	D2→D1	D2→D3	D3→D1	D3→D2	Average	S→T
TSN [5]	ERM	33.6±0.6	31.3±0.5	32.5±0.9	36.1±1.2	31.7±0.5	40.2±0.7	34.2±0.4	47.5±0.7
Mixup [72]	33.6±0.2	29.1±0.3	31.2±0.7	36.5±0.4	33.0±0.5	39.7±0.1	33.9±0.3	47.8±0.5
IRM [62]	34.6±0.1	30.3±1.3	31.2±1.2	36.3±0.2	32.3±0.1	40.0±0.3	34.1±0.5	47.6±0.5
ADA [71]	33.8±0.2	30.3±0.9	31.1±1.4	36.4±0.6	33.3±0.4	40.1±0.8	34.2±0.6	47.6±0.3
COP [100]	34.7±0.1	29.7±1.6	31.5±0.8	36.7±0.5	31.5±0.1	40.3±0.5	34.1±0.5	47.3±0.9
TSM [7]	ERM	34.6±0.5	32.3±1.2	30.2±0.9	34.8±0.7	31.2±1.9	39.9±1.5	33.8±0.6	47.1±0.4
Mixup [72]	34.1±0.7	29.3±0.6	28.1±0.2	32.4±0.3	31.7±0.7	39.1±0.1	32.4±0.3	47.3±0.4
IRM [62]	34.3±0.2	29.3±1.5	31.1±0.8	36.3±0.7	31.6±1.0	38.5±0.5	33.5±0.7	46.8±0.7
ADA [71]	34.3±0.8	30.6±1.0	30.0±1.0	34.6±1.3	31.8±0.6	38.8±0.9	33.4±0.6	47.1±0.5
COP [100]	34.9±0.3	32.1±1.1	30.5±0.2	32.2±0.6	31.8±1.5	37.5±0.9	33.2±0.4	46.4±0.6
APN [15]	ERM	35.2±0.8	30.2±2.1	35.3±1.0	43.2±1.2	36.4±0.3	45.3±0.9	37.6±0.7	46.9±0.5
Mixup [72]	34.7±0.2	30.9±0.7	33.8±0.7	44.7±0.5	36.4±0.3	44.4±0.7	37.5±0.4	47.8±0.1
IRM [62]	33.6±0.5	28.5±1.6	34.3±0.4	42.8±0.8	34.6±0.6	44.0±0.7	36.3±0.5	47.0±0.3
ADA [71]	36.0±0.3	29.5±0.2	35.0±0.4	43.0±0.3	37.6±0.4	45.6±2.3	37.8±0.6	47.8±0.4
COP [100]	36.5±0.8	32.1±0.7	37.6±1.8	41.8±0.8	37.9±0.2	49.9±1.3	39.3±0.6	47.2±0.3
TRN [6]	ERM	36.8±1.4	32.1±1.2	34.2±1.0	44.7±0.5	37.6±0.9	48.9±0.1	39.1±0.7	46.2±0.4
Mixup [72]	37.5±1.0	31.2±1.6	35.3±1.3	43.2±1.4	39.0±0.6	48.1±0.2	39.0±0.6	46.7±0.1
IRM [62]	37.8±2.3	30.5±0.6	37.0±2.6	42.6±1.3	40.3±0.5	47.9±0.6	39.3±1.1	46.5±0.5
ADA [71]	38.4±1.0	30.4±1.2	35.9±1.3	41.2±0.6	38.8±0.5	47.5±1.1	38.7±0.8	46.1±0.4
COP [100]	36.8±1.1	34.2±2.7	35.8±1.1	39.6±2.4	38.1±0.5	49.0±2.0	38.9±1.1	47.8±0.5
VideoDG [15]	36.2±0.4	31.9±0.2	36.5±0.5	40.5±0.8	39.5±0.5	49.1±0.7	39.0±0.6	47.5±0.1
STDN (Ours)	40.5±0.5	38.6±0.1	38.5±2.6	44.0±1.3	40.4±1.6	47.2±1.2	41.6±1.0	49.8±0.4
1e-3 and a weight decay of 5e-4. By default, the trade-off hyperparameters are set as  $\lambda_{\mathrm{ent}} = 0.1$  and  $\lambda_{\mathrm{rel}} = 0.5$ . We adopt an efficient feature augmentation technique namely MixStyle [74] for simulating novel target domains during training. We modify the original MixStyle to adapt the video data, i.e., we calculate the mean and standard deviation across both space and time dimensions within each channel of each instance (instead of only space dimension for image data). All experiments are conducted by PyTorch [99] with four NVIDIA GTX 1080Ti GPUs. The code is released at https://github.com/KunyuLin/STDN/.
Comparison with State-of-the-arts: We compare our proposed STDN with two types of state-of-the-arts: 1) general domain generalization (DG) methods cooperated with different video classification architectures; 2) state-of-the-art video domain generalization methods. For the two newly designed benchmarks (i.e., Epic-Kitchens-DG and Jester-DG), we adopt five different types of general domain generalization methods for comparison, including Empirical Risk Minimization (ERM), Mixup [72], Invariant Risk Minimization (IRM) [62], Adversarial Data Augmentation (ADA) [71], Clip Order Prediction (COP) [100]. All results are summarized in Table 1 and 2. On all the three benchmarks, our STDN outperforms all the state-of-the-art methods. Specifically, our STDN obtains performance improvement by  $2.2\%$  and  $2.3\%$  on  $\mathrm{HMDB} \rightarrow \mathrm{UCF}$  and Epic-Kitchens-DG respectively, which is significant compared with previous state-of-the-arts. In addition, VideoDG [15] obtains lower performance than their proposed architecture APN [15] on Epic-Kitchens-DG and Jester-DG. By contrast, our superiority on three benchmarks of different types verifies the effectiveness and versatility of our proposed STDN, demonstrating the effectiveness of perceiving diverse spatial-temporal cues. In the supplemental material, we make an attempt to compare with two variants of Planamente et al. [16] to show our effectiveness, following some works in the domain adaptation field [101, 102, 103].
Ablation Study: We analyze the effects of each component in our proposed STDN, as shown in Table 3. Following the training scheme of TSN [5], we apply a classifier on top of the backbone as our baseline. By stacking our proposed Spatial Grouping Module (SGM) on top of the backbone, we obtain significant improvement over the baseline (i.e.,  $2.2\%$  on  $\mathrm{UCF} \rightarrow \mathrm{HMDB}$  and  $2.0\%$  on  $\mathrm{HMDB} \rightarrow \mathrm{UCF}$ ), demonstrating the effectiveness of extracting different types of spatial cues within individual frames. Then, by introducing the temporal de
Figure 4: Diversity analysis on UCF  $\rightarrow$  HMDB. We use the normalized Mean Square Error (MSE) to evaluate the feature diversity of variants of TRM, i.e., measuring difference between temporal relation features of different time scales. A higher value of normalized MSE indicates higher diversity.
Figure 5: T-SNE visualization of spatial features. For both the baseline and SGM, we cluster the set of spatial features into  $K = 4$  clusters by  $K$ -means before visualization. In the figure, dots stand for spatial feature vectors and stars stand for cluster centers, and different colors denote different clusters.
pendency modeling of our Spatial-Temporal Relation Module (denoted by TRM), we obtain  $1.8\%$  and  $1.4\%$  improvement on UCF  $\rightarrow$  HMDB and HMDB  $\rightarrow$  UCF, respectively. It should be noted that the relation discrimination loss  $L_{\mathrm{rel}}$  is an important part in temporal dependency
modeling, since we obtain very minor performance improvement without the loss in temporal dependency modeling. By introducing the full Spatial-Temporal Relation Module (STRM), we obtain  $3.4\%$  and  $2.3\%$  improvement over "Backbone+SGM" on UCF  $\rightarrow$  HMDB and HMDB  $\rightarrow$  UCF, respectively. These results demonstrate the effectiveness of modeling dependencies between various video cues in both
Table 3: Ablation study on UCF-HMDB.
Method	UCF→HMDB	HMDB→UCF
Backbone	52.7±0.3	71.9±0.3
+SGM	54.9±0.3	73.9±0.4
+TRM	56.7±0.2	75.3±0.4
+STRM	58.3±0.4	76.2±0.3
+MixStyle	59.3±0.3	76.6±0.2
Full STDN	60.2±0.5	77.1±0.4
space and time dimensions, which enriches the diversity in spatial-temporal relation modeling. Moreover, by introducing the feature augmentation technique MixStyle, we obtain further improvement. Finally, our full model aggregates diverse spatial-temporal features, leading to better generalization performance on both UCF  $\rightarrow$  HMDB and HMDB  $\rightarrow$  UCF.
Diversity Analysis: We make a quantitative analysis to the diversity of learned video features for our model. Specifically, we evaluate the difference between temporal relation features of different time scales, measured by the normalized Mean Square Error (MSE) between feature vectors. A higher value of normalized MSE indicates a large difference. As shown in Figure 4, without our relation discrimination loss  $L_{\mathrm{rel}}$ , learned temporal relation features at different time scales hold very small difference (implying feature collapse). By introducing  $L_{\mathrm{rel}}$ , our TRM improves the diversity, indicated by the higher MSE value. By introducing our Spatial Grouping Module, the diversity is further improved as various spatial cues are extracted from each frame. Moreover, by modeling spatial dependencies, our model further enlarges the difference between features across scales.
Analysis of Spatial Grouping: We make a qualitative analysis to the grouping process of our proposed Spatial Grouping Module (SGM). Specifically, we use t-SNE [104] for visualizing feature distributions of spatial features, and we adopt the model trained without our SGM as the baseline (admits average pooling to extract an integrated feature for each frame) for comparison. Also, we use the Davies-Bouldin Index<sup>1</sup> as a quantitative metric to measure the clustering performance, i.e., a lower value of the Davies-Bouldin Index indicates better separation between clusters. As shown by the qualitative and quantitative results in Figure 5, our SGM extracts spatial features with better cluster separation than the baseline, which is attributed to that our SGM enhances the distinction between features in different spatial groups. These results indicate that our proposed spatial grouping process forces the model to learn features encoding more different information. In the supplemental material, we also show Grad-CAM examples to qualitatively compare our SGM with the baseline.
<sup>1</sup>The Davies-Bouldin Index [105] measures a ratio between the intra-cluster distance and inter-cluster distance.
Source Domain (Training)
Target Domain (Test)
Figure 6: Grad-CAM visualization on UCF-HMDB. As shown in the figure, compared with the baseline, our proposed STDN captures more diverse class-correlated cues in the source domain, i.e., including domain-specific backboards and domain-invariant basketballs. As a result, our proposed STDN generalizes better in the target domain, where backboards are invisible and thus our STDN uses the basketball for recognition instead. Best viewed in color.
Grad-CAM Visualization: We compare our proposed STDN with a TRN [6] model (the baseline) by Grad-CAM [106]. As shown in Figure 6, the baseline prefers to use the domain-specific backboard for recognition, which causes recognition errors in the target domains as backboards are invisible. In contrast to the baseline, our proposed STDN perceives more diverse class-correlated cues from the source domain, including some domain-invariant cues such as basketballs. As a result, our STDN can predict the correct video class by recognizing the basketball in the target video. These results demonstrate that, our proposed diversity-based approach can discover some potential domain-invariant cues, which alleviates the overfitting to domain-specific cues and leads to better generalization in the target domain.
In this work, we propose to explore spatial-temporal diversity to address the video domain generalization task. Our proposed Spatial-Temporal Diversification Network learns diverse spatial-temporal features in videos, which discovers potential domain-invariant cues and thus alleviates the heavy reliance on domain-specific cues. We conduct extensive quantitative and qualitative experiments on three benchmarks (including two newly designed benchmarks), and the results demonstrate the effectiveness and versatility of our approach.
Acknowledgements. This work was supported partially by the NSFC (U21A20471,U1911401), Guangdong NSF Project (No. 2023B1515040025, 2020B1515120085). The authors would like to thank Zhilin Zhao, Yi-Xing Peng, and Yu-Ming Tang for their valuable suggestions on model design or writing.
[1] Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3D convolutional networks. In IEEE International Conference on Computer Vision, 2015.
[2] João Carreira and Andrew Zisserman. Quo Vadis, Action Recognition? A new model and the kinetics dataset. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[3] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[4] Xiaolong Wang, Ross B. Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[5] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaou Tang, and Luc Van Gool. Temporal Segment Networks: Towards good practices for deep action recognition. In European Conference on Computer Vision, 2016.
[6] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In European Conference on Computer Vision, 2018.