SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kun-Yu Lin $^{1*}$
Jia-Run  $\mathbf{D}\mathbf{u}^{1*}$
Yipeng Gao
Jiaming Zhou
Wei-Shi Zheng $^{1,2\dagger}$
$^{1}$ School of Computer Science and Engineering, Sun Yat-sen University, China
$^{2}$ Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China
{linky5,dujr6,gaoyp23,zhoujm55}@mail2.sysu.edu.cn
wszheng@ieee.org
Video domain generalization aims to learn generalizable video classification models for unseen target domains by training in a source domain. A critical challenge of video domain generalization is to defend against the heavy reliance on domain-specific cues extracted from the source domain when recognizing target videos. To this end, we propose to perceive diverse spatial-temporal cues in videos, aiming to discover potential domain-invariant cues in addition to domain-specific cues. We contribute a novel model named Spatial-Temporal Diversification Network (STDN), which improves the diversity from both space and time dimensions of video data. First, our STDN proposes to discover various types of spatial cues within individual frames by spatial grouping. Then, our STDN proposes to explicitly model spatial-temporal dependencies between video contents at multiple space-time scales by spatial-temporal relation modeling. Extensive experiments on three benchmarks of different types demonstrate the effectiveness and versatility of our approach.
Recently, advanced deep network architectures have achieved competitive results for video classification [1, 2, 3, 4, 5, 6, 7, 8], leading to wide applications in surveillance systems, sport analysis, health monitoring, etc [9, 10, 11]. However, existing video classification models rely on the i.i.d. assumption, i.e., training and test videos are independently and identically distributed. This assumption would be easily violated, since models often face unfamiliar scenarios in real-world applications. For example, a housework robot will work in a new house, and a surveillance system will encounter illumination change caused by camera viewpoint or weather [12, 13, 14]. Holding such an assumption, the performance of video classification models would drop significantly in unfamiliar test scenarios.
To alleviate the above problem, our work studies the video domain generalization task, which aims to learn a video classification model that is generalizable in unseen target domains by training in a source domain [15, 16]. In this task, videos from the source and target domains follow different distributions though with an identical label space. For example, as shown in Figure 1, humans in the source domain play basketball shooting on indoor basketball courts while those in the target domain play outdoors. Different from the video domain adaptation task with available unlabeled target videos for training [17, 18, 19, 20], video domain generalization can only access the source domain during training, which is much more challenging but more practical.
Figure 1: Video classification models suffer from the misguidance of domain-specific cues when generalizing to unseen domains. As shown in the figure, in the source domain, the static backboard provides a clearer cue compared with the blurred basketball in motion, thus prevailing video classification models are prone to recognize the class "Shoot Ball" by the backboard. However, the backboard is invisible in the target domain due to viewpoint change, and thus previous models learned in the source domain would make mistakes in recognition. Videos in the figure are from the UCF-HMDB benchmark. Best viewed in color.
A critical challenge of video domain generalization is to defend against the reliance on domain-specific cues in the source domain that are correlated with class labels. For example, as shown in Figure 1, video classification models prefer to leverage the backboard for recognizing the class "shoot ball" in the source domain, since the static backboard provides a clearer cue compared with the blurred basketball in motion (static patterns are usually easy-to-fit [21, 22, 23, 24]). However, in the target domain, the backboard is occluded due to the viewpoint, thus recognizing the class by the backboard would cause recognition errors. It is challenging to address this problem in lack of any knowledge of the target domain. Typically, existing works explore invariance across domains for learning generalizable video features [25, 15, 16]. For example, Yao et al. propose to learn generalizable temporal features by encoding information of local features into global features, assuming that local temporal features are more invariant across domains compared with global ones [15].
In this work, we propose a novel approach for video domain generalization, which explores spatial-temporal diversity in videos. Our approach aims to perceive diverse class-correlated cues from abundant video contents, and thus we would leverage not only easy-to-fit domain-specific cues but also other potential domain-invariant cues for recognizing videos in target domains (e.g., we expect that our model can capture not only static backboards but also dynamic basketballs in the source domain). As a result, our approach can alleviate the overfitting of domain-specific cues in the source domain and generalize better in target domains by leveraging those potential domain-invariant cues. Specifically, we propose to explore the diversity from both space and time dimensions of video data, leading to a novel architecture named Spatial-Temporal Diversification Network (STDN). Our contributions are summarized as follows:
- We propose Spatial Grouping Module to discover various groups of spatial cues within individual frames by embedding a clustering-like process, enriching the diversity from a spatial modeling perspective.
- We propose Spatial-Temporal Relation Module to explicitly model spatial-temporal dependencies between video contents at multiple space-time scales, enriching the diversity from a spatial-temporal relation modeling perspective.
- Extensive experiments are conducted on three benchmarks of different types, including two newly designed benchmarks, and the results demonstrate the effectiveness and versatility of our proposed method.
Video Classification aims to recognize actions or events in videos. Recently, many advanced deep learning architectures have been proposed for video classification. 3D CNNs extend the 2D convolution to 3D convolution for video feature learning [1, 2, 3, 26, 27, 28, 29, 30]. Another
type of models first applies 2D convolution for frame-level spatial modeling and then conducts temporal modeling based on frame features [5, 6, 31, 32, 33]. Some works propose to couple explicit shifts along the time dimension for efficient temporal modeling [7, 34, 35]. More recently, pioneer works have made efforts in adapting Vision Transformer [36] for video classification [37, 38, 39, 40, 41, 42, 43, 44, 45]. Although these advanced architectures achieve appealing performance, they usually assume an identical test distribution to the training one, which is not practical in real-world applications.
Video Domain Generalization aims to train video classification models in a source domain for generalizing to unseen target domains. With target videos inaccessible during training, existing works usually assume different types of invariance across domains to defend against the reliance on domain-specific cues [25, 15, 16]. For example, Yao et al. propose to learn generalizable temporal features according to an assumption from empirical findings, i.e., local temporal features are more invariant across domains compared with the global ones [15]; Planamente et al. propose to constrain a consistency across visual and audio modalities by relative norm alignment for addressing domain generalization of egocentric action recognition [16]. In this work, we propose to perceive diverse class-correlated spatial-temporal cues in source videos, which alleviates the misguidance of domain-specific cues in a way that is orthogonal to previous works.
Video Domain Adaptation aims to learn transferable video classification models for a label-free target domain by transferring knowledge from a label-sufficient source domain [17, 18]. Different from video domain generalization, video domain adaptation is oriented to a specific seen unlabeled target domain. Typically, existing works learn invariance between labeled source videos and unlabeled target videos to tackle video domain adaptation. A class of representative works propose to learn domain-invariant temporal features by designing temporal modeling modules [18, 19, 46, 47]. In addition, Choi et al. [20, 48] propose self-supervised methods adaptive to video data. Furthermore, multi-modal works explore information interaction between different modalities (e.g., RGB, Flow, Audio) for domain-invariant feature learning [49, 50, 51, 52, 53].
General Domain Generalization, also known as out-of-distribution generalization, studies learning models generalizable to out-of-distribution data for the image classification task. In recent years, a plethora of methods have been proposed to address domain generalization [25, 54, 55, 56]. Prevailing methods are mainly based on feature alignment [57, 58, 59], domain adversarial learning [60, 61], invariant risk minimization [62, 63, 64, 65], meta learning [66, 67, 68, 69, 70], data augmentation [71, 72, 73, 74, 75], etc. In addition, ensemble learning is an effective way to learn generalizable models [76, 77, 78]. And recently, Zhu et al. develop a theory showing that ensemble learning can provably improve test accuracy by discovering the "multi-view" structure of data [79], which partially inspires our approach. Among architecture-based methods [80, 81], Meng et al. propose to redesign attention modules for learning diverse task-related features [80]. Different from existing general domain generalization methods, we propose a domain generalization method specific to video classification, which explores diverse class-correlated information in intrinsic space and time dimensions of video data. There are some works that study the identification of out-of-distribution data of different categories from training data [82, 83, 84, 85, 86, 87, 88], but this topic is not within the scope of our work.
3 Spatial-Temporal Diversification Network
In this section, we illustrate our proposed Spatial-Temporal Diversification Network (STDN) in detail, which perceives diverse class-correlated spatial-temporal cues from video contents for generalization in unseen target domains.
In video domain generalization, a set of labeled videos  $\mathcal{D} = \{(x,y)\}$  from a source domain are given for training, where  $x\in \mathcal{X}$  and  $y\in \mathcal{V}$  denote a source video and its corresponding class label. Given only the source video set, the goal of video domain generalization is to learn a video classification model that is generalizable in unseen target domains. The source and target domains follow different but related distributions with the same label space  $\mathcal{Y} = \{0,1,\dots,C - 1\}$ , where  $C$  denotes the number of video classes. Following the standard video domain generalization setting [15], each video is evenly divided into  $N$  segments, and one frame is sampled from each segment as the model input
Figure 2: An overview of our proposed Spatial-Temporal Diversification Network (STDN). We use a video of  $N = 5$  segments with  $K = 4$  spatial groups for example. After backbone feature extraction, our STDN extracts spatial cues of  $K$  types for each frame by the Spatial Grouping Module, enriching the diversity in spatial modeling. Then, our STDN explicitly models spatial-temporal dependencies at multiple space-time scales, enriching the diversity in spatial-temporal relation modeling. Best viewed in color.
during training and testing, i.e.,  $x = \{x_{1}, x_{2}, \ldots, x_{N}\}$  and  $x_{n}$  denotes the  $n$ -th sampled frame from the  $n$ -th segment.
Aiming at generalization in unseen target domains, our idea is to perceive rich and diverse class-correlated cues in the source domain. In this way, our model would leverage not only easy-to-fit domain-specific cues but also other potential domain-invariant cues for recognizing videos in the target domain, alleviating the misguidance of domain-specific cues. Considering the intrinsic space and time dimensions of video data, we propose to explore the diversity in both spatial and temporal modeling. An overview of our proposed STDN is shown in Figure 2. Firstly, given the video  $x$ , our STDN takes  $N$  sampled frames as input and separately extracts  $N$  spatial feature maps  $\{z_1, z_2, \dots, z_N\}$  by the backbone (e.g., ResNet [89]), where  $z_n \in \mathbb{R}^{H \times W \times D}$  denotes the feature map of the  $n$ -th frame,  $D$  denotes the feature dimension and  $H \times W$  denotes the size of feature maps. Then, we extract spatial cues of  $K$  types (groups) from each spatial feature map by our proposed Spatial Grouping Module, aiming to enrich the spatial diversity. In the Spatial Grouping Module, two entropy-based losses are introduced to enhance the distinction between different spatial cues. On top of the Spatial Grouping Module, we propose to explicitly model spatial-temporal dependencies between video contents at multiple space-time scales by our proposed Spatial-Temporal Relation Module. The learning of the Spatial-Temporal Relation Module is guided by a relation discrimination loss, which ensures the diversity of the extracted spatial-temporal relation features. Finally, diverse spatial-temporal features are aggregated for video domain generalization.
3.3 Spatial Grouping Module
Our proposed Spatial Grouping Module aims to discover diverse class-correlated spatial cues from abundant contents of individual frames, which enriches the diversity from a spatial modeling perspective for video domain generalization. Our Spatial Grouping Module extracts various spatial cues of different types by partitioning features from different spatial positions into several groups within individual frames. In this way, our Spatial Grouping Module discovers more diverse spatial cues, compared with prevailing approaches that extract an integrated feature for each frame (e.g., by average pooling).
As shown in Figure 3 (a), given the spatial feature map  $z_{n} \in \mathbb{R}^{H \times W \times D}$  of the  $n$ -th frame, our proposed Spatial Grouping Module learns to extract  $K$  spatial cues by aggregating the  $HW$  spatial features. Specifically, the proposed spatial grouping process is conducted based on  $K$  learnable anchor features  $\{a_{n,1}, a_{n,2}, \ldots, a_{n,K}\}$ , where  $a_{n,k} \in \mathbb{R}^D$  denotes the anchor feature of the  $k$ -th spatial group for the  $n$ -th frame. Then, we calculate the probability of assigning a spatial feature to each spatial group, which is formulated as follows:
$$
p _ {n, i, k} = \frac {\exp \left(- \operatorname {d i s t} \left(z _ {n , i} , a _ {n , k}\right) / \tau\right)}{\sum_ {j = 1} ^ {K} \exp \left(- \operatorname {d i s t} \left(z _ {n , i} , a _ {n , j}\right) / \tau\right)}, \tag {1}
$$
Figure 3: Overviews of our proposed (a) Spatial Grouping Module and (b) Spatial-Temporal Relation Module. Best viewed in color.
where  $z_{n,i} \in \mathbb{R}^D$  denotes the  $i$ -th spatial feature in the feature map  $z_{n}$  ( $i \in [1,2,\dots,HW]$ ),  $\mathrm{dist}(\cdot, \cdot)$  denotes the Euclidean distance metric and  $\tau$  is the temperature factor. According to Eq. (1), if the spatial feature  $z_{n,i}$  is closer to the anchor feature  $a_{n,k}$ , then the  $z_{n,i}$  will be assigned to the  $k$ -th spatial group with higher probability. After group partition, our Spatial Grouping Module produces  $K$  integrated features representing  $K$  different spatial cues by aggregating spatial features in each group. The integration process is formulated as follows:
$$
z _ {n, k} ^ {s} = \frac {1}{\sum_ {i = 1} ^ {H W} p _ {n , i , k}} \sum_ {i = 1} ^ {H W} p _ {n, i, k} * z _ {n, i}, \tag {2}
$$
where  $z_{n,k}^{s}$  denotes the spatial cues integrated from the  $k$ -th group within the  $n$ -th frame.
In order to extract spatial cues of diverse types, we introduce two entropy-based losses to enhance the distinction between different spatial groups. The first one is an entropy minimization loss to enhance the confidence of group assignment for each spatial feature. The loss is formulated as follows:
$$
L _ {\text {e m i n}} = - \frac {1}{N H W} \sum_ {n = 1} ^ {N} \sum_ {i = 1} ^ {H W} \sum_ {k = 1} ^ {K} p _ {n, i, k} \log \left(p _ {n, i, k}\right). \tag {3}
$$
For the assignment probability vector  $p_{n,i} = [p_{n,i,1},p_{n,i,2},\dots ,p_{n,i,K}]^T\in \mathbb{R}^{K\times D}$ , if the entropy is minimized, then the feature  $z_{n,i}$  will be confidently assigned to a specific spatial group. The second loss is an entropy maximization loss for the mean assignment probability vector, which guarantees that those  $HW$  spatial features are assigned to different spatial groups. Specifically, the loss is formulated as follows:
$$
L _ {\mathrm {e m a x}} = \frac {1}{N} \sum_ {n = 1} ^ {N} \sum_ {k = 1} ^ {K} \bar {p} _ {n, k} \log (\bar {p} _ {n, k}), \tag {4}
$$
where  $\bar{p}_{n,k} = \frac{1}{HW}\sum_{i = 1}^{HW}p_{n,i,k}$  denotes the mean probability of assigning features to the  $k$ -th group within the  $n$ -th frame. For the mean assignment probability vector  $\bar{p}_n = [\bar{p}_{n,1},\bar{p}_{n,2},\dots,\bar{p}_{n,K}]^T\in$ $\mathbb{R}^{K\times D}$ , if the entropy is maximized, then the spatial features  $\{z_{n,i}\}$  will be uniformly assigned to  $K$  spatial groups. By using the two entropy-based losses, we guarantee that spatial features are different from each other across different spatial groups, enriching the diversity of extracted spatial cues.
In the Spatial Grouping Module, the learnable anchor feature for each group is extracted by weighted combining those  $HW$  spatial features, and the weights are calculated conditioned on the feature map  $z_{n}$  by using a lightweight two-layer convolutional network. In this way, the spatial grouping process can be regarded as conducting clustering over spatial features within individual frames. All involved parameters in the module are end-to-end trained together with the main network, i.e., we contribute a parametric clustering module to group spatial features for improving the spatial diversity.
3.4 Spatial-Temporal Relation Module
Our proposed Spatial-Temporal Relation Module aims to discover diverse class-correlated spatial-temporal cues from abundant video contents, which enriches the diversity from a spatial-temporal relation modeling perspective for video domain generalization. As demonstrated by previous works [4, 6, 90], there are rich dependencies between entities over space and time dimensions in videos, which
is crucial for video classification. Accordingly, we propose to explicitly model spatial-temporal dependencies between video cues at multiple space-time scales. Our proposed Spatial-Temporal Relation Module conducts dependency modeling at space and time dimensions separately, and an overview of the module is shown in Figure 3 (b).
First, based on the spatial cues extracted by our Spatial Grouping Module, we conduct spatial dependency modeling between these spatial cues at multiple space scales. Specifically, given the representations of spatial cues  $z_{n}^{s} = [z_{n,1}^{s},z_{n,2}^{s},\ldots ,z_{n,K}^{s}]^{T}\in \mathbb{R}^{K\times D}$  for the  $n$ -th frame, we extract the spatial relation feature at the  $l$ -th space scale by the spatial dependency modeling function  $R_{l}^{s}(\cdot)$  as follows:
$$
R _ {l} ^ {s} \left(z _ {n} ^ {s}\right) = \mathbb {E} _ {k _ {1}, k _ {2}, \dots , k _ {l}} \left[ H _ {l} ^ {s} \left(z _ {n, k _ {1}} ^ {s}, z _ {n, k _ {2}} ^ {s}, \dots , z _ {n, k _ {l}} ^ {s}\right) \right] \in \mathbb {R} ^ {D _ {s}}, \tag {5}
$$
where  $\mathbb{E}[\cdot ]$  denotes the expectation calculation and  $H_{l}^{s}(\cdot ,\cdot ,\dots ,\cdot)$  denotes a linear projection function after feature concatenation. The index set  $\{k_1,k_2,\ldots ,k_l\}$  denotes the index of spatial features uniformly sampled from the  $K$  spatial features, where the index  $l\in \{2,3,\ldots ,K\}$  indicates the space scale,  $k_{1}\neq k_{2}\neq \dots \neq k_{l}$  and  $k_{i}\in \{1,2,\ldots ,K\}$ . For each frame, we extract  $K - 1$  spatial relation features by dependency modeling at  $K - 1$  space scales separately. And, we concatenate these spatial relation features and produce an integrated feature for each frame, which is given by  $\hat{z}_n = [R_2^s (z_n^s)^T,R_3^s (z_n^s)^T,\dots ,R_K^s (z_n^s)^T,G(z_n)^T ]^T\in \mathbb{R}^{KD_s}$ . In the integrated feature  $\hat{z}_n$ ,  $G(z_{n})\in \mathbb{R}^{D_{s}}$  denotes the global feature extracted from the feature map  $z_{n}$  by a convolution layer.
Second, based on the frame-level integrated features, we conduct temporal dependency modeling between frames at multiple time scales. Specifically, given  $N$  frame-level features denoted by  $\hat{z} = [\hat{z}_1,\hat{z}_2,\dots ,\hat{z}_N]$ , we extract the temporal relation feature at the  $m$ -th time scale by the temporal dependency modeling function  $R_{m}^{t}(\cdot)$  as follows:
$$
R _ {m} ^ {t} (\hat {z}) = \mathbb {E} _ {n _ {1} <   n _ {2} <   \dots <   n _ {m}} \left[ H _ {m} ^ {t} \left(\hat {z} _ {n _ {1}}, \hat {z} _ {n _ {2}}, \dots , \hat {z} _ {n _ {m}}\right) \right] \in \mathbb {R} ^ {D _ {t}}, \tag {6}
$$
where  $H_{m}^{t}(\cdot ,\cdot ,\dots ,\cdot)$  denotes a linear projection function after feature concatenation. The index set  $\{n_1,n_2,\ldots ,n_m\}$  denotes the index of frame features randomly sampled from the  $N$  frame features, where the index  $m\in \{2,3,\ldots ,N\}$  indicates the time scale and  $n_i\in \{1,2,\dots ,N\}$ . Note that we keep the relative order of sampled frames for temporal modeling. By using  $N - 1$  temporal dependency modeling functions, we extract  $N - 1$  temporal relation features at  $N - 1$  time scales for each video.
To ensure the diversity of temporal relation features, we propose a relation discrimination loss to constrain that different temporal dependency modeling functions (i.e., different time scales) capture different temporal cues. This loss constrains that a relation classifier can distinguish one relation feature from not only relation features of other classes but also relation features of the same class but of other time scales. Thus, it avoids the feature collapse of learned temporal relation features. Specifically, the loss is formulated as follows:
$$
L _ {\text {r e l}} = \frac {1}{N - 1} \sum_ {m = 2} ^ {N} \mathrm {C E} \left(F _ {\text {r e l}} \left(\tilde {z} _ {m}\right), \tilde {y} _ {m}\right), \tag {7}
$$
where  $\tilde{z}_m = R_m^t (\hat{z})$  denotes the temporal relation feature at the  $m$ -th time scale,  $F_{\mathrm{rel}}(\cdot)$  denotes a relation classifier that classifying  $(N - 1)*C$  classes, and  $\mathrm{CE}(\cdot ,\cdot)$  denotes the cross-entropy loss. The  $\tilde{y}_m$  denotes the relation label of the video  $x$  with label  $y$ , i.e.,  $\tilde{y}_m = y*(N - 1) + (m - 2)\in \{0,1,2,\ldots ,(N - 1)*C - 1\}$ . In this way, the loss forces different temporal dependency modeling functions to capture different class-correlated temporal cues in the video since the captured temporal cues are discriminative across scales. By incorporating the Spatial-Temporal Relation Module with the relation discrimination loss  $L_{\mathrm{rel}}$ , we extract rich and diverse spatial-temporal cues.
Feature Aggregation: After exploring spatial-temporal diversity by our proposed Spatial Grouping Module and Spatial-Temporal Relation Module, our model discovers diverse class-correlated spatial-temporal cues from abundant video contents. Then, we aggregate these diverse spatial-temporal features for video classification. Specifically, the feature aggregation is formulated as  $\check{z} = \sum_{m=2}^{N} H_m^a(\tilde{z}_m)$ , where  $H_m^a(\cdot)$  denotes a small SE-based block [91] for modulating the  $m$ -th temporal relation features.