SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dongho Lee*
Jongseo Lee*
Jinwoo Choi†
Kyung Hee University, Republic of Korea
{kide004,jong980812，jinwoochoi}@khu.ac.kr
Recognizing human actions in videos requires spatial and temporal understanding. Most existing action recognition models lack a balanced spatio-temporal understanding of videos. In this work, we propose a novel two-stream architecture, called Cross-Attention in Space and Time (CAST), that achieves a balanced spatio-temporal understanding of videos using only RGB input. Our proposed bottleneck cross-attention mechanism enables the spatial and temporal expert models to exchange information and make synergistic predictions, leading to improved performance. We validate the proposed method with extensive experiments on public benchmarks with different characteristics: EPIC-KITCHENS-100, Something-Something-V2, and Kinetics-400. Our method consistently shows favorable performance across these datasets, while the performance of existing methods fluctuates depending on the dataset characteristics. The code is available at https://github.com/KHU-VLL/CAST.
To accurately recognize human actions in videos, a model must understand both the spatial and temporal contexts. A model that lacks fine-grained spatial understanding is likely to fail in predicting the correct action. For example, as shown in Figure 1 (a), a model that understands temporal context such as hand motion across frames but not the fine-grained spatial context may confuse whether an object in the hand a ketchup, or a cheese, or a milk carton. Consequently, the model fails to predict the correct action, Put down a cheese. Similarly, a model that lacks temporal context understanding may also fail to predict the correct action. In Figure 1 (b), let us suppose a model understands spatial context but does not understand temporal context, e.g., the model is confused about whether the hand is moving from outside the fridge to the inside or vice versa. Then the model fails to predict the correct action of Take out a sauce. Therefore, for accurate action recognition, models need to comprehend both the spatial and temporal contexts of videos.
Despite the recent progress in action recognition through the use of Transformers [60, 11, 3], achieving a balanced spatio-temporal understanding remains a challenging problem. Compared to images, the additional temporal dimension in videos makes spatio-temporal representation learning computationally intensive and requires a significant amount of training data [3]. Consequently, most action recognition models lack a balanced spatio-temporal understanding of videos. Notably, models that perform well on static-biased [32, 8, 50] datasets, such as Kinetics-400, may not perform as well on temporal-biased [3, 28] datasets, such as Something-Something-V2, and vice versa. For instance, as shown in Figure 4 (a), on the EPIC-KITCHENS-100 dataset, VideoMAE [56] outperforms ST-Adapter [42] on the verb prediction task, while ST-Adapter outperforms VideoMAE on the noun prediction task. Similarly, BEVT [62] outperforms AIM [72] on the Something-Something-V2
*Equally contributed first authors.
Figure 1: The importance of spatio-temporal understanding. If a model lacks fine-grained spatial understanding, the model may predict an incorrect action. E.g., the model fails to predict Put down a cheese in (a) due to subtle appearance differences between the objects. On the other hand, if a model lacks temporal context understanding, the model may predict an incorrect action. E.g., the model fails to predict Take out a sauce in (b) due to the ambiguity of the action. Therefore, both spatial and temporal understanding are crucial in action recognition. Best viewed with zoom and color.
dataset, while BEVT underperforms AIM on the Kinetics-400 dataset. We observe a similar trend for other methods as reported in Table 2.
One possible solution to the challenge of balanced spatio-temporal understanding is to use multimodal learning. For example, two-stream networks [51, 14] employ both RGB and optical flow streams to learn both spatial and temporal contexts. However, this approach can be computationally expensive due to optical flow estimation.
In this work, we introduce a two-stream architecture, Cross-Attention in Space and Time (CAST), to address the challenge of balanced spatio-temporal understanding using only RGB input. In Figure 2, we show a high-level illustration of the proposed method. Our architecture employs two expert models - a spatial expert model and a temporal expert model - which exchange information to make a synergistic collective prediction. We realize the information exchange by cross-attention between the two experts. We empirically validate that placing cross-attention in a bottleneck architecture facilitates more effective learning. To validate the effectiveness of the proposed method, we conduct extensive experiments on multiple datasets with distinct characteristics, including the temporal-biased Something-Something-V2, static-biased Kinetics-400, and fine-grained EPIC-KITCHENS-100. Our results demonstrate that CAST achieves balanced spatio-temporal understanding and shows favorable performance across these different datasets.
In this work, we make the following significant contributions.
- We introduce a two-stream architecture, CAST, which addresses the challenge of balanced spatio-temporal understanding that has been largely overlooked by previous works.
- We conduct extensive experiments on multiple datasets with distinct characteristics to demonstrate the effectiveness of CAST. In terms of balanced spatio-temporal understanding, CAST shows favorable performance, while existing methods show more imbalanced performance.
- We conduct an extensive ablation study and analysis to validate the design choices of the proposed method. We show that employing spatial expert and temporal expert and placing cross-attention in a bottleneck architecture is crucial for achieving effective spatio-temporal representation learning.
Video Action Recognition. CNN-based approaches have been widely used for action recognition, including 2D CNNs [61, 74, 33, 52, 27], 3D CNNs [58, 5, 59, 64, 13], 2D and 1D separable CNNs [59, 70], or two-stream CNNs [14, 15]. These methods have achieved great progress thanks to the strong inductive biases. Recently, Transformer-based approaches [1, 3, 21, 43, 68, 12, 71] become popular in the community due to the long-term context modeling capabilities. Similar to the two-stream CNNs, we propose a two-stream transformer architecture consisting of two expert models: a spatial expert and a temporal expert. However, unlike traditional two-stream CNNs, we use RGB input only, instead of RGB and flow.
Figure 2: High-level illustration of the proposed method. In this work, we employ spatial and temporal expert models. The two experts exchange information with each other using cross-attention. Initially, the experts may predict incorrect actions due to the lack of information. For example, the temporal expert may predict reach out to something while the ground truth is Pick up a fork. Similarly, the spatial expert may predict utensil holder instead of fork in the shallower layers. However, after using cross-attention to exchange information multiple times, the proposed method can collectively predict the correct action Pick up a fork. Best viewed with zoom and color.
Cross-attention. Cross-attention has been widely utilized in multi-modal learning to facilitate information exchange between different modalities such as audio, visual, and text [34, 67, 40, 30, 18]. Recently, cross-attention between different views of the same video has shown impressive results [71, 75, 6, 26]. Similar to these, we propose a cross-attention method using a single RGB input, but with two distinct expert models: a spatial expert and a temporal expert. The two experts attend to each other through cross-attention to achieve a balanced spatio-temporal understanding.
Foundation model. Trained on web-scale datasets using self-supervised learning, foundation models [25, 4, 48, 45, 44] are highly adaptable and versatile. Foundation models show impressive performance on various tasks in computer vision [65, 62], natural language processing [49, 57], and audio recognition [17]. In this work, we employ CLIP [44] as our spatial expert as it shows impressive performance on more than 30 computer vision tasks.
Parameter-efficient transfer learning. Although the "pre-training and fine-tuning" paradigm with strong foundation models has demonstrated impressive performance on several computer vision tasks, it is computationally expensive and often unnecessary to fine-tune the full model [72]. Several works have demonstrated that learning only a small subset of parameters and keeping the remaining parameters frozen is effective for NLP tasks [23, 29] and computer vision tasks [36, 66, 54, 46, 47]. Extending image foundation models by adding adapter architectures has shown favorable performance on action recognition [35, 72, 42]. The proposed method also employs adapter architecture with cross-attention between two experts. We empirically demonstrate that the proposed method outperforms existing adapter-based video models in terms of achieving balanced spatio-temporal understanding.
3 Method: Cross-Attention in Space and Time
We introduce CAST, a method for balanced spatio-temporal representation learning for action recognition, as shown in Figure 3. We employ frozen spatial and temporal expert models that can be any vision transformer, consisting of 12 transformer blocks each. To facilitate information exchange between the experts, we introduce the bottleneck cross-attention in space and time (B-CAST) module on top of the frozen layers. This module enables the experts to exchange information and learn more balanced spatio-temporal contexts than separate experts. To improve adaptation to downstream tasks, we use adapter layers with a small number of learnable parameters, following AIM [72]. In the following subsections, we provide a detailed description of each component of our proposed CAST.
3.1 Input embeddings
CAST takes only RGB videos as inputs. The input is a mini-batch of videos,  $\mathbf{I} \in \mathbb{R}^{B \times 2T \times H \times W \times C}$ , consisting of  $B$  videos of  $2T$  frames,  $H \times W$  spatial dimensions, and  $C$  channels. We apply patch tokenization to the input videos for the spatial expert and the temporal expert. For the spatial expert, we decompose every even frame of each video in  $\mathbf{I}$  into  $N$  non-overlapping patches of  $p \times p$  pixels [11].
Figure 3: Overview of CAST. (a) CAST employs frozen spatial and temporal expert models. On top of the experts, we add a cross-attention module B-CAST to enable the exchange of information between the two experts. Additionally, we employ adapters with a small number of learnable parameters to the experts for better adaptation. (b) The proposed B-CAST consists of temporal-to-spatial (T2S) and spatial-to-temporal (S2T) cross-attentions to allow for a better understanding of the spatio-temporal features in the video data. For efficient and effective learning, we incorporate cross-attention into the bottleneck adcater. We employ separate position embedding for each expert. (c) We visualize T2S and S2T cross-attentions. Given a query, the model attends along the temporal axis only in T2S while the model attends along the spatial axes only in S2T.
Then we pass the patches through a frozen linear layer and add position embeddings to obtain spatial embeddings,  $\mathbf{X}_s\in \mathbb{R}^{BT\times N\times D}$ . For the temporal expert, we decompose every two frames of each video in  $\mathbf{I}$  into  $2\times p\times p$  pixels non-overlapping tubes [1]. Then we pass the tubes through a frozen linear layer and add position embeddings to obtain temporal embeddings,  $\mathbf{X}_t\in \mathbb{R}^{B\times TN\times D}$ .
3.2 CAST architecture
The model architecture of each expert is the same as the ViT except for adapters and the B-CAST module. All the other parameters are frozen, while the adapter and B-CAST parameters are learnable.
For completeness, we first define the operations used and then describe the entire model architecture. Given an input  $\mathbf{X}$ , we define Multi-Head Self Attention (MHSA) operation as follows:
$$
\operatorname {M H S A} (\mathbf {X}) = \operatorname {S o f t m a x} \left(\left(\mathbf {X W} _ {Q}\right) \left(\mathbf {X W} _ {K}\right) ^ {\top}\right) \left(\mathbf {X W} _ {V}\right), \tag {1}
$$
$\mathbf{W}_Q, \mathbf{W}_K,$  and  $\mathbf{W}_V$  are the query, key, and value projection matrices, respectively. We also define the adapter operation with linear down and up projection matrices  $\mathbf{W}_D$  and  $\mathbf{W}_U$  as follows:
$$
\operatorname {A D A P} (\mathbf {X}) = \sigma \left(\mathbf {X} \mathbf {W} _ {D}\right) \mathbf {W} _ {U}, \tag {2}
$$
where  $\sigma (\cdot)$  is the GELU activation function [20].
For each attention block  $l$ , we apply independent Multi-Head Self Attention (MHSA) for each expert along with a skip connection as follows:
$$
\mathbf {Y} ^ {(l)} = \mathbf {X} ^ {(l)} + \operatorname {A D A P} (\operatorname {M H S A} (\operatorname {L N} (\mathbf {X} ^ {(l)}))) + \operatorname {M H S A} (\operatorname {L N} (\mathbf {X} ^ {(l)})), \tag {3}
$$
where  $\mathrm{LN}(\cdot)$  denotes the Layer Normalization operation. The spatial path undergoes spatial attention, while the temporal path undergoes space-time attention following TimeSformer [3].
As shown in Figure 3 (b), to exchange information between the two experts, we apply the B-CAST operation  $\Phi (\cdot)$  to  $\mathbf{Y}_{e_1}$  and  $\mathbf{Y}_{e_2}$  from the expert  $e_1$  and  $e_2$  as follows along with a skip connection:
$$
\mathbf {B} ^ {(l)} = \mathbf {Y} ^ {(l)} + \Phi \left(\mathbf {Y} _ {e _ {1}} ^ {(l)}, \mathbf {Y} _ {e _ {2}} ^ {(l)}\right). \tag {4}
$$
We describe the B-CAST operation  $\Phi (\cdot)$  in detail in Section 3.3.
Finally, we pass the output, denoted as  $\mathbf{B}^{(l)}$ , through a two-layer feed forward network (FFN) [11] with the GELU activation function in between the layers and another adapter to obtain the next layer input  $\mathbf{X}^{(l + 1)}$  as follows along with a skip connection:
$$
\mathbf {X} ^ {(l + 1)} = \mathbf {B} ^ {(l)} + \operatorname {F F N} \left(\operatorname {L N} \left(\mathbf {B} ^ {(l)}\right)\right) + \operatorname {A D A P} \left(\operatorname {L N} \left(\mathbf {B} ^ {(l)}\right)\right). \tag {5}
$$
Classification head. To produce the final prediction, we need to aggregate the outputs of both spatial and temporal experts. For the spatial expert, we average the frame-level class tokens from the last attention block,  $\mathbf{X}_s^{(12)}$ , to obtain a single class token. We denote this operation as  $\mathrm{CLS}(\cdot)$ . To obtain temporal expert features, we aggregate all the tokens from the last attention block of the temporal expert,  $\mathbf{X}_t^{(12)}$ , using the global average pooling  $\mathrm{GAP}(\cdot)$  operation. Then we add the adapter output of the CLS token and the adapter output of the GAP token to produce a fused token  $\mathbf{Z}$ :
$$
\mathbf {Z} = \operatorname {A D A P} \left(\operatorname {C L S} \left(\mathbf {X} _ {s} ^ {(1 2)}\right)\right) + \operatorname {A D A P} \left(\operatorname {G A P} \left(\mathbf {X} _ {t} ^ {(1 2)}\right)\right). \tag {6}
$$
Finally, we feed the fused token  $\mathbf{Z}$  a classification layer followed by a softmax function to obtain the predicted class probabilities. We train the model using the standard cross-entropy loss.
3.3 B-CAST module architecture
Multi-Head Cross-Attention. Multi-Head Cross-Attention (MHCA) is a variant of the MHSA operation (1), where query tokens come from one expert  $(e_1)$  and key and value tokens come from another expert  $(e_2)$ . This allows the experts to exchange information and benefit from the strengths of each other. We define the MHCA operation as follows:
$$
\operatorname {M H C A} \left(\mathbf {Y} _ {e _ {1}}, \mathbf {Y} _ {e _ {2}}\right) = \operatorname {S o f t m a x} \left(\left(\mathbf {Y} _ {e _ {1}} \mathbf {W} _ {Q}\right) \left(\mathbf {Y} _ {e _ {2}} \mathbf {W} _ {K}\right) ^ {\top}\right) \left(\mathbf {Y} _ {e _ {2}} \mathbf {W} _ {V}\right), \tag {7}
$$
where  $\mathbf{W}_Q$ ,  $\mathbf{W}_K$ , and  $\mathbf{W}_V$  are learnable query, key, and value parameter matrices respectively.
Temporal-to-Spatial Cross-Attention. In Temporal-to-Spatial (T2S) cross-attention, query tokens come from the spatial expert  $s$ , and key and value tokens come from the temporal expert  $t$ : MHCA  $(\mathbf{Y}_s^{(l)},\mathbf{Y}_t^{(l)})$ . We depict the attention window in Figure 3 (c). Given a query, the model attends along the temporal dimension only. By using T2S cross-attention, the spatial expert can learn to attend to temporal features from the temporal expert. T2S MHCA leads to capturing spatio-temporal dependencies and improves the model performance in action recognition.
Spatial-to-Temporal Cross-Attention. In Spatial-to-Temporal (S2T) cross-attention, query tokens come from the temporal expert  $t$ , and key and value tokens come from the spatial expert  $s$ : MHCA( $\mathbf{Y}_t^{(l)}$ ,  $\mathbf{Y}_s^{(l)}$ ). We illustrate the attention window in Figure 3 (c). Given a query, the model attends along the spatial dimension only. By using S2T cross-attention, the temporal expert can attend to fine-grained spatial features from the spatial expert. S2T MHCA leads to a more balanced spatio-temporal understanding and improves the performance in fine-grained action recognition.
Bottleneck Cross-Attention in Space and Time. To achieve efficient and effective learning, we incorporate the T2S and S2T MHCA into bottleneck-shaped adapters. We illustrate B-CAST architecture in Figure 3 (b). We plug the MHCA modules into adapters and add new learnable positional embeddings for each MHCA. We define the B-CAST operation for T2S  $\Phi_S(\cdot)$  as follows:
$$
\Phi_ {S} \left(\mathbf {Y} _ {s} ^ {(l)}, \mathbf {Y} _ {t} ^ {(l)}\right) = \sigma \left(\operatorname {M H C A} \left(\mathbf {E} _ {s} + \operatorname {L N} \left(\mathbf {Y} _ {s} ^ {(l)} \mathbf {W} _ {D, s}\right), \mathbf {E} _ {t} + \operatorname {L N} \left(\mathbf {Y} _ {t} ^ {(l)} \mathbf {W} _ {D, t}\right)\right)\right) \mathbf {W} _ {U, s}, \tag {8}
$$
where  $\mathbf{W}_{D,s}$  and  $\mathbf{W}_{U,s}$  are linear down- and up-projection matrices for the spatial expert, and  $\mathbf{E}_s$  and  $\mathbf{E}_t$  are new positional embeddings for the spatial and temporal experts,  $\sigma(\cdot)$  is the GELU activation function, respectively. We can define the B-CAST operation for S2T,  $\Phi_T(\cdot)$  in a similar manner. The output of B-CAST goes into a feed forward network using (5). Our empirical validation shows that the B-CAST architecture is efficient and effective. (See Table 1.)
Figure 4: Balanced spatio-temporal understanding performance. We visualize the action recognition accuracies of existing methods and the proposed method. (a) We show the Top-1 accuracies of ST-Adapter and VideoMAE on the EK100 verb and noun prediction tasks. (b) We show the Top-1 accuracies of AIM and BEVT on the SSV2, and K400. (c) For each method, we show the harmonic mean of Top-1 accuracies on the EK100 noun, EK100 verb, SSV2, and K400. CAST shows a more balanced spatio-temporal understanding capability compared to the existing methods. Best viewed with zoom and color.
In this section, we present the experimental results that answer the following research questions: (1) Do existing methods show a balanced spatio-temporal understanding of videos? (Section 4.3) (2) What are the ingredients for a balanced spatio-temporal understanding? (Section 4.3) (3) Is the proposed method effective? (Section 4.3, Section 4.4) (4) How can we effectively combine spatial and temporal models to achieve such balance? (Section 4.5) (5) Does the proposed method outperform state-of-the-art methods in terms of balanced spatio-temporal understanding? (Section 4.6) To this end, we first provide details about the datasets and implementation in Section 4.1 and Section 4.2, respectively.
Action recognition. We evaluate the CAST on two public datasets for conventional action recognition: Something-Something-V2 (SSV2) [19] and Kinetics-400 (K400) [24]. The SSV2 requires more temporal reasoning [3, 28] while the K400 is relatively static biased [32, 8, 50].
Fine-grained action recognition. We evaluate the CAST on the fine-grained action recognition task: EPIC-KITCHENS-100 (EK100) [10]. In contrast to conventional action recognition, EK100 defines an action as a combination of a verb and a noun. Therefore, we refer to the action recognition in EK100 as fine-grained action recognition. Since fine-grained action recognition requires correctly predicting both the verb and the noun to recognize an action it is more challenging than conventional action recognition, which requires predicting a single action label: e.g., K400 or SSV2.
In this section, we briefly provide our experimental setup and implementation details. Please refer to the Appendix § B for complete implementation details. We conduct all the experiments with 16 NVIDIA GeForce RTX 3090 GPUs. We implement CAST using PyTorch and build upon the existing codebase of VideoMAE [56].
Training. We sample 16 frames from each video to construct an input clip. For the K400 dataset, we apply dense sampling [15], while for SSV2 and EK100, we use uniform sampling [61]. We then perform random cropping and resizing every frame into  $224 \times 224$  pixels. We use the AdamW [39] optimizer with momentum betas of (0.9, 0.999) [7] and a weight decay of 0.05. By default, we train the model for 50 epochs, with the cosine annealing learning rate scheduling [38] and a warm-up period of 5 epochs. The default base learning rate, layer decay [2], and drop path are set to 0.001, 0.8, and 0.2, respectively. We freeze all the parameters of each expert, except for the B-CAST layer, adapters, and the last layer normalization. We set the batch size per GPU as 6 with update frequency of 2.
Inference. Given an input video, we randomly sample frames multiple times to construct input clips with multiple temporal views with multiple spatial crops. After the temporal frame sampling, we resize every frame so that the shorter side has 224 pixels. Then we perform spatial cropping to get multiple  $224 \times 224$  crops for each clip. We get the final prediction by averaging the predictions on (temporal views)  $\times$  (spatial crops). For the K400 dataset, we use (5 clips)  $\times$  (3 crops) views, while for the other datasets, we use (2 clips)  $\times$  (3 crops) views for the inference.
Figure 5: Improvements of CAST over each expert on EK100 noun classes. We show the super-category-wise weighted average F1 score improvement of CAST over each expert. (Left) Improvement over CLIP. CAST outperforms CLIP for every super-category except meat and substitute. (Right) Improvement over VideoMAE. CAST outperforms VideoMAE for every super-category except furniture and prepared food. Best viewed with zoom and color.
4.3 Balanced spatio-temporal understanding
In Figure 4 (a), we present the top-1 accuracies of several existing models. In the EK100 verb prediction task, VideoMAE outperforms ST-Adapter with a margin of 2.9 points (70.5% vs. 67.6%), while in the EK100 noun prediction task, ST-Adapter [42] outperforms VideoMAE [56] with a margin of 3.6 points (55.0% vs. 51.4%). As shown in Figure 4 (b), BEVT [62] outperforms AIM [72] with a margin of 2.5 points (70.6% vs. 68.1%) on the SSV2 dataset, while on the K400 dataset, AIM outperforms BEVT with a margin of 3.9 points (84.5% vs. 80.6%). We observe similar trends for other methods as well. Please refer to Section 4.6 for a detailed comparison. Our findings indicate that the performance of many existing models is significantly imbalanced toward either spatial or temporal understanding.