SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ingredients for balanced spatio-temporal understanding. To achieve a more balanced spatio-temporal understanding, we can employ two expert models: a spatial expert and a temporal expert. For the spatial expert, we use CLIP [44], which has demonstrated impressive performance on various computer vision tasks. For the temporal expert, we use VideoMAE [56], which has shown favorable performance on temporal-biased tasks such as SSV2 and EK100 verb prediction tasks. (Please refer to Section 4.6 for the accuracy details.) While each expert is highly specialized in its own domain, we aim to create synergy between them by exchanging information to improve the balanced spatio-temporal understanding performance.
Effect of CAST. In Figure 4 (c), we gauge the balanced spatio-temporal understanding performance of our spatial expert, temporal expert, and CAST. For each method, we calculate the harmonic mean of top-1 accuracies for EK100 noun, EK100 verb, SSV2, and K400. The harmonic mean is an effective metric for gauging balanced performance because it gives more weight to lower-performing tasks. A higher harmonic mean value indicates that the performance over the different tasks is more balanced. Our spatial expert achieves an accuracy of  $56.5\%$ , while the temporal expert achieves an accuracy of  $66.6\%$ , and our CAST achieves an accuracy of  $71.6\%$ . These results validate the effectiveness of our proposed method, CAST, which allows our spatial and temporal experts to make synergistic predictions by exchanging information with each other through cross-attention.
4.4 Analysis on fine-grained action recognition
In this section, we provide a detailed analysis of how the proposed CAST improves the balanced spatio-temporal understanding in the fine-grained action recognition task: EK100.
Category-level performance analysis. In Figure 5, We present the EK100 noun super-category-wise weighted average F1 score improvement of CAST over our spatial expert (CLIP) and temporal expert (VideoMAE). In Figure 5 left, we observe that CAST significantly improves upon the spatial expert, CLIP, in several super-categories such as cutlery, utensils, and vegetables. These results indicate that the spatial expert achieves a more accurate understanding of fine-grained small objects interacting with the actors by leveraging the temporal context from the temporal expert. Similarly, in Figure 5 right, we observe that CAST significantly improves upon the temporal expert, VideoMAE, in several categories such as vegetables and cutlery. The trend is similar to the comparison with the spatial expert: CAST achieves more accurate understanding of fine-grained small objects by leveraging the fine-grained spatial context from CLIP.
Qualitative analysis. To better understand the effectiveness of CAST, we provide qualitative analysis on a few sample frames from the EK100 dataset in Figure 6. We show the predictions of CLIP, VideoMAE, and CAST. As expected, each expert model provides more accurate prediction in their respective tasks of expertise but shows weaker performance in the other task. In contrast,
Table 1: Ablation study. To validate the effect of each component, we show experimental results on the EPIC-Kitchens-100 dataset. In every experiment, we use the ViT-B/16 backbone for every expert. The best numbers are highlighted in gray.
(a) Effect of information exchange.
Method	Top-1 Acc.
Verb	Noun	Act.
Indep. experts w/o adapter	70.7	50.1	40.0
Indep. experts w/adapter	68.1	54.2	41.7
Ensemble of experts w/adapter	68.2	55.3	42.9
CAST	72.5	60.3	48.7
(b) Different information exchange methods.
Method	Late	Layer-wise	Top-1 Acc.
Verb	Noun	Act.
Add	✓	68.9	56.6	44.2
Concat	✓	69.2	56.4	44.5
Lateral	✓	68.9	49.1	39.0
CAST	✓	72.5	60.3	48.7
(c) B-CAST architecture.
Method	Tune
Param(M)	Top-1 Acc.
Verb	Noun	Act.
Identity	18.1	68.1	54.2	41.7
w/o adapter	85.9	69.3	49.4	39.4
X-attn->adapter	93.0	71.3	60.1	47.9
B-CAST	44.8	72.5	60.3	48.7
(d) Effect of projection ratio.
Ratio	Top-1 Acc.
Verb	Noun	Act.
1/8	70.7	59.9	47.4
1/4	71.3	59.8	47.4
1/2	72.5	60.3	48.7
1	72.1	59.8	48.6
(e) Effect of cross-attention window shape.
Window shape	Top-1 Acc.
T2S	S2T	Verb	Noun	Act.
space-time	space-time	71.0	59.3	47.2
space-time	space	71.9	60.3	48.4
space	space	72.3	60.2	48.5
time	space	72.5	60.3	48.7
(f) Effect of the number of cross-attention layers.
X-attention layer	Top-1 Acc.
1-3	4-6	7-9	10-12	Verb	Noun	Act.
✓	71.2	59.4	47.4
✓	✓	71.3	59.9	47.9
✓	✓	✓	71.8	60.0	48.2
✓	✓	✓	✓	72.5	60.3	48.7
(g) Effect of bi-directional cross-attention.
Method	Top-1 Acc.
Verb	Noun	Act.
Indep. experts w/ adapter	68.1	54.2	41.7
S2T only	71.2	55.0	43.7
T2S only	68.7	60.5	46.7
CAST	72.5	60.3	48.7
(h) Role of each expert.
Expert	Top-1 Acc.
Spatial	Temporal	Verb	Noun	Act.
CLIP	CLIP	69.3	58.8	46.0
VideoMAE	CLIP	72.2	58.8	47.8
VideoMAE	VideoMAE	69.8	49.9	40.3
CLIP	VideoMAE	72.5	60.3	48.7
CAST consistently shows correct predictions for both noun and verb prediction tasks, such as spoon and open. The qualitative examples demonstrate the effectiveness of CAST in achieving balanced spatio-temporal understanding, which is essential for fine-grained action recognition.
4.5 Ablation study on CAST architecture
We conduct comprehensive ablation studies to examine the design choices for the proposed CAST architecture. Here we conduct all experiments on the EK100 [10] dataset with 16-frame input videos and report the top-1 accuracy on the validation set. We employ CLIP [44] as a spatial expert model and VideoMAE [56] as a temporal expert model. For a fair ablation study, we use the same hyperparameters for each experiment unless explicitly mentioned.
Effect of information exchange. We investigate whether CAST effectively achieves a synergistic effect by exchanging information between the two expert models. In Table 1 (a), we compare CAST with three baselines. i) A baseline using two independent expert models without any information exchange (fully fine-tuned). ii) The same baseline as i), but we add adapters and fine-tune the adapters and head only, iii) A test-time ensemble of two independent experts (with adapters and heads fine-tuning only). The baselines predict nouns using the spatial model and verbs using the temporal model. We observe that the two expert models using ensembling achieve an improvement in Action accuracy by at least 1.2 points compared to the baselines without any information exchange. Furthermore, CAST achieves a best Action accuracy of  $48.7\%$ . These results suggest that information exchange is crucial for achieving balanced spatio-temporal understanding.
Comparison with simple information exchange baselines. We compare CAST with simple information exchange baselines: i) late fusion with addition, ii) late fusion with concatenation, iii) layer-wise fusion using the bidirectional lateral connection (element-wise addition) with linear projection. We add adapters and fine-tune the adapters and head only in all three baselines, using our training recipe in Section 4.2 For the details of baseline fusion methods, please see Figure 7. We show the results in Table 1 (b). It is worth noting that both the late fusion and layer-wise lateral connection baselines result in a significant performance drop. Furthermore, we observe that layer-wise fusion without cross-attention yields inferior performance compared to the simple late fusion baselines. The results indicate that cross-attention in the bottleneck architecture is crucial for effective information exchange between spatial and temporal experts.
Design of B-CAST module. To explore the most effective and efficient way to integrate adapters for information exchange between the two expert models, we conduct an ablation study and present the results in Table 1 (c). For the details of the baselines, please see Figure 8. The first row of the table represents a baseline without the B-CAST module, which is equivalent to the identity function. Compared to this baseline, B-CAST achieves a significant improvement of 7.0 points in Action accuracy. The second row shows the performance of a baseline with cross-attention but without the
Figure 6: Qualitative examples from EK100 comparing CLIP, VideoMAE, and the proposed CAST. Each expert model shows more accurate predictions in their expertise but shows weaker performance on the other task. However, CAST consistently shows correct predictions for both tasks, demonstrating the effectiveness of the proposed spatio-temporal cross-attention mechanism.
bottleneck adapters. The 9.3-point gap between this baseline and B-CAST highlights the importance of bottleneck adapters for effective information exchange between the two expert models. The third row  $(X\text{-}\text{attn.}\rightarrow\text{adapter})$  is a baseline with the adapters after cross-attention. Compared to B-CAST, this baseline shows a 0.8 points drop in Action accuracy while having more than double the number of learnable parameters (44.9M vs. 93.0M). The results indicate that cross-attention in bottleneck is more effective and more efficient than the baseline. In summary, by placing cross-attention in the middle of the bottleneck adapter, B-CAST facilitates effective information exchange between the two experts and achieves a synergistic effect.
Effect of projection ratio in bottleneck. In this study, we investigate the impact of the down projection ratio in the bottleneck architecture presented in Table 1 (d). The results demonstrate that a ratio of  $1/2$  yields the best performance. Notably, a ratio of 1 results in inferior performance, which we attribute to overfitting caused by the addition of more parameters.
Effect of cross-attention window shape. We investigate the impact of the window shape in the cross-attention mechanism in the T2S and S2T modules in Table 1 (e). Please refer to Figure 3 (c) for the details of the window size. We maintain the same model capacity across different methods. Using space-time attention for both T2S and S2T modules results in the worst performance. We conjecture that learning joint space-time attention is challenging with the given model capacity [3]. On the other hand, using time attention in T2S and space attention in S2T yields the best performance. Consequently, we adopt this configuration throughout the paper.
Effect of the number of cross-attention layers. We investigate the impact of the number of cross-attention layers used. To this end, we gradually increase the number of cross-attention layers starting from the top three layers, and report the results in Table 1 (f). As we increase the number of cross-attention layers, we observe a corresponding improvement in performance, as expected.
Effect of bi-directional cross-attention. To validate the effectiveness of bi-directional information exchange, we ablate each cross attention at a time. We compare CAST with unidirectional information exchange baselines equipped with S2T or T2S cross-attention only. Each unidirectional information exchange baseline still has both experts. In Table 1 (g), compared to our CAST  $(48.7\%)$ , the S2T only baseline shows 5.0 points drop  $(43.7\%)$  and the T2S only baseline shows 2.0 points drop  $(46.7\%)$  in accuracy. The results validate the effectiveness of the proposed bi-directional cross-attention.
Role of each expert. In Table 1 (h), we investigate the role of experts within CAST by controlling the assignment of models to each expert. We observe that we can achieve the best performance of  $48.7\%$  when we employ CLIP as our temporal expert and VideoMAE as our spatial expert. When we employ one VideoMAE as our spatial expert and another VideoMAE as our temporal expert, we obtain  $40.3\%$  accuracy. When we employ one CLIP as our spatial expert and another CLIP as our temporal expert, we obtain  $46.0\%$  accuracy.
Interestingly, when we revert the role of CLIP and VideoMAE, i.e., we employ VideoMAE as the spatial and CLIP as the temporal expert, we achieve a good performance of  $47.8\%$ . The results demonstrate that the B-CAST architecture facilitates effective information exchange between the two experts. Through the stacked B-CAST, the experts can learn high-quality spatio-temporal representations by exchanging information, even when the roles are reverted.
In summary, these findings suggest that CAST achieves optimal performance when models are assigned to expert roles that align with their strengths. CLIP serves as an effective spatial expert, whereas VideoMAE is more effective as a temporal expert. The B-CAST architecture encourages these experts to leverage their respective strengths through information exchange, resulting in enhanced spatio-temporal balanced understanding.
Table 2: Comparison with the state-of-the-arts on the EK100, SSV2 and K400 datasets. We show the Top-1 accuracy on each dataset and the harmonic mean (H.M.) of the Top-1 accuracies. The best performance is in bold and the second best is underscored.
Method	GFLOPs/ View	EK100 Top-1	SSV2 & K400 Top-1	All
Verb	Noun	Act.	SSV2	K400	H.M.	H.M.
CLIP* [44]	140	54.9	52.7	33.8	47.8	78.9	59.5	56.5
EVL [35]	592	-	-	-	62.4	82.9	71.2	-
ST-Adapter [42]	607	67.6	55.0	-	69.5	82.7	75.5	67.3
AIM [72]	404	64.8	55.5	41.3*	68.1	84.5	75.4	66.7
MBT [40]	936	64.8	58.0	43.4	-	80.8	-	-
ViViT FE [1]	990	66.4	56.8	44.0	65.9	81.7	73.0	66.6
TimeSformer [3]	2380	-	-	-	62.4	80.7	70.4	-
MViT [12]	170	-	-	-	67.7	80.2	73.4	-
MFormer [43]	1185	67.1	57.6	44.1	68.1	80.2	73.7	67.3
ORViT MF [21]	-	68.4	58.7	45.7	67.9	-	-	-
Video Swin [37]	282	-	-	-	69.6	82.7	75.8	-
BEVT [62]	282	-	-	-	70.6	80.6	75.3	-
VideoMAE [56]	180	70.5	51.4	41.7*	70.8	81.5	75.8	66.6
MeMViT [68]	59	70.6	58.5	46.2	-	-	-	-
OMNIVORE [16]	-	69.5	61.7	49.9	71.4	84.0	77.2	70.8
MTV-HR [71]	930	68.0	63.1	48.6	68.5	82.4	74.8	69.8
CAST	391	72.5	60.9	49.3	71.6	85.3	77.9	71.6
*We conduct experiments with our own implementation.
4.6 Comparison with state-of-the-art
In this section, we evaluate the performance of CAST and state-of-the-art methods in terms of balanced spatio-temporal understanding on multiple datasets, as shown in Table 2. For each method, in addition to reporting the top-1 accuracy of each task, we report the harmonic mean of top-1 accuracies for i) SSV2, and K400, and ii) EK100 verb, EK100 noun, SSV2, and K400. For comparison with state-of-the-art models, we have set different hyperparameters than those used in our ablation study. Please refer to Table 4 for the details. For fair comparisons of the computation complexity, we show the GFLOPs/View. In cases where a compared method shows various GFLOPs/View depending on the dataset, we specifically note the lowest GFLOPs/View value for reference. For more detailed comparison of computation complexity, please refer to the Appendix § E.
We observe that among the CLIP-based methods (the second group in Table 2), AIM [72] achieves favorable performance on the static-biased K400 dataset, with  $84.5\%$  accuracy. However, AIM shows a relatively lower performance of  $68.1\%$  on the temporal-biased SSV2. On the other hand, VideoMAE [56], one of the state-of-the-art methods, shows  $70.8\%$  accuracy on the SSV2 dataset, which is more competitive than AIM. However, VideoMAE shows a lower accuracy of  $81.5\%$  on the K400 dataset, less competitive than AIM. Our proposed method, CAST, demonstrates favorable performance on both the SSV2  $(71.6\%)$  and K400  $(85.3\%)$  datasets, resulting in a harmonic mean of  $77.9\%$ , which is higher than that of AIM  $(75.4\%)$  and VideoMAE  $(75.8\%)$ . CAST shows a more balanced spatio-temporal understanding than the existing methods. Additionally, CAST shows favorable performance in fine-grained action recognition on the EK100 dataset. CAST achieves a competitive Action accuracy of  $49.3\%$ , which is the second best among the compared methods.
In terms of the overall harmonic mean of EK100 verb, EK100 noun, SSV2, and K400 accuracies, CAST shows the best performance of  $71.6\%$ . The results highlight the effectiveness of CAST. By exchanging information between spatial and temporal experts, our CAST shows a favorable balanced spatio-temporal understanding performance.
In this paper, we present a solution to the problem of action recognition models lacking a balanced spatio-temporal understanding of videos. The proposed method, CAST, incorporates a spatial expert and a temporal expert that exchange information through cross-attention to achieve synergistic predictions. Our extensive experiments on datasets with varying characteristics demonstrate that CAST outperforms both individual expert models and existing methods in terms of a balanced spatio-temporal understanding measure: the harmonic mean of accuracies on the datasets. The results highlight the effectiveness of CAST in achieving a balanced spatio-temporal understanding of videos, and suggest that CAST could have broad applicability in the field of video understanding.
Acknowledgment. This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.RS-2022-00155911, Artificial Intelligence Convergence Innovation Human Resources Development (Kyung Hee University)); by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea Government (MSIT) (Artificial Intelligence Innovation Hub) under Grant 2021-0-02068; by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. 2022R1F1A1070997).
[65] Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191, 2022.
[66] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, 2022.
[67] Xi Wei, Tianzhu Zhang, Yan Li, Yongdong Zhang, and Feng Wu. Multi-modality cross attention network for image and sentence matching. In CVPR, 2020.
[68] Chao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition. In CVPR, 2022.
[69] Wenhao Wu, Zhun Sun, and Wanli Ouyang. Revisiting classifier: Transferring vision-language models for video recognition. In AAAI, 2023.
[70] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning for video understanding. In ECCV, 2018.
[71] Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview transformers for video recognition. In CVPR, 2022.
[72] Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. Aim: Adapting image models for efficient video understanding. In ICLR, 2023.
[73] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2017.
[74] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In ECCV, 2018.
[75] Haowei Zhu, Wenjing Ke, Dong Li, Ji Liu, Lu Tian, and Yi Shan. Dual cross-attention learning for fine-grained visual categorization and object re-identification. In CVPR, 2022.
In this appendix, we provide additional architecture/implementation/dataset details, experimental settings, quantitative/qualitative results, limitations of our method, and broader impact of our method to complement the main paper. We organize the appendix as follows:
A. Architecture details of our framework
B. Implementation details and experimental settings
C. Details of datasets in our experiments
D. Additional quantitative and qualitative results
E. Comparison with State-of-the-Art with additional information
F. Class-wise F1 score on EK100 verb and noun classes
G. Additional qualitative analysis on EK100
H. Limitations
I. Broader impacts
A Architecture Details
In this section, we provide details of our B-CAST architecture. Let us assume we employ CLIP [44] as a spatial expert and VideoMAE [56] as a temporal expert.
Table 3: Stage-wise details of the two experts in B-CAST. We provide a detailed description of each operation performed in each expert. The input for this example is a video consisting of 16 frames. MHCA represents multi-head cross-attention applied with a specific window shape: either time-attention or space-attention. In the description, B, D, T, and N represent the batch size, embedding dimension, temporal sequence length, and spatial sequence length, respectively. We omit the Layer Normalization and activation functions for simplicity.
Spatial Expert	Temporal Expert
B-CAST Stage	Remark	Output Tensor Shape	Remark	Output Tensor Shape
Up Projection	Linear projection with ratio = 2.0	YS:(196+1)×B·8×768	Linear projection with ratio = 2.0	Yt:B×8·196×768
Post Processing	Attach CLS token of YsReshape: N×B·T×D	YS:(196+1)×B·8×384	Reshape: B×T·N×D	Yt:B×8·196×384
Cross-Attention	T2SMHCA(Ys,Yt)Window shape: time	YS:B·196×8×384	S2TMHCA(Yt,Ys)Window shape: space	Yt:B·8×196×384
Positional Embeddings	# parameters: 8×384	YS:B·196×8×384Yt:B·196×8×384	# parameters: 196×384	Yt:B·8×196×384Ys:B·8×196×384
Pre processing	Detach CLS token of YsReshape:B·N×T×D	YS:B·196×8×384Yt:B·196×8×384	Detach CLS token of YsReshape: B·T×N×D	Yt:B·8×196×384Ys:B·8×196×384
Gather Features	Gather Ytfrom Temporal Expert	YS:(196+1)×B·8×384Yt:B×8·196×384	Gather Ysfrom Spatial Expert	Yt:B×8·196×384Ys:(196+1)×B·8×384
Down Projection	Linear projection with ratio = 0.5	YS:(196+1)×B·8×384	Linear projection with ratio = 0.5	Yt:B×8·196×384
Input of B-CAST	-	YS:(196+1)×B·8×768	-	Yt:B×8·196×768
B-CAST architecture. In Table 3, we provide stage-wise details of the two experts in B-CAST. Given an input from multi-head self-attention (MHSA) layer, we first pass it through the linear projection layer of each expert. Subsequently, the two experts exchanges their features each other. The multi-head cross-attention (MHCA) layer of each expert enables the effectively exchange of information between the two experts. Afterward, we pass the output tensors from the B-CAST
through the Feed-Forward Network (FFN). We repeat this process in a stacked manner, consisting of 12 blocks, each comprising the MHSA module along with adapters, B-CAST, and FFN module along with adapters. Finally, we feed the resulting tensors to a classification head to predict action.
Architecture details of different information exchange methods. In Figure 7, we visualize the architectures of the simple information exchange baselines presented in Table 1 (b) of the main paper. These baselines involve fully fine-tuning the two expert models without the use of adapters, following the training recipe outlined in Appendix § B. In Figure 7 (a), we present the add baseline, which facilitates information exchange between the spatial and temporal experts through late fusion using element-wise addition. In Figure 7 (b), we present the concat baseline, which exchanges the information between the spatial and the temporal experts through late fusion using concatenation. In Figure 7 (c), we show the lateral connection baseline, which enables information exchange between the spatial and temporal experts through layer-wise lateral connections, incorporating linear projections.
Figure 7: Architecture visualization of the different information exchange baselines.
Architecture details of the baselines in the B-CAST architecture ablation study. In Figure 8, we visualize the architectures of the baselines used in the B-CAST architecture ablation study, as presented in Table 1 (c) of the main paper. In Figure 8 (a), we illustrate the identity baseline, which serves as a baseline without the B-CAST module. In Figure 8 (b), we present the w/o adapter baseline, which includes cross-attention but does not incorporate the bottleneck adapters. In Figure 8 (c), we show the X-attn.→adapter baseline, which uses adapters positioned after the cross-attention stage. For reference, we include our B-CAST architecture in Figure 8 (d), which represents the final model used in our study.
Figure 8: Architecture visualization of the baselines used in the B-CAST ablation study.