CAST: Cross-Attention in Space and Time for Video Action Recognition
Authors: Dongho Lee
Jongseo Lee
Jinwoo Choi
Venue: NeurIPS 2023 poster (2023)
Keywords: action recognition
video understanding
cross-attention
balanced spatio-temporal understanding
TL;DR: The proposed bottleneck cross-attention module enables a balanced spatio-temporal understanding in video action recognition

Abstract: Recognizing human actions in videos requires spatial and temporal understanding. Most existing action recognition models lack a balanced spatio-temporal understanding of videos. In this work, we propose a novel two-stream architecture, called Cross-Attention in Space and Time (CAST), that achieves a balanced spatio-temporal understanding of videos using only RGB input. Our proposed bottleneck cross-attention mechanism enables the spatial and temporal expert models to exchange information and make synergistic predictions, leading to improved performance. We validate the proposed method with extensive experiments on public benchmarks with different characteristics: EPIC-Kitchens-100, Something-Something-V2, and Kinetics-400. Our method consistently shows favorable performance across these datasets, while the performance of existing methods fluctuates depending on the dataset characteristics. The code is available at https://github.com/KHU-VLL/CAST.

CAST: Cross-Attention in Space and Time for Video Action Recognition
Dongho Lee*
Jongseo Lee"
Jinwoo Choi
Kyung Hee University, Republic of Korea
(kide004, jong980812, jinwoochoi] ( 88ku. ac kr
Abstract
Recognizing human actions in videos requires spatial and temporal understand- ing. Most existing action recognition models lack balanced spatio-temporal understanding of videos, In this work, we propose novel two-stream architec- ture, called Cross-Attention in Space and Time (CAST), that achieves a balanced spatio-temporal understanding of videos using only RGB input. Our proposed bottleneck cross-attention mechanism enables the spatial and temporal expert models to exchange information and make synergistic predictions, leading to improved performance. We validate the proposed method with extensive experi- ments on public benchmarks with different characteristics: EPIC- WIIC-KITCHNNS-IT V2, and Kinetics-400. Our method consistently shows favorable performance across these datasets, while the performance of existing methods fluctuates depending the dataset characteristics. The code is available https: / /gi thub. com KHU-KHUVVII VLL.VVLLL.S....
Introduction
To accurately recognize human actions in videos, model must understand both the spatial and temporal contexts. model that lacks fine-grained spatial understanding is likely fail predicting the correct action. For example, as shown in Figure (a), model that understands temporal context such hand motion across frames but the fine-grained spatial context may confuse whether an object in the hand a ketchup, or a cheese, or milk carton. Consequently, the model fails to predict the correct action, Put down cheese. Similarly, model that lacks temporal context understanding may also fail to predict the correct action. Figure (b), let us suppose a model understands spatial context but does not understand temporal context, e.g., the model is confused about whether the hand is moving from outside the fridge to the inside vice versa. Then the model fails to predict correct action Take out sauce. Therefore, for accurate action recognition, models need to comprehend both the spatial and temporal contexts of videos.
Despite the recent progress in action recognition through the of Transformers [60, 11, 3]. achieving balanced spatio-temporal understanding remains a challenging problem. Compared to images, the additional temporal dimension in videos makes spatio-tense temporal representation learning computationally intensive and requires significant amount of training data [3]. Consequently. most action recognition models lack a balanced apatio-tempora understanding videos. Notably, models that perform well on stttc-biased [32, 8, 50] datasets, such Kinetics-4000, may not perform as well on temporal-biasee [3. 28] datasets, such as Something Something- V2. and vice versa. For instance, as shown in Figure 4 (a), on the EPIC- PPIC-KITCHENSSITT 100 dataset, VideoMAF. [56] outperforms ST- Adapter [42] on the verb prediction task. while ST-Adapter outperforms VideoMAE the noun prediction task. Similarly, BEVT [62] outperforms AIM [72] on the Something-Sone Sooeinn---mmmt--VVV
Something-Somettingg
"Equally contributed first authors. 1 'Corresooddng author.
37th Conference Neural Information Processing Systems (NeurIPS 2023).

---

down cheese
truihe
Take out?
Ketchup?
Cheese??
Milk carton?
(a) Importance of spatial understanding
(b) Importance of temporal understanding
Figure I: The importance of spatio-temporal understanding. If model lacks fine-grained spatial understanding, the model may predict an incorrect action. E.g., the model fails to predict Put down cheese in (a) due to subtle appearance differences between the objects. On the other hand, a model lacks temporal context understanding, the model may predict incorrect action. E.g., the model fails to predict Take out a sauce in (b) due to the ambiguity of the action. Therefore, both spatial and temporal understanding are crucial in action recognition. Best viewed with zoom and color.
dataset, while BEVT underperforms AIM on the Kinetics-400 dataset. We observe similar trend for other methods as reported in Table 2.
One possible solution to the challenge of balanced spatio-temporal understanding is to use multi- modal learning. For example, two-stream networks [51, 14] employ both RGB and optical flow streams to learn both spatial and temporal contexts. However, this approach can be computationally expensive due to optical flow estimation.
In this work, we introduce two-stream architecture, Cross-Attention in Space and Time (CAST), to address the challenge of balanced spatio-temporal understanding using only RGB input. In Figure we show a high-level illustration of the proposed method. Our architecture employs two expert models- a spatial expert model and temporal expert model which exchange information to make synergistic collective prediction. We realize the information exchange by cross-attention between the two experts. We empirically validate that placing cross-attention in a bottleneck architecture facilitates more effective learning. To validate the effectiveness of the proposed method, conduct extensive experiments on multiple datasets with distinct characteristics, including the temporal-biant Something- Soieinnin-ttttiig--gnee V2, bbised Kinetics-400, and fine- -ee--gaiindd EPIC- KITCCKIN--SIIIII 100. Our results demonstrate that CAST achieves balanced spatio-temporal understanding and shows favorable performance across these different datasets.
In this work, we make the following significant contributions.
We introduce two-stream architecture, CAST, which addresses the challenge of balanced spatio-temporal understanding that has been largely overlooked by previous works.
We conduct extensive experiments on multiple datasets with distinct characteristics to demon- strate the effectiveness of CAST. In terms of balanced spatio-temporal understanding, CAST shows favorable performance, while existing methods show more imbalanced performance.
We conduct an extensive ablation study and analysis to validate the design choices of the proposed method. We show that employing spatial expert and temporal expert and placing cross-attention in a bottleneck architecture is crucial for achieving effective spatio-temporal representation learning.
2 Related Work
Video Action Recognition. CNN-based approaches have been widely used for action recognition,
including 2D CNNs [61, 74, 33. 52, 27], 3D CNNs [58, 5, 59, 64, 13], 2D and 1D separable
CNNs [59, 70], or two-stream CNNs [14, 15]. These methods have achieved great progress thanks approaches [1. 3, 21, 43, 68, 12, 71] become popular in the community due to the long-term context modeling capabilities. Similar to the two-stream CNNs, we propose a two-stream transformer architecture consisting of two expert models: a spatial expert and a temporal expert. However, unlike traditional two-stream CNNs, we use RGB input only, instead of RGB and flow.
to the strong inductive biases. Recently, Transformer-basee
2

---

Collecive prediction: up
holding
The actor le dlskingg upa cutlery.
Spaila expert
port expert
There are hand, dishes, and utensil holder.
The actor and bringing something
Next conversation
Spilia
Temperal export
Ground Truth: Pick up fonk
Figure 2: High-level illustration of the proposed method. In this work, we employ spatial and temporal expert models. The two experts exchange information with each other using cross-attention. Initially, the experts may predict incorrect actions due to the lack of information. For example, the temporal expert may predict reach out to something while the ground truth Pick up a fork. Similarly, the spatial expert may predict utensil holder instead of fork in the shallower layers. However, after using Pros--atenttion to exchange information multiple times, the proposed method can collectively predict the correct action Pick up fork. Best viewed with zoom and color.
Cross-atfention. Cross-attention has been widely utilized in multi-modal learning to facilitate information exchange between different modalities such audio, visual, and text [34, 67, 40, 30, 18]. Recently, cross-attention between different views of the same video has shown impressive results [71, 75, 6, 26]. Similar to these, we propose cross-attention method using single RGB input, but with two distinct expert models: a spatial expert and a temporal expert. The two experts attend each other through cross-attention to achieve a balanced spatio-temporal understanding.
Foundation model. Trained web-scale datasets using self-supervised learning, foundation models [25, 4, 48, 45, 44] are highly adaptable and versatile. Foundation models show impressive performance on various tasks in computer vision [65, 62]. natural language processing [49, 57], and audio recognition [17]. In this work, we employ CLIP [44] as our spatial expert as shows impressive performance on more than 30 computer vision tasks.
Parameter-efficient transfer learning. Although the spreitraining and fine-tuning" paradigm with strong foundation models has demonstrated impressive performance on several computer vision tasks, it is computationally expensive and often unnecessary to fine-tune the full model [72]. Several works have demonstrated that learning only a small subset of parameters and keeping the remaining parameters frozen effective for NLP tasks [23, 29] and computer vision tasks [36, 66, 54, 46, 47]. Extending image foundation models by adding adapter architectures has shown favorable performance on action recognition [35, 72, 42]. The proposed method also employs adapter architecture with cross- attention between two experts. We empirically demonstrate that the proposed method outperforms existing adapter-based video models in terms of achieving balanced spatio-temporal understanding.
3 Method: Cross-Attentin in Space and Time
We introduce CAST, a method for balanced spatio-temporal representation learning for action recognition, as shown in Figure 3. We employ frozen spatial and temporal expert models that can be any vision transformer, consisting of 12 transformer blocks each. To facilitate information exchange between the experts, we introduce the bottleneck cross-attention in space and time (B-CAST) module on top of the frozen layers. This module enables the experts to exchange information and learn more balanced spatio-temporal contexts than separate experts. To improve adaptation to downstream tasks, we use adapter layers with small number of learnable parameters, following AIM [72]. In the following subsections, we provide a detailed description of each component of our proposed CAST.
3.1 Input embeddings
CAST takes only RGB videos inputs. The input is mini-batch of videos, I E X consisting of B videos of 2T frames, H x W spatial dimensions, and C channels. We apply patch tokenization to the input videos for the spatial expert and the temporal expert. For the spatial expert, we decompose every even frame of each video in into N non-overlapping patches of p x P pixels [11].
3

---

Clees frotion Hoad
Spatial Expert
Temporal Expert
Positional
soven
122 Blocks
sokan
Projestion
Projection
Food Forward
Food Forward
on sonal la Souutu//22
Straa tancona X011)
Layerhorm
LayerNorm
K
1V f
Information Eucharge
B-CAST
B-CAST
Layerfiom
Laverhoom
Down Projection
Down Projection
Adrapter
Adapter
Proposed
Huad
Mala Heed
T2S Cross-attention (Window shape: Timaa)
S21 Cross-attentio Wildoww shapw Spacs)
Layerhloem
Patich & Emboed
Paiich Pos
Embaed
Input Video
architecture
(d) Visusization of cross-etther
Figure 3: Overview of CAST. (a) CAST employs frozen spatial and temporal expert models. On top of the experts, we add a cross- atoss-attention module B-CAST to enable the exchange of information between the two experts. Additionally, parameters to the experts for better adaptation. (b) The proposed B-CAST consists temporal-to- spatial (T2S) and mnttial-to-temprrll (S2T) tross-attention to allow for better understanding of the spatio-temporal features in the video data. For efficient and effective learning, we incorporate cross-attention into the bottleneck adpater. We employ separate position embedding for each expert. (c) We visualize T2S and S2T cross-attentions Given a query, the model attends along the temporal axis only in T2S while the model attends along the spatial axes only in S2T.
employ adapters with small number of learnable
Then we pass the patches through frozen linear layer and add position embeddings obtain spatial embeddings, X, E BT For the temporal expert, decompose every two frames of each video into x p x p pixels non-overlapping tubes [1]. Then we pass the tubes through frozen linear layer and add position embeddings to obtain temporal embeddings, X, BBTNNDD
3.2 CAST architecture
The model architecture of each expert is the same as the ViT except for adapters and the B-CAST module. All the other parameters are frozen, while the adapter and B-CAST parameters are learnable.
For completeness, we first define the operations used and then describe the entire model architecture. Given an input X, we define Multi-Head Self Attention (MHSA) operation as follows:
MHSA(X) Softmax (XW e) (XWx) (XWv),
(1)
We. Wx. and Wv are the query, key, and value projection matrices, respectively. We also define the adapter operation with linear down and up projection matrices WD and Wy as follows:
ADAP(X) (XWD)
where (-)) is the GELU activation function [20].
For each attention block I, apply independent Multi-Head Self Attention (MHSA) for each expert along with skip connection follows:
(1) (1) + ADAP(MHSA(LLL)
MHSA(LN(XX))
where LN(-) denotes the Layer Normalization operation. The spatial path undergoes spatial attention, while the temporal path undergoes space-time attention following TimeSformer [3].

---

As shown in Figure 3 (b), to exchange information between the two experts, we apply the B-CAST operation (-) to Ye, and Yez from the expert e1 and e2 follows along with a skip connection:
B Y
We describe the B-CAST operation (-) in detail in Section 3.3.
Finally, we pass the output, denoted as B through a two-layer feed forward network (FFN) [1]]
with the GELU activation function in between the layers and another adapter obtain the next layer
input ((++)) as follows along with a skip connection:
x = B +FN((L(B")) + ADAP(LN((")).
(5)
Classification head. To produce the final prediction, we need to aggregate the outputs of both spatial and temporal experts. For the spatial expert, we average the frame-level class tokens from
the last attention block, x(!2), to obtain a single class token. We denote this operation as CLS(.). To obtain temporal expert features, we aggregate all the tokens from the last attention block of the
temporal expert. x(12) using the global average pooling GAP(-) operation. Then we add the adapter output of the CLS token and the adapter output of the GAP token produce fused token Z:
Z = ADAP(CLS(XX
( )2 ADAP(GAP(X, (2))).
(6)
Finally, we feed the fused token Z classification layer followed by softmax function obtain the predicted class probabilities. We train the model using the standard cross-entropy loss.
3.3 B-CAST module architecture
(MHCA) is a variant of the MHSA operation (1), where query tokens come from one expert (e1) and key and value tokens come from another expert (e2). This allows the experts to exchange information and benefit from the strengths of each other. We define the MHCA operation as follows:
Multi-Head Cross-Attention. Multi-Heate
MHCA(Y,;; Ye,) Softmax((Ye, (Y) W&) (Y,,, Wv),
where We. WK. and Wy are learnable query, key, and value parameter matrices respectively.
Temporal-to- caapporrl-t--Spatiee to-Spatial Cross-Attention. In Temporal-to-Spatii (T2S) cross-attention, query to- kens come from the spatial expert s, and key and value tokens come from the temporal expert MHCA(Y!)) y(!)) We depict the attention window in Figure 3 (c). Given a query, the model attends along the temporal dimension only. By using T2S aotssnatt,n,oon, the spatial expert can learn to attend to temporal features from the temporal expert. T2S MHCA leads to capturing spatio-temporal dependencies and improves the model performance in action recognition.
Spatial-to- teme-oora Cross-Attention. In Dpatial-to-Tempprr (S2T) cross-attention, query to- kens come from the temporal expert 1, and key and value tokens come from the spatial expert s: MHCA(Y," ;
(1) Y(!).. We illustrate the attention window in Figure 3 (c). Given query, the model attends along the spatial dimension only. By using S2T cross-attention, the temporal expert can attend to fineggrined spatial features from the spatial expert. S2T MHCA leads to a more balanced spatio-temporal understanding and improves the performance in fine-grained action recognition.
Bottleneck Pross-Attention in Space and Time. To achieve efficient and effective learning, we incorporate the T2S and S2T MHCA into bottleneck-shape adapters. We illustrate B-CAST architecture in Figure 3 (b). We plug the MHCA modules into adapters and add new learnable positional embeddings for each MHCA. We define the B-CAST operation for T2S &s(() as follows:
43 Y OMHCA + 1 WD.). E...).E.A. +L(Y))) WD.)) WD.)))
(8) where Wp. and WUs are linear down- and up- up-projection matrices for the spatial expert, and E, and E, are new positional embeddings for the spatial and temporal experts, o-)) is the GELU activation function, respectively. We can define the B-CAST operation for S2T, Φr(-) in similar manner. The output of B-CAST goes into a feed forward network using (5). Our empirical validation shows that the B B-CAST architecture is efficient and effective. (See Table L.)
5

---

PKtonn Uerth
FK100 Nour
K400
Harmanic Mesn PK100 Vert houn 552, Kroo
70.
66.0
70L6
54.5
71.6
00.8
07.6
00.6
WST-Asupeer AYdooMME
AST-Aclaptrr ViideoWAE
VHM
BEVT
AAAM WeeVT
CUP
V-DooMMC
DAST
(5)
[b]
(4)
Figure 4: Balanced spatio-temporal understanding performance. We visualize the action recogn- tion accuracies of existing methods and the proposed method. (a) We show the Top-1 accuracies of ST-Adapter and VideoMAE the EK EK100 verb and noun prediction tasks. (b) We show the Top-1 accuracies of AIM and BEVT on the SSV2, and K400. (c) For each method, we show the harmonic mean of Top-1 accuracies on the 100 noun, EK100 verb, SSV2, and K400. CAST shows more balanced spatio-temporal understanding capability compared to the existing methods. Best viewed with zoom and color.
Experimental Results
In this section, we present the experimental results that answer the following research questions:
(1) Do existing methods show balanced spatio-temporal understanding of videos? (Section 4.3) (2) What are the ingredients for a balanced spatio-tempont
understanding? (Section 4.3) (3) Is the proposed method effective? (Section 4.3, Section 4.4) (4) How can we effectively combine spatial and temporal models to achieve such balance? (Section 4.5) (5) Does the proposed method outperform statc-of-the-arr methods in terms of balanced spatio-temporal understanding? (Section 4.6) To this end, we first provide details about the datasets and implementation in Section 4.1 and Section 4.2, respectively.
4.1 Datasets
Action recognition. We evaluate the CAST on two public datasets for conventional action recogni- tion: Somethingg-Something- V2 (2S [19] and Kinetics-400 (K400) [24]. The SSV2 requires more temporal reasoning [3, 28] while the K400 is relatively static biased [32, 50].
Fine-grained action recognition. We evaluate the CAST on the fine- mne-grained action recognition task:
100 (EK 100) [10]. In contrast to conventional action recognition, EK 100 defines action combination of a verb and noun. Therefore, EK 100 as fine-grained action recognition. Since fine-grained action recognition requires correctly predicting both the verb and the noun to recognize an action it is more challenging than conventional action recognition, which requires predicting a single action label: e.g., K400 or SSV2.
refer to the action recognition
4.2 Implementation details
In this section, we briefly provide our experimental setup and implementation details. Please refer to the Appendix $ B for complete implementation details. We conduct all the experiments with 16 NVIDIA GeForce RTX 3090 GPUs. We implement CAST using PyTorch and build upon the existing codebase of VideoMAR [56].
Training. We sample 16 frames from each video to construct an input clip. For the K400 dataset, we apply dense sampling [15], while for SSV2 and EK 100, we use uniform sampling [61]. We then perform random cropping and resizing every frame into 224 x 224 pixels. We use the AdamW [39] optimizer with momentum betas of (0.9, 0.999) [7] and a weight decay of 0.05. By default, we train the model for 50 epochs, with the cosine annealing learning rate scheduling [38] and a warm-up period of 5 epochs. The default base learning rate, layer decay [2], and drop path are set 0.001, 0.8, and 0.2, respectively. We freeze all the parameters of each expert, except for the B-CAST layer, adapters, and the last layer normalization. We set the batch size per GPU with update frequency of 2.
Inference. Given an input video, we randomly sample frames multiple times construct input clips with multiple temporal views with multiple spatial crops. After the temporal frame sampling, we resize every frame so that the shorter side has 224 pixels. Then we perform spatial cropping get multiple 224 x 224 crops for each clip. We get the final prediction by averaging the predictions on (temporal views) x (spatial crops). For the K400 dataset, we use (5 clips) x (3 crops) views, while for the other datasets, we use (2 clips) (3 crops) views for the inference.

---

00009 ovesd Inrowseer avr 83%%
021
Figure 5: Improvements of CAST over each expert on EK100 noun classes We show the super-category-weer weighted average F1 score improvement of CAST over each expert. (Left) Improvement over CLIP. CAST outperforms CLIP for every super-category except meat and substitute. (Right) Improvement over VideoMAF. CAST outperforms VideoMAE for every super-category except furrittrre and prepared food. Best viewed with zoom and color.
4.3 Balanced spatio-tempore understanding
In Figure 4 (a), we present the top-1 1 accuracies of several existing models. In the EK100 verb prediction task, VideoMAE outperforms ST-Adapter with margin of 2.9 points (70.5% vs. 67.6%), while in the EK noun prediction task, ST-Adapte [42] outperforms VideoMAE [56] with a margin of 3.6 points (55.0% vs. 51.4%). As shown in Figure (b), BEVT [62] outperforms AIM [72] with a margin of 2.5 points (70.6% vs. 68. 6) on the SSV2 dataset, while on the K400 dataset, AIM outperforms BEVT with margin of 3.9 points (84.5% vs. 80.6%). We observe similar trends for other methods as well. Please refer to Section 4.6 for a detailed comparison. Our findings indicate that the performance of many existing models is significantly imbalanced toward either spatial or temporal understanding.
Ingredients for balanced spatio-temporal understanding. To achieve a more balanced spatio- temporal understanding, we can employ two expert models: a spatial expert and a temporal expert. For the spatial expert, we use CLIP [44], which has demonstrated impressive performance on various computer vision tasks. For the temporal expert, we use VideoMAE [56]. which has shown favorable tasks such as SSV2 and EK 100 verb prediction tasks. (Please refer to Section 4.6 for the accuracy details.) While each expert is highly specialized in its own domain, we aim to create synergy between them by exchanging information to improve the balanced spatio-temporal understanding performance.
performance on temporal-biased
Effect of CAST. In Figure 4 (c), we gauge the balanced spatio-temporal understanding performance of our spatial expert, temporal expert, and CAST. For each method, we calculate the harmonic mean of top-1 accuracies for EK 100 noun, 15000 verb, SSV2, and K400. The harmonic mean is an effective metric for gauging balanced performance because it gives more weight to lower-perforingg tasks. A higher harmonic mean value indicates that the performance over the different tasks is more balanced. Our spatial expert achieves an accuracy of 56.5%, while the temporal expert achieves an accuracy of 66.6%, and our CAST achieves an accuracy of 71.6%. 'hese results validate the effectiveness of our proposed method, CAST, which allows our spatial and temporal experts to make synergistic predictions by exchanging information with each other through cross-attention
4.4 Analysis on fine-grained action recognition
In this section, we provide detailed analysis of how the proposed CAST improves the balanced spatio- understanding in the fine-grained action recognition task: EK100.
Category-level performance analysis. In Figure 5, We present the 100 noun super-category- wise weighted average Fl score improvement of CAST over our spatial expert (CLIP) and temporal expert (VideoMAE). In Figure 5 left, we observe that CAST significantly improves upon the spatial expert, CLIP, in several super-categories such cutlery, utensils, and vegetables. These results indicate that the spatial expert achieves a more accurate understanding of fine-grained small objects interacting with the actors by leveraging the temporal context from the temporal expert. Similarly, in Figure right, we observe that CAST significantly improves upon the temporal expert, VA, in several categories such as vegetables and curlery. The trend is similar to the comparison with the spatial expert: CAST achieves more accurate understanding of fine- finengranedd small objects by leveraging the fine-grained spatial context from CLIP.
Qualitative analysis. To better understand the effectiveness of CAST, we provide qualitative analysis few sample frames from the EK 100 dataset in Figure 6. We show the predictions of CLIP, VideOMAE, and CAST. As expected, each expert model provides more accurate prediction in their respective tasks of expertise but shows weaker performance in the other task. In contrast,
7

---

Table Ablation study. To validate the effect of each component, we show experimental results on 100 dataset. In every experiment, we use the ViT-B/16 backbone for every expert. The best numbers are highlighted gray. (a) Effoct of information exchange.
the
Different information exchange methods.
(c) B-CAST architecture.
Pop-1
Acc
Top-l
Time
Top
Methasl
Veb
Noun
Act
Methed
Laie Layer-wise
Verb
Noun
Act.
Mettod
Param(0
Vorb
Nown
Act
Indop, expert wis udapter
Ensemble expers odapter
CAST
cagett whapes
725
55.3
60.3
42.9
48.7
Coneat
Lutrrl
CAST
Add
68.9
69.3
72.5
68.9
60.3
56.4
56.6
49.1
39.0
44.5
48.7
64.2
X-ain Kulpier
we
BAST
93.0
443
725
71.3
90.1
08.
45.7
47.9
(d) Effect of projection ratio.
(e) Effect of cross-attention window shape.
(1) Effect of the number of cross-atiention layers.
Top-1 Acc.
Window shape
Top-
Acc.
X X-amention layer
Top-1 Ace.
Ratio
Verb
Noun
Act.
T25
S2T
Verb
Noun
Act.
133
4-6
7-9
10-12
Verb
Noun
Act.
1/8
70.7
59.9
47.4
gpuce-lime
space-time
71.0
59.3
47.2
71.2
59.4
47.4
1/4
71.3
59.8
47.4
space-lime
space
601.
48.4
31.3
47.9
1/2
72.5
60.3
48.7
space
space
60.2
71.8
48.2
72.1
59.8
time
space
72.5
60.3
48.7
72.5
60.3
48.7
Effect of bi-directional cross -ate-nnttenionn
(h) Role of each expert.
Top-1 Acc.
Expert
Top-1 Acc.
Method
Verb
Noun
Act.
Spatial
Temporal
Verh
Noun
Indep. experts w/ adapter
68.1
54.2
41.7
CLIP
CLIP
69.3
58.8
46.0
S2T only
71.2
55.0
43.7
VideoMAB
CLIP
72.2
58.8
47.8
T28 only
60.5
46.7
VideoMAE
VideoMAE
69.8
49.9
40.3
CAST
72.5
60.3
48.7
CLIP
VideoMAE
72.5
60.3
48.7
CAST consistently shows correct predictions for both noun and verb prediction tasks, such as spoon and open. The qualitative examples demonstrate the effectiveness of CAST in achieving balanced spatio-tempora understanding, which is essential for fine-grate aie-ggrined action recognition.
4.5 Ablation study on CAST architecture
We conduct comprehensive ablation studies examine the design choices for the proposed CAST architecture. Here we conduct all experiments on the EK100 [10] dataset with 16-frame input videos and report the top-1 accuracy on the validation set. We employ CLIP [44] as spatial expert model and VideoMAE [56] as temporal expert model. For fair ablation study, we use the same hyperparameters for each experiment unless explicitly mentioned.
Effeet of information exchange. We investigate whether CAST effectively achieves a synergistic effect by exchanging information between the two expert models. In Table 1 (a), we compare CAST with three baselines. A baseline using two independent expert models without any information exchange (fully fine-tuned), ii) The same baseline as i), but we add adapters and fine-tune the adapters and head only, iii) A test-time ensemble of two independent experts (with adapters and heads fine-tuning only). The baselines predict nouns using the spatial model and verbs using the temporal model. We observe that the two expert models using ensembling achieve an improvement in Action accuracy by least 1.2 points compared the baselines without any information exchange. Furthermore, CAST achieves a best Action accuracy of 48..% These results suggest that information exchange is crucial for achieving balanced spatio-temporal understanding.
Comparison with simple information exchange baselines. We compare CAST with simple information exchange baselines: i) late fusion with addition, ii) late fusion with concatenation, úii) layer-wise fusion using the bidirectional lateral connection (element-wise addition) with linear projection. We add adapters and fine-tune the adapters and head only in all three baselines, using our training recipe in Section 4.2 For the details of baseline fusion methods, please see Figure 7. We show the results in Table 1 (b). It is worth noting that both the late fusion and layer-wise lateral connection baselines result in significant performance drop. Furthermore, without cross-attention yields inferior performance compared to the simple late fusion baselines. The results indicate that cross- -otss-attention in the bottleneck architecture is crucial for effective information exchange between spatial and temporal experts.
observe that layer- layer-wise fusion
Design of B-CAST module. To explore the most effective and efficient way integrate adapters for information exchange between the two expert models, we conduct an ablation study and present the results in Table 1 (c). For the details of the basclines, please see Figure 8. The first row of the table represents a baseline without the B-CAST module, which is equivalent to the identity function. Compared to this bascline, B-CAST achieves a significant improvement of 7.0 points in Action accuracy. The second row shows the performance of bascline with coos-aattntion but without the
8

---

Verb
Ollh X
CUP.
VideMMAE
WowetAr-Pan
Gown
VidasMAE Bart
CAST
Spoon
CASI:
Rank
DAST Brroos
CAST
Open
CAST
Scoop
CAST.
Mix
Figure 6: Qualitative examples from EK100 comparing CLIP, VideoMAE, and the proposed
CAST. Each expert model shows more accurate predictions in their expertise but shows weaker performance on the other task. However, CAST consistently shows correct predictions for both tasks, demonstrating the effectiveness of the proposed spatio-tempore cross-attention mechanism.
bottleneck adapters. The 9.3-point gap between this baseline and B-CAST highlights the importance of bottleneck adapters for effective information exchange between the two expert models. The third row (xdaate-a
a baseline with the adapters after cross- attention. Compared this baseline shows a 0.8 points drop in Action accuracy while having more than double the number of learnable parameters (44, 44 93. The results indicate that cross-attention in bottleneck is more effective and more efficient than the baseline. In summary, by placing cross-attention in the middle of the bottleneck adapter, B-CAST facilitates effective information exchange between the two experts and achieves synergistic effect.
B-CAST,
Effect of projection ratio in bottleneck. In this study, we investigate the impact of the down projection ratio in the bottleneck architecture presented in Table (d). The results demonstrate that a ratio of 1/2 yields the best performance. Notably, ratio of results in inferior performance, which we attribute to overfitting caused by the addition of more parameters.
Effect of cross-attention window shape. We investigate the impact of the window shape in the cross-attention mechanism in the T2S and S2T modules in Table (c). Please refer to Figure 3 (c) for the details of the window size. We maintain the same model capacity across different methods. Using space-time attention for both T2S and S2T modules results in the worst performance. We conjecture that learning joint space-time attention is challenging with the given model capacity [3]. On the other hand, using time attention in T2S and space attention S2T yields the best performance. Consequently, we adopt this configuration throughout the paper.
Effect of the number of cross-attention layers. We investigate the impact of the number of cosss-tttentin layers used. To this end, gradually increase the number of cross-attention layers starting from the top three layers, and report the results in Table (f). As we increase the number of cross-attention layers, we observe corresponding improvement in performance, as expected.
Effect of bi-directional cross-attention. To validate the effectiveness of bi-directional information exchange, ablate each cross attention time. We compare CAST with unidirectional information exchange baselines equipped with S2T or T2S aross-attention only. Each unidirectional information exchange baseline still has both experts. In Table (g), compared to CAST (48.7%). the S2T only baseline shows 5.0 points drop (43.7%) and the T2S only baseline shows 2.0 points drop (46.7%) in accuracy. The results validate the effectiveness of the proposed bi-directional cross-attention.
Role of each expert. In Table I (h), we investigate the role of experts within CAST by controlling the assignment of models to each expert. We observe that we can achieve the best performance of 48.7% when we employ CLIP as our temporal expert and VideoMAB as our spatial expert. When we employ one VideoMAE our spatial expert and another VideoMAE our temporal expert, we obtain 40.3% accuracy. When we employ one CLIP as our spatial expert and another CLIP as our temporal expert, we obtain 46.0% accuracy.
Interestingly, when we revert the role of CLIP and VideoMAE, i.e., we employ VideoMAE as the spatial and CLIP as the temporal expert, achieve a good performance of 47.8%. The results demonstrate that the B-CAST architecture facilitates effective information exchange between the two experts. Through the stacked B B-CAST, the experts can learn high-quality spatio-temporal representations by exchanging information, even when the roles are reverted.
In summary, these findings suggest that CAST achieves optimal performance when models are assigned to expert roles that align with their strengths. CLIP serves as effective spatial expert, whereas VideoMAE is more effective as a temporal expert. The B-CAST architecture encourages these experts to leverage their respective strengths through information exchange, resulting in enhanced spatio-temporal balanced understanding.

---

Table 2: Comparison with the state-of-the-arts on the EK100, SSV2 and K400 datasets. We show the Top-1 accuracy on each dataset and the harmonic mean (H.M.) of the Top-1 accuracies. The best performance in bold and the second best underscored.
GFLOP/
EK100 Top-1
SSV2 & K400 Top-1
All
Method
View
Verb
Noun
Act.
SSV2
K400
H.M.
H.M.
CLIP* [44]
140
54.9
52.7
33.8
47.8
78.9
59.5
56.5
EVL [35]
592
62.4
82.9
71.2
ST-Adapter [42]
607
67.6
55.0
69.5
82.7
75.5
67.3
AIM [72]
404
64.8
55.5
41.3*
68.1
84.5
75.4
66.7
MBT [40]
936
64.8
58.0
43.4
80.8
ViVIT PE [1]
990
66.4
56.8
44.0
65.9
81.7
73.0
66.6
TimeSformer [3]
2380
62.4
80.7
70.4
MViT [12]
170
67.7
80.2
73.4
MFoommr [43]
1185
67.1
57.6
44.1
68.1
80.2
73.7
67.3
ORVIT MF [21]
68.4
58.7
45.7
67.9
Vidco Swin [37]
282
69.6
82.7
75.8
BEVT [62]
282
70.6
80.6
75.3
VideoMAR [56]
180
70.5
51.4
41.7*
70.8
81.5
75.8
66.6
McMViT [68]
59
58.5
46.2
OMNIVORE [16]
69.5
61.7
49,9
71.4
84.0
77.2
70.8
MTV-HR [71]
930
68.0
63.1
48.6
68.5
82.4
74.8
69.8
CAST
391
72.5
60.9
49.3
71.6
85.3
77.9
71.6
"W conduct experiments with our own implementation.
4.6 Comparison with state-of-the-art
In this section, we evaluate the performance of CAST and state-of-ther methods terms of balanced spatio-temporal understanding on multiple datasets, as shown in Table 2. For each method, in addition reporting the top-1 accuracy of each task, report the harmonic mean of top-1 accuracies for i) SSV2, and K400, and ) EK100 verb, 1K0 noun, SSV2, and K400. For comparison with state-of-the-an art models, we have set different hyperparameters than those used in our ablation study. Please refer Table 4 for the details. For fair comparisons of the computation complexity, we show the GFLOPs/View. In cases where a compared method shows various GFLOPs/View depending on the dataset, we specifically note the lowest GFLOPs/View value for reference. For more detailed comparison of computation complexity, please refer to the Appendix
We observe that among the CLIP-based methods (the second group in Table 2). AIM [72] achieves favorable performance on the static-biased K400 dataset, with 84.5% accuracy. However, AIM shows a relatively lower performance of 68. %% on the cemporal-biased SSV2. On the other hand, VideoMAE [56], one of the state-of-the-an methods, shows 70.8% accuracy on the SSV2 dataset, which is more competitive than AIM. However, VideoMAE shows lower accuracy of 81.5% on the K400 dataset, less competitive than AIM. Our proposed method, CAST, demonstrates favorable performance on both the SSV2 (71 .6%) and K400 (85.3%) datasets, resulting in harmonic mean of 77.9%, which is higher than that of AIM (75.4%) and VideoMAE (75.8%). CAST shows more balanced favorable performance in fine-grained action recognition on the EK000 dataset. CAST achieves competitive Action accuracy of 49.3%, which is the second best among the compared methods.
understanding than the existing methods. Additionally, CAST shows
In terms of the overall harmonic mean of EK100 verb, 1K100 noun, SSV2, and K400 accuracies, CAST shows the best performance of 71 1.6. The results highlight the effectiveness of CAST. By exchanging information between spatial and temporal experts, our CAST shows a favorable balanced spatio-temporal understanding performance.
Conclusions
In this paper, we present a solution to the problem of action recognition models lacking balanced spatio-temporal understanding of videos. The proposed method, CAST, incorporates a spatial expert and a temporal expert that exchange information through cross-attention to achieve synergistic predictions. Our extensive experiments on datasets with varying characteristics demonstrate that CAST outperforms both individual expert models and existing methods in terms of a balanced spatio-temporal understanding measure: the harmonic mean of accuracies the datasets. The results highlight the effectiveness of CAST in achieving a balanced spatio-temporal understanding of videos, and suggest that CAST could have broad applicability in the field of video understanding.
10

---

Acknowledgment. This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (ITP)) grant funded by the Korea goVe)nment(MSIT) (No.RS- 2022-00155911, Artificial Intelligence Convergence Innovation Human Resources Development (Kyung Hee University)); by the Institute of Information & Communications Technology Planning & Evaluation (ITTP) grant funded by the Korea Government (MSIT) (Artificial Intelligence Innovation Hub) under Grant 2021 -02--000 by the National Research Foundation of Korea(NRF) grant funded by the Korea (SST))) (No. 192RIFFA117099)
References
[1] Anurag Arnah, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In ICCV, 2021.
[2] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei, Beit: Bert pre-training of image transformers, arXiv preprint axXiv2206.08255: 2021.
[3] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICMI., 2021.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot leamners. In NeurlPS, 2020.
[5] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
[6] Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In /CCV, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020.
[8] Jinwoo Choi, Chen Gao, Joseph CE Messou, and Jia-Bin Huang. Why can't i dance in the mall? learning to mitigate scene bias in action recognition. In NeurIPS, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced scarch space. CVPR workshops, 2020.
[10] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision. IJV, 130(1):33 (30(1))33-55, 2022.
[11] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[12] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, 2021.
[13] Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In CVPR, 2020.
[14] Christoph Feichtenhofer. Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In CVPR, 2016.
[15] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019.
[16] Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A Single Model for Many Visual Modalities. CVPR, 2022.
[17] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. arXiv preprint arXiv:2305.05665 2023.
[18] Satya Krishna Gorti, Noël Vouitsis, Junwei Ma, Keyvan Golestan, Maksims Volkovs, Animesh Garg, and Guangwei Yu. X-pool: Cross-modal language-video attention for text-video retrieval. CVPR, 2022.
[19] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
11

---

[20] Dan Hendrycks and Kevin Gimpel. Gaussian emror linear units (gelus). arXiv preprint arXiv: ww0.vv1606.0000 www..10000..418 08415, 2016.
[21] Roci Herzig, Flad Ben-Avraham, Karttikeya Mangalam, Amir Bar, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Object-region vidco transformers. In CVPR, 2022.
[22] Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoetter, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In CVPR, 2020.
[23] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder.
Compacter: Efficient low-rank
hypercomplex adapter layers. In NeurlPS, 2021.
[24] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier. Sudheendra Vijaynnarasimhan. Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kincties human action vidco dataset. arXiv preprint X 2017.
[25] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova, Bert: Pre-training of deep hidirectional transformers for language understanding. In NAACL-HLT, 2019.
[26] Hannah Halin Kim, Shuzhi Yu, Shuai Yuan, and Carlo Tomasi. Cross-attention transformer for video interpolation. In ACCV, 2022.
[27] Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong. Movinets: Mobile video networks for efficient video recognition. In CVPR, 2021.
[28] Matthew Kowal, Mennatullah Siam, Md Amirul Islam, Neil DB Bruce, Richard P Wildes, and Konstanti- nos G Derpanis. A deeper dive into what deep spatiotemporal networks encode: Quantifying static vs. dynamic information. In CVPR, 2022.
[29] Brian Lester, Rami AI-Rfou, and Noah Constant. The power scale for parameter-efficient prompt tuning. In EMNLP, 2021.
[30] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align hefore fuse: Vision and language representation learning with momentum distillation. NeurlPS, 2021.
[31] Kunchang Li. Yali Wang. Peng Gao, Guanglu Song. Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatiotemporal representation learning. In ICLR. 2022.
[32] Yingwei Li, Yi Li, and Nuno Vasconcelos. Resound: Towards action recognition without representation bias. In ECCV, 2018.
[33] Ji Lin, Chuang Gan, and Song Han, Tsm: Temporal shift module for efficient video understanding. In ICCV, 2019.
[34] Yan-Bo Bo Lin, Jie Lei, Mohit Bansal, and Gedas Bertasius. Felipse: Efficient long-range video retrieval using sight and sound. In ECCV, 2022.
[35] Ziyi Lin, Shijic Geng, Renrui Zhang, Peng Gao, Gerard Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video leameer. ECCV, 2022.
[36] Yen-Cheng Liu, Chih-Yao Ma, Junjiao Tian, Zijian He, and Zsolt Kira. Polyhistor: Parameter-efficient multi-task adaptation for dense vision tasks. In NeurlPS, 2022.
[37] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022.
[38] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 2016.
[39] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[40] Arsha Nagrani, Shan Yang, Anurag Amab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. In NeurlPS, 2021.
[41] Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng. Jianlong Fu, Shiming Xiang. and Haibin Ling. Expanding language-image pretrained models for general video recognition. In ECCV. 2022.
[42] Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter-efficient tonde-to-video transfer learning. In NewrIPS, 2022.
12

---

[43] Mandela Patrick, Dylan Campbell, Yuki M. Asano, Ishan Misra Florian Metze, Christoph Feichtenhofer. Andrea Vedaldi, and João F. Henriques. Keeping your eye the ball: Trajectory attention in vidco transformers. In NeurIPS, 2021.
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In JCML, 2021.
[45] Aditya Ramesh, Mikhail Pavlow, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Nlya Sutskever. Zero-shot text-to-image generation. In ICMI., 2021.
[46] Sulvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In NeurlPS, 2017.
[47] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of multi domti-domain deep neural networks. In CVPR, 2018
[48] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffiusion models. In CVPR, 2022.
[49] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzama Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter apen-sccess multilingual language model. arXiv preprint arXiv:2211.05100, 2022.
[50] Laura Sevilla-Lara, Shengxin Zha, Zhicheng Yan, Vedanuj Goswami, Matt Feiszli, and Lorenzo Torresani. Only time can tell: Discovering temporal data for temporal modeling. In WACV, 2021.
[51] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in vidcos. In NeurlPS, 2014.
[52] Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz. Gate-shift-fuse for vidco action recognition. arXiv:2203.08897 2022.
[53] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint 2r3 2023.
[54] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. In NeurlPS, 2022.
[55] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zhigniew Wojna. Rethinking the inception architecture for computer vision. CVPR, 2016.
[56] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-cfficient learners for self-supervised video pre-traning. In NeurlPS, 2022.
[57] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 13971, 2023.
[58] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri, Learning spatiotemporal features with 3d convolutional networks. ICCV, 2015.
[59] Du Tran, Heng Wang. Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look spatiotemporal convolutions for action recognition. CVPR, 2018
[60] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NewriPS, 2017.
[61] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Lue Van Gool, Temporal segment networks for action recognition in videos. TPAMI, (1 (1)2::0222222555 2018.
[62] Rui Wang, Dongdong Chen, Zuxuan Wa, Yinpeng Chen, Xiyang Dai, Mengehen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. Bevt: Bert pretraining of video transformers. In CVPR, 2022.
[63] Rui Wang. Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengehen Liu, Lu Yuan, and Yu-Gang Jiang. Masked video distillation: Rethinking masked feature modeling for seff-supervised video representation learning. In CVPR, 2023.
[64] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
13

---

[65] Yi Wang, Kunchang Li. Yizhuo Li. Yinan He. Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General vidco foundation models via generative and discriminative learning. arXiv preprint 1rXiv:2212.33191, 2022.
[66] Zifeng Wang, Zizhao Zhang, Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, 2022.
[67] Xi Wei, Tianzhu Zhang, Yan Li, Yongdong Zhang, and Feng Wu. Multi-modality cross attention network for image and sentence matching. In CVPR, 2020.
[68] Chao Yuan Wu, Yanghao Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Memvit: Memory-augmentee multiscale vision transformer for efficient long-term video recognition. In CVPR, 2022.
[69] Wenhao Wu, Zhun Sun, and Wanli Ouyang. Revisiting classifier: Transferring vision-language models for video recognition. In AAAI, 2023.
[70] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning for video understanding. In ECCV, 2018.
[71] Shen Yan, Xuchan Xiong, Anurag Arna,, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview transformers for vidco recognition. In CVPR, 2022.
[72] Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Li. Aim: Adapting image models for efficient video understanding. In ICLR, 2023.
[73] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2017.
[74] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In ECCV, 2018
[75] Haowei Zhu, Wenjing Kc, Dong Li, Ji Liu, Lu Tian, and Yi Shan. Dual cross-attention lcarning for fine-grained visual categorization and object re-identification. In CVPR, 2022.
14

---

Appendix
In this appendix, provide additional acchiecuur//mmeeeenttttdddaan details, experimental settings, quanitative//ualitttvee results, limitations of our method, and broader impact of our method to complement the main paper. We organize the appendix as follows:
A. Architecture details our framework
B. Implementation details and experimental settings
C. Details of datasets in our experiments
D. Additional quantitative and qualitative results
E. Comparison with State-of-the-Ar with additional information
F. Class-wise F1 score on EK 100 verb and noun classes
G. Additional qualitative analysis on EK100
H. Limitations
I. Broader impacts
A Architecture Details
In this section, provide details of our B-CAST architecture. Let us assume employ CLIP [44] as spatial expert and VideoMAE [56] as temporal expert.
Table 3: Stage-wise details of the two experts in B-CAST. We provide detailed description of each operation performed in each expert. The input for this example is video consisting of 16 frames. MHCA represents multi-head cross-attention applied with a specific window shape: either time-attention or space-attention. In the description, B, D, and represent the batch size, embedding dimension, temporal sequence length, and spatial sequence length, respectively. We omit the Layer Normalization and activation functions for simplicity.
Spatial Expert
Temporal Expert
B-CAST Stage
Remark
Output Tensor Shape
Remark
Output Tensor Shape
Linear projection
Linear projection
Up Projection
116678-196x668
with ratio 2.0
with ratio 2.0
Attach CLS token of Y,
Post Processing
(161)))))B888
Reshape: BxT-NxD
Y;:Bx8
Reshupe: NxB-TxD
T2S MHCA(Y,, Y,)
S2T MHCA(Y, Y,)
Cross-Attention
Y,B--16x883884
Yy:B-8x196x384
Window shope: time
Window shape: spacc
Positional
Y,:B.196x8x384
Y;1B-8x196x3884
# parameters: 8x384
parameters: 196x384
Embeddings
Y,:B-196x8x3884
Y,:B-8x196x384
Detuch CLS token of Y,
Y,:B-196x8x384
Detach CLS token of
Y,:B-8x196xx884
Pre processing
Reshape:B-Nx
Y;:B-19xx8x384
Reshape: B-TxNxD
Y,:B-8x 196x384
Gather
Gather Y,
1=:(1994))B--8x338
Gather Y, from Spatial Expert
Y xxBx8-199xX3
Features
from Temporal Expert
Y;BBx8-96x3844
Y (196+1)xB-8x3388
Linear projection
Linear projection
Down Projection
(196+))))B888
XrBx8-196x384
with ratio 0.5
with ratio = 0.5
Input of B-CAST
(19691)))BB887766
Y,Bx 181666666768
B-CAST architecture. Table 3, we provide stage-wise details of the two experts B-CAST. Given an input from multi-head self-attention (MHSA) layer, we first pass it through the linear projection layer of each expert. Subsequently, the two experts exchanges their features each other. The multi-head cross-attention (MHCA) layer each expert enables the effectively exchange of information between the two experts. Afterward, pass the output tensors from the B-CAST
15

---

through the Feed-FFe Forward Network (FFN). We repeat this process in stacked manner, consisting of 12 blocks, each comprising the MHSA module along with adapters, B-CAST, and FFN module along with adapters. Finally, we feed the resulting tensors to classification head to predict action.
Architecture details of different information exchange methods. In Figure 7, we visualize the architectures of the simple information exchange baselines presented in Table (b) of the main paper. These baselines involve fully fine-tuning the two expert models without the use of adapters, following the training recipe outlined in Appendix $ B. In Figure 7 (a), present the add baseline, which facilitates information exchange between the spatial and temporal experts through late fusion using element- wise addition. In Figure 7 (b), we present the concat baseline, which exchanges the information between the spatial and the temporal experts through late fusion using concatenation. In Figure 7 (c), we show the lateral connection baseline, which enables information exchange between the spatial and temporal experts through layer-wise lateral connections, incorporating linear projections.
Noun
Verb
Noun
Verb
Noun
Verb
Classificatio Head
Classiicatio Head
Clasefication Head
Cressiictiio Hennd
Head
CAP
E
Down Projection
xtl B coke
19 diocos
[Late
Peed Forvard
Pead hawars
CLS Toloon 8
CAP token
Thkknn 3
Concatenate
(Late fusices)
GAP Token 2"
Spatial Expert
Tamporal Expert
Spatial Expert
Temporai Export
Mus-hea
Nats-tcc
(a) Add
(b) Concat
Lateral
Figure 7: Architecture visualization of the different information exchange baselines.
Architecture details of the baselines in the B-CAST architecture ablation study. In Figure 8, we visualize the architectures of the baselines used in the B-CAST architecture ablation study, as presented in Table (c) of the main paper. In Figure (a), we illustrate the identity baseline, which serves as a baseline without the B-CAST module. In Figure 8 (b), we present the w/o adapter baseline, which includes cross-attention but does not incorporate the bottleneck adapters. In Figure 8 (c), show the X-attn. -adatee- baseline, which uses adapters positioned after the cross-atts catsssttttttionn stage. For reference, we include our B-CAST architecture in Figure 8 (d), which represents the final model used our study.
Up Sualle to
Pinjeciion
Project
Down Irojection
Farponal Souttl
Idunrity
ideerity
T25
Attrtion
Project on scr1
Alverion
TDa
K V1
TV 1K
Exper/eem
Expenfor
Dxper4oom
Liyarboom
adapter
Xaattn.-acaapoor
C-CAST
Figure 8: Architecture visualization of the baselines used the B-CAST ablation study.
Classification head. We use different classification strategies for conventional action recognition and mnn-grained action recognition as shown in Figure 9. i) For conventional action recognition datasets, i.e., Kinetics-400 (K400) and CLS and GAP tokens from the two experts to predict actions, as depicted in Figure 9 (a). The spatial expert, CLIP [44], generates one CLS token for each frame. To obtain single CLS token,
Something-V2 (SSV2), CAST combines the
16

---

Action
Action
Noun
Verb:
Classification
Classilication Head
Classification
Head
Adapter
Adapter
CLS Token
GAP Token
12"
Taken
Token
Ri
Average
Average
Average
Average
ag exiiee stoce
Spetial Path
Temporal Path x12 Blocks
Spatial Path
Temporal Path x12 Blocks
Blocks
x12 Blocks
(a) Conventional action recognition
(b) Fine-grained action recognition
Figure 9: Classification head architecture. We use different classification strategies for conventional action recognition shown (a) and fine-grained action recognition shown in (b). T, N, and D denote the temporal sequence length, spatial sequence length, and embedding dimension respectively. The output from the spatial expert consists of frame-level feature vectors. Each frame-level feature vector consists of N patches and one CLS token. The output of the temporal expert is video-level feature vector, consisting of N patches.
we take the average of the CLS tokens from all frames of the input video. The temporal expert, VideoMAE [56], performs global average pooling on all output features to obtain single GAP token. After passing through adapters, we add the CLS and GAP tokens together and feed the token into linear classification head. ii) In fine-grained action recognition dataset, i.e., EPICCKITCCHNSS-O0 two separate classification
(EK100), where model needs to predict both verb and noun, heads instead of single head shown in Figure 9 (b). Specifically, we feed the CLS token from the spatial expert into linear classification head for noun prediction and the GAP token from the temporal expert into another linear classification head for verb prediction.
B Implementation Details
In this section, provide more details of our experimental setup and implementation of each dataset. We conduct the experiments with 16 NVIDA GeForee RTX 3090 GPUs. We implement CAST using PyTorch and build upon the existing codebase of VideoMAE [56].
Data preprocessing. After sampling the videos 16 frames, We randomly crop each frame the video and resize it to 224 224. We also apply data augmentation techniques, including mixup [73]. label smoothing [55], horizontal flip, color jitter, and randaugment [9], repeated augmentation [22] to diversify the training data. We do not use horizontal flip on SSV2. Note that if the patch embedding layer of the temporal expert has a time stride value of 2, e.g., VideoMAE [56], we only take even frames for the spatial pathway. After the patch embedding layers, both experts take an input of channels x frames x 224 width x 224 height. We use the same data preprocessing protocol in all the experiments.
Model training. We conduct experiments using 2 nodes, each equipped with GPUs. To ensure efficient multi-node training, we utilize the DeepSpeed library. Additionally, we increase the effective batch size by implementing gradient accumulation to update the model weights. For the EK100 datasct, we set the update frequency to 4 iterations 4 resulting in a total batch size of 24
Pttpps:/ / thub. com / crosoft /teepSpeed We use the update frequency of 2 iterations for the additional quantitative analysis experiments in Appendix
S D.
17

---

Table 4: Hyperparameters used and the fine-tuning configuration for each dataset.
Config
EK100
K400
SSV2
Optimizer
AdamW [39]
Adam W
AdamW W le-3
Base lcarning rate
le-3
1c-3
Weight decay
0.05
0.05
0.05
Optimizer momentum [7]
B1 B2 029,0.9..999
P M2 11.22=0.900999 0.9.0.999
B1 P1.82=0.990.0 0.999
Gpu per batch size
6
6
Update frequency
4
4
Learning rate schedule
cosine decay [38]
cosine decay
Cosine decay
Warmup epochs
5
5
Training epochs
50
70
50
Flip augmentation
yes
yes
no
Color jitter
0.4
0.4
RandAug [9]
(9, 0.5)
(9, 0.5)
(9, 0.5)
Label smoothing [5S]
0.1
0.1
0.1
Mixup [73]
0.8
0.8
Drop path
0.2
0.2
0.3
Layer-wise Ir decay [2]
0.75
0.75
0.75
per GPU. We linearly scale the base learning rate, then actual Ir = base Ir x total batch size/ 256. In Table 4, summarize the fine-tuning configuration and the hyperparameters used in Table 2, Table 9, Table 10, and Table 11.
Pre-trained weights. We take the off-the--shel pre-trained weights of the two expert models. For our main temporal expert, we take the VideoMAE [56] weights pre-trained on the K400, and SSV2 datasets from the official repository Since VideoMAE does not provide pre-trained weights specifically for the EK100, pre-train VideoMAE on the EK100 without incorporating extra video datasets. The pre-training process follows the recipe described in the VideoMAE paper [56]. For all other experiments, make of the pre-trained weights provided by the respective model repositories to ensure consistency and reliability our results.
C Datasets.
Action recognition We evaluate our B-CAST module on two video datasets:Kineticsss Kinetics400 (K400) [24], dataset for action recognition that contains around 300K video clips and 400 human action classes. The dataset is split into train/val/test, with 240K/20K/40K video clips. The videos are all trimmed to around 10 seconds from different You'ube video. ii) SSV2 contains over 220K short video clips labeled video clips of humans performing pre-defined, basic actions with everyday objects. The dataset is split into train/val/test, with 168K/24K/27K and have 174 human-objects interaction categories.
(SSV2) [19]. i) K400 is large-scale third-person video
Fine-grained action recognition We evaluate our B-CAST module on a Compositional Action dataset: APIC-KITCHENS-IT (EK 100) [10]. EK100 is a (ane-scale((0hhorss) egocentric video dataset that records several days of kitchen unseripted activities. It consists of 90K action segments, which are split into train/val/test sets of 67K//0K/13K Differ from preceding two datasets, EK100 define an action and nouns to recognize an action, it is more challenging than recognizing actions a dataset where actions are represented by single label e.g., Kinetics-400,
combination of a verb and a noun. Because it requires matching both verbs
D Additional Quantitative Analysis
In this section, we provide additional results to complement the main paper. We demonstrate (1) the generality of CAST with different ViT architectures and pre-trained weights in Appendix $ D.I, and (2) the effect of B B-CAST-specific positional embeddings in Appendix D.2.
Shtpps: / //g/ thub.
18

---

D.1 Generality
In this section, we showcase the generality of the proposed method. We demonstrate that CAST works well with any ViT backbone and pre-trained weights. We conduct all the experiments on the EK100 dataset.
CAST is pre-training dataset agnostic. CAST is pre-training dataset agnostic. In Table 5, we compare CAST to a CAST variant where we replace CLIP with ViT-B model pre-trained on the ImageNet-21K (IN21K) dataset. As a reference, we also show the performance of the independent experts using two separate expert models without any information exchange. When equipped with IN21K pre-trained ViT-B, CAST still achieves reasonable performance with an Action accuracy of 45.5%, while the baseline of independent experts achieves 40.4% accuracy.
Table 5: Effeec of CLIP pre-trained weights.
Pre-training dataset
Information
Top-1 Acc.
Method
Spatial expert
Temporal expert
Exchange
Verb
Noun
Action
Independent experts
IN21K IN21K
EK100
X
69.7
50.9
40.4
CAST
EK100
/
70.9
56.8
45.5
CAST
CLIP
EK
V
72.5
60.3
48.7
In Table 6, we analyze the effect of pre-training datasets the temporal expert, VideoMAE, in CAST. We show the results of using EK100, SSV2, and K400 pre-trained VideoMAE weights. We also investigate the impact of pre-training datasets on the spatial expert, CLIP. We present the results of using IN21K and CLIP pre-trained weights. When using CLIP pre-trained weights for the spatial expert, we observe stable Action accuracy ranging from a minimum of 48.7% to a maximum of 49. 19.4%. We observe similar trend when use IN21K pre-trained CLIP weights. These results demonstrate that the proposed method agnostic to the pre-training datasets.
Table 6: Effect of pre-trained weights.
Pre-training dataset
Top-1 Acc.
Spatial expert
Temporal expert
Verb
Noun
Act.
IN-21K
EK 100
70.9
56.8
45.5
IN-21K
SSV2
71.6
56.1
45.3
IN-21K
K400
72.2
56.4
45,9
CLIP
EK100
72.5
60.3
48.7
CLIP
SSV2
73.3
60.0
49.0
CLIP
K400
72.9
60.4
49.4
CAST model-agnostic. In Table 7, we demonstrate that CAST is model-agnostic. In this experiment, we replace our spatial and temporal expert models with other existing models. We employ EVA [53], an extension of CLIP that has shown excellent performance in various vision tasks, as our spatial expert. We employ MVD [63] as our temporal expert. The results show that CAST achieves similar performance when we employ different models as the experts. For example, EVA + MVD achieves an Action accuracy of 49.2%, while CLIP + VideoMAE achieves 48.7% Action accuracy.
Table 7: Effect of employing different models as experts.
Model architecture
Top-I I Acc.
Spatial expert
Temporal expert
Verb
Noun
Act.
CLIP
VidcoMAE
72.5
60.3
48.7
CLIP
MVD
73.1
60.1
49.3
EVA
VideoMAE
73.1
59.8
49.1
EVA
MVD
73.7
60.1
49.2
19

---

Patch R²
query
K key
K
K
Q
K
K
(a) Attention along temporal axes
(b) Attention along spatial
(c) Attention along apatio-tempont axes
Figure 10: Visualization of attention window shape.
D.2 B-CAST-specific positional embeddings
We investigate the impact of B-CAST-specific positional embeddings, as depicted in Figure (d). In the sempora--to-ppatin (T2S) cross-attention, the spatial expert attends along the temporal axis, as shown Figure 10 (a), while in the self-attention stage during pre-training, the spatial expert attends along the spatial axes, as depicted in Figure 10 (b). Consequently, the spatial expert lacks information about temporal patch sequences. Similarly, in the spatial-to-temprral (S2T) cross-attention, the temporal expert attends along the spatial axis, as illustrated in Figure 10 (b), while during self- attention stage during pre-training, the temporal expert attends along the spatio-temporal axes, as shown in Figure 10 (c). Consequently, the temporal expert lacks information about spatial-only patch sequences. To address this limitation, we introduce new learnable positional embeddings that are specific to T2S and S2T cross-attention.
As shown in Table 8, adding the B-CAST-ppecfic positional embeddings boost the performance by 1.2 points compared to without using the B-CAST-specific positional embeddings. The results indicate the effectiveness of the B- B-CAff-specific positional embeddings.
Table Effect of B-CAST-specific positional embeddings.
Top-1 I Acc.
Method
Verb
Noun
Action
CAST w/o B-CAST-specific positional embeddings
71.3
59.7
47.5
CAST w/ B-CAST--ppeciffe positional embeddings
72.5
60.3
48.7
E
Comparison with State-of-the-Art
To provide more comprehensive information, augment the tables for comparison with state-of- the-art in the main paper. Table 9, Table 10, and Table 11, corresponding to the respective datasets, include additional details such as the number of frames per clip, the number of temporal and spatial views used for inference, the computation complexity, and the number of learnable parameters for each model. For the details of the hyperparameters used, please refer to Table 4.
F
Class- Wise Performance Comparison
We provide class-wise Fl score improvement of CAST over our spatial expert (CLIP) and our temporal expert (VideoMAF). We show the verb-class-wise F1 score improvement over CLIP in Figure 1l and the improvement over VideoMAE in Figure 12. We show the noun-class class-wise Fl score improvement over CLIP in Figure 13 and the improvement over VidcoMAE in Figure 14.
20

---

Table Comparison with the state-of-the-arts on the EPIC-Kitchens-100 dataset. We show the Top-1 accuracy for Action, Noun, and Verb prediction tasks as well as the number of frames per clip, the number temporal and spatial views used for inference, the computation complexity, and the number of learnable parameters for each model. The best performance is in bold and the second best underscored.
Learnable
Top Acc.
Method
Backbone
Frames
Views
TFLOPs
Puram (M)
Verb
Noun
Action
SlowFast [15]
ResNo150
50.0
58.5
(12)
Ne100
MoViNet [27]
MeViNet-AG
72.2
57.3
47.7
CLIF [64]
VIT-B
2x3
0.84
86
55.5
52.3
33.9
AIM [72]
WI-B
16
2x3
2.42
64.8
55.5
41.3
ST.Adapeer [422]
VITBB
3xc
67.5
55.0
[40]
VIT-B
43.4
[1]
VIT-I
56.8
MiFormee [45]
VIT-
3x
3.56
67.1
57.6
44.1
ORVIT-MF-HR [2]]
VIT-B
16
10x3
68.4
58.7
45.7
VideoSwin [37]
Swin 9
89
67.8
57.0
46.1
VideoMAE [56]
VIT-B
2x3
1.08
70.5
51.4
41.7
McMVIT [6]]]
VIBB
1xl
46.2
OMNIVORE [16]
61.2
MTV-JIR [71]
MIV-B
32
4x1
3.72
310
68.0
63.1
48.6
CAST w/ CLIP & VideOAF peetmined on MICH-IITHHENSI00
CAST-R
2x3
2.35
72.5
60.9
49.3
"We conduct experiments with own implementation.
Table 10: Comparison with the state-of-the-are on the Saanething-Somethingg
dataset. We show the Top-1 accuracy as well as the number of frames per clip, the number of temporal and spatial views used for inference, the computation complexity, and the number of learnable parameters for each model. The best performance is in bold and the second best is underscored.
Method
Backbone
Frames
Views
TFLOP:
Learnable Param (M) Top-I Acc.
SlowFast [15]
ResNet101
8--32
1x3
0.32
63,1
CLIP* [44]
WIT-B
2x3
43,2
EVL [35]
VIT-B
32
1x3
2,05
29
62.4
ST-Adapter
VIT
32
3x1
1,96
69.5
AIM [?]]
VIVIT FE [1]
VIT-L
32
4x3
11.89
311
65.9
TimeSformer [3]
VIT-L
64
1x3
7.14
62.4
MVT [12]
VIT-B
1x3
1.37
67.7
MFormee [3]
VIT-L
32
1x3
3,56
68.1
Video Swin [37]
Swin-B B
32
1x3
0.96
69.6
REVT [62]
Swin-B
1x3
0.96
70.6
VideoMAE [56]
VIT-B
16
2x3
1.08
87
70.8
ORVIT-MF-L
ViT-L
OMNIVORE [16]
Swin-B
32
71.4
MTV.HR [7]]
MTV-B
32
4x3
11.16
310
68.5
CAST VideoMAE peetrained on Somethingg-Somettingg
CAST-B
2x3
2.35
71.6
We conduct experiments with our own implementation.
G Qualitative Analysis
To better understand the effectiveness of CAST, we provide qualitative analysis on more sample frames from the EK100 dataset in Figure 15. Each expert model provides more accurate prediction in their respective tasks of expertise but shows weaker performance in the other task. In contrast, CAST consistently shows correct predictions for both noun and verb prediction tasks. The qualitative exam- ples demonstrate the effectiveness of CAST in achieving balanced spatio-temporal understanding. which is essential for fine-grained action recognition.
H Limitations
Despite achieving a good balanced spatio-temporal understanding performance, CAST has a few limitations well. CAST has small number of learnable parameters since freeze the expert models except for the adapters. However, the computational complexity of CAST not negligible due to the utilization of two expert models. Due to resource limitations, we are unable to conduct experiments various input video lengths and model sizes. Lastly, cross-attention layers require features the same dimension for attention operation. Therefore, if the two model have significantly different architectures, it might be challenging for CAST to employ the two models.
21

---

Table 11: Comparison with the state-of-the-arts on the Kinetics400 dataset. We show the Top-1 accuracy as well as the number of frames per clip, the number of temporal and spatial views used for inference, the computation complexity, and the number of learnable parameters for each model. The best performance is in bold and the second best is underscored.
Method
Backbone
Frames
Views
TFLOPs
Learnable Param (M)
Top-1 Acc.
SlowFast [15]
ResNet101
10x3
7.02
79.8
X3D [13]
X3D-XL
16
10x3
145
79.1
MoVINet [27]
MoViNet-A6
1x1
0.39
81.5
UniFormer [31]
Hybrid-B
32
4x3
3.12
50
83.0
CLIP* [44]
VIT-B
8
5x33
2.2
77.3
[VS[
VIT-B
32
3x
1.78
84.2
ST-Adapter [22]
VIT-B
32
3x1
1.82
82.7
TexidVis [69]
ViT-B
83.6
AIM [72]
VIT-B
32
3x1
2.43
84.7
[41]
ViE-B
ViVIT FE
WTL
128
1x3
11.94
311
81.7
TimeSformer [3]
VIHL
96
1x3
25.06
430
80.7
MVL [12]
ViF-B
32
5x1
0.85
80.2
BEVT [62]
Swin
32
4x3
3.38
80.6
MFForeer [43]
VTL.
10x3
35.55
80.2
Video Swin [37]
Swin
32
4x3
7.25
83.1
VideoMAF [56]
VIT-B
5x3
2.7
81.5
OMNIVORE [16]
Swin-B
32
84.0
MTV-HR [7]]
MTV-R
32
4x3
11.16
310
82.4
CAST w/ VideoMAF. pretrained on Kinetics 400
CAST
16
5x3
5.87
45
85.3
Wee conduct experiments with our own implementation.
I Broader Impacts
Our work is the task of human action recognition from videos. Surveillance could be one application, which might have privacy related concerns when the technology deployed. Other consumer applications like personal or internet video search and tagging is expected to benefit individuals and organizations alike by helping them more efficiently maintain human centered data.
22

---

Drahh
sercch
Inead
dose
move
Bam
hold
coo
Fold
harg
wath
thow
tum-coon
Stepen
mecare
dsp
bush
sirk
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
F1 score Improvement
Figure 11: Improvements of CAST over CLIP on EK100 verb classes. We show the class-wiss wise Fl score improvement of CAST over CLIP. CAST achieves improvement of 17.8 points average. Best viewed with zoom and color.
23

---

Drah
lww
4986
sercch
Fee
pather
wrsp
pu1
Insot
shate
180
Drow
pour
bun
ma
walh
sma
Soupe
rel
lsnr
dsp
dock
hard
s1rk
Jumdoom
rondtton
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
improvement
Figure 12: Improvements of CAST over VideoMAE on EK100 verb classes. We show the class- wise Fl score improvement of CAST over VideoMAE. CAST achieves an improvement of 2.2 points on average. Best viewed with zoom and color.

---

Figure 13: Improvements of CAST over CLIP on EK100 noun classes. We show the class-wird Fl score improvement of CAST over CLIP. CAST achieves an improvement of 7.9 points on average. Best viewed with zoom and color.
25

---

Figure 14: Improvements of CAST VideoMAE on EK100 noun classes. We show the class-wise Fl score improvement of CAST over VideoMAE. CAST achieves an improvement of 9.2 points on average. Best viewed with zoom and color.
26

---

Noun Classes
Verb Classes
CLIP:
Carrot
CLIP:
Pan Mushroom
CLIP:
Take
X
CLIP:
Pour
VideoMAE:
VideoMAE:
VideoMAE: Tum-on
VideoMAE: Put
CAST:
Carrot
CAST:
CAST:
Tum--n
CAST:
Pour
CLIP:
Knife Bag Knife
CLIP:
Potato
CLIP:
Turn-on / VideoMAE: Turn-on Tum-on
CLIP:
Take X
VideoMAE:
VideoMAE:
Pepper Potato
VideoMAE: Take X
CAST:
CAST:
CAST:
CAST:
Cut
CLIP:
Fork
CLIP:
Peach / Knife Peach
Hang VideoMAE: Throw
CLIP
CLIP:
Put
VideoMAE: Galic
X
VideoMAE:
VideoMAE: Move
CAST:
Been:green
CAST:
CAST:
Throw
CAST:
Shake
GT:
Asparagus
Package VideoMAE: Package
CLIP:
CLIP:
Nuts
CLIP:
Wash
CUP:
Take X
VideoMAE:
Bag X
VideoMAE: Dry
X
VideoMAE: Mix
CAST:
Blueberry
CAST:
Nuts /
CAST:
Wash
CAST: Mix
J01
CLIP: Recipe v
CLIP:
Sponge
CLIP: Scoop
CLIP: Take
VideoMAE:
Box X
VideoMAE: Sponge
VideoMAE: Insert
VideoMAE: Dry /
Recipe
CAST:
Liquid
CAST: Scoop
CAST: Dry
Figure 15: Qualitative examples from EK100 comparing CLIP, VideoMAE, and the proposed CAST. Each expert model shows more accurate predictions in their expertise but shows weaker performance on the other task. However, the proposed CAST consistently shows correct predictions for both noun and verb classes, demonstrating the effectiveness of the proposed spatio-temporal cross-attention mechanism. Best viewed with zoom and color.
27


Reviews:
[{"review_id": "odaAURFEt8", "reviewer": "Reviewer_sJb4", "summary": "This paper proposes a method, namely CAST, for video action recognition based on adapting from large-scale pre-trained models (e.g., CLIP and VideoMAE). The main motivation is to balance and exchange information between spatial and temporal information of two different experts: spatial and temporal. The proposed CAST is an adapter architecture, which adopts CLIP as a spatial expert and VideoMAE as a temporal expert and introduces the cross attention to encourage interaction between these two streams. Experiments are conducted on Epic-KITCHEN-100, Kinetics-400, and Something-Something-v2 with good results compared with current methods. Written presentation is good and mostly easy to follow.", "questions": "* Some minor comments / typos:\n- line 104: \"p pixels\" -> p x p pixels for more consistent with that in line 107? \n- line 106: \"BT x N x D\" -> should be \"B x TN x D\" for more consistent with that in line 108?\n- Figure 4 caption has no explanation for (c).\n- line 204: Figure 4 (c) instead?", "limitations": "The reviewer does not foresee any potential negative societal impact of this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The motivation of balancing between spatial and temporal information of video action recognition makes sense and the proposed architecture of CAST technically sounds.\n\n* Experimental results are strong: (i) good results compared with existing methods; (ii) various ablation to justify the model design choices.", "weaknesses": "* Although the good improvements are shown, the downside may be: there may be an unfair comparison with some other adapter-methods, e.g., AIM where they built up on only one expert model. This means CAST needs more data / pretraining time (of CLIP and VideoMEA);\n\n* There is no FLOPs (or runtime) comparison in table 2. As CAST employs a 2-stream architecture, there will be more computation compared with previous methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the great questions. We address the issues raised by the reviewer below.\n\n**Unfair comparison with adapter-methods**\n\nFor a fair comparison, we conducted an additional experiment, as shown in Table 4 of the supplementary material. Below, we have included the table for your convenience.\n\n| Method         | Spatial Expert     | Temporal Expert     | EK100 Verb | EK100 Noun | EK100 Action |GFLOPs/View|\n|---------------|------------------|-------------------|----------|----------|----------|-|\n| AIM-B          | CLIP w/ WIT400M    | -                   | 64.8       | 55.5       | 41.3       |405|\n| ST-Adapter-B   | CLIP w/ WIT400M    | -                   | 67.6       | 55.0       | -          |-|\n| CAST-B         | ViT w/ IN-1K       | VideoMAE w/ EK100   | 70.9       | 56.8       | 45.5       |391|\n| CAST-B         | CLIP w/ WIT400M    | VideoMAE w/ EK100   | 72.5       | 60.3       | 48.7       |391|\n\nIn this experiment, we use a variant of CAST with a spatial expert pre-trained on the ImageNet-1K and a temporal expert pre-trained and fine-tuned on the EK100. This variant achieved an action accuracy of 45.5% on EK100, surpassing both AIM (41.3%) and VideoMAE (41.7%). Furthermore, this CAST variant also outperforms ST-Adapter, with accuracy values of 70.9% versus 67.6% for verb predictions and 56.8% versus 55.0% for noun predictions. Importantly, this variant requires less pre-training data compared to AIM and ST-Adapter, making it more efficient in terms of data requirements. It is worth noting that while pre-training a ViT on ImageNet-1K requires slightly more data (EK100 consists of 11.5M frames while ImageNet-1K consists of 1M frames only, which corresponds to less than 10% of the EK100 dataset) and time compared to VideoMAE, the overall performance gain achieved by the CAST variant justifies this additional effort. These results demonstrate the versatility and effectiveness of CAST, as it can leverage pre-trained weights from different sources and still achieve competitive performance.\n\n**FLOPs comparison**\n\nWe show FLOPs, the number of frames per clip, the number of test views, and learnable parameters in Tables 8, 9, and 10 of the supplementary material. We found the error in GFLOPs of CAST in the supplementary material and show the fixed GFLOPs in the following Table. The GFLOPs of the other methods compared in the supplementary material are correct.\n\n|Dataset|View|GFLOPs/View|GFLOPs|\n|-|-|-|-|\n|EK100|2x3|391|2346|\n|SSV2|2x3|391|2346|\n|SSV2|5x3|391|5865|\n\nCAST shows favorable spatio-temporal balanced understanding performance on the EK100, SSV2, and K400 even with lightweight configurations.\n\n**Tensor shape in lines 106 and 108**\n\nThe notation for line 106 and line 108 is different: line 106 is \"BT x N x D\" because it is an image passing through a spatial path, while line 108 is \"B x TN x D\" because it is a video passing through a temporal path. The difference occurs because the spatial path treats each frame as an independent image. We show the process of cross-attention of different dimensions of tensors in Supplementary Table 1.\n\n**Typo and missing caption**\n\nThank you for your suggestions. We will fix the typo and add the missing caption in the final version."}, {"review_id": "bjiavhsdPw", "reviewer": "Reviewer_5TeX", "summary": "The manuscript proposes an approach of action recognition. The key idea behind the proposed approach is the usage of a two stream network where one network is specialized to encode the spatial details while the other to encode the temporal details. The approach also presents additional cross connections to effectively integrate the information from one stream to another. Experiments on three benchmark datasets shows that the proposed approach is capable of outperforming existing approaches on datasets requiring spatial reasoning, temporal reasoning and spatio-temporal reasoning.", "questions": "Please see the weaknesses section for detailed queries.\n\nWhat is the CLS(.) operation in line 126?\n", "limitations": "The manuscript adequately addressed the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is written clearly explaining the various contributions and the underlying motivations. Majority of the existing approaches consider action recognition as a generic problem. However, the analysis presented in the manuscript showing the performance variations of existing approaches on datasets requiring spatial reasoning, temporal reasoning and spatio-temporal reasoning is quite intuitive and useful to the research community. The various design choices are validated with an extensive set of ablation studies. This includes the impact of the various backbones, pretraining strategies and the contributions. The improved performance on the different datasets compared to existing approaches showcases the effectiveness of the proposed approach. Moreover, this work could inspire the action recognition research community to move away from the paradigm of one model for all.", "weaknesses": "Two stream networks for action recognition is not a novel concept. For instance there are works that use RGB input in one network and optical flow in the other [1,2], two RGB input streams sampled at different frame rates [3] or spatial resolution [4], etc. However, there is no discussion regarding these approaches in the manuscript. The work of [5] is also similar to the proposed approach as it uses two different encoders for extracting spatial and temporal features. Even though the proposed approach is not exactly derivative of the few above-mentioned approaches, it is highly recommended to include a discussion comparing these approaches. \n\nIn order to predict the final action class, the adapter output of the CLS token of spatial expert and the adapter output of the global average pooled features of temporal expert is added followed by a classification layer. This addition operation across the features may diminish the discriminative information present in the individual streams. Instead, one may predict the action logits separately followed by an average fusion of the logits. Even though this operation is the same as the one presented in the paper, the compute graph is different and hence the gradient flow.\n\nEven though the proposed approach results in improved performance compared to existing approaches, the comparison sometimes seems unfair. For example, the spatial expert of the best CAST model is initialized with CLIP embedding while the temporal expert is initialized with VideoMAE pretrained on SSV2. The same goes for other datasets as well. Could the performance improvement be due to this additional data? Agreeing the fact this is the main concept of the proposed approach, the compute requirements needed for this additional pretraining also needs to be taken into context. Even though the computational complexity in terms of FLOPs is reported in the supplementary material, run time latencies could tell a different story. The reviewer recommends the authors to compare the run time latencies of the proposed approach with existing approaches. The number of trainable parameters reported in the supplementary material is also not indicative of how heavy the model is. It is recommended to report the total number of parameters present in the final model. This will be helpful to researchers while selecting a model by taking into consideration the compute/size vs accuracy tradeoff.\n\n[1] Simonyan, Karen, and Andrew Zisserman. \"Two-stream convolutional networks for action recognition in videos.\" Advances in neural information processing systems 27 (2014).\n\n[2] Feichtenhofer, Christoph, Axel Pinz, and Andrew Zisserman. \"Convolutional two-stream network fusion for video action recognition.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.\n\n[3] Feichtenhofer, Christoph, et al. \"Slowfast networks for video recognition.\" Proceedings of the IEEE/CVF international conference on computer vision. 2019.\n\n[4] Fan, Quanfu, et al. \"More is less: Learning efficient video representations by big-little network and depthwise temporal aggregation.\" Advances in Neural Information Processing Systems 32 (2019).\n\n[5] Jiang, Bo, et al. \"Two-pathway transformer network for video action recognition.\" 2021 IEEE International Conference on Image Processing (ICIP). IEEE, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Related work**\n\nCAST is similar to two-stream networks [1,2] as it learns two streams for spatial and temporal context. Unlike these approaches involving optical flow estimation, CAST uses only RGB streams. Similar to CAST, SlowFast networks [3] and bLVNet-TAM [4] also use only RGB streams. SlowFast employs slow and fast streams for spatial and temporal information but differs from CAST's transformer-based architecture.\n\nCAST and bLVNet-TAM [4] have distinctions. CAST aims for balanced spatio-temporal understanding, while bLVNet-TAM prioritizes computation efficiency, using different stream resolutions. CAST's unique design features bi-directional cross-attention, while bLVNet-TAM relies on temporal shifting.\n\nJiang et al. [5] propose uni-directional cross-attention between RGB and edge encoders. CAST's bi-directional cross-attention in bottleneck architecture achieves effective fusion. This CAST architecture outperforms discussed methods [3-5], as shown in the tables below.\n \n| Method       | Backbone    | SSV2  | K400 |\n|-|-|-|-|\n| SlowFast     | ResNet-101  | -     | 79.8 |\n| bLVNet-TAM   | bLResNet-50 | 65.2  | 73.5 |\n| CAST         | CAST-B      | 71.6  | 85.4 |\n\nWe will add the missing references [4,5] as suggested, building upon the reviewer's insightful feedback that enhances our analysis and comparisons with related works [1-3].\n\n**Classification head architecture**\n\nWe have adopted the recommended head architecture, wherein we independently predict action logits for each expert and then average the logits, as depicted in Fig. 3 of the global response PDF. The results using this architecture are presented in the following table. For K400 experiments, we utilize a VideoMAE pre-trained on the hybrid dataset [34] as the temporal expert for both the methods compared.\n\n| Method     | SSV2          | K400        |\n|-|-|-|\n| add token  | 71.6          | 85.6        |\n| avg logit  | 71.8   | 85.4 |\n\nThe suggested head architecture yields a minor 0.2-point improvement on the SSV2 dataset and a 0.2% point decline on K400 compared to the architecture used in the main paper. These results imply that discriminative information might already be learned through cross-attention in each layer.\n\n**Fair comparison: data-efficiency**\n\nCompared to AIM and ST-Adapter, CAST does not require extra training data. In our main setup, similar to AIM and ST-Adapter, we use the pre-trained CLIP model as the spatial expert. However, we utilize VideoMAE as the temporal expert, training it from scratch on the target dataset. For example, on EK100, we first self-train VideoMAE and then fine-tune CAST end-to-end. This means CAST only needs additional VideoMAE pre-training time on the target dataset, without any other extra requirements compared to AIM and ST-Adapter.\n\nIn contrast to non-adapter-based methods like VideoMAE, CAST indeed demands more pre-training time and additional training data. For a fair comparison, we conducted an additional experiment, shown in Table below.\n\n| Method | Spatial Expert | Temporal Expert | EK100 Verb | EK100 Noun | EK100 Act. |\n|-|-|-|-|-|-|\n| VideoMAE | - | VideoMAE w/ EK100 | 70.5 | 51.4 | 41.7 |\n| AIM-B | CLIP w/ WIT400M    | - | 64.8 | 55.5 | 41.3 |\n| ST-Adapter-B | CLIP w/ WIT400M    | - | 67.6 | 55.0 | - |\n| CAST-B | ViT w/ IN-1K | VideoMAE w/ EK100   | 70.9 | 56.8  | 45.5 |\n| CAST-B | CLIP w/ WIT400M | VideoMAE w/ EK100   | 72.5 | 60.3  | 48.7 |\n\nIn this experiment, we used a CAST variant with a spatial expert pre-trained on ImageNet-1K and a temporal expert pre-trained and fine-tuned on EK100. This variant achieved 45.5% action accuracy on EK100, surpassing AIM (41.3%) and VideoMAE (41.7%). Furthermore, this CAST variant also outperformed ST-Adapter, with accuracy values of 70.9% versus 67.6% for verb predictions and 56.8% versus 55.0% for noun predictions. Importantly, this variant required less pre-training data compared to AIM and ST-Adapter, making it more data-efficient. Although pre-training a ViT on ImageNet-1K took slightly more data and time than VideoMAE (EK100 consists of 11.5M frames while ImageNet-1K consists of 1M frames only, which corresponds to less than 10% of the EK100 dataset), the overall performance gain by the CAST variant justified this additional effort. These results highlight CAST's versatility and effectiveness, leveraging diverse pre-trained weights for competitive performance.\n\n**Fair comparison: computation efficiency**\n\nFor a fair comparison with existing methods, we show the number of total parameters, inference-time throughput, and latency in the Table below.  \n\n|Method|Spatial|Temporal|Total Parameters(M)|Throughput(V/s)|Latency(ms)|SSV2|SSV2 GFLOPs|K400|K400 GFLOPs|EK100 Verb|EK100 Noun|EK100 Act.|EK100 GFLOPs|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|AIM-B|CLIP w/ WIT400M|-|97|37|31.5|68.1|1238|84.5|1214|64.8|55.5|41.3|2430|\n|AIM-L|CLIP w/ WIT400M|-|354|9|122.5|69.4|5754|87.3|5604|-|-|-|-|\n|ST-Adapter-B|CLIP w/ WIT400M|-|93|49|22.9|69.3|977|82.5|911|67.6|55.0|-|-|\n|ST-Adatper-L|CLIP w/ WIT400M|-|-|-|-|71.9|4124|86.9|4124|-|-|-|-|\n|CAST-B|CLIP w/ WIT400M|VideoMAE w/ target data|217|28|48.2|71.6|2346|85.4|5865|72.5|60.3|48.7|2346|\n\nCompared to AIM and ST-Adapter, which employ a single expert model, our proposed CAST does not require additional pre-training data. The efficiency of the temporal expert, VideoMAE, is a key factor. VideoMAE's data-efficient nature is well-documented in its paper. Consequently, when compared to AIM and ST-Adapter, which rely on a sole expert, CAST surpasses AIM-L by +2.3% in performance. Additionally, CAST uses 137 million fewer parameters and computes ~3400 GFLOPs less than AIM-L on the SSV2. Similarly, compared to S-T Adapter-L, CAST achieves comparable performance with only a 0.3 point difference while computes ~1800 GFLOPs less on the SSV2.\n\n**CLS operation**\n\nThe CLS operation refers to the process of extracting the CLS token.\n"}, {"review_id": "EuuzFdDVYG", "reviewer": "Reviewer_PRLf", "summary": "This paper presents two-stream vision transformers, dubbed CAST, for balanced spatiotemporal video representation learning. Given the two experts, CLIP [36] and VideoMAE [46] for spatial and temporal expert, the proposed B-CAST module allows the exchange of complementary information across the separte experts via MHCA. Such information exchange appears to be crucial for achieving balanced performances on appearance-centric (K400, Noun prediction in EK) and motion-centric benchmarks (SSv2, Verb prediction in EK).", "questions": "1.\tIs bi-directional information fusion necessary? What would be the result if we drop one of T2S MHCA or S2T MHCA?\n2.\tHas any analysis been conducted on verb prediction results similar to the approach taken in Fig. 5?\n3.\t(Minor) There is a discrepancy between the captions in Figure 4 and the actual subfigures. As a result, the text in lines 188-195 and 204-212 should be adjusted to refer to the correct subfigures.\n", "limitations": "The authors have provided a comprehensive discussion on the potential limitations and social impact of their work in Supp. 8 and 9. If the proposed model is found to require more computational cost or slows the processing speed down due to two-stream architecture, this inefficiency should also be considered as a limitation, which would pave the way for future improvements.\n\n[Justification of the rating]\nI acknowledge that this paper deliberately integrates two separate methods, adapter [34, 60] and feature fusion [12, 33, 59], into one module for balance spatiotemporal video representation learning. However, I’m concerned with the fairness of the experiment (refer to W1, W2, W3, and W6) and efficiency (W4). Consequently, my preliminary rating leans towards “borderline reject,” but I remain open to changing the score if the authors provide adequate discussions during the rebuttal period.\n\n[Post-rebuttal justification]\nMost of my concerns are well addressed by the rebuttal. I've gone through the rebuttal and found the provided experimental results seems to be reasonable and solid. I strongly recommend adding these results and the related discussions to the final manuscript. Consequently, I raise my rating to \"weak accept.\"", "rating": 6, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1.\tThe paper deliberately engineers to integrate two separate research streams, adapter [34, 60] and representation fusion [12, 33, 59], into one framework for balance spatiotemporal video representation learning.\n2.\tThe proposed method achieves strong performances on both appearance- and motion-centric benchmarks.\n3.\tExtensive investigation with several instantiations for CAST conducted in the ablation studies.\n", "weaknesses": "1.\tIn Tab. 1b, why are the three baseline models fully fine-tuned without adapters? Given that a partially tuned model with adapters is reported to be more effective than a fully fine-tuned model [34, 60], the comparison seems unfair. Reevaluating these models with adapters would make for a more fair comparison.\n2.\tConcerning the first point, are the model \"independent experts\" in Tab. 1a partially tuned with adapters or fully fine-tuned as in Tab. 1b? If it's the latter, for a fair comparison, CAST should be compared with an ensemble of two experts partially trained with adapters rather than with fully fine-tuned ones.\n3.\tFor a more in-depth understanding, I suggest including the number of parameters, FLOPs, and, if possible, inference throughput in Tab. 1, particularly in Tabs. 1a, 1b, and 1d..\n4.\tThe proposed method appears to increase computational cost and training & inference speed. Please provide a comparison of FLOPs, training & inference throughput with the baseline experts [36, 46], their ensemble (“independent experts” in Tab.1a), and other adapter-based methods [34, 60]. This would allow readers to comprehensively evaluate the applicability of the proposed method.\n5.\tIn Tab.1e, there is only a small performance gap between between CAST (4th row) and the model in 3rd row, i.e, (T2S, S2T) = (space, time). Does this imply that, under conditions where attention is optimized (L278), the important factor is layer-wise mixing of the features from the two experts, rather than the specific method of mixing? For your reference, I consider the \"Lateral\" model in Tab.1b, which linearly fuses each other, to be an unfair comparison to the models in Tab.1e due to its less computation. It would be better to elaborate the discussion including results of (T2S, S2T) = (space, time), (time, time) or other variants.\n6.\tTable 2 should include information on the pretraining dataset, FLOPs, # trainable parameters, # frames, and, if possible, inference speed for a comprehensive comparison.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Fair comparison**\n\nFor a fair comparison with baselines, we augment Table 1 (a), (b), and (d) of the main paper with total parameters, GFLOPs/View, and throughput.\n\n***Fusion baselines***\nFor a fair comparison with baseline information exchange methods, we add adapters to add, concat, and lateral fusion baselines. We fine-tune the adapters only. We show the results in Table below.\n\n|Method|Late|Layer-wise|Total Param(M)|Tuneable Param(M)|GFLOPs/View|Throughput(V/s)|EK100 Verb|EK100 Noun|EK100 Act|\n|-|-|-|-|-|-|-|-|-|-|\n|Add|✅||187|15|343|34|68.9|56.6|44.2|\n|Concat|✅||188|16|343|34|69.2|56.4|44.5|\n|Lateral||✅|201|29|366|32|68.9|49.1|39.0|\n|CAST||✅|217|45|391|28|72.5|60.3|48.7|\n\nCAST shows significant improvement over all the baselines with adapters. CAST shows more than 4 points improvement compared to the baselines.\n\n***Indep. experts baselines***\nFor a comprehensive comparison, we present the results of independent experts with and without adapters in the table below. \n|Method|Total Param(M)|Tuneable Param(M)|GFLOPs/View|Throughput (V/s)|EK100 Verb|EK100 Noun|EK100 Act|\n|-|-|-|-|-|-|-|-|\n|Indep. experts w/o adapter|172|172|321|38|70.7|50.1|40.0|\n|Indep. experts w/ adapter|187|15|343|34|68.1|54.2|41.7|\n|Ensemble of experts w/ adapters|188|15|343|33|68.2|55.3|42.9|\n|CAST|217|45|391|28|72.5|60.3|48.7|\n\nCAST surpasses all the baselines compared. Most importantly, CAST achieves an impressive 5.8-point boost over the ensemble experts with adapters baseline. \n\nWe observe that CAST shows a quite good trade-off between the performance and computation, compared to the baselines compared.\n\n***Projection ratio***\nWe choose the projection ratio of 1/2 as it yields the best trade-off between the computation and action accuracy.\n\n|Method|Total|Tuneable|GFLOPs/View|Throughput(V/s)|EK100 Verb|EK100 Noun|EK100 Act|\n|-|-|-|-|-|-|-|-|\n|1/8|191|19|351|31|70.7|59.9|47.4|\n|1/4|198|26|361|30|71.3|59.8|47.4|\n|1/2|217|45|391|28|72.5|60.3|48.7|\n|1|275|103|483|24|72.1|59.8|48.6|\n\n\n**Comprehensive comparison with existing methods**\n\nWe compare the computational cost of CAST with independent expert models and other adapter-based models, presenting the results in the table below. Among these, CLIP-B/16 with adapter represents a partially tuned spatial expert, while VideoMAE-B/16 with adapter signifies the temporal expert.\n\n***Training:***\nWe report the single-step time (including forward pass, backward pass, and parameter update) using the EK100 dataset on a single GPU (RTX 3090) with 24GB of memory. To ensure a fair comparison, we exclude data loading and distributed communication from the step time measurement. We maintain an equal batch size of 6/GPU across all models.\n\n***Inference:***\nWe measure latency with a batch size of 1, focusing solely on the forward pass time. We also measure throughput with a batch size of 32.\n\n|Method| Total Param(M) | Tune Param(M) | Frames | Step Time (S)| VRAM (GB)|GFLOPs/View|Latency (S)|Throughput(V/s)|EK100 Verb|EK100 Noun|EK100 Act.\n|-|-|-|-|-|-|-|-|-|-|-|-|\n| CLIP-B w/ adapter |93|7|8|0.14|6.6|152|13.7|97|54.8|54.8|35.0|\n| VideoMAE-B w/ adapter|94|7|16|0.21|13.0|192|20.1|57|68.4|48.1|38.2|\n| ST-Adapter-B|93|7|16|0.31|12.6|-|22.9|49|67.6|55.0|-|\n| AIM-B|97|11|16|0.35|12.7|405|31.5|37|64.8|55.0|41.3|\n| CAST-B|217|45|16|0.44|21.0|391|48.2|28|72.5|60.3|48.7|\n\n**Attention window shape**\n\nWe show the exhaustive search results in the table below.\n\n| Window shape T2S  | Window shape S2T | Tune param(M) |  Verb| Noun  | Act. |\n|-|-|-|-|-|-|\n| time | space-time | 57 | 71.5 | 60.9 | **48.7** |\n| time | space | 45 | 72.5 | 60.3 | **48.7** |\n| time | time | 43 | 72.1 | 60.6 | 48.6 |\n| space | space-time | 59 | 71.5 | 60.3 | 48.4 |\n| space | space | 47 | 72.3 | 60.2 | 48.5 |\n| space | time | 45 | 72.0 | 60.2 | 48.5 |\n| space-time | space-time | 72 | 71.0 | 59.3 | 47.2 |\n| space-time | space | 59 | 71.9 | 60.3 | 48.4 |\n| space-time | time | 57 | 71.3 | 60.3 | 48.2 |\n\nThe experimental results show that the performance is not quite sensitive to the window shape. We choose the time attention for T2S and space attention for S2T, as this configuration offers a good trade-off between the number of learnable parameters and action accuracy.\n\n\n**Augmenting Table 2.**\n\nWe will add the information on the pretraining dataset, FLOPs, the number of trainable parameters, the number of frames, and inference speed in Table 2 of the revised paper.\n\n\n**Bi-directional cross-attention**\n\nTo validate the effectiveness of bi-directional cross-attention, we compare it with uni-directional cross-attention by individually omitting S2T and T2S in the table below.\n\n| Method | Total Param(M) | Tune Param(M) | Throughput (V/s) | Verb | Noun | Act. |\n|-|-|-|-|-|-|-|\n| Indep. experts w/ adapter | 187 | 15 | 34 | 68.1 | 54.2 | 41.7 |\n| S2T only | 210 | 38 | 30 | 71.2 | 55.0 | 43.7  |\n| T2S only | 208 | 36 | 30 | 68.7 | 60.5 | 46.7  |\n| CAST | 217 | 45 | 28 | 72.5 | 60.3 | 48.7  |\n\nCAST's bi-directional cross-attention outperforms uni-directional cross-attention. CAST achieves a 5-point enhancement over the S2T-only baseline and a 2-point improvement over the T2S-only baseline.\n\n**Category-wise analysis of verb prediction performance**\n\nWe have conducted an analysis of the verb prediction performance across different verb categories, not super classes, and we present the results are in Figures 5 and 6 as well as in Section 6 of the supplementary material. There is no super class definition for verbs in the EK100 dataset.\n\n\n**Missing caption in Figure 4.**\n\nWe will include the missing caption for Figure 4 (c) and make the necessary revisions to the text in lines 188-195 and 204-212 accordingly in the final version.\n\n\n**Limitations**\n\nIn the final version, we will acknowledge the efficiency reduction as a potential limitation."}, {"review_id": "9zKrsGc1lW", "reviewer": "Reviewer_TFT9", "summary": "This paper presents an approach to action recognition that is based on the fusion of two streams of analysis. This involves a spatial stream and a temporal stream that interact through a novel mechanism to improve classification performance.", "questions": "1. Compared to other models, the proposed model appears to be better at verbs and weaker at nouns. Is it possible for the authors to comment on that trade-off?\n\n2. Figure 6 shows a few good illustrated examples. Would the authors consider presenting more examples (e.g. as Supplementary Material)?", "limitations": "There is recognition of some of the limitations of the work throughout the paper, but not a dedicated statement.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is generally well written\n\n2. The cross-attention mechanism is quite novel with respect to methods for 2 stream fusion.\n\n3. Results appear to be quite good on standard benchmarks, and best when considering average performance", "weaknesses": "1. The B-CAST mechanism is quite complex and hard to unpack. With that said, I'm not sure there is a simpler way of framing this, but I would encourage the authors to consider simplifying the description (if possible). Figure 3 is fantastic in this regard.\n2. The results only marginally improve the state of the art and are second best on the two main benchmarks considered, but bested by different models in each case. I wouldn't hold the paper back due to this, but it makes the contribution a bit weaker.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive suggestions. We address the issues raised by the reviewer below.\n\n* Straightforward unpacking of B-CAST mechanism\n\n    To facilitate a straightforward understanding of the B-CAST architecture, we present two additional figures in the global response PDF attached. Figure 1 illustrates how the model transforms and computes features in detail with tensor dimensions. After passing the previous layer output through the MHSA of each expert, we reduce the feature dimension by half with a down projection layer followed by layer normalization. Subsequently, the two experts exchange features, reshaping them according to the window shape. After the cross-attention, we upscale the feature by a factor of two using an up-projection layer. For an in-depth understanding of tensor dimensions, we direct the reviewer to Table 1 of the supplementary material.\n\n    Figure 2 demonstrates where the proposed cross-attention mechanism attends to. In Temporal-to-Spatial (T2S) cross-attention, the query corresponds to a spatial patch of the spatial expert. In T2S, the query attends to temporal patches at the same position across all frames. We term this cross-attention window shape as “time”. Conversely, in Spatial-to-Temporal (S2T) cross-attention, the query represents a patch of the temporal expert. In S2T, a temporal query attends to all spatial patches of the same frame. We term this cross-attention window shape as “space”. We empirically find that using T2S and S2T shows favorable performance compared to alternative choices in Table 1 (e) of the main paper.\n\n* Performance gap\n\n    We have investigated the reason for the relatively marginal performance improvement of CAST compared to state-of-the-art methods on the K400 dataset. We found that the reason is mainly due to differences in the preprocessing steps. In prior works such as VideoMAE, the videos are resized so that the shorter side is 320 pixels, and then random crops are taken from the resized video. However, in our previous implementation, we resized the videos so that the shorter side is 256 pixels. This discrepancy in preprocessing accounts for the lower performance of CAST compared to AIM on the K400 dataset. When we use the same video resizing as prior works, we achieve 85.4% accuracy on the K400 dataset and 77.9% harmonic mean of SSV2 and K400 accuracies surpassing the previous state-of-the-art (77.2%). On the EK100, the previous state-of-the-art method OMNIVORE trained with an external video dataset, K400, shows 49.9% action accuracy. When we train CAST with an external video dataset (Wang et al., 2022 [34] of supplementary material), we achieve 50.2% action accuracy as shown in Table 8 of supplementary material. \n\n* Verb-Noun performance trade-off on EK100\n\n    In comparison to OMNIVORE (with 69.5% accuracy for verbs and 61.7% for nouns) and MTV-HR (with 68.0% accuracy for verbs and 63.1% for nouns), CAST exhibits a larger gap between verb and noun prediction accuracies, achieving 72.7% accuracy for verbs and 60.6% for nouns. This can be attributed to the fact that the noun prediction task in the EK100 dataset entails a highly detailed 300-way classification of kitchen objects, representing a fine-grained classification challenge. Both MTV-HR and OMNIVORE are pre-trained on image datasets as well as video datasets (MTV-HR pre-trained on the ImageNet-21K and K400, and OMNIVORE pre-trained on the ImageNet-21K, ImageNet-1K, SUN-RGBD, and K400), which contributes to their superior performance in fine-grained noun classification. In contrast, CAST utilizes a spatial expert that is based on CLIP, an image-text contrastively pre-trained model. It is plausible that CLIP, despite its pre-training, may exhibit relative weaknesses in fine-grained noun classification compared to MTV-HR and OMNIVORE. Importantly, MTV-HR employs high-resolution image pre-training. Notably, similar to CAST, other CLIP-based methods like ST-Adapter and AIM also demonstrate relatively wider gaps between verb and noun accuracies. ST-Adapter yields 67.6% accuracy for verbs and 55.0% for nouns, while AIM achieves 64.8% accuracy for verbs and 55.5% for nouns. CAST surpasses these approaches through the effective bi-directional cross-attention-based information exchange between spatial and temporal experts, ultimately resulting in stronger performance across both verb and noun prediction tasks. We believe that this gap can be reduced by employing a spatial expert model with enhanced fine-grained classification capabilities, for instance, one pre-trained with higher-resolution images as seen in MTV-HR.\n\n* More qualitative examples\n\n    We show more qualitative examples in Fig. 9 of supplementary material.\n\n* Limitations\n    \n    We have dedicated limitation and broader impacts sections in the supplementary material."}, {"review_id": "rbPTqMInEu", "reviewer": "Reviewer_rpNG", "summary": "This paper proposes a novel two-stream architecture called Cross-Attention in Space and Time (CAST) for video action recognition. The proposed architecture achieves a balanced spatio-temporal understanding of videos using only RGB input. ", "questions": "See weakness", "limitations": "The authors have addressed the limitations", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-written and well-structured.\n2. The implementation of Adapters in the cross-attention mechanism is impressive.\n3. The paper presents sufficient experiments.", "weaknesses": "In Table 2, when compared to VideoVAE, CAST only achieve minor improvement on the SSv2 dataset. The authors are encouraged to discuss why the cross-attention mechanism appears to have a limited impact on enhancing performance for motion-dominated datasets, like SSv2.  Likewise, it would be valuable to have an analysis explaining why the proposed method underperforms compared to AIM (CLIP pretrained only) on the K400 dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the great questions. We address the issues raised by the reviewer below.\n\n* Why does CAST show minor improvement on the SSV2 dataset compared to VideoMAE?\n\n    The main reason for the minor improvement is that the SSV2 is an object appearance agnostic action dataset. SSV2 consists of actions such as “pushing something from left to right”, “pushing something from right to left”, “throwing something”, “squeezing something”, and so on. Temporal experts could show strong performance in discriminating such movements, not specific objects. Given that VideoMAE already serves as a strong temporal expert, adding fine-grained spatial information from the spatial expert (CLIP) may not provide significant additional benefits for this specific dataset.\n\n    It is crucial to clarify that the primary objective of our work is not to achieve the best performance on a particular dataset. Instead, our focus is on achieving a balanced spatio-temporal understanding, which results in good performance across datasets with diverse characteristics. As evident in our results, CAST demonstrates strong performance in this respect, achieving a harmonic mean of 71.5% across EK100 verb, noun, SSV2, and K400 datasets. This demonstrates the effectiveness of CAST in achieving balanced performance across different action recognition tasks and datasets.\n\n* Why the proposed method underperforms compared to AIM on the K400 dataset?\n\n    We have investigated the reason for the lower performance of CAST compared to AIM on the K400 dataset. We found that the reason is mainly due to the preprocessing steps. In prior works such as VideoMAE, the videos are resized so that the shorter side is 320 pixels, and then random crops are taken from the resized video. However, in our previous implementation, we resized the videos so that the shorter side is 256 pixels. This discrepancy in preprocessing accounts for the lower performance of CAST compared to AIM on the K400 dataset. When we use the same video resizing as prior works, we achieve 85.4% accuracy on the K400 dataset, outperforming AIM-B (84.5%)."}]

Author rebuttal:
We express our gratitude to the reviewers for their valuable and constructive comments. We have taken careful consideration of the points raised by each reviewer. We address these concerns comprehensively. Additionally, we have included a supplementary PDF containing figures that further support our responses and explanations for each reviewer's comments.






Meta-review: The paper received unanimous accept reviews, including 4x Weak accept and 1x accept. The core idea is to leverage two expert models, CLIP for spatial understanding, and VideoMAE for temporal understanding, along with an adapter architecture that introduces the cross attention between these two streams. The resulting model obtains strong performance on 3 standard datasets. All reviewers appreciate the clearly written paper, with solid experimentation and good results. ACs concur with reviewer consensus.
Decision: Accept (poster)