Title: CAST: Cross-Attention in Space and Time for Video Action Recognition
Venue: NeurIPS 2023
Accepted: yes
Keywords: action recognition

video understanding

cross-attention

balanced spatio-temporal understanding
Abstract: Recognizing human actions in videos requires spatial and temporal understanding. Most existing action recognition models lack a balanced spatio-temporal understanding of videos. In this work, we propose a novel two-stream architecture, called Cross-Attention in Space and Time (CAST), that achieves a balanced spatio-temporal understanding of videos using only RGB input. Our proposed bottleneck cross-attention mechanism enables the spatial and temporal expert models to exchange information and make synergistic predictions, leading to improved performance. We validate the proposed method with extensive experiments on public benchmarks with different characteristics: EPIC-Kitchens-100, Something-Something-V2, and Kinetics-400. Our method consistently shows favorable performance across these datasets, while the performance of existing methods fluctuates depending on the dataset characteristics. The code is available at https://github.com/KHU-VLL/CAST.

CAST: Cross-Attention in Space and Time for Video Action Recognition

Abstract

Recognizing human actions in videos requires spatial and temporal understanding. Most existing action recognition models lack a balanced spatio-temporal understanding of videos. In this work, we propose a novel two-stream architecture, called Cross-Attention in Space and Time (CAST), that achieves a balanced spatio-temporal understanding of videos using only RGB input. Our proposed bottleneck cross-attention mechanism enables the spatial and temporal expert models to exchange information and make synergistic predictions, leading to improved performance. We validate the proposed method with extensive experiments on public benchmarks with different characteristics: EPIC-KITCHENS-100, Something-Something-V2, and Kinetics-400. Our method consistently shows favorable performance across these datasets, while the performance of existing methods fluctuates depending on the dataset characteristics. The code is available at https://github.com/KHU-VLL/CAST .

1 Introduction

To accurately recognize human actions in videos, a model must understand both the spatial and temporal contexts. A model that lacks fine-grained spatial understanding is likely to fail in predicting the correct action. For example, as shown in Figure 1 (a), a model that understands temporal context such as hand motion across frames but not the fine-grained spatial context may confuse whether an object in the hand a ketchup , or a cheese , or a milk carton . Consequently, the model fails to predict the correct action, Put down a cheese . Similarly, a model that lacks temporal context understanding may also fail to predict the correct action. In Figure 1 (b), let us suppose a model understands spatial context but does not understand temporal context, e.g., the model is confused about whether the hand is moving from outside the fridge to the inside or vice versa. Then the model fails to predict the correct action of Take out a sauce . Therefore, for accurate action recognition, models need to comprehend both the spatial and temporal contexts of videos.

Despite the recent progress in action recognition through the use of Transformers [60, 11, 3], achieving a balanced spatio-temporal understanding remains a challenging problem. Compared to images, the additional temporal dimension in videos makes spatio-temporal representation learning computationally intensive and requires a significant amount of training data [3]. Consequently, most action recognition models lack a balanced spatio-temporal understanding of videos. Notably, models that perform well on static-biased [32, 8, 50] datasets, such as Kinetics-400, may not perform as well on temporal-biased [3, 28] datasets, such as Something-Something-V2, and vice versa. For instance, as shown in Figure 4 (a), on the EPIC-KITCHENS-100 dataset, VideoMAE [56] outperforms STAdapter [42] on the verb prediction task, while ST-Adapter outperforms VideoMAE on the noun prediction task. Similarly, BEVT [62] outperforms AIM [72] on the Something-Something-V2

t

Ground truth

: Put down a cheese

(a) Importance of spatial understanding

Milk carton?

Cheese?

Take out?

Ground truth

:

Take out a sauce

(b) Importance of temporal understanding

t

Put

in?

t

t

Put in?

dataset, while BEVT underperforms AIM on the Kinetics-400 dataset. We observe a similar trend for other methods as reported in Table 2.

One possible solution to the challenge of balanced spatio-temporal understanding is to use multimodal learning. For example, two-stream networks [51, 14] employ both RGB and optical flow streams to learn both spatial and temporal contexts. However, this approach can be computationally expensive due to optical flow estimation.

In this work, we introduce a two-stream architecture, Cross-Attention in Space and Time (CAST), to address the challenge of balanced spatio-temporal understanding using only RGB input. In Figure 2, we show a high-level illustration of the proposed method. Our architecture employs two expert models - a spatial expert model and a temporal expert model - which exchange information to make a synergistic collective prediction. We realize the information exchange by cross-attention between the two experts. We empirically validate that placing cross-attention in a bottleneck architecture facilitates more effective learning. To validate the effectiveness of the proposed method, we conduct extensive experiments on multiple datasets with distinct characteristics, including the temporal-biased Something-Something-V2, static-biased Kinetics-400, and fine-grained EPIC-KITCHENS-100. Our results demonstrate that CAST achieves balanced spatio-temporal understanding and shows favorable performance across these different datasets.

In this work, we make the following significant contributions.

We introduce a two-stream architecture, CAST, which addresses the challenge of balanced spatio-temporal understanding that has been largely overlooked by previous works.

Weconduct extensive experiments on multiple datasets with distinct characteristics to demonstrate the effectiveness of CAST. In terms of balanced spatio-temporal understanding, CAST shows favorable performance, while existing methods show more imbalanced performance.

We conduct an extensive ablation study and analysis to validate the design choices of the proposed method. We show that employing spatial expert and temporal expert and placing cross-attention in a bottleneck architecture is crucial for achieving effective spatio-temporal representation learning.

2 Related Work

Video Action Recognition. CNN-based approaches have been widely used for action recognition, including 2D CNNs [61, 74, 33, 52, 27], 3D CNNs [58, 5, 59, 64, 13], 2D and 1D separable CNNs [59, 70], or two-stream CNNs [14, 15]. These methods have achieved great progress thanks to the strong inductive biases. Recently, Transformer-based approaches [1, 3, 21, 43, 68, 12, 71] become popular in the community due to the long-term context modeling capabilities. Similar to the two-stream CNNs, we propose a two-stream transformer architecture consisting of two expert models: a spatial expert and a temporal expert. However, unlike traditional two-stream CNNs, we use RGB input only, instead of RGB and flow.

The actor is

picking up a cutlery.

Ground Truth

:

Pick up a fork

The actor is reaching out

and bringing something.

t

t

Next conversation

Temporal expert

Temporal expert

There are hand, dishes,

and utensil holder.

Spatial expert

This is a hand holding a

fork.

Spatial expert

Collective prediction:

Pick up a fork

Cross-attention. Cross-attention has been widely utilized in multi-modal learning to facilitate information exchange between different modalities such as audio, visual, and text [34, 67, 40, 30, 18]. Recently, cross-attention between different views of the same video has shown impressive results [71, 75, 6, 26]. Similar to these, we propose a cross-attention method using a single RGB input, but with two distinct expert models: a spatial expert and a temporal expert. The two experts attend to each other through cross-attention to achieve a balanced spatio-temporal understanding.

Foundation model. Trained on web-scale datasets using self-supervised learning, foundation models [25, 4, 48, 45, 44] are highly adaptable and versatile. Foundation models show impressive performance on various tasks in computer vision [65, 62], natural language processing [49, 57], and audio recognition [17]. In this work, we employ CLIP [44] as our spatial expert as it shows impressive performance on more than 30 computer vision tasks.

Parameter-efficient transfer learning. Although the 'pre-training and fine-tuning' paradigm with strong foundation models has demonstrated impressive performance on several computer vision tasks, it is computationally expensive and often unnecessary to fine-tune the full model [72]. Several works have demonstrated that learning only a small subset of parameters and keeping the remaining parameters frozen is effective for NLP tasks [23, 29] and computer vision tasks [36, 66, 54, 46, 47]. Extending image foundation models by adding adapter architectures has shown favorable performance on action recognition [35, 72, 42]. The proposed method also employs adapter architecture with crossattention between two experts. We empirically demonstrate that the proposed method outperforms existing adapter-based video models in terms of achieving balanced spatio-temporal understanding.

3 Method: Cross-Attention in Space and Time

We introduce CAST, a method for balanced spatio-temporal representation learning for action recognition, as shown in Figure 3. We employ frozen spatial and temporal expert models that can be any vision transformer, consisting of 12 transformer blocks each. To facilitate information exchange between the experts, we introduce the bottleneck cross-attention in space and time (B-CAST) module on top of the frozen layers. This module enables the experts to exchange information and learn more balanced spatio-temporal contexts than separate experts. To improve adaptation to downstream tasks, we use adapter layers with a small number of learnable parameters, following AIM [72]. In the following subsections, we provide a detailed description of each component of our proposed CAST.

3.1 Input embeddings

CAST takes only RGB videos as inputs. The input is a mini-batch of videos, I ∈ R B × 2 T × H × W × C , consisting of B videos of 2 T frames, H × W spatial dimensions, and C channels. We apply patch tokenization to the input videos for the spatial expert and the temporal expert. For the spatial expert, we decompose every even frame of each video in I into N non-overlapping patches of p × p pixels [11].

Temporal Expert

Input Video

Information Exchange

Spatial Expert

(a) Proposed architecture overview

Classification

Head

Adapter

Feed Forward

Adapter

Patch & Pos

Embed

LayerNorm

CLS token

GAP token

Patch & Pos

Embed

Adapter

Feed Forward

Multi-Head

Self-Attention

LayerNorm

t

(c) Visualization of cross-attention window shape

S2T Cross-attention

(Window shape: Space)

K

K

K

K

K

K

K

k

K

K

K

K

Q

K

K

k

K

K

Q

T2S Cross-attention

(Window shape: Time)

T

H

W

T

H

W

Adapter

×

12

Blocks

B-CAST

LayerNorm

LayerNorm

Multi-Head

Self-Attention

(b) Proposed B-CAST architecture

Update

Positional embedding

Frozen

LayerNorm

Q

K

V

V

Up

Projection

LayerNorm

Q

K

Up

Projection

Temporal to Spatial (T2S)

Multi-Head Cross-Attention

Spatial to Temporal (S2T)

Multi-Head Cross-Attention

Down

Projection

Down

Projection

Adapter

Adapter

B-CAST

n

h

0

中

Then we pass the patches through a frozen linear layer and add position embeddings to obtain spatial embeddings, X s ∈ R BT × N × D . For the temporal expert, we decompose every two frames of each video in I into 2 × p × p pixels non-overlapping tubes [1]. Then we pass the tubes through a frozen linear layer and add position embeddings to obtain temporal embeddings, X t ∈ R B × TN × D .

3.2 CAST architecture

The model architecture of each expert is the same as the ViT except for adapters and the B-CAST module. All the other parameters are frozen, while the adapter and B-CAST parameters are learnable.

For completeness, we first define the operations used and then describe the entire model architecture. Given an input X , we define Multi-Head Self Attention (MHSA) operation as follows:

W Q , W K , and W V are the query, key, and value projection matrices, respectively. We also define the adapter operation with linear down and up projection matrices W D and W U as follows:

where σ ( · ) is the GELU activation function [20].

For each attention block l , we apply independent Multi-Head Self Attention (MHSA) for each expert along with a skip connection as follows:

where LN ( · ) denotes the Layer Normalization operation. The spatial path undergoes spatial attention, while the temporal path undergoes space-time attention following TimeSformer [3].

As shown in Figure 3 (b), to exchange information between the two experts, we apply the B-CAST operation Φ ( · ) to Y e 1 and Y e 2 from the expert e 1 and e 2 as follows along with a skip connection:

We describe the B-CAST operation Φ ( · ) in detail in Section 3.3.

Finally, we pass the output, denoted as B ( l ) , through a two-layer feed forward network (FFN) [11] with the GELU activation function in between the layers and another adapter to obtain the next layer input X ( l + 1 ) as follows along with a skip connection:

Classification head. To produce the final prediction, we need to aggregate the outputs of both spatial and temporal experts. For the spatial expert, we average the frame-level class tokens from the last attention block, X ( 12 ) s , to obtain a single class token. We denote this operation as CLS ( · ) . To obtain temporal expert features, we aggregate all the tokens from the last attention block of the temporal expert, X ( 12 ) t , using the global average pooling GAP ( · ) operation. Then we add the adapter output of the CLS token and the adapter output of the GAP token to produce a fused token Z :

Finally, we feed the fused token Z a classification layer followed by a softmax function to obtain the predicted class probabilities. We train the model using the standard cross-entropy loss.

3.3 B-CAST module architecture

Multi-Head Cross-Attention. Multi-Head Cross-Attention (MHCA) is a variant of the MHSA operation (1), where query tokens come from one expert ( e 1) and key and value tokens come from another expert ( e 2). This allows the experts to exchange information and benefit from the strengths of each other. We define the MHCA operation as follows:

where W Q , W K , and W V are learnable query, key, and value parameter matrices respectively.

Temporal-to-Spatial Cross-Attention. In Temporal-to-Spatial (T2S) cross-attention, query tokens come from the spatial expert s , and key and value tokens come from the temporal expert t : MHCA ( Y ( l ) s , Y ( l ) t ) . We depict the attention window in Figure 3 (c). Given a query, the model attends along the temporal dimension only. By using T2S cross-attention, the spatial expert can learn to attend to temporal features from the temporal expert. T2S MHCA leads to capturing spatio-temporal dependencies and improves the model performance in action recognition.

Spatial-to-Temporal Cross-Attention. In Spatial-to-Temporal (S2T) cross-attention, query tokens come from the temporal expert t , and key and value tokens come from the spatial expert s : MHCA ( Y ( l ) t , Y ( l ) s ) . We illustrate the attention window in Figure 3 (c). Given a query, the model attends along the spatial dimension only. By using S2T cross-attention, the temporal expert can attend to fine-grained spatial features from the spatial expert. S2T MHCA leads to a more balanced spatio-temporal understanding and improves the performance in fine-grained action recognition.

Bottleneck Cross-Attention in Space and Time. To achieve efficient and effective learning, we incorporate the T2S and S2T MHCA into bottleneck-shaped adapters. We illustrate B-CAST architecture in Figure 3 (b). We plug the MHCA modules into adapters and add new learnable positional embeddings for each MHCA. We define the B-CAST operation for T2S Φ S ( · ) as follows:

where W D , s and W U , s are linear down- and up-projection matrices for the spatial expert, and E s and E t are new positional embeddings for the spatial and temporal experts, σ ( · ) is the GELU activation function, respectively. We can define the B-CAST operation for S2T, Φ T ( · ) in a similar manner. The output of B-CAST goes into a feed forward network using (5). Our empirical validation shows that the B-CAST architecture is efficient and effective. (See Table 1.)

40

50

60

70

80

CLIP

VideoMAE

CAST

Harmonic Mean of EK100 Verb, Noun, SSV2, K400

(b)

56.5

66.6

71.6

(c)

48

50

52

54

56

ST-Adapter

VideoMAE

55.0

51.4

64

66

68

70

72

ST-Adapter

VideoMAE

67.6

70.5

EK100 Verb

78

80

82

84

86

AIM

BEVT

84.5

80.6

K400

64

66

68

70

72

AIM

BEVT

68.1

70.6

SSV2

(a)

4 Experimental Results

In this section, we present the experimental results that answer the following research questions: (1) Do existing methods show a balanced spatio-temporal understanding of videos? (Section 4.3) (2) What are the ingredients for a balanced spatio-temporal understanding? (Section 4.3) (3) Is the proposed method effective? (Section 4.3, Section 4.4) (4) How can we effectively combine spatial and temporal models to achieve such balance? (Section 4.5) (5) Does the proposed method outperform state-of-the-art methods in terms of balanced spatio-temporal understanding? (Section 4.6) To this end, we first provide details about the datasets and implementation in Section 4.1 and Section 4.2, respectively.

4.1 Datasets

Action recognition. We evaluate the CAST on two public datasets for conventional action recognition: Something-Something-V2 (SSV2) [19] and Kinetics-400 (K400) [24]. The SSV2 requires more temporal reasoning [3, 28] while the K400 is relatively static biased [32, 8, 50].

Fine-grained action recognition. We evaluate the CAST on the fine-grained action recognition task: EPIC-KITCHENS-100 (EK100) [10]. In contrast to conventional action recognition, EK100 defines an action as a combination of a verb and a noun. Therefore, we refer to the action recognition in EK100 as fine-grained action recognition . Since fine-grained action recognition requires correctly predicting both the verb and the noun to recognize an action it is more challenging than conventional action recognition, which requires predicting a single action label: e.g., K400 or SSV2.

4.2 Implementation details

In this section, we briefly provide our experimental setup and implementation details. Please refer to the Appendix § B for complete implementation details. We conduct all the experiments with 16 NVIDIA GeForce RTX 3090 GPUs. We implement CAST using PyTorch and build upon the existing codebase of VideoMAE [56].

Training. We sample 16 frames from each video to construct an input clip. For the K400 dataset, we apply dense sampling [15], while for SSV2 and EK100, we use uniform sampling [61]. We then perform random cropping and resizing every frame into 224 × 224 pixels. We use the AdamW [39] optimizer with momentum betas of (0.9, 0.999) [7] and a weight decay of 0.05. By default, we train the model for 50 epochs, with the cosine annealing learning rate scheduling [38] and a warm-up period of 5 epochs. The default base learning rate, layer decay [2], and drop path are set to 0.001, 0.8, and 0.2, respectively. We freeze all the parameters of each expert, except for the B-CAST layer, adapters, and the last layer normalization. We set the batch size per GPU as 6 with update frequency of 2.

Inference. Given an input video, we randomly sample frames multiple times to construct input clips with multiple temporal views with multiple spatial crops. After the temporal frame sampling, we resize every frame so that the shorter side has 224 pixels. Then we perform spatial cropping to get multiple 224 × 224 crops for each clip. We get the final prediction by averaging the predictions on (temporal views) × (spatial crops). For the K400 dataset, we use (5 clips) × (3 crops) views, while for the other datasets, we use (2 clips) × (3 crops) views for the inference.

materials

cutlery

utensils

vegetables

baked goods and grains

fruits and nuts

rubbish

dairy and eggs

drinks

appliances

crockery

prepared food

cookware

spices and herbs and sauces

furniture

hand

other

containers

cleaning equipment and material

storage

meat and substitute

0.025

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

F1 Score improvement

EK100 Noun, overall improvement over CLIP :

8.3

%

vegetables

other

fruits and nuts

materials

cutlery

dairy and eggs

baked goods and grains

utensils

drinks

containers

spices and herbs and sauces

crockery

cookware

rubbish

hand

cleaning equipment and material

appliances

storage

meat and substitute

furniture

prepared food

0.00

0.05

0.10

0.15

0.20

0.25

F1 Score improvement

EK100 Noun, overall improvement over VideoMAE :

9.2

%

4.3 Balanced spatio-temporal understanding

In Figure 4 (a), we present the top-1 accuracies of several existing models. In the EK100 verb prediction task, VideoMAE outperforms ST-Adapter with a margin of 2 . 9 points (70 . 5% vs. 67 . 6%), while in the EK100 noun prediction task, ST-Adapter [42] outperforms VideoMAE [56] with a margin of 3 . 6 points (55 . 0% vs. 51 . 4%). As shown in Figure 4 (b), BEVT [62] outperforms AIM [72] with a margin of 2 . 5 points (70 . 6% vs. 68 . 1%) on the SSV2 dataset, while on the K400 dataset, AIM outperforms BEVT with a margin of 3 . 9 points (84 . 5% vs. 80 . 6%). We observe similar trends for other methods as well. Please refer to Section 4.6 for a detailed comparison. Our findings indicate that the performance of many existing models is significantly imbalanced toward either spatial or temporal understanding.

Ingredients for balanced spatio-temporal understanding. To achieve a more balanced spatiotemporal understanding, we can employ two expert models: a spatial expert and a temporal expert. For the spatial expert, we use CLIP [44], which has demonstrated impressive performance on various computer vision tasks. For the temporal expert, we use VideoMAE [56], which has shown favorable performance on temporal-biased tasks such as SSV2 and EK100 verb prediction tasks. (Please refer to Section 4.6 for the accuracy details.) While each expert is highly specialized in its own domain, we aim to create synergy between them by exchanging information to improve the balanced spatio-temporal understanding performance.

Effect of CAST. In Figure 4 (c), we gauge the balanced spatio-temporal understanding performance of our spatial expert, temporal expert, and CAST. For each method, we calculate the harmonic mean of top-1 accuracies for EK100 noun, EK100 verb, SSV2, and K400. The harmonic mean is an effective metric for gauging balanced performance because it gives more weight to lower-performing tasks. A higher harmonic mean value indicates that the performance over the different tasks is more balanced. Our spatial expert achieves an accuracy of 56 . 5%, while the temporal expert achieves an accuracy of 66 . 6%, and our CAST achieves an accuracy of 71 . 6%. These results validate the effectiveness of our proposed method, CAST, which allows our spatial and temporal experts to make synergistic predictions by exchanging information with each other through cross-attention.

4.4 Analysis on fine-grained action recognition

In this section, we provide a detailed analysis of how the proposed CAST improves the balanced spatio-temporal understanding in the fine-grained action recognition task: EK100.

Category-level performance analysis. In Figure 5, We present the EK100 noun super-categorywise weighted average F1 score improvement of CAST over our spatial expert (CLIP) and temporal expert (VideoMAE). In Figure 5 left, we observe that CAST significantly improves upon the spatial expert, CLIP, in several super-categories such as cutlery , utensils , and vegetables . These results indicate that the spatial expert achieves a more accurate understanding of fine-grained small objects interacting with the actors by leveraging the temporal context from the temporal expert. Similarly, in Figure 5 right, we observe that CAST significantly improves upon the temporal expert, VideoMAE, in several categories such as vegetables and cutlery . The trend is similar to the comparison with the spatial expert: CAST achieves more accurate understanding of fine-grained small objects by leveraging the fine-grained spatial context from CLIP.

Qualitative analysis. To better understand the effectiveness of CAST, we provide qualitative analysis on a few sample frames from the EK100 dataset in Figure 6. We show the predictions of CLIP, VideoMAE, and CAST. As expected, each expert model provides more accurate prediction in their respective tasks of expertise but shows weaker performance in the other task. In contrast,

CAST consistently shows correct predictions for both noun and verb prediction tasks, such as spoon and open . The qualitative examples demonstrate the effectiveness of CAST in achieving balanced spatio-temporal understanding, which is essential for fine-grained action recognition.

4.5 Ablation study on CAST architecture

We conduct comprehensive ablation studies to examine the design choices for the proposed CAST architecture. Here we conduct all experiments on the EK100 [10] dataset with 16-frame input videos and report the top-1 accuracy on the validation set. We employ CLIP [44] as a spatial expert model and VideoMAE [56] as a temporal expert model. For a fair ablation study, we use the same hyperparameters for each experiment unless explicitly mentioned.

Effect of information exchange. We investigate whether CAST effectively achieves a synergistic effect by exchanging information between the two expert models. In Table 1 (a), we compare CAST with three baselines. i) A baseline using two independent expert models without any information exchange (fully fine-tuned). ii) The same baseline as i), but we add adapters and fine-tune the adapters and head only, iii) A test-time ensemble of two independent experts (with adapters and heads fine-tuning only). The baselines predict nouns using the spatial model and verbs using the temporal model. We observe that the two expert models using ensembling achieve an improvement in Action accuracy by at least 1.2 points compared to the baselines without any information exchange. Furthermore, CAST achieves a best Action accuracy of 48.7%. These results suggest that information exchange is crucial for achieving balanced spatio-temporal understanding.

Comparison with simple information exchange baselines. We compare CAST with simple information exchange baselines: i) late fusion with addition, ii) late fusion with concatenation, iii) layer-wise fusion using the bidirectional lateral connection (element-wise addition) with linear projection. We add adapters and fine-tune the adapters and head only in all three baselines, using our training recipe in Section 4.2 For the details of baseline fusion methods, please see Figure 7. We show the results in Table 1 (b). It is worth noting that both the late fusion and layer-wise lateral connection baselines result in a significant performance drop. Furthermore, we observe that layer-wise fusion without cross-attention yields inferior performance compared to the simple late fusion baselines. The results indicate that cross-attention in the bottleneck architecture is crucial for effective information exchange between spatial and temporal experts.

Design of B-CAST module. To explore the most effective and efficient way to integrate adapters for information exchange between the two expert models, we conduct an ablation study and present the results in Table 1 (c). For the details of the baselines, please see Figure 8. The first row of the table represents a baseline without the B-CAST module, which is equivalent to the identity function. Compared to this baseline, B-CAST achieves a significant improvement of 7.0 points in Action accuracy. The second row shows the performance of a baseline with cross-attention but without the

CLIP:

VideoMAE:

CAST:

Pan

Pan

Broccoli

Cloth

Lid

Fork

Spoon

Fork

Spoon

Trun-on

Open

Open

Scoop

CLIP:

VideoMAE:

CAST:

CLIP:

VideoMAE:

CAST:

CLIP:

VideoMAE:

CAST:

CLIP:

VideoMAE:

CAST:

CLIP:

VideoMAE:

CAST:

Take

Sort

Mix

bottleneck adapters. The 9.3-point gap between this baseline and B-CAST highlights the importance of bottleneck adapters for effective information exchange between the two expert models. The third row ( X-attn. → adapter ) is a baseline with the adapters after cross-attention. Compared to B-CAST, this baseline shows a 0.8 points drop in Action accuracy while having more than double the number of learnable parameters (44 . 9 M vs. 93 . 0 M ). The results indicate that cross-attention in bottleneck is more effective and more efficient than the baseline. In summary, by placing cross-attention in the middle of the bottleneck adapter, B-CAST facilitates effective information exchange between the two experts and achieves a synergistic effect.

Effect of projection ratio in bottleneck. In this study, we investigate the impact of the down projection ratio in the bottleneck architecture presented in Table 1 (d). The results demonstrate that a ratio of 1 / 2 yields the best performance. Notably, a ratio of 1 results in inferior performance, which we attribute to overfitting caused by the addition of more parameters.

Effect of cross-attention window shape. We investigate the impact of the window shape in the cross-attention mechanism in the T2S and S2T modules in Table 1 (e). Please refer to Figure 3 (c) for the details of the window size. We maintain the same model capacity across different methods. Using space-time attention for both T2S and S2T modules results in the worst performance. We conjecture that learning joint space-time attention is challenging with the given model capacity [3]. On the other hand, using time attention in T2S and space attention in S2T yields the best performance. Consequently, we adopt this configuration throughout the paper.

Effect of the number of cross-attention layers. We investigate the impact of the number of cross-attention layers used. To this end, we gradually increase the number of cross-attention layers starting from the top three layers, and report the results in Table 1 (f). As we increase the number of cross-attention layers, we observe a corresponding improvement in performance, as expected.

Effect of bi-directional cross-attention. To validate the effectiveness of bi-directional information exchange, we ablate each cross attention at a time. We compare CAST with unidirectional information exchange baselines equipped with S2T or T2S cross-attention only. Each unidirectional information exchange baseline still has both experts. In Table 1 (g), compared to our CAST (48.7%), the S2T only baseline shows 5.0 points drop (43.7%) and the T2S only baseline shows 2.0 points drop (46.7%) in accuracy. The results validate the effectiveness of the proposed bi-directional cross-attention.

Role of each expert. In Table 1 (h), we investigate the role of experts within CAST by controlling the assignment of models to each expert. We observe that we can achieve the best performance of 48 . 7% when we employ CLIP as our temporal expert and VideoMAE as our spatial expert. When we employ one VideoMAE as our spatial expert and another VideoMAE as our temporal expert, we obtain 40 . 3% accuracy. When we employ one CLIP as our spatial expert and another CLIP as our temporal expert, we obtain 46 . 0% accuracy.

Interestingly, when we revert the role of CLIP and VideoMAE, i.e., we employ VideoMAE as the spatial and CLIP as the temporal expert, we achieve a good performance of 47 . 8%. The results demonstrate that the B-CAST architecture facilitates effective information exchange between the two experts. Through the stacked B-CAST, the experts can learn high-quality spatio-temporal representations by exchanging information, even when the roles are reverted.

In summary, these findings suggest that CAST achieves optimal performance when models are assigned to expert roles that align with their strengths. CLIP serves as an effective spatial expert, whereas VideoMAE is more effective as a temporal expert. The B-CAST architecture encourages these experts to leverage their respective strengths through information exchange, resulting in enhanced spatio-temporal balanced understanding.

4.6 Comparison with state-of-the-art

In this section, we evaluate the performance of CAST and state-of-the-art methods in terms of balanced spatio-temporal understanding on multiple datasets, as shown in Table 2. For each method, in addition to reporting the top-1 accuracy of each task, we report the harmonic mean of top-1 accuracies for i) SSV2, and K400, and ii ) EK100 verb, EK100 noun, SSV2, and K400. For comparison with state-of-the-art models, we have set different hyperparameters than those used in our ablation study. Please refer to Table 4 for the details. For fair comparisons of the computation complexity, we show the GFLOPs/View. In cases where a compared method shows various GFLOPs/View depending on the dataset, we specifically note the lowest GFLOPs/View value for reference. For more detailed comparison of computation complexity, please refer to the Appendix § E.

We observe that among the CLIP-based methods (the second group in Table 2), AIM [72] achieves favorable performance on the static-biased K400 dataset, with 84 . 5% accuracy. However, AIM shows a relatively lower performance of 68 . 1% on the temporal-biased SSV2. On the other hand, VideoMAE [56], one of the state-of-the-art methods, shows 70 . 8% accuracy on the SSV2 dataset, which is more competitive than AIM. However, VideoMAE shows a lower accuracy of 81 . 5% on the K400 dataset, less competitive than AIM. Our proposed method, CAST, demonstrates favorable performance on both the SSV2 (71 . 6%) and K400 (85 . 3%) datasets, resulting in a harmonic mean of 77 . 9%, which is higher than that of AIM (75 . 4%) and VideoMAE (75 . 8%). CAST shows a more balanced spatio-temporal understanding than the existing methods. Additionally, CAST shows favorable performance in fine-grained action recognition on the EK100 dataset. CAST achieves a competitive Action accuracy of 49 . 3%, which is the second best among the compared methods.

In terms of the overall harmonic mean of EK100 verb, EK100 noun, SSV2, and K400 accuracies, CAST shows the best performance of 71 . 6%. The results highlight the effectiveness of CAST. By exchanging information between spatial and temporal experts, our CAST shows a favorable balanced spatio-temporal understanding performance.

5 Conclusions

In this paper, we present a solution to the problem of action recognition models lacking a balanced spatio-temporal understanding of videos. The proposed method, CAST, incorporates a spatial expert and a temporal expert that exchange information through cross-attention to achieve synergistic predictions. Our extensive experiments on datasets with varying characteristics demonstrate that CAST outperforms both individual expert models and existing methods in terms of a balanced spatio-temporal understanding measure: the harmonic mean of accuracies on the datasets. The results highlight the effectiveness of CAST in achieving a balanced spatio-temporal understanding of videos, and suggest that CAST could have broad applicability in the field of video understanding.

Acknowledgment. This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.RS2022-00155911, Artificial Intelligence Convergence Innovation Human Resources Development (Kyung Hee University)); by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea Government (MSIT) (Artificial Intelligence Innovation Hub) under Grant 2021-0-02068; by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. 2022R1F1A1070997).

References

Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Luˇ ci´ c, and Cordelia Schmid. Vivit: A video vision transformer. In ICCV , 2021.

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 , 2021.

Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML , 2021.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS , 2020.

Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR , 2017.

Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In ICCV , 2021.

Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML , 2020.

Jinwoo Choi, Chen Gao, Joseph CE Messou, and Jia-Bin Huang. Why can't i dance in the mall? learning to mitigate scene bias in action recognition. In NeurIPS , 2019.

Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In CVPR workshops , 2020.

Dima Damen, Hazel Doughty, Giovanni Maria Farinella, , Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision. IJCV , 130(1):33-55, 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV , 2021.

Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In CVPR , 2020.

Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In CVPR , 2016.

Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV , 2019.

Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A Single Model for Many Visual Modalities. In CVPR , 2022.

Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. arXiv preprint arXiv:2305.05665 , 2023.

Satya Krishna Gorti, Noël Vouitsis, Junwei Ma, Keyvan Golestan, Maksims Volkovs, Animesh Garg, and Guangwei Yu. X-pool: Cross-modal language-video attention for text-video retrieval. In CVPR , 2022.

Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In ICCV , 2017.

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016.

Roei Herzig, Elad Ben-Avraham, Karttikeya Mangalam, Amir Bar, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Object-region video transformers. In CVPR , 2022.

Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In CVPR , 2020.

Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. In NeurIPS , 2021.

Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 , 2017.

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT , 2019.

Hannah Halin Kim, Shuzhi Yu, Shuai Yuan, and Carlo Tomasi. Cross-attention transformer for video interpolation. In ACCV , 2022.

Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong. Movinets: Mobile video networks for efficient video recognition. In CVPR , 2021.

Matthew Kowal, Mennatullah Siam, Md Amirul Islam, Neil DB Bruce, Richard P Wildes, and Konstantinos G Derpanis. A deeper dive into what deep spatiotemporal networks encode: Quantifying static vs. dynamic information. In CVPR , 2022.

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP , 2021.

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS , 2021.

Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatiotemporal representation learning. In ICLR , 2022.

Yingwei Li, Yi Li, and Nuno Vasconcelos. Resound: Towards action recognition without representation bias. In ECCV , 2018.

Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In ICCV , 2019.

Yan-Bo Lin, Jie Lei, Mohit Bansal, and Gedas Bertasius. Eclipse: Efficient long-range video retrieval using sight and sound. In ECCV , 2022.

Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. In ECCV , 2022.

Yen-Cheng Liu, Chih-Yao Ma, Junjiao Tian, Zijian He, and Zsolt Kira. Polyhistor: Parameter-efficient multi-task adaptation for dense vision tasks. In NeurIPS , 2022.

Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR , 2022.

Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 , 2016.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR , 2019.

Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. In NeurIPS , 2021.

Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for general video recognition. In ECCV , 2022.

Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter-efficient image-to-video transfer learning. In NeurIPS , 2022.

Mandela Patrick, Dylan Campbell, Yuki M. Asano, Ishan Misra Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and João F. Henriques. Keeping your eye on the ball: Trajectory attention in video transformers. In NeurIPS , 2021.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML , 2021.

Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In NeurIPS , 2017.

Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of multi-domain deep neural networks. In CVPR , 2018.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 , 2022.

Laura Sevilla-Lara, Shengxin Zha, Zhicheng Yan, Vedanuj Goswami, Matt Feiszli, and Lorenzo Torresani. Only time can tell: Discovering temporal data for temporal modeling. In WACV , 2021.

Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In NeurIPS , 2014.

Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz. Gate-shift-fuse for video action recognition. arXiv:2203.08897 , 2022.

Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 , 2023.

Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. In NeurIPS , 2022.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR , 2016.

Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In NeurIPS , 2022.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV , 2015.

Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In CVPR , 2018.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks for action recognition in videos. TPAMI , 41(11):2740-2755, 2018.

Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. Bevt: Bert pretraining of video transformers. In CVPR , 2022.

Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, and Yu-Gang Jiang. Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning. In CVPR , 2023.

Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR , 2018.

Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191 , 2022.

Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR , 2022.

Xi Wei, Tianzhu Zhang, Yan Li, Yongdong Zhang, and Feng Wu. Multi-modality cross attention network for image and sentence matching. In CVPR , 2020.

Chao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition. In CVPR , 2022.

Wenhao Wu, Zhun Sun, and Wanli Ouyang. Revisiting classifier: Transferring vision-language models for video recognition. In AAAI , 2023.

Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning for video understanding. In ECCV , 2018.

Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview transformers for video recognition. In CVPR , 2022.

Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. Aim: Adapting image models for efficient video understanding. In ICLR , 2023.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR , 2017.

Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In ECCV , 2018.

Haowei Zhu, Wenjing Ke, Dong Li, Ji Liu, Lu Tian, and Yi Shan. Dual cross-attention learning for fine-grained visual categorization and object re-identification. In CVPR , 2022.

Appendix

In this appendix, we provide additional architecture/implementation/dataset details, experimental settings, quantitative/qualitative results, limitations of our method, and broader impact of our method to complement the main paper. We organize the appendix as follows:

Architecture details of our framework

Implementation details and experimental settings

Details of datasets in our experiments

Additional quantitative and qualitative results

Comparison with State-of-the-Art with additional information

Class-wise F1 score on EK100 verb and noun classes

Additional qualitative analysis on EK100

Limitations

Broader impacts

A Architecture Details

In this section, we provide details of our B-CAST architecture. Let us assume we employ CLIP [44] as a spatial expert and VideoMAE [56] as a temporal expert.

B-CAST architecture. In Table 3, we provide stage-wise details of the two experts in B-CAST. Given an input from multi-head self-attention (MHSA) layer, we first pass it through the linear projection layer of each expert. Subsequently, the two experts exchanges their features each other. The multi-head cross-attention (MHCA) layer of each expert enables the effectively exchange of information between the two experts. Afterward, we pass the output tensors from the B-CAST

through the Feed-Forward Network (FFN). We repeat this process in a stacked manner, consisting of 12 blocks, each comprising the MHSA module along with adapters, B-CAST, and FFN module along with adapters. Finally, we feed the resulting tensors to a classification head to predict action.

Architecture details of different information exchange methods. In Figure 7, we visualize the architectures of the simple information exchange baselines presented in Table 1 (b) of the main paper. These baselines involve fully fine-tuning the two expert models without the use of adapters, following the training recipe outlined in Appendix § B. In Figure 7 (a), we present the add baseline, which facilitates information exchange between the spatial and temporal experts through late fusion using element-wise addition. In Figure 7 (b), we present the concat baseline, which exchanges the information between the spatial and the temporal experts through late fusion using concatenation. In Figure 7 (c), we show the lateral connection baseline, which enables information exchange between the spatial and temporal experts through layer-wise lateral connections, incorporating linear projections.

Temporal Expert

×

12 Blocks

Spatial Expert

×

12 Blocks

(c) Lateral

Multi-Head

Self-Attention

Feed Forward

Network

Feed Forward

Network

Classification

Head

Classification

Head

Noun

Verb

Linear

Linear

(b) Concat

Spatial Expert

Temporal Expert

Noun

Verb

Concatenate

(Late fusion)

Down

Projection

(a) Add

Spatial Expert

Temporal Expert

Noun

Verb

Add

(Late fusion)

Classification

Head

Classification

Head

Classification

Head

Classification

Head

CLS Token

ℝ

!

GAP Token

ℝ

!

CLS Token

ℝ

"

GAP Token

ℝ

"

Multi-Head

Self-Attention

CLS Token

ℝ

"

GAP Token

ℝ

"

Architecture details of the baselines in the B-CAST architecture ablation study. In Figure 8, we visualize the architectures of the baselines used in the B-CAST architecture ablation study, as presented in Table 1 (c) of the main paper. In Figure 8 (a), we illustrate the identity baseline, which serves as a baseline without the B-CAST module. In Figure 8 (b), we present the w/o adapter baseline, which includes cross-attention but does not incorporate the bottleneck adapters. In Figure 8 (c), we show the X-attn. → adapter baseline, which uses adapters positioned after the cross-attention stage. For reference, we include our B-CAST architecture in Figure 8 (d), which represents the final model used in our study.

Q

Down

Projection

Q

K

V

V

Down

Projection

Up

Projection

K

Up

Projection

Identity

Identity

(a) Identity

(b) W/o adapter

(c) X-attn.

→

adapter

(d) B-CAST

Down

Projection

LayerNorm

Temporal to Spatial

T2S

Cross-Attention

Q

K

V

V

Down

Projection

Up

Projection

LayerNorm

Spatial to Temporal

S2T

Cross-Attention

Q

K

Up

Projection

Q

K

V

V

Q

K

Temporal to Spatial

T2S

Cross-Attention

Spatial to Temporal

S2T

Cross-Attention

Temporal to Spatial

T2S

Cross-Attention

Spatial to Temporal

S2T

Cross-Attention

LayerNorm

LayerNorm

Classification head. We use different classification strategies for conventional action recognition and fine-grained action recognition as shown in Figure 9. i) For conventional action recognition datasets, i.e., Kinetics-400 (K400) and Something-Something-V2 (SSV2), CAST combines the CLS and GAP tokens from the two experts to predict actions, as depicted in Figure 9 (a). The spatial expert, CLIP [44], generates one CLS token for each frame. To obtain a single CLS token,

×

12 Blocks

Temporal Path

×

12 Blocks

CLS Token

GAP Token

ℝ

!

ℝ

!

Classification

Head

Average

Average

ℝ

"×!

ℝ

$%"×!

ℝ

"×($'()×!

Classification

Head

Noun

Verb

Action

×

12 Blocks

Temporal Path

×

12 Blocks

CLS Token

GAP Token

ℝ

!

ℝ

!

Average

Average

ℝ

"×!

ℝ

$%"×!

ℝ

"×($'()×!

Classification

Head

Action

Adapter

Adapter

(a) Conventional action recognition

(b) Fine-grained action recognition

we take the average of the CLS tokens from all frames of the input video. The temporal expert, VideoMAE [56], performs global average pooling on all output features to obtain a single GAP token. After passing through adapters, we add the CLS and GAP tokens together and feed the token into a linear classification head. ii) In a fine-grained action recognition dataset, i.e., EPIC-KITCHENS-100 (EK100), where a model needs to predict both verb and noun, we use two separate classification heads instead of a single head shown in Figure 9 (b). Specifically, we feed the CLS token from the spatial expert into a linear classification head for noun prediction and the GAP token from the temporal expert into another linear classification head for verb prediction.

B Implementation Details

In this section, we provide more details of our experimental setup and implementation of each dataset. We conduct the experiments with 16 NVIDA GeForce RTX 3090 GPUs. We implement CAST using PyTorch and build upon the existing codebase of VideoMAE [56].

Data preprocessing. After sampling the videos to 16 frames, We randomly crop each frame of the video and resize it to 224 × 224. We also apply data augmentation techniques, including mixup [73], label smoothing [55], horizontal flip, color jitter, and randaugment [9], repeated augmentation [22] to diversify the training data. We do not use horizontal flip on SSV2. Note that if the patch embedding layer of the temporal expert has a time stride value of 2, e.g., VideoMAE [56], we only take even frames for the spatial pathway. After the patch embedding layers, both experts take an input of 3 channels × 8 frames × 224 width × 224 height. We use the same data preprocessing protocol in all the experiments.

Model training. We conduct experiments using 2 nodes, each equipped with 8 GPUs. To ensure efficient multi-node training, we utilize the DeepSpeed 3 library. Additionally, we increase the effective batch size by implementing gradient accumulation to update the model weights. For the EK100 dataset, we set the update frequency to 4 iterations 4 , resulting in a total batch size of 24

per GPU. We linearly scale the base learning rate, then actual lr = base lr × total batch size / 256. In Table 4, we summarize the fine-tuning configuration and the hyperparameters used in Table 2, Table 9, Table 10, and Table 11.

Pre-trained weights. We take the off-the-shelf pre-trained weights of the two expert models. For our main temporal expert, we take the VideoMAE [56] weights pre-trained on the K400, and SSV2 datasets from the official repository 5 . Since VideoMAE does not provide pre-trained weights specifically for the EK100, we pre-train VideoMAE on the EK100 without incorporating extra video datasets. The pre-training process follows the recipe described in the VideoMAE paper [56]. For all other experiments, we make use of the pre-trained weights provided by the respective model repositories to ensure consistency and reliability in our results.

C Datasets.

Action recognition We evaluate our B-CAST module on two video datasets:Kinetics400 (K400) [24], Something-Something-V2 (SSV2) [19]. i) K400 is a large-scale third-person video dataset for action recognition that contains around 300K video clips and 400 human action classes. The dataset is split into train/val/test, with 240K/20K/40K video clips. The videos are all trimmed to around 10 seconds from different YouTube video. ii) SSV2 contains over 220K short video clips labeled video clips of humans performing pre-defined, basic actions with everyday objects. The dataset is split into train/val/test, with 168K/24K/27K and have 174 human-objects interaction categories.

Fine-grained action recognition We evaluate our B-CAST module on a Compositional Action dataset: EPIC-KITCHENS-100 (EK100) [10]. EK100 is a large-scale(100hours) egocentric video dataset that records several days of kitchen unscripted activities. It consists of 90K action segments, which are split into train/val/test sets of 67K/10K/13K. Differ from preceding two datasets, EK100 define an action as a combination of a verb and a noun. Because it requires matching both verbs and nouns to recognize an action, it is more challenging than recognizing actions in a dataset where actions are represented by a single label e.g., Kinetics-400, Something-Something-V2.

D Additional Quantitative Analysis

In this section, we provide additional results to complement the main paper. We demonstrate (1) the generality of CAST with different ViT architectures and pre-trained weights in Appendix § D.1, and (2) the effect of B-CAST-specific positional embeddings in Appendix § D.2.

D.1 Generality

In this section, we showcase the generality of the proposed method. We demonstrate that CAST works well with any ViT backbone and pre-trained weights. We conduct all the experiments on the EK100 dataset.

CAST is pre-training dataset agnostic. CAST is pre-training dataset agnostic. In Table 5, we compare CAST to a CAST variant where we replace CLIP with a ViT-B model pre-trained on the ImageNet-21K (IN21K) dataset. As a reference, we also show the performance of the independent experts using two separate expert models without any information exchange. When equipped with IN21K pre-trained ViT-B, CAST still achieves reasonable performance with an Action accuracy of 45 . 5%, while the baseline of independent experts achieves 40 . 4% accuracy.

In Table 6, we analyze the effect of pre-training datasets on the temporal expert, VideoMAE, in CAST. We show the results of using EK100, SSV2, and K400 pre-trained VideoMAE weights. We also investigate the impact of pre-training datasets on the spatial expert, CLIP. We present the results of using IN21K and CLIP pre-trained weights. When using CLIP pre-trained weights for the spatial expert, we observe stable Action accuracy ranging from a minimum of 48 . 7% to a maximum of 49 . 4%. We observe a similar trend when we use IN21K pre-trained CLIP weights. These results demonstrate that the proposed method is agnostic to the pre-training datasets.

CAST is model-agnostic. In Table 7, we demonstrate that CAST is model-agnostic. In this experiment, we replace our spatial and temporal expert models with other existing models. We employ EVA [53], an extension of CLIP that has shown excellent performance in various vision tasks, as our spatial expert. We employ MVD [63] as our temporal expert. The results show that CAST achieves similar performance when we employ different models as the experts. For example, EVA + MVD achieves an Action accuracy of 49 . 2%, while CLIP + VideoMAE achieves 48 . 7% Action accuracy.

(b) Attention along spatial axes

K

K

K

K

K

K

K

K

K

K

K

K

Q

K

K

K

K

K

Q

(a) Attention along temporal axes

K

K

K

K

K

K

K

K

K

K

K

K

K

K

K

K

K

K

K

K

K

K

K

K

K

K

K

Q

K

K

K

(c) Attention along spatio-temporal axes

!

"

#

!

"

#

!

"

#

: "#$%ℎ ∈ ℝ

!

!

∶ *+,-.

"

∶ /,.

D.2 B-CAST-specific positional embeddings

We investigate the impact of B-CAST-specific positional embeddings, as depicted in Figure 8 (d). In the temporal-to-spatial (T2S) cross-attention, the spatial expert attends along the temporal axis, as shown in Figure 10 (a), while in the self-attention stage during pre-training, the spatial expert attends along the spatial axes, as depicted in Figure 10 (b). Consequently, the spatial expert lacks information about temporal patch sequences. Similarly, in the spatial-to-temporal (S2T) cross-attention, the temporal expert attends along the spatial axis, as illustrated in Figure 10 (b), while during selfattention stage during pre-training, the temporal expert attends along the spatio-temporal axes, as shown in Figure 10 (c). Consequently, the temporal expert lacks information about spatial-only patch sequences. To address this limitation, we introduce new learnable positional embeddings that are specific to T2S and S2T cross-attention.

As shown in Table 8, adding the B-CAST-specific positional embeddings boost the performance by 1 . 2 points compared to without using the B-CAST-specific positional embeddings. The results indicate the effectiveness of the B-CAST-specific positional embeddings.

E Comparison with State-of-the-Art

To provide more comprehensive information, we augment the tables for comparison with state-ofthe-art in the main paper. Table 9, Table 10, and Table 11, corresponding to the respective datasets, include additional details such as the number of frames per clip, the number of temporal and spatial views used for inference, the computation complexity, and the number of learnable parameters for each model. For the details of the hyperparameters used, please refer to Table 4.

F Class-Wise Performance Comparison

We provide class-wise F1 score improvement of CAST over our spatial expert (CLIP) and our temporal expert (VideoMAE). We show the verb-class-wise F1 score improvement over CLIP in Figure 11 and the improvement over VideoMAE in Figure 12. We show the noun-class-wise F1 score improvement over CLIP in Figure 13 and the improvement over VideoMAE in Figure 14.

G Qualitative Analysis

To better understand the effectiveness of CAST, we provide qualitative analysis on more sample frames from the EK100 dataset in Figure 15. Each expert model provides more accurate prediction in their respective tasks of expertise but shows weaker performance in the other task. In contrast, CAST consistently shows correct predictions for both noun and verb prediction tasks. The qualitative examples demonstrate the effectiveness of CAST in achieving balanced spatio-temporal understanding, which is essential for fine-grained action recognition.

H Limitations

Despite achieving a good balanced spatio-temporal understanding performance, CAST has a few limitations as well. CAST has a small number of learnable parameters since we freeze the expert models except for the adapters. However, the computational complexity of CAST is not negligible due to the utilization of two expert models. Due to resource limitations, we are unable to conduct experiments on various input video lengths and model sizes. Lastly, cross-attention layers require features of the same dimension for attention operation. Therefore, if the two model have significantly different architectures, it might be challenging for CAST to employ the two models.

I Broader Impacts

Our work is on the task of human action recognition from videos. Surveillance could be one application, which might have privacy related concerns when the technology is deployed. Other consumer applications like personal or internet video search and tagging is expected to benefit individuals and organizations alike by helping them more efficiently maintain human centered data.

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

F1 score improvement

transition

drink

set

wear

scrape

crush

add

check

peel

cut

turn

attach

coat

drop

flip

form

grate

increase

lift

measure

pat

rip

roll

rub

search

sharpen

smell

sort

turn-down

use

wait

press

throw

fill

wash

squeeze

dry

pour

hang

scrub

adjust

break

fold

remove

cook

turn-on

mix

filter

scoop

wrap

empty

insert

spray

open

hold

look

turn-off

pull

sprinkle

unscrew

put

take

move

shake

divide

gather

close

feel

knead

apply

stretch

water

unroll

lower

eat

slide

brush

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

transition

turn-down

set

wear

drink

cook

hang

search

scrub

crush

knead

add

check

fill

attach

coat

drop

flip

form

grate

hold

increase

lift

measure

pat

rip

roll

rub

scrape

sharpen

smell

sort

use

wait

wash

turn-on

mix

adjust

turn

peel

put

open

pour

squeeze

throw

close

take

cut

shake

move

dry

turn-off

insert

press

eat

scoop

empty

remove

break

filter

pull

sprinkle

wrap

fold

unscrew

apply

gather

look

feel

spray

stretch

water

divide

unroll

lower

slide

brush

F1 score improvement

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

mat

chair

juice

pie

fan:extractor

burger

phone

cream

avocado

grape

squash

banana

parsley

foil

pizza

fish

cereal

lemon

plug

yoghurt

cooker:slow

wrap

leek

liquid

kale

vegetable

towel:kitchen

ginger

salmon

chopstick

corn

button

salad

recipe

spreads

microwave

meat

rack:drying

scissors

dishwasher

filter

package

leaf

blueberry

top

container

apron

asparagus

basil

bean:green

broccoli

cake

caper

choi:pak

clothes

coconut

coriander

drink

fishcakes

floor

handle

ice

juicer

ladle

lentil

mixture

pancake

paprika

pea

pin:rolling

presser

processor:food

sandwich

stalk

tablet

teapot

tofu

tuna

vinegar

wine

wire

yeast

jar

peeler:potato

bag

cloth

bread

can

rice

garlic

glass

box

lid

paper

peach

skin

maker:coffee

plate

oil

sink

oven

cup

hob

sponge

pot

pan

fridge

chilli

tap

coffee

water

cupboard

bottle

onion

kettle

bowl

lettuce

drawer

alarm

tray

sauce

rubbish

board:chopping

pasta

salt

milk

liquid:washing

machine:washing

food

curry

egg

pepper

carrot

towel

hand

paste

cucumber

spoon

cheese

dough

chicken

heat

kitchen

jug

colander

spatula

cutlery

knife

mushroom

shell:egg

clip

spice

glove

flour

butter

wrap:plastic

aubergine

fork

nut

pith

hummus

omelette

potato

basket

cellar:salt

cap

olive

tea

biscuit

apple

rest

oatmeal

sugar

spinach

onion:spring

courgette

brush

freezer

sausage

orange

light

powder

pork

lighter

noodle

oregano

seed

soup

tongs

grater

stock

bacon

scale

opener:bottle

chocolate

lime

mango

book

cover

holder

spinner:salad

thermometer

blender

toaster

F1 score improvement

F1 score improvement

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

mat

cellar:salt

juicer

fan:extractor

chair

avocado

cereal

leek

fish

plug

foil

salmon

parsley

juice

microwave

lemon

grape

liquid

pizza

chopstick

towel:kitchen

cooker:slow

pie

kale

wrap

banana

towel

spreads

meat

filter

yoghurt

squash

button

food

top

dough

sink

apron

asparagus

basil

bean:green

broccoli

cake

caper

choi:pak

clothes

coconut

coriander

corn

drink

fishcakes

floor

ginger

handle

ice

ladle

lentil

mixture

pancake

paprika

pea

pin:rolling

pith

presser

processor:food

sandwich

stalk

tablet

teapot

tofu

tuna

vinegar

wine

wire

yeast

drawer

scissors

salad

peeler:potato

heat

fridge

tap

can

kettle

cupboard

plate

cloth

peach

container

rubbish

hand

hob

pan

vegetable

oven

kitchen

coffee

skin

tray

dishwasher

sauce

sponge

lid

package

cup

alarm

water

oil

box

bag

cream

maker:coffee

liquid:washing

jar

board:chopping

leaf

onion

machine:washing

glass

chilli

burger

pot

salt

bottle

cutlery

pasta

mushroom

curry

bowl

cheese

spatula

milk

garlic

paste

spoon

spice

knife

paper

shell:egg

colander

rice

jug

lettuce

clip

bread

pepper

glove

butter

phone

wrap:plastic

carrot

light

egg

fork

basket

blueberry

cap

oatmeal

olive

biscuit

chicken

tea

apple

nut

rest

hummus

aubergine

spinach

onion:spring

courgette

freezer

lighter

potato

brush

rack:drying

sausage

cucumber

orange

powder

pork

recipe

noodle

oregano

seed

soup

tongs

grater

sugar

stock

bacon

scale

omelette

opener:bottle

chocolate

flour

lime

mango

book

cover

holder

spinner:salad

thermometer

blender

toaster

1.00

Take

Turn-on

Turn-on

Pour

Put

Pour

CLIP:

VideoMAE:

CAST:

CLIP:

VideoMAE:

CAST:

Carrot

Box

Carrot

CLIP:

VideoMAE:

CAST:

Pan

Pasta

Mushroom

CLIP:

VideoMAE:

CAST:

Knife

Bag

Knife

CLIP:

VideoMAE:

CAST:

Potato

Pepper

Potato

CLIP:

VideoMAE:

CAST:

Fork

Galic

Been:green

Asparagus

CLIP:

VideoMAE:

CAST:

GT:

Peach

Knife

Peach

CLIP:

VideoMAE:

CAST:

CLIP:

VideoMAE:

CAST:

Package

Package

Blueberry

CLIP:

VideoMAE:

CAST:

Nuts

Bag

Nuts

CLIP:

VideoMAE:

CAST:

Recipe

Box

Recipe

CLIP:

VideoMAE:

CAST:

Sponge

Sponge

Liquid

Turn-on

Turn-on

Turn-on

CLIP:

VideoMAE:

CAST:

Take

Take

Cut

CLIP:

VideoMAE:

CAST:

Hang

Throw

Throw

CLIP:

VideoMAE:

CAST:

Put

Move

Shake

CLIP:

VideoMAE:

CAST:

Wash

Dry

Wash

CLIP:

VideoMAE:

CAST:

Take

Mix

Mix

CLIP:

VideoMAE:

CAST:

Scoop

Scoop

CLIP:

VideoMAE:

CAST:

Take

Dry

Dry

CLIP:

VideoMAE:

CAST:

CHIErEE

CORN

FUUES

SERIAL

HLETEK

FLICUI

Peer reviews:
comments: SUMMARY:
This paper proposes a novel two-stream architecture called Cross-Attention in Space and Time (CAST) for video action recognition. The proposed architecture achieves a balanced spatio-temporal understanding of videos using only RGB input.

STRENGTHS:
1. The paper is well-written and well-structured.
2. The implementation of Adapters in the cross-attention mechanism is impressive.
3. The paper presents sufficient experiments.

WEAKNESSES:
In Table 2, when compared to VideoVAE, CAST only achieve minor improvement on the SSv2 dataset. The authors are encouraged to discuss why the cross-attention mechanism appears to have a limited impact on enhancing performance for motion-dominated datasets, like SSv2.  Likewise, it would be valuable to have an analysis explaining why the proposed method underperforms compared to AIM (CLIP pretrained only) on the K400 dataset.

QUESTIONS:
See weakness

LIMITATIONS:
The authors have addressed the limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

comments: SUMMARY:
This paper presents an approach to action recognition that is based on the fusion of two streams of analysis. This involves a spatial stream and a temporal stream that interact through a novel mechanism to improve classification performance.

STRENGTHS:
1. The paper is generally well written

2. The cross-attention mechanism is quite novel with respect to methods for 2 stream fusion.

3. Results appear to be quite good on standard benchmarks, and best when considering average performance

WEAKNESSES:
1. The B-CAST mechanism is quite complex and hard to unpack. With that said, I'm not sure there is a simpler way of framing this, but I would encourage the authors to consider simplifying the description (if possible). Figure 3 is fantastic in this regard.
2. The results only marginally improve the state of the art and are second best on the two main benchmarks considered, but bested by different models in each case. I wouldn't hold the paper back due to this, but it makes the contribution a bit weaker.

QUESTIONS:
1. Compared to other models, the proposed model appears to be better at verbs and weaker at nouns. Is it possible for the authors to comment on that trade-off?

2. Figure 6 shows a few good illustrated examples. Would the authors consider presenting more examples (e.g. as Supplementary Material)?

LIMITATIONS:
There is recognition of some of the limitations of the work throughout the paper, but not a dedicated statement.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents two-stream vision transformers, dubbed CAST, for balanced spatiotemporal video representation learning. Given the two experts, CLIP [36] and VideoMAE [46] for spatial and temporal expert, the proposed B-CAST module allows the exchange of complementary information across the separte experts via MHCA. Such information exchange appears to be crucial for achieving balanced performances on appearance-centric (K400, Noun prediction in EK) and motion-centric benchmarks (SSv2, Verb prediction in EK).\n\nSTRENGTHS:\n1.\tThe paper deliberately engineers to integrate two separate research streams, adapter [34, 60] and representation fusion [12, 33, 59], into one framework for balance spatiotemporal video representation learning.\n2.\tThe proposed method achieves strong performances on both appearance- and motion-centric benchmarks.\n3.\tExtensive investigation with several instantiations for CAST conducted in the ablation studies.\n\nWEAKNESSES:\n1.\tIn Tab. 1b, why are the three baseline models fully fine-tuned without adapters? Given that a partially tuned model with adapters is reported to be more effective than a fully fine-tuned model [34, 60], the comparison seems unfair. Reevaluating these models with adapters would make for a more fair comparison.\n2.\tConcerning the first point, are the model \"independent experts\" in Tab. 1a partially tuned with adapters or fully fine-tuned as in Tab. 1b? If it's the latter, for a fair comparison, CAST should be compared with an ensemble of two experts partially trained with adapters rather than with fully fine-tuned ones.\n3.\tFor a more in-depth understanding, I suggest including the number of parameters, FLOPs, and, if possible, inference throughput in Tab. 1, particularly in Tabs. 1a, 1b, and 1d..\n4.\tThe proposed method appears to increase computational cost and training & inference speed. Please provide a comparison of FLOPs, training & inference throughput with the baseline experts [36, 46], their ensemble (“independent experts” in Tab.1a), and other adapter-based methods [34, 60]. This would allow readers to comprehensively evaluate the applicability of the proposed method.\n5.\tIn Tab.1e, there is only a small performance gap between between CAST (4th row) and the model in 3rd row, i.e, (T2S, S2T) = (space, time). Does this imply that, under conditions where attention is optimized (L278), the important factor is layer-wise mixing of the features from the two experts, rather than the specific method of mixing? For your reference, I consider the \"Lateral\" model in Tab.1b, which linearly fuses each other, to be an unfair comparison to the models in Tab.1e due to its less computation. It would be better to elaborate the discussion including results of (T2S, S2T) = (space, time), (time, time) or other variants.\n6.\tTable 2 should include information on the pretraining dataset, FLOPs, # trainable parameters, # frames, and, if possible, inference speed for a comprehensive comparison.\n\nQUESTIONS:\n1.\tIs bi-directional information fusion necessary? What would be the result if we drop one of T2S MHCA or S2T MHCA?\n2.\tHas any analysis been conducted on verb prediction results similar to the approach taken in Fig. 5?\n3.\t(Minor) There is a discrepancy between the captions in Figure 4 and the actual subfigures. As a result, the text in lines 188-195 and 204-212 should be adjusted to refer to the correct subfigures.\n\nLIMITATIONS:\nThe authors have provided a comprehensive discussion on the potential limitations and social impact of their work in Supp. 8 and 9. If the proposed model is found to require more computational cost or slows the processing speed down due to two-stream architecture, this inefficiency should also be considered as a limitation, which would pave the way for future improvements.\n\n[Justification of the rating]\nI acknowledge that this paper deliberately integrates two separate methods, adapter [34, 60] and feature fusion [12, 33, 59], into one module for balance spatiotemporal video representation learning. However, I’m concerned with the fairness of the experiment (refer to W1, W2, W3, and W6) and efficiency (W4). Consequently, my preliminary rating leans towards “borderline reject,” but I remain open to changing the score if the authors provide adequate discussions during the rebuttal period.\n\n[Post-rebuttal justification]\nMost of my concerns are well addressed by the rebuttal. I've gone through the rebuttal and found the provided experimental results seems to be reasonable and solid. I strongly recommend adding these results and the related discussions to the final manuscript. Consequently, I raise my rating to \"weak accept.\"", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript proposes an approach of action recognition. The key idea behind the proposed approach is the usage of a two stream network where one network is specialized to encode the spatial details while the other to encode the temporal details. The approach also presents additional cross connections to effectively integrate the information from one stream to another. Experiments on three benchmark datasets shows that the proposed approach is capable of outperforming existing approaches on datasets requiring spatial reasoning, temporal reasoning and spatio-temporal reasoning.\n\nSTRENGTHS:\nThe paper is written clearly explaining the various contributions and the underlying motivations. Majority of the existing approaches consider action recognition as a generic problem. However, the analysis presented in the manuscript showing the performance variations of existing approaches on datasets requiring spatial reasoning, temporal reasoning and spatio-temporal reasoning is quite intuitive and useful to the research community. The various design choices are validated with an extensive set of ablation studies. This includes the impact of the various backbones, pretraining strategies and the contributions. The improved performance on the different datasets compared to existing approaches showcases the effectiveness of the proposed approach. Moreover, this work could inspire the action recognition research community to move away from the paradigm of one model for all.\n\nWEAKNESSES:\nTwo stream networks for action recognition is not a novel concept. For instance there are works that use RGB input in one network and optical flow in the other [1,2], two RGB input streams sampled at different frame rates [3] or spatial resolution [4], etc. However, there is no discussion regarding these approaches in the manuscript. The work of [5] is also similar to the proposed approach as it uses two different encoders for extracting spatial and temporal features. Even though the proposed approach is not exactly derivative of the few above-mentioned approaches, it is highly recommended to include a discussion comparing these approaches. \n\nIn order to predict the final action class, the adapter output of the CLS token of spatial expert and the adapter output of the global average pooled features of temporal expert is added followed by a classification layer. This addition operation across the features may diminish the discriminative information present in the individual streams. Instead, one may predict the action logits separately followed by an average fusion of the logits. Even though this operation is the same as the one presented in the paper, the compute graph is different and hence the gradient flow.\n\nEven though the proposed approach results in improved performance compared to existing approaches, the comparison sometimes seems unfair. For example, the spatial expert of the best CAST model is initialized with CLIP embedding while the temporal expert is initialized with VideoMAE pretrained on SSV2. The same goes for other datasets as well. Could the performance improvement be due to this additional data? Agreeing the fact this is the main concept of the proposed approach, the compute requirements needed for this additional pretraining also needs to be taken into context. Even though the computational complexity in terms of FLOPs is reported in the supplementary material, run time latencies could tell a different story. The reviewer recommends the authors to compare the run time latencies of the proposed approach with existing approaches. The number of trainable parameters reported in the supplementary material is also not indicative of how heavy the model is. It is recommended to report the total number of parameters present in the final model. This will be helpful to researchers while selecting a model by taking into consideration the compute/size vs accuracy tradeoff.\n\n[1] Simonyan, Karen, and Andrew Zisserman. \"Two-stream convolutional networks for action recognition in videos.\" Advances in neural information processing systems 27 (2014).\n\n[2] Feichtenhofer, Christoph, Axel Pinz, and Andrew Zisserman. \"Convolutional two-stream network fusion for video action recognition.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.\n\n[3] Feichtenhofer, Christoph, et al. \"Slowfast networks for video recognition.\" Proceedings of the IEEE/CVF international conference on computer vision. 2019.\n\n[4] Fan, Quanfu, et al. \"More is less: Learning efficient video representations by big-little network and depthwise temporal aggregation.\" Advances in Neural Information Processing Systems 32 (2019).\n\n[5] Jiang, Bo, et al. \"Two-pathway transformer network for video action recognition.\" 2021 IEEE International Conference on Image Processing (ICIP). IEEE, 2021.\n\nQUESTIONS:\nPlease see the weaknesses section for detailed queries.\n\nWhat is the CLS(.) operation in line 126?\n\nLIMITATIONS:\nThe manuscript adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method, namely CAST, for video action recognition based on adapting from large-scale pre-trained models (e.g., CLIP and VideoMAE). The main motivation is to balance and exchange information between spatial and temporal information of two different experts: spatial and temporal. The proposed CAST is an adapter architecture, which adopts CLIP as a spatial expert and VideoMAE as a temporal expert and introduces the cross attention to encourage interaction between these two streams. Experiments are conducted on Epic-KITCHEN-100, Kinetics-400, and Something-Something-v2 with good results compared with current methods. Written presentation is good and mostly easy to follow.\n\nSTRENGTHS:\n* The motivation of balancing between spatial and temporal information of video action recognition makes sense and the proposed architecture of CAST technically sounds.\n\n* Experimental results are strong: (i) good results compared with existing methods; (ii) various ablation to justify the model design choices.\n\nWEAKNESSES:\n* Although the good improvements are shown, the downside may be: there may be an unfair comparison with some other adapter-methods, e.g., AIM where they built up on only one expert model. This means CAST needs more data / pretraining time (of CLIP and VideoMEA);\n\n* There is no FLOPs (or runtime) comparison in table 2. As CAST employs a 2-stream architecture, there will be more computation compared with previous methods.\n\nQUESTIONS:\n* Some minor comments / typos:\n- line 104: \"p pixels\" -> p x p pixels for more consistent with that in line 107? \n- line 106: \"BT x N x D\" -> should be \"B x TN x D\" for more consistent with that in line 108?\n- Figure 4 caption has no explanation for (c).\n- line 204: Figure 4 (c) instead?\n\nLIMITATIONS:\nThe reviewer does not foresee any potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}