SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Classification head. We use different classification strategies for conventional action recognition and fine-grained action recognition as shown in Figure 9. i) For conventional action recognition datasets, i.e., Kinetics-400 (K400) and Something-Something-V2 (SSV2), CAST combines the CLS and GAP tokens from the two experts to predict actions, as depicted in Figure 9 (a). The spatial expert, CLIP [44], generates one CLS token for each frame. To obtain a single CLS token,
(a) Conventional action recognition
Figure 9: Classification head architecture. We use different classification strategies for conventional action recognition shown in (a) and fine-grained action recognition shown in (b).  $T$ ,  $N$ , and  $D$  denote the temporal sequence length, spatial sequence length, and embedding dimension respectively. The output from the spatial expert consists of frame-level feature vectors. Each frame-level feature vector consists of  $N$  patches and one CLS token. The output of the temporal expert is a video-level feature vector, consisting of  $N \cdot T$  patches.
(b) Fine-grained action recognition
we take the average of the CLS tokens from all frames of the input video. The temporal expert, VideoMAE [56], performs global average pooling on all output features to obtain a single GAP token. After passing through adapters, we add the CLS and GAP tokens together and feed the token into a linear classification head. ii) In a fine-grained action recognition dataset, i.e., EPIC-KITCHENS-100 (EK100), where a model needs to predict both verb and noun, we use two separate classification heads instead of a single head shown in Figure 9 (b). Specifically, we feed the CLS token from the spatial expert into a linear classification head for noun prediction and the GAP token from the temporal expert into another linear classification head for verb prediction.
In this section, we provide more details of our experimental setup and implementation of each dataset. We conduct the experiments with 16 NVIDIA GeForce RTX 3090 GPUs. We implement CAST using PyTorch and build upon the existing codebase of VideoMAE [56].
Data preprocessing. After sampling the videos to 16 frames, We randomly crop each frame of the video and resize it to  $224 \times 224$ . We also apply data augmentation techniques, including mixup [73], label smoothing [55], horizontal flip, color jitter, and randaugment [9], repeated augmentation [22] to diversify the training data. We do not use horizontal flip on SSV2. Note that if the patch embedding layer of the temporal expert has a time stride value of 2, e.g., VideoMAE [56], we only take even frames for the spatial pathway. After the patch embedding layers, both experts take an input of 3 channels  $\times$  8 frames  $\times$  224 width  $\times$  224 height. We use the same data preprocessing protocol in all the experiments.
Model training. We conduct experiments using 2 nodes, each equipped with 8 GPUs. To ensure efficient multi-node training, we utilize the DeepSpeed library. Additionally, we increase the effective batch size by implementing gradient accumulation to update the model weights. For the EK100 dataset, we set the update frequency to 4 iterations<sup>4</sup>, resulting in a total batch size of 24
<sup>3</sup>https://github.com/microsoft/DeepSpeed
$^{4}$ We use the update frequency of 2 iterations for the additional quantitative analysis experiments in Appendix  $\S$  D.
Table 4: Hyperparameters used and the fine-tuning configuration for each dataset.
Config	EK100	K400	SSV2
Optimizer	AdamW [39]	AdamW	AdamW
Base learning rate	1e-3	1e-3	1e-3
Weight decay	0.05	0.05	0.05
Optimizer momentum [7]	β1,β2=0.9,0.999	β1,β2=0.9,0.999	β1,β2=0.9,0.999
Gpu per batch size	6	6	6
Update frequency	4	6	4
Learning rate schedule	cosine decay [38]	cosine decay	Cosine decay
Warmup epochs	5	5	5
Training epochs	50	70	50
Flip augmentation	yes	yes	no
Color jitter	0.4	0.4	0.4
RandAug [9]	(9,0.5)	(9,0.5)	(9,0.5)
Label smoothing [55]	0.1	0.1	0.1
Mixup [73]	0.8	0.8	0.8
Drop path	0.2	0.2	0.3
Layer-wise lr decay [2]	0.75	0.75	0.75
per GPU. We linearly scale the base learning rate, then actual  $lr = \text{base} lr \times \text{total batch size} / 256$ . In Table 4, we summarize the fine-tuning configuration and the hyperparameters used in Table 2, Table 9, Table 10, and Table 11.
Pre-trained weights. We take the off-the-shelf pre-trained weights of the two expert models. For our main temporal expert, we take the VideoMAE [56] weights pre-trained on the K400, and SSV2 datasets from the official repository<sup>5</sup>. Since VideoMAE does not provide pre-trained weights specifically for the EK100, we pre-train VideoMAE on the EK100 without incorporating extra video datasets. The pre-training process follows the recipe described in the VideoMAE paper [56]. For all other experiments, we make use of the pre-trained weights provided by the respective model repositories to ensure consistency and reliability in our results.
C Datasets.
Action recognition We evaluate our B-CAST module on two video datasets: Kinetics400 (K400) [24], Something-Something-V2 (SSV2) [19]. i) K400 is a large-scale third-person video dataset for action recognition that contains around 300K video clips and 400 human action classes. The dataset is split into train/val/test, with 240K/20K/40K video clips. The videos are all trimmed to around 10 seconds from different YouTube video. ii) SSV2 contains over 220K short video clips labeled video clips of humans performing pre-defined, basic actions with everyday objects. The dataset is split into train/val/test, with 168K/24K/27K and have 174 human-objects interaction categories.
Fine-grained action recognition We evaluate our B-CAST module on a Compositional Action dataset: EPIC-KITCHENS-100 (EK100) [10]. EK100 is a large-scale(100hours) egocentric video dataset that records several days of kitchen unscripted activities. It consists of 90K action segments, which are split into train/val/test sets of 67K/10K/13K. Differ from preceding two datasets, EK100 define an action as a combination of a verb and a noun. Because it requires matching both verbs and nouns to recognize an action, it is more challenging than recognizing actions in a dataset where actions are represented by a single label e.g., Kinetics-400, Something-Something-V2.
D Additional Quantitative Analysis
In this section, we provide additional results to complement the main paper. We demonstrate (1) the generality of CAST with different ViT architectures and pre-trained weights in Appendix § D.1, and (2) the effect of B-CAST-specific positional embeddings in Appendix § D.2.
5https://github.com/MCG-NJU/VideoMAE
D.1 Generality
In this section, we showcase the generality of the proposed method. We demonstrate that CAST works well with any ViT backbone and pre-trained weights. We conduct all the experiments on the EK100 dataset.
CAST is pre-training dataset agnostic. CAST is pre-training dataset agnostic. In Table 5, we compare CAST to a CAST variant where we replace CLIP with a ViT-B model pre-trained on the ImageNet-21K (IN21K) dataset. As a reference, we also show the performance of the independent experts using two separate expert models without any information exchange. When equipped with IN21K pre-trained ViT-B, CAST still achieves reasonable performance with an Action accuracy of  $45.5\%$ , while the baseline of independent experts achieves  $40.4\%$  accuracy.
Table 5: Effect of CLIP pre-trained weights.
Method	Pre-training dataset	Information Exchange	Top-1 Acc.
Spatial expert	Temporal expert	Verb	Noun	Action
Independent experts	IN21K	EK100	×	69.7	50.9	40.4
CAST	IN21K	EK100	✓	70.9	56.8	45.5
CAST	CLIP	EK100	✓	72.5	60.3	48.7
In Table 6, we analyze the effect of pre-training datasets on the temporal expert, VideoMAE, in CAST. We show the results of using EK100, SSV2, and K400 pre-trained VideoMAE weights. We also investigate the impact of pre-training datasets on the spatial expert, CLIP. We present the results of using IN21K and CLIP pre-trained weights. When using CLIP pre-trained weights for the spatial expert, we observe stable Action accuracy ranging from a minimum of  $48.7\%$  to a maximum of  $49.4\%$ . We observe a similar trend when we use IN21K pre-trained CLIP weights. These results demonstrate that the proposed method is agnostic to the pre-training datasets.
Table 6: Effect of pre-trained weights.
Pre-training dataset	Top-1 Acc.
Spatial expert	Temporal expert	Verb	Noun	Act.
IN-21K	EK100	70.9	56.8	45.5
IN-21K	SSV2	71.6	56.1	45.3
IN-21K	K400	72.2	56.4	45.9
CLIP	EK100	72.5	60.3	48.7
CLIP	SSV2	73.3	60.0	49.0
CLIP	K400	72.9	60.4	49.4
CAST is model-agnostic. In Table 7, we demonstrate that CAST is model-agnostic. In this experiment, we replace our spatial and temporal expert models with other existing models. We employ EVA [53], an extension of CLIP that has shown excellent performance in various vision tasks, as our spatial expert. We employ MVD [63] as our temporal expert. The results show that CAST achieves similar performance when we employ different models as the experts. For example, EVA + MVD achieves an Action accuracy of  $49.2\%$ , while CLIP + VideoMAE achieves  $48.7\%$  Action accuracy.
Table 7: Effect of employing different models as experts.
Model architecture	Top-1 Acc.
Spatial expert	Temporal expert	Verb	Noun	Act.
CLIP	VideoMAE	72.5	60.3	48.7
CLIP	MVD	73.1	60.1	49.3
EVA	VideoMAE	73.1	59.8	49.1
EVA	MVD	73.7	60.1	49.2
(a) Attention along temporal axes
(b) Attention along spatial axes
(c) Attention along spatio-temporal axes
Figure 10: Visualization of attention window shape.
D.2 B-CAST-specific positional embeddings
We investigate the impact of B-CAST-specific positional embeddings, as depicted in Figure 8 (d). In the temporal-to-spatial (T2S) cross-attention, the spatial expert attends along the temporal axis, as shown in Figure 10 (a), while in the self-attention stage during pre-training, the spatial expert attends along the spatial axes, as depicted in Figure 10 (b). Consequently, the spatial expert lacks information about temporal patch sequences. Similarly, in the spatial-to-temporal (S2T) cross-attention, the temporal expert attends along the spatial axis, as illustrated in Figure 10 (b), while during self-attention stage during pre-training, the temporal expert attends along the spatio-temporal axes, as shown in Figure 10 (c). Consequently, the temporal expert lacks information about spatial-only patch sequences. To address this limitation, we introduce new learnable positional embeddings that are specific to T2S and S2T cross-attention.
As shown in Table 8, adding the B-CAST-specific positional embeddings boost the performance by 1.2 points compared to without using the B-CAST-specific positional embeddings. The results indicate the effectiveness of the B-CAST-specific positional embeddings.
Table 8: Effect of B-CAST-specific positional embeddings.
Method	Top-1 Acc.
Verb	Noun	Action
CAST w/o B-CAST-specific positional embeddings	71.3	59.7	47.5
CAST w/ B-CAST-specific positional embeddings	72.5	60.3	48.7
E Comparison with State-of-the-Art
To provide more comprehensive information, we augment the tables for comparison with state-of-the-art in the main paper. Table 9, Table 10, and Table 11, corresponding to the respective datasets, include additional details such as the number of frames per clip, the number of temporal and spatial views used for inference, the computation complexity, and the number of learnable parameters for each model. For the details of the hyperparameters used, please refer to Table 4.
F Class-Wise Performance Comparison
We provide class-wise F1 score improvement of CAST over our spatial expert (CLIP) and our temporal expert (VideoMAE). We show the verb-class-wise F1 score improvement over CLIP in Figure 11 and the improvement over VideoMAE in Figure 12. We show the noun-class-wise F1 score improvement over CLIP in Figure 13 and the improvement over VideoMAE in Figure 14.
Table 9: Comparison with the state-of-the-arts on the EPIC-Kitchens-100 dataset. We show the Top-1 accuracy for Action, Noun, and Verb prediction tasks as well as the number of frames per clip, the number of temporal and spatial views used for inference, the computation complexity, and the number of learnable parameters for each model. The best performance is in bold and the second best is underscored.
Method	Backbone	Frames	Views	TFLOPs	Learnable
Param (M)	Top-1 Acc.
Verb	Noun	Action
SlowFast [15]	ResNet50	-	-	-	-	54.9	50.0	38.5
GSF [52]	ResNet50	16	3×2	0.4	-	68.8	52.7	44.0
MoViNet [27]	MoViNet-A6	-	-	-	31	72.2	57.3	47.7
CLIP* [44]	ViT-B	8	2×3	0.84	86	55.5	52.3	33.9
AIM* [72]	ViT-B	16	2×3	2.42	14	64.8	55.5	41.3
ST-Adapter [42]	ViT-B	8	3×1	-	-	67.6	55.0	-
MBT [40]	ViT-B	32	-	-	-	64.8	58.0	43.4
ViViT FE [1]	ViT-L	32	4×1	15.92	311	66.4	56.8	44.0
MFormer [43]	ViT-L	32	3×1	3.56	-	67.1	57.6	44.1
ORViT-MF-HR [21]	ViT-B	16	10×3	-	-	68.4	58.7	45.7
VideoSwin [37]	Swin-B	-	-	-	89	67.8	57.0	46.1
VideoMAE* [56]	ViT-B	16	2×3	1.08	87	70.5	51.4	41.7
MeMViT [68]	ViT-B	16	1×1	0.06	-	70.6	58.5	46.2
OMNIVORE [16]	Swin-B	32	-	-	-	69.5	61.7	49.9
MTV-HR [71]	MTV-B	32	4×1	3.72	310	68.0	63.1	48.6
CAST w/ CLIP & VideoMAE pretrained on EPIC-KITCHENS-100	CAST-B	16	2×3	2.35	45	72.5	60.9	49.3
*We conduct experiments with our own implementation.
Table 10: Comparison with the state-of-the-arts on the Something-Something-V2 dataset. We show the Top-1 accuracy as well as the number of frames per clip, the number of temporal and spatial views used for inference, the computation complexity, and the number of learnable parameters for each model. The best performance is in bold and the second best is underscored.
Method	Backbone	Frames	Views	TFLOPs	Learnable Param (M)	Top-1 Acc.
SlowFast [15]	ResNet101	8+32	1×3	0.32	-	63.1
CLIP* [44]	ViT-B	8	2×3	0.84	-	43.2
EVL [35]	ViT-B	32	1×3	2.05	29	62.4
ST-Adapter [42]	ViT-B	32	3×1	1.96	-	69.5
AIM [72]	ViT-B	32	1×3	2.50	14	69.1
ViViT FE [1]	ViT-L	32	4×3	11.89	311	65.9
TimeSformer [3]	ViT-L	64	1×3	7.14	-	62.4
MViT [12]	ViT-B	64	1×3	1.37	37	67.7
MFormer [43]	ViT-L	32	1×3	3.56	-	68.1
Video Swin [37]	Swin-B	32	1×3	0.96	89	69.6
BEVT [62]	Swin-B	32	1×3	0.96	-	70.6
VideoMAE [56]	ViT-B	16	2×3	1.08	87	70.8
ORViT-MF-L [21]	ViT-L	32	1×3	-	-	69.5
OMNIVORE [16]	Swin-B	32	-	-	-	71.4
MTV-HR [71]	MTV-B	32	4×3	11.16	310	68.5
CAST w/ VideoMAE pretrained on Something-Something-V2	CAST-B	16	2×3	2.35	45	71.6
*We conduct experiments with our own implementation.
G Qualitative Analysis
To better understand the effectiveness of CAST, we provide qualitative analysis on more sample frames from the EK100 dataset in Figure 15. Each expert model provides more accurate prediction in their respective tasks of expertise but shows weaker performance in the other task. In contrast, CAST consistently shows correct predictions for both noun and verb prediction tasks. The qualitative examples demonstrate the effectiveness of CAST in achieving balanced spatio-temporal understanding, which is essential for fine-grained action recognition.
Despite achieving a good balanced spatio-temporal understanding performance, CAST has a few limitations as well. CAST has a small number of learnable parameters since we freeze the expert models except for the adapters. However, the computational complexity of CAST is not negligible due to the utilization of two expert models. Due to resource limitations, we are unable to conduct experiments on various input video lengths and model sizes. Lastly, cross-attention layers require features of the same dimension for attention operation. Therefore, if the two model have significantly different architectures, it might be challenging for CAST to employ the two models.
Table 11: Comparison with the state-of-the-arts on the Kinetics400 dataset. We show the Top-1 accuracy as well as the number of frames per clip, the number of temporal and spatial views used for inference, the computation complexity, and the number of learnable parameters for each model. The best performance is in bold and the second best is underscored.
Method	Backbone	Frames	Views	TFLOPs	Learnable Param (M)	Top-1 Acc.
SlowFast [15]	ResNet101	80	10×3	7.02	-	79.8
X3D [13]	X3D-XL	16	10×3	1.45	-	79.1
MoViNet [27]	MoViNet-A6	120	1×1	0.39	-	81.5
UniFormer [31]	Hybrid-B	32	4×3	3.12	50	83.0
CLIP* [44]	ViT-B	8	5×3	2.2	86	77.3
EVL [35]	ViT-B	32	3×1	1.78	29	84.2
ST-Adapter [42]	ViT-B	32	3×1	1.82	-	82.7
Text4Vis [69]	ViT-B	16	4×3	-	-	83.6
AIM [72]	ViT-B	32	3×1	2.43	11	84.7
X-CLIP [41]	ViT-B	16	4×3	3.44	-	84.7
ViViT FE [1]	ViT-L	128	1×3	11.94	311	81.7
TimeSformer [3]	ViT-L	96	1×3	25.06	430	80.7
MViT [12]	ViT-B	32	5×1	0.85	37	80.2
BEVT [62]	Swin-B	32	4×3	3.38	88	80.6
MFormer [43]	ViT-L	32	10×3	35.55	-	80.2
Video Swin [37]	Swin-L	32	4×3	7.25	197	83.1
VideoMAE [56]	ViT-B	16	5×3	2.7	87	81.5
OMNIVORE [16]	Swin-B	32	-	-	-	84.0
MTV-HR [71]	MTV-B	32	4×3	11.16	310	82.4
CAST w/ VideoMAE pretrained on Kinetics-400	CAST-B	16	5×3	5.87	45	85.3
*We conduct experiments with our own implementation.
I Broader Impacts
Our work is on the task of human action recognition from videos. Surveillance could be one application, which might have privacy related concerns when the technology is deployed. Other consumer applications like personal or internet video search and tagging is expected to benefit individuals and organizations alike by helping them more efficiently maintain human centered data.
Figure 11: Improvements of CAST over CLIP on EK100 verb classes. We show the class-wise F1 score improvement of CAST over CLIP. CAST achieves an improvement of 17.8 points on average. Best viewed with zoom and color.
Figure 12: Improvements of CAST over VideoMAE on EK100 verb classes. We show the class-wise F1 score improvement of CAST over VideoMAE. CAST achieves an improvement of 2.2 points in on average. Best viewed with zoom and color.
Figure 13: Improvements of CAST over CLIP on EK100 noun classes. We show the class-wise F1 score improvement of CAST over CLIP. CAST achieves an improvement of 7.9 points on average. Best viewed with zoom and color.
Figure 14: Improvements of CAST over VideoMAE on EK100 noun classes. We show the class-wise F1 score improvement of CAST over VideoMAE. CAST achieves an improvement of 9.2 points on average. Best viewed with zoom and color.
Noun Classes
CLIP: Carrot  
VideoMAE: Box  
CAST: Carrot
CLIP: Pan X  
VideoMAE: Pasta X  
CAST: Mushroom
Verb Classes
CLIP: Take  
VideoMAE: Turn-on  
CAST: Turn-on
CLIP: Pour  
VideoMAE: Put  
CAST: Pour
CLIP: Knife √  
VideoMAE: Bag ×  
CAST: Knife √
CLIP: Potato  
VideoMAE: Pepper  
CAST: Potato
CLIP: Turn-on  
VideoMAE: Turn-on  
CAST: Turn-on
CLIP: Take  
VideoMAE: Take  
CAST: Cut
CLIP: Fork  
VideoMAE: Galic  
CAST: Been:green  
GT: Asparagus
CLIP: Peach  
VideoMAE: Knife  
CAST: Peach
CLIP: Hang X  
VideoMAE: Throw  
CAST: Throw
CLIP: Put X VideoMAE: Move X CAST: Shake Y
CLIP: Package  
VideoMAE: Package  
CAST: Blueberry
CLIP: Nuts  
VideoMAE: Bag  
CAST: Nuts
CLIP: Wash  
VideoMAE: Dry  
CAST: Wash
CLIP: Take  
VideoMAE: Mix  
CAST: Mix
CLIP: Recipe  
VideoMAE:Box  
CAST:Recipe
Figure 15: Qualitative examples from EK100 comparing CLIP, VideoMAE, and the proposed CAST. Each expert model shows more accurate predictions in their expertise but shows weaker performance on the other task. However, the proposed CAST consistently shows correct predictions for both noun and verb classes, demonstrating the effectiveness of the proposed spatio-temporal cross-attention mechanism. Best viewed with zoom and color.
CLIP: Sponge X  
VideoMAE: Sponge X  
CAST: Liquid X
CLIP: Scoop  
VideoMAE: Insert  
CAST: Scoop
CLIP: Take  
VideoMAE: Dry  
CAST: Dry