SlowGuess/NeurIPS_2023_OCR huggingface dataset:
HMDB-51 and UCF-101 are two relatively small datasets, which contain 6766 videos from 51 action categories and 13,320 videos from 101 categories, respectively. Something-Something v2 (SSv2) is a large-scale motion-centric video dataset, including 168,913 videos for training and 24,777 videos for validation from 174 categories. As a typical video understanding task, we choose action recognition and report top-1 accuracy as the evaluation metric in all experiments.
By following [45, 29], we convert the raw videos into the MP4 compressed format with a GOP size of 12, where each GOP consists of 1 I-frame and 11 P-frames. We collect I-frames, Motion Vectors and Residuals from 4 consecutive GOPs, totally containing 4 I-frames and 12 P-frames. Within each GOP, we uniformly sample 3 P-frames. In regards of the backbone network, we utilize two representative video Transformer architectures based on ViT [10] and Swin Transformer [34] pre-trained on Kinetics-400 with raw videos, respectively. We adopt the original model configurations and train the prompt parameters using the AdamW optimizer [35] on 12 NVIDIA V100 GPUs. We apply random scaling, corner cropping, and horizontal flipping as data augmentation. The base learning rate, weight decay and batch size are set to  $1 \times 10^{-3}$ ,  $1 \times 10^{-4}$  and 240, respectively. Additionally, we adopt a warm-up strategy within the first 5 training epochs.
4.3 Comparison with the State-of-the-art Approaches
Compressed video-based methods. We firstly compare our method with the following full fine-tuning compressed video-based models: 1) self-supervised pre-training based ones including CoViAR [45], IMRNet [48], MVCGC [21] and the Full Fine-tuning baseline; and 2) supervised pre-training based ones including Refined-MV [2], IPTSN [20], SIFP-Net [30], MEACI-Net [29] and MM-ViT [5]. Table 1 and Table 2 summarize the comparison results on the HMDB-51, UCF-101 and SSv2 datasets. Generally, the proposed CVPT method achieves comparable or even better performance than the full fine-tuning ones, by tuning less than  $1\%$  of the overall parameters. Notably, when adopting the same ViT-B backbone, CVPT improves the top-1 accuracy of MM-ViT/Full Fine-tuning by  $2.1\% / 0.9\%$  on UCF-101, and  $2.9\% / 0.8\%$  on SSv2, via tuning only  $0.6\%$  of the parameters.
Besides the full fine-tuning methods, we also compare with representative efficient fine-tuning models, including the Linear Probe baseline, VPT [23] and AdaptFormer [7]. Since VPT and AdaptFormer only report results on raw videos, we re-implement and evaluate them on compressed videos, denoted by  $\mathrm{VPT^{*}}$  and  $\mathrm{Adaptformer^{*}}$ , respectively. As displayed, Linear Probe exhibits the lowest performance as it only optimizes the classification head.  $\mathrm{VPT^{*}}$  slightly outperforms  $\mathrm{Adaptformer^{*}}$ , but both of their performance are clearly inferior to the fine-tuning counterparts. This suggests that these methods struggle to effectively align the upstream data for pre-training the downstream data for fine-tuning. In contrast, our approach leverages inserted prompt blocks to facilitate the interaction of multimodal information, boosting the accuracy of  $\mathrm{VPT^{*}}$  and  $\mathrm{Adaptformer^{*}}$  by  $10.8\%$  and  $12.0\%$  on HMDB-51,  $5.4\%$  and  $7.1\%$  on UCF-101, and  $4.8\%$  and  $5.6\%$  on SSv2, respectively, when pre-training with the self-supervised learning.
In addition to the accuracy, our approach also has advantages in its training efficiency and facilitation. Concretely, unlike IPTSN that adopts a complex knowledge distillation model based on a raw video-based teacher during fine-tuning, or MVCGC that mines cross-modality information from pretrained RGB models through self-supervised learning, our method explores the efficient tuning and reaches a promising accuracy. Moreover, as shown in Table 1 and Table 2, our approach consistently promotes the performance by different pre-training strategies and network architectures (e.g. ViT-L), highlighting its scalability.
Raw video-based methods. As summarized in Table 1 and Table 2, raw video-based fine-tuning methods generally reach higher performance than compressed video-based counterparts, since the raw videos convey more coherent and consistent information. Nevertheless, CVPT significantly reduces this gap, and even surpasses both VPT and Adaptformer that are fine-tuned on raw videos. The reason lies in that CVPT explores the Motion Vectors and Residuals as prompts to complement the missing motion cues caused by sparse sampling of RGB frames in a more comprehensive way.
Table 1: Comparison with the state-of-the-art approaches w.r.t. the top-1 accuracy  $(\%)$  on HMDB-51 and UCF-101 pre-trained on Kinetics-400. \* indicate the results by our re-implementation using the same configuration. PT' stands for pre-training. 'SL' and SSL' refer to supervised learning and self-supervised learning, respectively.
Method	Modality	Backbone	Input [M]	GFLOPs	PT	Tunable Params. [M]	HMDB-51	UCF-101
Raw	CoCLR [17]	RGB+OF	S3D	26.2	182.0	SSL	17.5 (100%)	62.9	90.6
RSPNet [6]	RGB	S3D-G	6.0	91.0	SSL	9.6 (100%)	64.7	93.7
CVRL [38]	RGB	R3D-50	48.2	402.0	SSL	46.9 (100%)	65.4	92.1
ρBYOL [13]	RGB	R3D-50	12.1	402.0	SSL	46.9 (100%)	73.6	95.5
STS [43]	RGB	S3D-G	96.3	689.0	SSL	9.6 (100%)	62.0	89.0
M³Video [41]	RGB	ViT-B	24.1	1080.0	SSL	86.4 (100%)	75.4	96.1
MVD-B [44]	RGB	ViT-B	24.1	1080.0	SSL	86.4 (100%)	76.4	97.0
MotionMAE [47]	RGB	ViT-B	24.1	1080.0	SSL	86.4 (100%)	-	96.3
VideoMAE [42]	RGB	ViT-B	24.1	1080.0	SSL	86.4 (100%)	73.3	96.1
VPT [23]	RGB	ViT-B	12.1	1089.6	SSL	0.1 (0.09%)	52.7	-
AdaptFormer [7]	RGB	ViT-B	12.1	1093.8	SSL	1.3 (1.46%)	55.7	-
Comp.	CoViAR [45]	I-Frame+MV+Res	ResNet152	3.8	1222.0	SSL	142.5 (100%)	37.1	63.7
IMRNet [48]	I-Frame+MV+Res	R3D-18	3.8	-	SSL	100.8 (100%)	45.0	76.8
MVCGC [21]	RGB+MV	S3D	15.7	-	SSL	17.5 (100%)	63.4	90.8
Refined-MV [2]	I-Frame+MV+Res	ResNet152	-	-	SL	142.5 (100%)	59.7	89.9
IPTS_N [20]	I-Frame+MV+Res	ResNet152	6.8	-	SL	130.8 (100%)	69.1	93.4
SIFP-Net [30]	I-Frame+MV+Res	I3D	8.1	-	SL	92.8 (100%)	72.3	94.0
MEACI-Net [29]	I-Frame+MV+Res	I3D	0.7	269.4	SL	50.7 (100%)	74.4	96.4
MM-ViT [5]	I-Frame+MV+Res	ViT-B	8.1	-	SL	86.4 (100%)	-	93.4
Full Fine-tuning	I-Frame+MV+Res	ViT-B	6.1	772.2	SSL	86.4 (100%)	60.3	88.1
Linear Probe VPT* [23]	I-Frame+MV+Res	ViT-B	6.1	772.2	SSL	0.1 (0.08%)	47.8	76.8
AdaptFormer* [7]	I-Frame+MV+Res	ViT-B	6.1	778.2	SSL	0.1 (0.09%)	52.1	83.6
I-Frame+MV+Res	ViT-B	6.1	780.6	SSL	1.3 (1.46%)	50.9	81.9
CVPT (Ours)	I-Frame+MV+Res	ViT-B	6.1	772.2	SSL	0.5 (0.6%)	62.9	89.0
CVPT (Ours)	I-Frame+MV+Res	ViT-B	6.1	772.2	SL	0.5 (0.6%)	69.7	95.5
CVPT (Ours)	I-Frame+MV+Res	ViT-L	6.1	2569.6	SL	0.1 (0.03%)	81.5	98.0
Table 2: Comparison with the state-of-the-art approaches w.r.t. the top-1 accuracy  $(\%)$  on SSv2 pretrained on Kinetics-400. \* indicate the results by our re-implementation using the same configuration. PT' stands for pre-training. 'SL' and SSL' refer to supervised learning and self-supervised learning, respectively.
Method	Modality	Backbone	Input [M]	GFLOPs	PT	Tunable Params. [M]	SSv2
Raw	RSPNet [6]	RGB	S3D-G	6.0	91.0	SSL	9.6 (100%)	55.0
ρBYOL [13]	RGB	R3D-50	12.1	402.0	SSL	46.9 (100%)	55.8
MVD-B [44]	RGB	ViT-B	24.1	1080.0	SSL	86.4 (100%)	72.5
M³Video [41]	RGB	ViT-B	24.1	1080.0	SSL	86.4 (100%)	70.5
MotionMAE [47]	RGB	ViT-B	24.1	1080.0	SSL	86.4 (100%)	71.8
VideoMAE [42]	RGB	ViT-B	24.1	1080.0	SSL	86.4 (100%)	70.3
VPT [23]	RGB	ViT-B	12.1	1089.6	SSL	0.1 (0.09%)	43.7
AdaptFormer [7]	RGB	ViT-B	12.1	1093.8	SSL	1.3 (1.46%)	59.0
Comp.	MM-ViT [5]	I-Frame+MV+Res	ViT-B	8.1	-	SL	86.4 (100%)	62.6
Full Fine-tuning	I-Frame+MV+Res	ViT-B	6.1	772.2	SSL	86.4 (100%)	57.6
Linear Probe	I-Frame+MV+Res	ViT-B	6.1	772.2	SSL	0.1 (0.08%)	33.8
VPT* [23]	I-Frame+MV+Res	ViT-B	6.1	778.2	SSL	0.1 (0.09%)	53.6
AdaptFormer* [7]	I-Frame+MV+Res	ViT-B	6.1	780.6	SSL	1.3 (1.46%)	52.8
CVPT (Ours)	I-Frame+MV+Res	ViT-B	6.1	772.2	SSL	0.5 (0.6%)	58.4
CVPT (Ours)	I-Frame+MV+Res	ViT-L	6.1	2569.6	SL	0.1 (0.03%)	65.5
Additionally, CVPT is capable of directly fine-tuning an off-the-shelf model pre-trained on raw videos to the domain of compressed videos, thus simultaneously saving extra computational cost in alternatively pre-training on the specific compressed data.
To evaluate the effectiveness of components in CVPT, we conduct extensive ablation study. Without loss of generality, we utilize ViT-B pre-trained by self-supervised learning as the backbone.
On the Main Components. We firstly evaluate the effect of the main components of the proposed method. Specifically, we employ linear probe as the baseline method. As shown in Table 3, the proposed CPR module without refinement, which only updates the conditional prompts in the first layer, clearly improve the baseline. By introducing the SCCP module, which enriches the cross-modal information between I-frames and conditional prompts of Motion Vectors and Residuals, the
Table 3: Ablation results (%) on the main components of CVPT.
CPR w/o refinement	SCCP	CPR	HMDB-51	UCF-101
✓	47.8	76.8
55.7	83.9
✓	✓	61.4	87.6
✓	✓	62.9	89.0
Figure 4: The influence of the number of SCCP blocks on the proposed method.
(a) SCCP w/o  $P_{\mathrm{R}}$  prompt
Figure 5: Illustrations of alternative structures of SCCP. (a) and (b) indicate SCCP without  $P_R$  prompt and without attentions, respectively.
(b) SCCP w/o attention
performance is remarkably boosted. Finally, applying the refinement module on CPR further obtains gain in performance, leading to the highest accuracy.
On the Selective Cross-modal Complementary Prompt (SCCP). To demonstrate the effect of the proposed SCCP module, we compare it to the following two alternative architecture designs as shown in Figure 5: 1) removing the  $P_{\mathrm{R}}$  prompt and separately embedding the Motion Vectors and Residuals into distinct two types of prompts as displayed in Figure 5(a); 2) abandoning the Residual-guided attention and directly adding to the branch of Motion Vectors as displayed in Figure 5(b). As summarized in Table 4, both of the two above alternative designs yield worse performance than the proposed SCCP module, thus demonstrating the effectiveness of employing the Residual prompt to facilitate extracting more informative dynamic cues for the downstream task.
Besides, we investigate the impact of incorporating different numbers of SCCP blocks into the backbone. The insertion process begins from the first layer and increments by three layers in each iteration. As illustrated in Figure 4, we observe that the overall performance on both datasets is consistently improved as the number of inserted SCCP blocks increases.
On Distinct Backbones. We evaluate the performance of our method by using different backbones, compared to the Full Fine-tuning baseline. As summarized in Table 5, our approach steadily outperforms Full Fine-tuning for various backbones by distinct pre-training strategies.
4.5 On the Efficiency of CVPT
We present a comprehensive comparison of inference time between the proposed CVPT method and the representative raw video-based approach VideoMAE. As shown in Table 6, our approach
Table 4: Ablation study (\%) on different structures of the proposed SCCP module.
Method	Total Params. [K]	HMDB-51	UCF-101
SCCP w/o attention	39.2	59.6	86.9
SCCP w/o PR prompt	19.2	57.6	85.5
SCCP	39.2	61.4	87.6
Table 5: Ablation study  $(\%)$  on different backbones.
Backbone	ViT-B (SSL)	ViT-B (SL)	Swin-B (SL)
HMDB-51	UCF-101	HMDB-51	UCF-101	HMDB-51	UCF-101
Full Fine-tuning	60.3	88.1	66.5	94.3	62.9	92.0
Ours	62.9 (+2.6)	89.0 (+0.9)	69.7 (+3.2)	95.5 (+1.2)	64.9 (+2.0)	94.0 (+2.0)
Table 6: Comparison of inference time (ms) per video.
Method	Pre-Process	Model Inference	Full Pipeline
VideoMAE [42]	2496.9	26.9	2523.8
CVPT (Ours)	238.3	25.0	263.3
BabyCrawling
Linear Probe
Adaptformer
Adaptformer*
VPT*
Figure 6: Visualization of the last layer feature map generated by various approaches on UCF-101.
Full Fine-tuning
CVPT
BaseballPitch
VPT*
Linear Probe
Full Fine-tuning
significantly reduces the pre-processing time without the need for video decoding, while maintaining a comparable model inference time, thus remarkably decreasing the overall time cost.
4.6 Qualitative Results
We visualize the classification maps generated by CVPT, compared to several state-of-the-art compressed video-based approaches including Linear Probe, Adaptformer*, VPT* and Full Fine-tuning. As shown in Figure 6, our method exhibits a broader focus on motion regions than those compared methods.
We propose a novel prompt tuning method, namely CVPT, for compressed videos. Our approach leverages Motion Vectors and Residuals to generate conditional prompts. Furthermore, we employ the SCCP blocks to fully explore the discriminative dynamic cues in Motion Vectors as complements to I-frames. Extensive experimental results on public benchmarks demonstrate the efficiency and effectiveness of CVPT, compared to the current state-of-the-art approaches for efficient fine-tuning.
Despite the promising performance, the scalability of the proposed CVPT method to various vision tasks such as compressed video-based segmentation and tracking has not been investigated. We will further study this problem in our future research.
This work is partly supported by the National Key R&D Program of China (2021ZD0110503), the National Natural Science Foundation of China (62022011 and 62202034), the Research Program of State Key Laboratory of Software Development Environment, and the Fundamental Research Funds for the Central Universities.
[44] Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, and Yu-Gang Jiang. Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning. In CVPR, 2023.
[45] Chao-Yuan Wu, Manzil Zaheer, Hexiang Hu, R. Manmatha, Alexander J. Smola, and Philipp Kähenbuhl. Compressed video action recognition. In CVPR, 2018.
[46] Kai Xu and Angela Yao. Accelerating video object segmentation with compressed video. In CVPR, 2022.
[47] Haosen Yang, Deng Huang, Bin Wen, Jiannan Wu, Hongxun Yao, Yi Jiang, Xiatian Zhu, and Zehuan Yuan. Self-supervised video representation learning with motion-aware masked autoencoders. arXiv preprint arXiv:2210.04154, 2022.
[48] Youngjae Yu, Sangho Lee, Gunhee Kim, and Yale Song. Self-supervised learning of compressed video representations. In ICLR, 2020.
[49] Bowen Zhang, Limin Wang, Zhe Wang, Yu Qiao, and Hanli Wang. Real-time action recognition with enhanced motion vector cnns. In CVPR, 2016.
[50] Yanan Zhang, Jiaxin Chen, and Di Huang. Cat-det: Contrastively augmented transformer for multi-modal 3d object detection. In CVPR, 2022.
[51] Yue Zhao and Philipp Krahenbuhl. Real-time online video detection with temporal smoothing transformers. arXiv preprint arXiv:2209.09236, 2022.
[52] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.
[53] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 130(9):2337-2348, 2022.
[54] Jiawen Zhu, Simiao Lai, Xin Chen, Dong Wang, and Huchuan Lu. Visual prompt multi-modal tracking. In CVPR, 2023.