Compressed Video Prompt Tuning
Authors: Bing Li
Jiaxin Chen
Xiuguo Bao
Di Huang
Venue: NeurIPS 2023 poster (2023)
Keywords: Compressed video
Action Recognition
Prompt Tuning
TL;DR: Our novel prompt learning method efficiently tunes pre-trained model for raw videos to various downstream tasks for compressed videos.

Abstract: Compressed videos offer a compelling alternative to raw videos, showing the possibility to significantly reduce the on-line computational and storage cost. However, current approaches to compressed video processing generally follow the resource-consuming pre-training and fine-tuning paradigm, which does not fully take advantage of such properties, making them not favorable enough for widespread applications. Inspired by recent successes of prompt tuning techniques in computer vision, this paper presents the first attempt to build a prompt based representation learning framework, which enables effective and efficient adaptation of pre-trained raw video models to compressed video understanding tasks. To this end, we propose a novel prompt tuning approach, namely Compressed Video Prompt Tuning (CVPT), emphatically dealing with the challenging issue caused by the inconsistency between pre-training and downstream data modalities. Specifically, CVPT replaces the learnable prompts with compressed modalities (\emph{e.g.} Motion Vectors and Residuals) by re-parameterizing them into conditional prompts followed by layer-wise refinement. The conditional prompts exhibit improved adaptability and generalizability to instances compared to conventional individual learnable ones, and the Residual prompts enhance the noisy motion cues in the Motion Vector prompts for further fusion with the visual cues from I-frames. Additionally, we design Selective Cross-modal Complementary Prompt (SCCP) blocks. After inserting them into the backbone, SCCP blocks leverage semantic relations across diverse levels and modalities to improve cross-modal interactions between prompts and input flows. Extensive evaluations on HMDB-51, UCF-101 and Something-Something v2 demonstrate that CVPT remarkably outperforms the state-of-the-art counterparts, delivering a much better balance between accuracy and efficiency.

Compressed Video Prompt Tuning
Bing Li
Jiaxin Chen² Xiuguo Bao Di Huang 1.2+
'SKLSDE, Beihang University, Beijing, China
IRIP Lab, SCSE, Beihang University. Beijing, China $CNCERT/CC, Beijing, China
(libingsy, jiaxinchen dhuang @ @buaa.edu.cn, baoxiuguo@ 139.com
Abstract
Compressed videos offer a compelling alternative to raw videos, showing the possi- bility to significantly reduce the on-line computational and storage cost. However, current approaches compressed video processing generally follow the resource- consuming pre-training and fine-tuning paradigm, which does not fully take ad- vantage of such properties, making them not favorable enough for widespread applications. Inspired by recent successes of prompt tuning techniques in computer vision, this paper presents the first attempt build prompt based representation learning framework, which enables effective and efficient adaptation of pre-trained video models to compressed video understanding tasks. To this end, pro- pose novel prompt tuning approach, namely Compressed Video Prompt Tuning (CVPT), emphatically dealing with the challenging issue caused by the inconsis- tency between pre-training and downstream data modalities. Specifically, CVPT replaces the learnable prompts with compressed modalities (e.g. Motion Vectors and Residuals) by re-parameterizing them into conditional prompts followed by layer-wise refinement. The conditional prompts exhibit improved adaptability and generalizability to instances compared to conventional individual learnable ones, and the Residual prompts enhance the noisy motion cues in the Motion Vector prompts for further fusion with the visual cues from I-frames. Additionally, we design Selective Cross-mode Complementary Prompt (SCCP) blocks. After in- serting them into the backbone, SCCP blocks leverage semantic relations across diverse levels and modalities to improve cross-modal interactions between prompts and input flows. Extensive evaluations on HMDB-51, UCF-101 and Something- Something v2 demonstrate that CVPT remarkably outperforms the state-of-the-art counterparts, delivering a much better balance between accuracy and efficiency.
Introduction
In recent years, there has been a surge of interest compressed video understanding 51 36
[1 due to its great potential in saving the computational and storage cost for on-line processing. which directly performs inference on compressed data and thus bypasses the resource-consuming decoding phase.
Raw videos consist of dense RGB frames conveying coherent and consistent content, while com- pressed videos are composed of sparsely decoded RGB frames with entire spatial information, i.e. Intra-frames (I-frames), and incompletely decoded RGB frames with free but coarse motion clues, i.e. Predicted frames (P (P-frames) or Bidirectionally predicted frames (B (B-frames). In this case, existing approaches to compressed video understanding generally follow the pre-training and fine-tuning paradigm, which initially pre- pre-trains models on compressed videos from a large database (e.g. Kinetics- 400 3 and fully fine-tunes them on downstream ones. According to different network structures,
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

IF I-Frames
Res: Residuals
Tunable Parameters
Sase : Pre- Raw Vidro Model Sr/ Beunch from Seniich for oe P/B-Frims PT: Prompt Tuning
FT: Fine- Tuming
MV Metion Vectons
Fusion Module
Frozen Parameters
Full Fine-Tuming
Full Fine- Tuning
Prompt-Tuning
Prompi-Tuning
Load Sean
Partially load Sucu
SRGB
Suur
MV
F
MV
Modified VIT
MV
MV
Res
Gr
Res
Res
(a) on Multi-streum CNNs
(b) FT on VIT
(c) Pluin
Compressed
(d) CVPT (Ours)
Figure I: Different types of approaches to representation learning in compressed videos. (a) and (b)
are the two main variants of the existing pre-training and fine-tuning paradigm, where the former is based on multi-stream CNNs and the latter is based on ViT, both of which require pre-training models on compressed videos. (c) plain prompt tuning scheme which can be considered as
a straightforward attempt to leverage frozen off-the-shelf models pre-trained on raw videos, but does not well adapt to the given task with degraded performance. (d) CVPT advances (c) by designing conditional prompts for Motion Vectors and Residuals, delivering novel paradigm which successfully learns powerful representations for compressed videos in a more effective and efficient manner.
they can be broadly classified into two categories. (1) [20 21 19 III apply multi-branch CNNs for individual modalities, which are then fused to deliver final results, as depicted in Figure (a). (ii) [5] employs a modified Vision Transformer (ViT) for all modalities, motivated by the advantage of VIT
in reasoning tasks and the ability to handle multi-modalities in text-image [5]] 37 50 and multi-task
domains [1, 18), as shown in Figure D (b). Despite their effectiveness, such a paradigm suffers two major drawbacks. Firstly, it requires pre-training networks compressed videos, resulting in high computational complexity in GPU-day in off-line mode. Several attempts are made to speed up pre-training by loading or partially loading off-the-shelff models built based on raw videos; however, is still not favorable enough for wide popularization. Secondly, it relies on fully fine-tuning backbones, which incurs large parameter storage burden, making it challenging for deployment in real-world applications. Furthermore, full fine-tuning tends to disrupt the knowledge learned by the pre-trained model, degrading the generalization performance (54). The limitations aforementioned highlight the need for more effective and efficient alternative to address these issues.
Inspired by the advancements of prompting techniques in Natural Language Processing (NLP), researchers from the field of Computer Vision (CV) have been discussing this representation learning paradigm. By freezing the pre- trained model and introducing small number of learnable parameters on input images or videos, they create decent visual prompts 4 information mining from pre-trained models and eliminates the necessity for fully fine-tuning networks. On the other side, raw videos contain richer spatial and temporal clues than compressed ones, and if sufficient and important knowledge can be mined from the models pre-trained on raw videos, it should strengthen representation learning of compressed videos. Both the facts trigger us to investigate prompt tuning paradigm to compressed video understanding with pre-trained raw video models as a more effective and efficient alternative to the pre-training and have achieved successes in various tasks, they generally work in the case that pre- trainininn and downstream data share the same modality and it remains problematic when they have different modalities, i.e. the pre-training data are in the raw video (dense RGB frames) modality and the downstream data are in the compressed video modality (I-frames, Motion Vectors and Residuals), as shown in Figure (c).
23 This paradigm enables efficient
one. Although
current prompt tuning approaches 32
24 4
In this paper, we present a novel paradigm to video understanding in the compressed domain, namely Compressed Video Soompt-Tuning (CVPT). As in Figure (d), our approach freezes the whole pre- trained raw video-based model and learns a few modal-specific visual prompts. It allows us to leverage the common prior knowledge in the feature and semantic spaces shared by the two types of videos, to facilitate representation learning of compressed videos. Specifically, VPT employs conditional prompt framework with refinement, in which Motion Vectors and Residuals are re-pan parameterized into conditional prompts. These prompts are more adaptable to individual instances and alleviate the impact of freezing these modalities which can lead to performance degradation. Additionally, CVPT incorporates simple and lightweight modality-complementan prompt blocks, namely Selective Cross-modal Complementary Prompt (SCCP), into the frozen pre-trained model, capturing cross- modal interactions for performance improvement. Concretely, SCCP firstly embeds the I-frame

---

tokens as well as the conditional Motion Vector and Residual prompts into a low-dimensional feature space, in pursuit of computational efficiency. Based on attention maps induced by the Residual prompts, the Motion Vector prompts are subsequently refined via suppressing the noisy motions, which are further integrated into the I-frame tokens as complementary motion cues. consequence, the cross-modal interaction is significantly strengthened, remarkably facilitating model tuning for efficient representation learning of compressed videos.
The main contributions of this paper are summarized in four-fold:
We propose CVPT, a novel visual prompt tuning framework, which enables pre-trained raw video models to adapt compressed video understanding tasks. To the best of our knowledge, this is the first work to address this issue.
We introduce modal-related conditional prompts to replace conventional learnable ones, proving more adaptable individual instances and alleviates the inconsistency between pre-training and downstream data modalities.
We design selective rross-modal complementary prompt blocks, and these blocks substan- tially strengthen cross-modal interactions and selectively complement dynamic cues in Motion Vectors to input flows.
We conduct extensive experiments on three widely used benchmarks and achieve newly state-of-the-ard results. Notably, our approach maintains the parameter--fficiennyy with less than 1% parameters being trainable.
2 Related Work
2.1 Compressed Video Representation Learning
Early studies commonly adopt multi-stream CNNs and focus on dedicatedly designing the model structures for P/B-frames and fusion modules. The pioneering work by [49] introduces the use of CNNs process compressed videos. They replace optical flows in the two-stream network 14 with Motion Vectors, resulting in higher efficiency. CoViAR all the modalities, including I-frames, Motion Vectors, and Residuals, completely bypassing video decoding. To improve the quality of dynamics, (2 applies image denoising techniques, while [40] employs GANs to generate Motion Vectors that approximate optical flows. [22] and [ adopt similar approach by feeding compressed modalities into CNN model to mimic raw video- based teacher. [22 presents Temporal Trilinear Pooling for fusing multiple modalities in lightweight models suitable for portable devices. Inspired by SlowFast 022 30 propose the Slow-I-Fast-P model, where pseudo optical flows are estimated by a specific loss function. MM-ViT (5) is the first work using ViT on compressed videos, factorizing the self-attention across the space, time and modality dimensions.
45 extends third stream to leverage
The advent of self-supervised learning. particularly contrastive learning and Masked AutoEncoder (MAE), has led to the exploration of self-supervised pretext tasks for compressed videos. In this context, IMRNet [48] stands out as the first work delve into self-supervised learning for compressed videos. It employs a three-stream network to individually encode I-frames, Motion Vectors, and Residuals, and introduces two pretext tasks that make use of motion statisties and internal GOP (Group of Pictures) structures in the compressed domain. Another notable approach is MVCGC 21], which delivers a cross-guidee goos--guidance contrastive learning framework with multi-views. MVCGG utilizes Motion Vectors supervision signals for RGB frames and vice versa.
To sum up, the existing approaches in compressed video understanding involve the design of spe- cialized modules pretext tasks, which makes the pre-training and fine-tuning paradigm resource- consuming, not favorable enough to various downstream applications. In contrast, our work presents a prompt tuning based alternative, where necessary knowledge are mined from the off-the-shelf model pre-trained on raw videos through only few modal-specifie visual prompts, and more effective and efficient.
2.2 Visual Prompt Tuning
Prompt tuning has emerged as a promising way to efficient representation learning in the field of NLP. By leveraging textual prompts, researchers have successfully reformulated downstream tasks
3

---

Mwv
P
Ei
Transformer Encoder
SCCP (Selective Cross modal Complementary Prompt)
Classification
head
* FFN
T
M
Embed
Embed
Embed
Multi-Head Self-Atteniinn
SCCP
FTN
T2
Multi-Head Self-Artentin
M
Motion Vectors Residuals
I-frames
SEFEN
CPR (Conditional Prompts
Multi-Hleud Self-Attenton
Refinement)
---frame Token
*
Frozen/Tunable Purameters
M,
SCCP
Motion Vector Prompt
+ Token Addition
Pav PR Ey
Residual Prompt
Prompt Refinement
Figure 2: Framework of the proposed CVPT. It firstly embed the input compressed videos, consisting of I-frames, Motion Vectors and Residuals, into the I-frame tokens Ep, the conditional Motion Vector prompts Pay and the Residual prompts PR. respectively. A frozen pre-trained Transformer with layers is adopt for feature extraction. The Selective Cross-modal Complementary Prompt (SCCP) blocks are employed to perform conditional prompt learning as well as facilitate the interactions between the prompts and the I-frame tokens for efficient fine-tuning.
This approach offers several advantages full fine-tuning, such as the high parameter efficiency and the reduced storage cost. has also shown promising results various CV applications. For instance, VPT [23] introduces learnable parameters Transformer encoders and achieves superior performance compared to full fine-tuning on multiple recognition tasks. AdaptFormer [7] enhances ViT models with lightweight modules and surpasses the performance of fully fine-tuned models in action recognition benchmarks. Convpass 25 proposes convolutional bypasses for prompt learning in pre-trained ViT models. Some prompting attempts have also been made to deal with multi-modal tasks, such as visual and language. CoOp 53 fine-tunes CLIP using continuous set of prompt vectors, while Co-CoOp (22 improves its generalization by conditioning prompts image instances. (26 strengthens CLIP for video understanding tasks by adding lightweight Transformers visual features and text prompts. MaPLe (27 optimizes the prompting mechanism both the vision and language branches improve the adaptability of CLIP.
to resemble pre- vre-training tasks that have already been solved 28 33
Different from current prompt-learning approaches in NLP and CV that primarily apply in the that pre-training and downstream data share the same modality, our work further addresses the challenge arising from the inconsistency between pre-training and downstream data modalities, i.e. the pre-training data are dense RGB frames while the downstream data are sparse I-frames and coarse Motion Vectors and Residuals.
The Proposed Approach
In this section, describe the details of the proposed Compressed Video Prompt Tuning (CVPT) approach. Briefly, CVPT employs a conditional prompt framework with refinement as shown in Figure [ by proposing a Selective Cross-modaa Complementary Prompt (SCCP), which are depicted in the rest part.
3.1 Conditional Prompt with Refinement
commonly utilized to efficiently fine-tune model pre-trained on upstream tasks to align with the downstream tasks. Existing prompt tuning approaches often assume
Prompt tuning 28 33

---

that the pre-trained task and the downstream task share the same data modality. However, when fine-tuning for compressed video-based applications, the data modality differs as the downstream tasks adopt the compressed videos, and the pre-trained tasks utilize the raw videos. A straightforward fine-tuning without considering the modality gap often severely deteriorates the performance, since the Motion Vectors and Residuals are unique to compressed videos and have distinct data distributions from the raw videos. To address this issue, existing works employ an extra feature extraction network that may negatively abate the inter-modal interactions; or modify the Transformer structure to process the multi-modal compressed video data, which however still requires pre-training and full fine-tuning.
In this paper, we propose an efficient prompt tuning approach for compassed sudeo-basets tasks. Motivated by [54]. we re- pe-aammeterize the Motion Vectors and Residuals into prompts and enhance the cross-mas modall interactions with I-frames for prompt learning. Different from [54]. we embed the Motion Vectors and Residuals into distinet types of prompts, and refine the Motion Vector prompts by leveraging the Residual counterparts, which are further integrated into the I-frame Motion Vectors as strengthened complements of motion cues.
Concretely, the input I-frames XI E xX are firstly encoded into the I-frame tokens Ef R by frozen patch embedding layer, where H. W, represent the frame height, frame width and the encoding dimension, respectively. The Motion Vectors X R2x and the Residuals XR E x23IpxxHX are encoded by learnable patch embedding layers, generating the conditional Motion Vector prompts PO MV E R and the Residual prompts P R.xxd respectively. TVIp denote the number of framess...
The I-frame tokens EJ together with the conditional prompts XMV and XR comprchensively integrated by the proposed Selective denoted by MM1-), where the output are fed into the Transformer block as complementary motion cues to the I-frame tokens, as well as inputs to the SCCP block from the subsequent layer. The SCCP blocks are successively stacked by aligning with the original ViT layers. The overall forward propagation process for prompt tuning can be briefly formulated as follows:
Complementary Prompt (SCCP) at the first layer
[PAv PR. R: E 1 =- Ti [[
P E RR Ei + 1)
I 1,
MV
y Lea_ ([Pliv. Pk E(]
(2)
where L is the number of Transformer layers, T:(-) and CLS (c.__)Had()) refer to the Transformer block the 1-th layer and the classification head, respectively. Here, [ : p E represents the R'
output of the SCCP block the 1-th layer, which will be elaborated Section 3.2
3.2 Selective Cross-modal Complementary Prompt
Given the I-frame tokens EP. the conditional Motion Vector prompts ph the Residual prompts PR- and frozen pre-trained encoder GRGB containing L Trransformer layers, SCCP aims to complement the motion cues from the conditional prompts to the I-frame branch. Generally, SCCP aggregates the tokens and prompts from multiple modalities of the previous layer to generate the new ones at the current layer, which is formulated as below:
[P Pk- 1...EE-11 -]= MM (P E|-)) 1 1
(3)
is worth noting that the Motion Vectors, which are block-wise motion cues estimated by matching. tend to be coarse and noisy compared to the optical flow counterparts displayed in Figure Ja). As depicted in (39], motion information near object boundaries plays a crucial role in representation learning. Inspired by this insight, we generate attention map to select refined motion cues from the Motion Vectors by leveraging the Residuals, based on the observation that the Residuals are generally well-aligned with the boundaries of moving objects and strongly correlated with Motion Vectors as observed in [4)] The refined motion cues are successively adopted as a informative complements to the RGB modality, thus boosting learning discriminative representations.
The detailed design of SCCP is illustrated in Figure (b). Specifically, at the (-th layer, the input of SCCP consists of the I-frame tokens E the conditional prompts P and pk In order to obtain compact representations and reduce the overall size of parameters, we firstly embed the C-dimensional input into (C/a) Cfaa)dimeesionnll feature space as follows:
E
g1 (E
P (P P (o(Kkk))
(4)
5

---

P
E
x 7HW)
THW)
7HW)
I-frame
Residual
Selective Cross-modal Complementary Prompt
1x Conv
1x Conv
Conn
Softmax(w P)
Optical flow
Motion Vector with
Residual attention mask
PbV
x THW)
Pk [ THW)
E x THW)
C
IxIConv
IxIConv
Ix IConv
I-frame
Residual
PMV
P
Ej
Motion Vector Prompts
Residual Prompts
I-frame Tokens
Optical flow
Motion Vector with Residual atlention mask
(b)
Figure 3: (a) Illustration on the optical flows representing accurate motion cues, the coarse Motion Vectors and the Residuals. The red rectangles highlight the areas that Motion Vectors contain noises, which however can be refined by leveraging the Residuals. (b) Detailed architecture of the proposed Selective Cross-modal Complemenaary Prompt.
where is the reduction factor, the embedding functions gl (-), 92((),, and gs(-) are simple 1 x 1 convolutions. In the ViT-B backbone, the input feature dimension 768, and the reduction factor is set to 96 in all prompt blocks. As the scale of backbone grows, usually increases proportionally. In the case of Swin Transformer, as the feature dimension C' becomes larger in deeper layers, we gradually increase it to maintain the same embedding feature size across different layers.
After feature embedding, we generate a on the embedded Residual prompts P strengthen smoothness of the attention map, we introduce learnable parameter. The attention map is subsequently applied to select representative motion cues from the embedded Motion Vector prompts P the following:
attention map by performing Softmax(-)
p A O P Softmax
P MV
(5)
The refined prompts P1-1 is further integrated into the intermediate embedding E El-1 by addition, in MV order complement the motion cues below
= E
P1-1 MV
(6)
After performing the above cross-modal interactions, the embedded tokens É the embedded prompts p1-1 and p!-1 are finally project back to the high-dimensional ones by using the following My
R
formulation:
= 1-1=91(E]
p!-1 95 (P p!-1 = 96 (o(Al-1).
where (+()), 95( (-). and (-) 1 x convolutions.
order to comprehensively enhance the representation learning, the SCCP block is applied each Transformer block, and successively stacked, i.e. the conditional prompts pi-1 and p!-1 are updated in each SCCP block and is fed into the next SCCP block as input.

---

Experimental Results and Analysis
4.1 Datasets and Evaluation Metric
HMDB-51 and UCF-101 are two relatively small datasets, which contain 6766 videos from 51 action categories and 13,320 videos from 101 categories, respectively. Something-Someehhngg v2 (SSv2) is large-scale motion-centric video dataset, including 168,913 videos for training and 24,777 videos for validation from 174 categories. As typical video understanding task, we choose action recognition and report top-1 accuracy as the evaluation metric in all experiments.
4.2 Implementation Details
By following 45 29) we convert the raw videos into the MP4 compressed format with a GOP size of 12, where each GOP consists of I-frame and 11 P-frames. We collect I-frames, Motion Vectors and Residuals from 4 consecutive GOPs, totally containing 4 I-frames and 12 P- P-frames. Within each GOP, we uniformly sample P-frames. In regards of the backbone network, we utilize two representative video Transformer architectures based on ViT [TO] and Swin Transformer [34 pre-trained on Kinetics-400 with raw videos, respectively. We adopt the original model configurations and train the prompt parameters using the AdamW optimizer [35] on 12 NVIDIA V100 GPUs. We apply random scaling, corner cropping, and horizontal flipping as data augmentation. The base learning rate, weight decay and batch size are set to 10 1 x 10-4 and 240, respectively. Additionally, we adopt a warm-up strategy within the first training epochs.
4.3 Comparison with the State-of-the-art Approaches
Compressed video-based methods. We firstly compare our method with the following full fine- tuning compressed video-based models: 1) self-supervised pre-training based ones including CoViAR IMRNet 48 MVCGC [21] and the Full Fine-tuning baseline; and 2) supervised pre-training
45..
based ones including Refined-MV 22, IPTSN 20) SIFP-Net [30], MEACI-Net [29] and MM-VIT
55. T Table summarize the comparison results on the HMDB-51, UCF-101 and SSv2 datasets. Generally, the proposed CVPT method achieves comparable or even better performance than the full fine-tuning ones, by tuning less than 1% of the overall parameters. Notably, when adopting the same ViT-B backbone, CVPT improves the top-1 accuracy of MiT/Fll Fine-tuning by 2.1% %/0%0% on UCF-101, and 2.9% SSv2, via tuning only 0.6% of the parameters.
Besides the full fine-tuning methods, we also compare with representative efficient fine-tuning models, including the Linear Probe baseline, VPT (23 and AdaptFormer 77). Since VPT and AdaptFormer only report results on raw videos, we re-implement and evaluate them on compressed videos, denoted by VPT* and Adaptformer", respectively. As displayed, Linear Probe exhibits the lowest performance as it only optimizes the classification head. VPT* slightly outperforms Adaptformer". but both of their performance struggle to effectively align the upstream data for pre-training the downstream data for tne-tuning, In contrast, our approach leverages inserted prompt blocks to facilitate the interaction of multimodal information, boosting the accuracy VPT* and Adaptformer' by 10.8% and 12.0% on HMDB-51, 5.4% and 7.1 % UCF-101, and 4.8% and 5.6% SSv2, respectively, when pre-training with the sule-super learning.
clearly inferior to the fine-tuning counterparts. This suggests that these methods
In addition to the accuracy, approach also has advantages in its training efficiency and facilitation. Concretely, unlike IPTSN that adopts a complex knowledge distillation model based on a raw video-based teacher during fine-tuning, or MVCGC that mines cross-modality information from pre- trained RGB models through self-supervised learning, our method explores the efficient tuning and reaches promising accuracy. Moreover, shown in Table D and Table our approach consistently promotes the performance by different pre-training strategies and network architectures (e.g. ViT-L), highlighting its scalability.
Raw viseo-based methods. As summarized in Table I and Table 2 raw video-based fine-tuning methods generally reach higher performance than compressed video-basee counterparts, since the raw videos convey more coherent and consistent information. Nevertheless, CVPT significantly reduces this gap, and even surpasses both VPT and Adaptformer that are fine-tuned on raw videos. The reason lies in that CVPT explores the Motion Vectors and Residuals as prompts to complement the missing motion cues caused by sparse sampling of RGB frames in a more comprehensive way.
7

---

Table 1: Comparison with the sfate-of-the-ee approaches w.r.t the top-1 accuracy (%) HMDB-51 and UCF- 101 pre-trained on Kinetics-400 **' indicate the results by our re-implementation using the
same configuration. 'PT" stands for pre- pre-training. 'SL' and 'SSL' refer to supervised learning and
self-supervised learning, respectively.
Method
Modality
Backbone
Input [M]
GFLOPs
PT
Tunable Params. [
HMDB-31
LCF-101
CoCLR
17
RGB+OF
S3D
26.2
182.0
SSL
17.5 (100%)
62.9
90.6
RSPNet
RGB
S3D-G
91.0
SSL
9.6 (100%)
64.7
CVRL [38
RGB
R3D-50
48.2
402.0
SSL
46.9 (100%)
65.4
92.1
(BYOL [13
RGB
R3D-50
12.1
402.0
SSL
46,9 (100%)
73.6
95.5
Raw
STS
RGB
S3D-G
96.3
689.0
SSL
9.6 (100%)
62.0
89.0
Viiidoo
4
RGB
VIT-B
24.1
1080.0
SSL
86,4 (100%)
75.4
96.1
MVD-B
RGB
VIT-B
24.1
1080.0
SSL
86.4 (100%)
76.4
97.0
MotionMAE
47
RGB
VIT-B
24.1
1080.0
SSL
86.4 (100%)
VideoMAE
42
RGB
VIT-B
24.1
1080.0
SSL
86,4 (100%)
73.3
96.1
VPT
RGB
VīT-B
12.1
1089.6
SSL
(0.09%)
52.7
AdaptFormer
RGB
VIT-B
12.1
1093.8
SSL
(1.46%)
55.7
CoViAR 45
L-Prame+MV+Re:
ResNet152
3.8
1222.0
SSL
142.5 (100%)
37.1
63.7
IMRNet
I-Frame+MV+Res
R3D-18
SSL
100.8 (100%)
45.0
76.8
MVCGC 21
RGB-MV
S3D
15.7
SSL
17.5 (100%)
63.4
90.8
Refined-MV
I-Frume+MV-Re
ResNet152
SL
142.5 (100%)
59.7
IPTSN 20
I-Frume-MV-Re:
ResNet152
SL
130.8 (100%)
69.1
93.4
SIPP Nel
30
Prame+MV+Re
B3D
8.1
SL.
92.8 (100%)
72.3
94.0
Comp.
MEACI-Net
FrameeMM++Rs
0.7
269.4
SL
50.7 (100%)
74.4
96.4
MM-VIT 5
---Reee-V--Res
ViT-B
SL
86.4 (100%)
93.4
Full Fine tuning
1- Hrame+M++Re
VIT-B
6.1
772.2
SSL
86.4 (1005))
60L3
88.1
Linear Probe
VIT-B
6.1
772.2
SSL
(0.08%)
47.8
VPT 23
Frame+MV+Res
VIT-B
6.1
778.2
SSI.
(0.09%)
52.1
83.6
AdaptFormer"
7
Frame+MM++Rs
VIT-B
6.1
780.6
SSL
1.3 (1.46%)
50,9
81.9
CVPT (Ours)
Frume+MV+Kes
VIT-B
6.1
772.2
SSL
0.5 (0.6%)
62.9
CVPT (Ours)
LVree
VIT-B
6.1
772.2
SL.
0.5 (0.646)
69.7
95.5
CVPT (Ours)
FrameeMM++Res
VIT-L
6.1
2569.6
SL
0.1 (0.03%)
81.5
98.0
Table Comparison with the state-of-the-ar approaches w.r.t. the top-1 accuracy (%) on SSv2 pre- trained on Kinetics-400. 'x' indicate the results by our re-implementation using the same configuration. 'PT' stands for pre-training. 'SL' and 'SSL' refer supervised learning and self-supervised learning.
respectively.
Method
Modality RGB
Backbone
Input [M]
GFLOPs
PT
Tunable Params, [M]
SSv2
RSPNet
6
S3D-G
6.0
91.0
SSL
9.6 (100%)
55.0
eBYOL
13
RGB
R3D-50
12.1
402.0
SSL
46.9 (100%)
55.8
MVD-B
ViT-B
24.1
1080.0
86.4 (100%)
Raw
Vidioo
VIT-B
24.1
1080.0
SSL
86.4 (100%)
70.5
MotionMAE
47
RGB
VYT-B
24.1
1080.0
SSL
86.4 (100%)
71.8
VidcoMAE
42
RGB
VIT-B
24.1
1080.0
SSL
86.4 (100%)
70.3
VPT
RGB
VIT-B
12.1
1089.6
SSL
0.1 (0.09%%
43.7
AdaptFormer
7
RGB
VIT-B
12.1
1093.8
SSL
1.3 (1.46%)
59.0
MM-VIT [S]
I-Framc+MV+Res
VIT-B
8.1
SL
(100%)
62.6
Full Fine-tuning
L-Frmc-MV+Res
VIT-B
6.1
772.2
SSL
86.4 (100%)
57.6
Comp.
Lineur Frobe
Frumc+MV+Roo
VIT-B
6.1
772.2
SSL
0.1 (0.08%)
33.8
VPT" 23
Framc+MV+RRs
VIT-B
6.1
778.2
SSL
0.1 (0.09%)
53.6
AdaptFormer'
I-Frme+MV MV+ Res
VIT-B
6.1
780.6
SSL
1.3 (1.46%)
52.8
CVPT (Ours)
Frame+
VIT-B
6.1
772.2
SSL
(0.6%)
38.4
CVPT (Ours)
Framc-MV+Res
VIT-L
6.1
2569.6
SL
0.1 (0.03%)
65.5
Additionally, CVPT is capable of directly fine-tuning an off-the-shelf model pre-trained on raw- videos to the domain of compressed videos, thus simultancously saving extra computational cost in alternatively pre-training on the specific compressed data.
4.4 Ablation Study
To evaluate the effectiveness of components in CVPT, we conduct extensive ablation study, Without loss of generality, we utilize ViT-B pre-trained by self-supervised learning as the backbone.
On the Main Components. We firstly evaluate the effect of the main components of the proposed method. Specifically, we employ linear probe as the baseline method. As shown Table 3 the proposed CPR module without refinement, which only updates the conditional prompts in the first layer, clearly improve the baseline. By introducing the SCCP module, which enriches the cross- modal information between I-frames and conditional prompts of Motion Vectors and Residuals, the

---

Table 3: Ablation results (%) on the main components of CVPT.
CPR w/o refinement
SCCP
CPR
HMDB-51
UCF-101
47.8
76.8
55.7
83,9
61.4
87.6
62,9
89.0
E
pi-d MVV
E
MV
R
550
#HKDB-S1
LCCF-101
Conv
Conv
ConviD
ConviD
Convi
pi-1 MV&R
p1-1
P!-1
RR
ConvID
ConvID
Conv1D
ConvID
ConvID
1-5 The number the SCCP blocks
1-12
P (a) SCCP w/o Px prompt
E (b) SCCP attention
MV
R
Figure 4: The influence of the number of SCCP blocks the proposed method.
Figure 5: Illustrations of alternative structures of SCCP. (a) and (b) indicate SCCP without Pr. prompt and without attentions, respectively.
performance is remarkably boosted. Finally, applying the refinement module on CPR further obtains gain in performance, leading to the highest accuracy.
On the Selective Cross-modal Complementary Prompt (SCCP). To demonstrate the effect of the proposed SCCP module, we compare to the following two alternative architecture designs as shown in Figure [ 1) removing the PR prompt and separately embedding the Motion Vectors and Residuals into distinct two types of prompts as displayed in Figure 2) abandoning the Residual- guided attention and directly adding the branch of Motion Vectors displayed in Figure As summarized in Table both the two above alternative designs yield worse performance than the proposed SCCP module, thus demonstrating the effectiveness of employing the Residual prompt to facilitate extracting more informative dynamic cues for the downstream task.
Besides, investigate the impact of incorporating different numbers of SCCP blocks into the backbone. The insertion process begins from the first layer and increments by three layers in each iteration. As illustrated in Figure 4 we observe that the overall performance both datasets is consistently improved as the number of inserted SCCP blocks increases.
On Distinct Backbones. We evaluate the performance of our method by using different backbones, compared to the Full Fine-tuning baseline. As summarized Table 5 our approach steadily outperforms Full Fine-tuning for various backbones by distinct pre-training strategies.
4.5 On the Efficiency of CVPT
We present comprehensive comparison of inference time between the proposed CVPT method and the representative raw video-based approach VideoMAE. As shown in Table our approach
Table Ablation study (%) on different structures of the proposed SCCP module.
Method
Total Params. [K]
HMDB-51
JCF-101
SCCP w/o atlention
39.2
59.6
86.9
SCCP w/o Ps prompt
19.2
57.6
85.5
SCCP
39.2
61.4
87.6

---

Table 5: Ablation study (%) on different backbones.
ViT-B (SSL)
VTT-B (SL)
Swin-B (SL)
Backbone
HMDB-51
UCF-101
HMDB-51
UCF-101 101
HMDB-51
UCF- 101
Full Fine tuning
60.3
88.1
66.5
94.3
62.9
92.0
Ours
62.9 (+2.6)
(40.9)
69.7 (43.2)
95.5 (+1.2)
64.9 (+2.0)
94.0 (+2.0)
Table 6: Comparison of inference time (ms) per video.
Method
Pre-Process
Model Inference
Full Pipeline
VideoMAE [42]
2496.9
26,9
2523.8
CVPT (Ours)
238.3
25.0
263.3
BabyCrawling
Linear Probe
Adaptformer"
BaseballPicch
Linear Probe
Adaptffonner
VPT"
Full F'ne--tuiigg
CVPT
VPT"
Full Fine- tuning
CVPT
Figure 6: Visualization of the last layer feature map generated by various approaches on UCF-101.
significantly reduces the pre- pre-proeessing time without the need for video decoding, while maintaining a comparable model inference time, thus remarkably decreasing the overall time cost.
4.6 Qualitative Results
We visualize the classification maps generated by CVPT, compared to several staeeoff-hhe-a com- pressed video-based approaches including Linear Probe, Adaptformer*, VPT* and Full Fine-tuning. As shown in Figure 6, our method exhibits a broader focus on motion regions than those compared methods.
5 Conclusion and Discussion
We propose novel prompt tuning method, namely CVPT, for compressed videos. Our approach leverages Motion Vectors and Residuals to generate conditional prompts. Furthermore, we employ the SCCP blocks to fully explore the discriminative dynamic cues Motion Vectors complements to I-frames. Extensive experimental results on public benchmarks demonstrate the efficiency and effectiveness of CVPT, compared the current state-of-the-arr approaches for efficient fine-tuning.
Despite the promising performance, the scalability of the proposed CVPT method to various vision tasks such compressed video-based segmentation and tracking has not been investigated. We will further study this problem in our future research.
Acknowledgment
This work partly supported by the National Key R&D Program of China (2021ZD01 10503), National Natural Science Foundation of China (62022011 I and 62202034), the Research Program of State Key Laboratory of Software Development Environment, and the Fundamental Research Funds for the Central Universities.
10

---

References
[1] Barak Battash, Haim Barad, Hanlin Tang, and Amit Bleiweiss. Mimic the raw domain: Accelerating action recognition in the compressed domain. In CVPR Workshop, 2020.
[2] Haoyuan Cao, Shining Yu, and Jiashi Feng. Compressed video action recognition with refined motion vector, arXiv preprint arXiv: 3XXiv:9900005333 2019.
[3] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics datasct. In CVPR, 2017.
[4] Acchuan Chen, Yuguang Yao, Pin-Yu Chen, Yihua Zhang, and Sijia Liu. Understanding and improving visual prompting: A label-mapping perspective. In CVPR, 2023.
[5] Jiawei Chen and Chiu Man Ho. Mm-vit: Multi-modal video transformer for compressed video action recognition. In WACV, 2022.
[6] Peihao Chen, Deng Huang, Dongliang He, Xiang Long, Runhao Zeng, Shilei Wen, Mingkui Tan, and Chuang Gan. Rspnet: Relative speed pereeption for unsupervised video representation learning. In AAAI, 2021.
[7] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yihing Song, Jue Wang, and Ping Luo. Adapt- former: Adapting vision transformers for scalable visual recognition. In NeurlPS, 2022.
[8] Weidong Chen, Dexiang Hong. Yuankai Qi. Zhenjun Han, Shuhui Wang, Laiyun Qing. Qingming Huang. and Guorong Li. Multi-attention network for compressed video referring object segmentation. In ACM MM, 2022.
[9] Rajshekhar Das, Yonatan Dukler, Avinash Ravichandran, and Ashwin Swaminathan. Learning expressive prompting with residuals for vision transformers. In CVPR, 2023.
[10] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint arXiv:2010. /929, 2020.
[11] Xiang Fang, Daizong Liu, Zhou, and Guoshun Nan. You can ground earlier than see: effective and efficient pipeline for temporal sentence grounding compressed videos. CVPR, 2023.
[12] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for vidco recognition. In ICCV, 2019.
[13] Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. large-scale study on unsupervised spatiotemporal representation learning. CVPR, 2021.
[14] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In CVPR, 2016.
[15] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In ECCV, 2020.
[16] Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. Ppt: Pre-trained prompt tuning for few-shot learning. In ACI, 2022.
[17] Tengda Han, Weidi Xie, and Andrew Zisserman. Self seff-supervieed co-training for video representation learning. In NeurlPS, 2020.
[18] Ronghang Hu and Amanprect Singh. Unit: Multimodal multitask learning with unified transformer. In ICCV, 2021.
[19] Yubin Hu, Yuze He, Yanghao Li, Jisheng Li, Yuxing Han, Jiangtao Wen, and Yong-Jin Liu. Efficient semantic segmentation by altering resolutions for compressed videos. In CVPR, 2023.
[20] Shiyuan Huang, Xudong Lin, Svebor Karaman, and Shih-Fu Chang. Flow-distilled ip two-stream networks for compressed video action recognition. arXiv preprint 19144499,, 2019.
[21] Yugi Huo, Mingyu Ding, Haoyu Lu, Nanyi Fei, Zhiwu Lu, Ji-Rong Wen, and Ping Luo. Compressed video contrastive learning. In NeuriPS, 2021.
[22] Yuqi Huo, Xiaoli Xu, Yao Lu, Yulei Niu, Zhiwu Lu, and Ji- Ji-Rong Wen. Mobile video action recognition. arXiv preprint 1908 10155, 2019.

---

[23] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan. and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
[24] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt uning. In ECCV, 2022.
[25] Shibo Jie and Zhi-Hong Deng. Convolutional bypasses are better vision transformer adapters. arXiv preprint Xiix:2207007033 2022.
[26] Chen Ju, Tengda Han, Kunhao Zheng. Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. In ECCV, 2022.
[27] Muhammad Uzair Khattak, Hanonna Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, 2023.
[28] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP, 2021.
[29] Bing Li, Jiaxin Chen, Dongming Zhang. Xiuguo Bao, and Di Huang. Representation learning for compressed video action recognition via attentive cross-modal interaction with motion enhancement. In IJCAI, 2022.
[30] Jiapeng Li, Ping Wei, Yongchi Zhang, and Nanning Zheng. A slow-i-fast-p architecture for compressed vidco action recognition. In ACM MM, 2020.
[31] Muchen Li and Leonid Sigal. Referring transformer: A one-step approach to multi-task visual grounding. In NeurlPS, 2021.
[32] Xiang Lisa Li and Percy Liang. Prefix-tuning: optimizing continuous prompts for generation. In ACL&IJCNLP, 2021.
[33] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveyx, 55(9). 2023.
[34] Ze Liu. Jia Ning, Yue Cao. Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022.
[35] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.0510,,2000 2017.
[36] Wenyang Luo, Yufan Liu, Bing Li, Weiming Hu, Yanan Miao, and Yangxi Li. Long-short term cross- transformer in compressed domain for few-shot video classification. arXiv preprint arXi::205.003569 2022.
[37] Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In CVPR, 2021.
[38] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang. Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In CVPR, 2021.
[39] Laura Sevilla-Lara, Yiyi Lia,, Fatma Güney, Varun Jampani, Andreas Geiger, and Michael J Black. On the integration of optical flow and action recognition. In GCPR, 2019.
[40] Zheng Shou, Xudong Lin, Yannis Kalantidis, Laura Sevilla-Lara, Marcus Rohrbach, Shih-Fu Chang. and Zhicheng Yan. Dmc-net: Generating discriminative motion cues for fast compressed vidco action recognition. In CVPR, 2019.
[41] Xinyu Sun, Peihao Chen, Liangwei Chen, Thomas Li, Mingkui Tan, and Chuang Gan. M³ video: Masked motion modeling for self-supervised video representation learning. arXiv preprint arXiv:2210.060%. 2022.
[42] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomac: Masked autoencoders are data-efficient learners for self-supervised video pre- trainngg arXiv preprint arXiv:2203. 22002, 2022.
[43] Jiangliu Wang. Jianbo Jiao, Linchao Bao, Shengfeng He, Wei Liu, and Yun- Liu. Self-super video representation learning by uncovering
statisties. IEEE TPAMI, 44:3791 -886, 2021.
12

---

[44] Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, and Yu-Gang Jiang. Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning. CVPR, 2023.
[45] Chao- Yuan Wu, Manzil Zaheer, Hexiang Hu, R. Manmatha, Alexander J. Smola, and Philipp Krähenbühh. Compressed video action recognition. In CVPR, 2018.
[46] Kai Xu and Angela Yao. Accelerating video object segmentation with compressed video. In CVPR, 2022.
[47] Haosen Yang, Deng Huang, Bin Wen, Jiannan Wu, Hongxun Yao, Yi Jiang, Xiatian Zhu, and Zehuan Yuan. Self-supervised video representation learning with motion-aware masked autoencoders. arXiv preprint urXiv:2210.04154. 2022.
[48] Youngjae Yu, Sangho Lee, Gunhee Kim, and Yale Song. Self-supervised learning of compressed video representations. ICLR, 2020.
[49] Bowen Zhang. Limin Wang, Zhe Wang, Yu Qiao, and Hanli Wang. Real-time action recognition with enhanced motion vector cnns, CVPR, 2016.
[50] Yanan Zhang, Jiaxin Chen, and Di Huang. Cat-det: Contrastively augmented transformer for multi-modal object detection. In CVPR, 2022.
[51] Yue Zhao and Philipp Krähenbühh.. Real-time online video detection with temporal smoothing transformers. arXiv preprint arXiv:2209.092336 2022.
[52] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for [53] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning prompt for vision-language models. I/C,, 130(9)23337-2388, 2022.
vision-language models. CVPR, 2022.
[54] Jiawen Zhu, Simiao Lai, Xin Chen, Dong Wang, and Huchuan Lu. Visual prompt multi-moda tracking. In CVPR, 2023.
13


Reviews:
[{"review_id": "YHPh74aJoN", "reviewer": "Reviewer_XoEQ", "summary": "This paper explores how to transfer pretrained RGB models to compressed videos with a parameter-efficient paradigm and introduces a prompt-tuning method named Compressed Video Prompt Tuning (CVPT). In CVPT, the learnable prompts are replaced with encoded compressed modalities that are refined in each layer. To improve cross-modal interactions between prompts and RGB input flow, this paper proposes a Selective Cross-modal Complementary Prompter block that refines the motion cues and complements other modalities to the RGB modality. Experimental results show that CVPT outperforms full fine-finetuning and other prompt-tuning methods on SSv2, UCF101 and HMDB51 dataset.", "questions": "1. The comparison of GLOPs should be provided in Table 1 & Table 2.\n2. It would be better to provide the inference time comparison between RGB models and the proposed method.", "limitations": "The authors have addressed the limitations of this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed prompt-tuning method designed for compressed videos can leverage the pretrained RGB models with a few\ntrainable parameters.\n2. On compressed videos, CVPT outperforms full fine-finetuning and other prompt-tuning methods (VPT, AdaptFormer) on SSv2, UCF101 and HMDB51 datasets.", "weaknesses": "1. The computational cost (GLOPs) of the proposed method may be higher than that of some previous works based on ViT. According to (1), the tokens of I-frames, motion vector prompts and residual prompts are all sent to each layer of pretrained ViT, so the number of input tokens may be larger than that of some previous ViT-based models. \n2. This paper says that compressed videos \"provide notable advantages in terms of processing efficiency\", but there is no efficiency comparison with previous works based on raw videos. It would be better to provide the inference time comparison between previous works (especially RGB models) and the proposed method.\n3. Some state-of-the-art methods[2,3] on the UCF101, HMDB51, and SSv2 dataset are not compared in this paper.\n4. The idea of this paper is mainly from [1], as mentioned in this paper, but the exploration of compressed video understanding is encouraged.\n\n- [1] Jiawen Zhu, Simiao Lai, Xin Chen, Dong Wang, and Huchuan Lu. Visual prompt multi-modal tracking. CVPR, 2023.\n- [2] Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. CVPR 2021.\n- [3] Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, and Yu-Gang Jiang. Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning. CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1&Q1:The computational cost (GLOPs) of the proposed method may be higher than that of some previous works based on ViT. According to (1), the tokens of I-frames, motion vector prompts and residual prompts are all sent to each layer of pre-trained ViT, so the number of input tokens may be larger than that of some previous ViT-based models.  & The comparison of GLOPs should be provided in Table 1 & Table 2.\n\nA1:In our proposed method, while other modalities are employed, each modality is sparsely sampled. In contrast to previous raw video ViT-based approaches (*e.g.* VideoMAE, MotionMAE, MVD), our method yields a reduction in the number of input tokens. Typically, these approaches sample either 16 or 32 frames from a 64 frames sequence with intervals of 4 or 2. In contrast, we sample only 4 I-frames from 4 continuous GOPs, spanning a  48 frames sequence.  In parallel, we selectively extract 3 P-frames (comprising motion vectors and residuals) from each GOP, which serve as the input to our model.  Furthermore, Our method also has reduced tokens compared to previously compressed video ViT-based methods (*e.g.* MM-ViT) which sample the same number of frames to form an input size equivalent to the raw video ViT-based methods. \nThank you for your suggestion. Due to time constraints, we currently present the GFLOPs of ViT-based approach below. A notable observation emerges that our method has lower GFLOPs compared to other methods due to the reduction in inputs. We intend to provide a comprehensive report of GFLOPs for all methods later. \n\n|Type of Method| Method |GFLOPs|\n| :-: |:-:|:-:|\n|Raw|$M^3$Video|1080.0|\n|Raw|MotionMAE|1080.0|\n|Raw|VideoMAE|1080.0|\n|Raw|MVD-B|1080.0|\n|Raw|VPT|1089.6|\n|Raw|Adaptformer|1093.8|\n|Compressed|CoViAR|1222.0|\n|Compressed|Full Fine-tune|772.2|\n|Compressed|Linear Probe|772.2|\n|Compressed|VPT$^*$|778.2|\n|Compressed|AdaptFormer$^*$|780.6|\n|Compressed|Ours|772.2|\n\nW2&Q2:This paper says that compressed videos \"provide notable advantages in terms of processing efficiency\", but there is no efficiency comparison with previous works based on raw videos. It would be better to provide the inference time comparison between previous works (especially RGB models) and the proposed method. & It would be better to provide the inference time comparison between RGB models and the proposed method.\n\nA2:Thanks for your suggestion. The efficiency inherent in compressed video analysis manifests through its direct evaluative capacity, obviating the need for video decoding. We present a comprehensive comparison of inference times between our proposed compressed video method and raw video method (*e.g.* VideoMAE) below. Remarkably, our approach demonstrates a significant reduction in pre-processing while maintaining a comparable inference time for model inference.\n\n|Method|Pre-Process (ms)|Model Inference (ms)|Full Pipeline (ms)|\n| :-: | :-: | :-: | :-: |\n|VideoMAE|2496.9|26.9|2523.8|\n|CVPT (Ours)|238.3|25.0|263.3|\n\nW3:Some state-of-the-art methods[2,3] on the UCF101, HMDB51, and SSv2 datasets are not compared in this paper.\t\n\nA3:We deeply value your suggestion. Both of the two methods are raw video-based methods which leverage a more consistent and comprehensive RGB modality information. Consequently, these approaches exhibit high performance in comparison to our method. Nonetheless, it is noteworthy that their reliance on video decoding, followed by subsequent analysis makes them relatively less efficient in terms of inference time compared to our approach. Furthermore, it is pertinent to highlight that both of these methods necessitate full fine-tuning, whereas our method focuses on tuning only a notably smaller subset of parameters. Finally, we will supplement the comparison with the aforementioned methods by incorporating them into both Table 1 and Table 2.\n|Method|Modality|Model|Input Size [M]|PT Manner|Tunable Params. [M]|HMDB-51|UCF-101|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|MVD-B|RGB|ViT-B|24.1|SSL|86.4 (100%)|76.4| 97.0|\n|$\\rho$BYOL|RGB|R3D-50|12.1|SSL|46.9 (100%)|73.6|95.5|\n|Ours|I-Frame+MV+Res|ViT-B|6.1|SSL|0.5 (0.6%)|62.9|89.0|\n\n|Method|Modality|Model|Input Size [M]|PT Manner|Tunable Params. [M]|SSv2|\n|-|-|-|-|-|-|-|\n|MVD-B|RGB|ViT-B|24.1|SSL|86.4 (100%)|72.5|\n|$\\rho$BYOL|RGB|R3D-50|12.1|SSL|46.9 (100%)|55.8|\n|Ours|I-Frame+MV+Res|ViT-B|6.1|SSL|0.5 (0.6%)|58.4|\n\nW4:The idea of this paper is mainly from [1], as mentioned in this paper, but the exploration of compressed video understanding is encouraged.\n\nA4:Indeed, our method draws inspiration from [1]. Nevertheless, when dealing with the prompt tuning challenge posed by compressed video, a unique complexity arises. **Firstly**,  each individual modality within compressed video is incomplete and exhibits significant differences among the modalities. Thus, we adopt distinct prompts for encoding motion vectors and residuals within the compressed video. This departure from the unified prompt approach in [1] enhances the efficiency of the entire tuning process. We adopt a similar architecture, as illustrated in Figure 5(a) for comparison. In Table 4, we present a comparative analysis, substantiating how our distinct prompt strategy enhances the efficacy of prompt tuning.  **Secondly**, motion cues in P-frames are coarse and noisy. Leveraging the inter-modality correlation, we introduce a novel operation that purifies motion cues within the SCCP module. This operation facilitates the integration of more accurate motion information into the inputs, consequently bolstering the performance. In contrast, [1] benefits from the utilization of comprehensive and accurate diverse modalities, thereby eliminating the necessity for such refinement.\n"}, {"review_id": "FEqyCvStUE", "reviewer": "Reviewer_5c9t", "summary": "This paper proposes an efficient fine-tuning method based on the prompting concept in the compressed video domain. The intuition is to freeze the backbone pre-trained on raw videos and use the proposed prompting techniques to query the required information for the compressed videos. To address the multi-modality of the compressed videos (RGB images, motion vectors, and residuals), the authors propose embedding them first and using a SCCP module to fuse and refine them. The SCCP module is designed based on the fact that the video tasks are more important and sensitive to the motion boundaries. Therefore, the residual map acts as a condition to attend the motion vector map. The results are then added back to the RGB image tokens. The outputs of SCCP are then passed to the next pre-trained layer. Superior performance is obtained and the gap between raw video is shortened.", "questions": "L3 should be: existing methods for compressed video classification/application … ?", "limitations": "Limitations are addressed.", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- The idea is simple and easy to follow.\n- The efficient design is important to the video tasks as the backbone can be frozen. The overall learnable parameter is significantly small compared to the backbone. Therefore, for different downstream tasks, the storing problem can be alleviated.\n- The idea of using the residual map to attend to the motion vectors sounds novel and interesting.", "weaknesses": "* Some operations in the proposed method are confusing:\n\t- In Eq. 6, what is the physical meaning of adding an attended motion vector to image embeddings/tokens? The motion vectors represent the relative movement information for each of the spatial blocks (movement in x and y directions in the form of vectors), while the RGB embedding contains spatial information/structure. The addition operation does not really make sense.\n\t- The processed \\mathcal{M}_l(E)_I^l) is added to E_I^l again, what is the intuition behind this operation? The overall sequence is: the motion vector is attended by the residual information and then added back to the raw image. The output is further processed and added back to the raw image. How to physically explain this?\n* The overall architecture is very similar to [51]. What is the fundamental difference?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1:Some operations in the proposed method are confusing:\n\nIn Eq. 6, what is the physical meaning of adding an attended motion vector to image embeddings/tokens? The motion vectors represent the relative movement information for each of the spatial blocks (movement in x and y directions in the form of vectors), while the RGB embedding contains spatial information/structure. The addition operation does not really make sense.\n\n\n\nA1:The I-frame tokens are embedded via sparsely sampled I-frame sequence which encompasses both spatial and motion information. However, the intrinsic motion details within these tokens are relatively subdued. To address this limitation, we enhance the I-frame features by incorporating motion vector attributes. This augmentation effectively enriches the motion-related aspects within the I-frame feature representation. Notably, this strategy is commonly employed in both the compressed video domain [42, 45, 28], where RGB and motion information are fused. Although we also considered concatenation as an alternative to addition, we found that it yielded similar performance while incurring higher computational complexity. In raw video domain, where RGB and optical flow information are merged similarly in [52, 53, 54].\n\n[52]. Simonyan K, Zisserman A. Two-stream convolutional networks for action recognition in videos. In NeurIPS, 2014.\n\n[53]. Wang Y, Long M, Wang J, et al. Spatiotemporal pyramid network for video action recognition. In CVPR, 2017.\n\n[54]. Xie D, Deng C, Wang H, et al. Semantic adversarial network with multi-scale pyramid attention for video classification.In AAAI,  2019.\n\n\n\nW2:The processed $\\mathcal{M}_l(E)_I^l)$ is added to $E_I^l$ again, what is the intuition behind this operation? The overall sequence is: the motion vector is attended by the residual information and then added back to the raw image. The output is further processed and added back to the raw image. How to physically explain this?\n\n\n\nA2:This operation aimed at incorporating prompts into input tokens, thus facilitating the efficient tuning of the backbone for adaptation to downstream tasks. Within the SCCP module, its objective is to enhance the I-frames feature with essential motion cues. After the SCCP processing, the updated conditional prompts are propagated to the subsequent SCCP layer. Simultaneously, augmented features are assimilated into the input token through an additive mechanism which is a routine operation of the prompt framework.\n\n\n\nW3:The overall architecture is very similar to [51]. What is the fundamental difference?\n\n\n\nA3:Indeed, our method draws inspiration from [51]. Nevertheless, when dealing with the prompt tuning challenge posed by compressed video, a unique complexity arises.  **Firstly**,  each individual modality within compressed video is incomplete and exhibits significant differences among the modalities. Thus, we adopt distinct prompts for encoding motion vectors and residuals within the compressed video. This departure from the unified prompt approach in [1] enhances the efficiency of the entire tuning process. We adopt a similar architecture, as illustrated in Figure 5(a) for comparison. In Table 4, we present a comparative analysis, substantiating how our distinct prompt strategy enhances the efficacy of prompt tuning.  **Secondly**, motion cues in P-frames are coarse and noisy. Leveraging the inter-modality correlation, we introduce a novel operation that purifies motion cues within the SCCP module. This operation facilitates the integration of more accurate motion information into the inputs, consequently bolstering the performance. In contrast, [1] benefits from the utilization of comprehensive and accurate diverse modalities, thereby eliminating the necessity for such refinement.\n\n\n\nQ1:L3 should be: existing methods for compressed video classification/application … ?\n\n\n\nA1:Thank you for your suggestion. We will make modifications to enhance precision and clarity within the text."}, {"review_id": "fm2TLw6Fbd", "reviewer": "Reviewer_Eouy", "summary": "The paper presents one alternative way of finetuning to work on compressed videos. Specifically, it designs a specific data flow within three modalities (RGB, residual, and motion vector). It also presents the way to make the model adapt to new compressed videos and provide a fair comparison. It demonstrates SOTA performance to understand the proposed setting.", "questions": "Can you also compare the training time & training epochs when you compare with finetune, linear probing, etc?\n\nCan you also compare the data scale between the data used to train the frozen model and the data used for your method training?\n\nIf we change the order of motion vector and residual, will the results be better than Fig5b?\n(motion vector gating residual and gating I-frame.)", "limitations": "NA", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The problem this paper studies is pretty interesting and useful. Their method design is also complete and thinks about the alternatives.\nThe presentation is pretty clear while the results are SOTA under their setting. \n", "weaknesses": "The motivation of residual gating motion vector and gating I-Frame information is still weak to me.\nIt would be better if the author can provide more evidence (exps and visualization).\n\nThe presentation of CPR is a little weak to me and needs to be a little more clear.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: The motivation of residual gating motion vector and gating I-Frame information is still weak to me. It would be better if the author can provide more evidence (exps and visualization).\n\nA1:In compressed video, motion vectors capture motion displacement between preceding and subsequent frames via block matching, while residuals represent the remaining error after motion estimation.  Earlier studies [38, 43] have utilized residuals to help rectify erroneous motion information within motion vectors，owing to the coarse and noisy nature of the latter. In this context, we propose the utilization of residuals to generate attention patterns, serving to refine the motion information. We extend the visualization through Figure A, which provides additional insight into the utilization of motion vectors to generate attention for residuals and utilization of residuals to generate attention for I-frames. The experimental results of these strategies are compared below. The generation of attention maps using motion vectors does not exhibit a notable filtering effect when observed through visualization. Consequently, the performance is notably akin to that w/o residual gating.\n\n|            Method             | Acc. |\n| :---------------------------: | :--: |\n|      w/o residual gating      | 86.9 |\n| Motion vector gating residual | 87.0 |\n|             Ours              | 87.6 |\n\n\n\nW2:The presentation of CPR is a little weak to me and needs to be a little more clear.\n\n\n\nA2:Thanks for your suggestion. Every modality within the compressed video exhibits incompleteness and significant inter-modality variations. Simultaneously, the presence of I-frames and the other two modalities is mutually exclusive. Thus, we consider employ distinct embedding functions to encode motion vectors and residuals into separate conditional prompts. Given the relatively longer temporal associated with motion vectors and residuals, an extended temporal tube is utilized to ensure uniform token number across all three modalities. Due to its primary emphasis on edge information within the motion [37], the residual plays a crucial role in enhancing recognition tasks within this specific region. Thus, our approach employs residuals to generate an attention map for filtering motion vectors. \n\n\n\nQ1:Can you also compare the training time & training epochs when you compare with finetune, linear probing, etc?\n\n\n\nA1:we present a comparative analysis of the training time for three distinct methods when conducting 100 epochs on the UCF-101 dataset. Our approach achieves a reduction of over 50% in training time compared to full fine-tuning. Additionally, the training time aligns comparably with the linear probe configuration.\n\n\n\n|     Method     | Training Time | Training Epoch |\n| :------------: | :-----------: | :------------: |\n| full fine-tune |      10h      |      100       |\n|  linear probe  |     4.2h      |      100       |\n|      CVPT      |     4.8h      |      100       |\n\n\n\nQ2:Can you also compare the data scale between the data used to train the frozen model and the data used for your method training?\n\n\n\nA2:The frozen model is pre-trained using the large dataset (Kinetics-400), employing raw videos in both supervised and self-supervised training manners. In contrast, we employed UCF-101, HMDB-51 and SSv2 datasets as distinct training data, respectively.\n\n| Data used for Frozen Model | Data used for our Method |\n| :------------------------: | :----------------------: |\n| Kinetics-400 (240k videos) |  UCF-101 (9.6k videos)   |\n| Kinetics-400 (240k videos) |  HMDB-51 (6.8k videos)   |\n| Kinetics-400 (240k videos) |    SSv2 (169k videos)    |\n\n\n\nQ3:If we change the order of motion vector and residual, will the results be better than Fig5b? (motion vector gating residual and gating I-frame.)\n\n\n\nA3:We present the experimental results of gating residuals with motion vectors in W1 &  A1, yielding a marginal increase of 0.1% compared to the result depicted in Figure 5(b). This finding suggests that the attention generated with motion vectors may not effectively perform the desired filtering, both in terms of visualization and experimental results. "}, {"review_id": "0axOuIAroj", "reviewer": "Reviewer_sECu", "summary": "The authors present a way to adapt pre-trained raw video models to compressed videos. They utilized the existing concept of prompt tuning from NLP and repurposed it for the compressed video domain. Their findings indicate that by fine-tuning just 0.1 percent of parameters for a downstream task such as video classification, the pre-trained model can be modified to cater to compressed videos.", "questions": "* Was a LoRA update option considered instead of prompt fine-tuning? If so, can the details of such experiments be mentioned in the supp. material", "limitations": ".", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* As per my knowledge, this is the first study to explore the prompt tuning method for the compressed video domain.\n* Enough experimental results are provided to back the claims made in paper\n* Paper is well written", "weaknesses": "* Only video classification is shown as a downstream task.\n* Why are the numbers reported in the submission differ from the numbers reported in the original paper(for eg: CoViAR) is there differences in the experimental setting?\n* Add Bold text in the tables to highlight the best-performing methods as it is really hard for the readers to sift through the tables in their current condition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1:Only video classification is shown as a downstream task.\n\n\n\nA1:Our designed modules are tailored to address a wide spectrum of tasks pertaining to compressed video. We follow [6] and provide experiments on video classification as a downstream task. Nonetheless, it is necessary in extending our validation to encompass additional tasks. We employed a self-supervised pre-trained ViT-B and subsequently conducted full fine-tuning or prompt tuning on the UCF-101 dataset for video retrieval task. In this context, we extracted the averaged tokens from the last transformer block to serve as the feature representation. the features of the test video queries were matched against the k-nearest-neighbors within the training set features. For assessment, we adopted the recall at k (R@k) metric, consistent with [15, 19]. R@k quantifies the proportion of queries where the top-k nearest neighbors include at least one video belonging to the same class. The experimental results also provide clear evidence of the improvements achieved by our method over full fine-tuning in the context of the retrieval task.\n\n|     Method      | R@1  | R@5  | R@10 | R@20 | R@50 |\n| :-------------: | :--: | :--: | :--: | :--: | :--: |\n| full fine-tune  | 91.4 | 94.3 | 95.3 | 96.5 | 96.7 |\n|    CVPT$^*$     | 87.6 | 90.5 | 92.4 | 95.3 | 96.4 |\n| AdaptFormer$^*$ | 82.3 | 85.3 | 89.9 | 93.7 | 97.0 |\n|      CVPT       | 92.1 | 96.3 | 97.5 | 98.7 | 99.5 |\n\n\n\nW2:Why are the numbers reported in the submission differ from the numbers reported in the original paper(for eg: CoViAR) Are there differences in the experimental setting?\n\n\n\nA2:The results reported for CoViAR in the original paper are grounded under supervised pre-training. In contrast, our study showcases the results under self-supervised pre-training, as reported by IMRNet. Given the prevalence of self-supervised pre-training among the methods being compared, we present CoViAR's results based on self-supervised pre-training, as outlined in the IMRNet. Additionally, we consider incorporating the result of CoViAR under supervised pre-training to enhance the comprehensiveness of our paper. \n\n\n\nW3:Add Bold text in the tables to highlight the best-performing methods as it is really hard for the readers to sift through the tables in their current condition. \n\n\n\nA3:Thanks for your suggestion. We will make improvements in next version.\n\n\n\n\n\nQ1:Was a LoRA update option considered instead of prompt fine-tuning? If so, can the details of such experiments be mentioned in the supp. material\n\n\n\nA1:LoRA introduces an auxiliary module that employs lower-order matrices through mapping weights within the multi-head attention layer. During training, only this auxiliary module is actively trained. This method shares similarities with the practice of enhancing a frozen model with adapters. In contrast to LoRA, the prompt tuning framework offers heightened flexibility in incorporating varying quantities of prompts at diverse positions. Furthermore, this approach simplifies the integration of a priori information into the prompt construction process. While the application of LoRA for compressed video falls outside the scope of this paper, we consider the prospect of exploring this avenue in future endeavors."}, {"review_id": "WhF0uTnn6P", "reviewer": "Reviewer_p8gQ", "summary": "This work studies the video classification task in the compressed video. With the motion vector and residual as the prompt, this work proposes selective cross-model complementary prompter idea to enhance the cross-model interactions, achieving promising results while maintaining a small number of trainable parameters.  ", "questions": "From the Ln124-125, what are the challenges, is there any examples? And is there any evidence to show the inconsistencies between upstream and downstream data? ", "limitations": "The scalability may be one of the limitations.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Most parts of this paper are well written which clearly demonstrates its motivation and methodology. Especially, the method of this paper is easy to follow.\n\nThe idea of using motion vector and residual in the compressed video as prompts is inspiring. These two modalities are computationally free to access, which is a significant advantage compared to traditional motion cues, such as optical flow.\n\nFrom the experimental results, the proposed CVPT achieves promising results while keeping the tunable parameters low. Visualization is a plus.\n", "weaknesses": "The evaluation benchmarks (UCF, HMDB, and SSv2) are all small scaled. One concern is the scalability of the proposed method.\n\nThe design of some key components is not well justified and the figure 2(right) is confusing. For example: 1. Will the prompt join the multi-head self-attention (MHSA)? If so, why the frozen MHSA is able to leverage prompt feature from other modalities. 2. In the SCCP, are the input prompts directly from the previous SCCP other than MHSA? What’s the motivation and effectiveness evidence of keep using the SCCP-processed prompts in SCCP of different layers? 3. In L_2, there is a closed loop for the I-frames embedding. What is the connection to the previous layer. 4. Not sure what are summed together by using the adding function in L_1 and L_2.\n\nThe experiments set-up in the ablation study is not clear enough. In the Table 3, CPR without refinement and CPR are both activated in the last row, which is confusing. Also, the linear probe is used as the baseline, but some implementation details are missing, for example: the number of input frames, how prompts attend the self-attention, and the architecture of classifier. In Table 5, what is the implementation details of fully fine-tune?\n\nLn 45 suggests the large parameter storage burden from previous methods. However, there is no related comparison in the following analysis. In addition to the trainable parameter, the throughput (samples/second), computational burden (GFLOPs), and memory cost (GB) are important metrics. \n\nMinor:\n\nLn 131 The resulting -> the resulting\n\nThere is no notation for g_4 to g_6 in Eq 7\n\nTable 4, E_R -> P_R \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1:The evaluation benchmarks (UCF, HMDB, and SSv2) are all small scaled. One concern is the scalability of the proposed method.\n\nA1:In fact, SSv2 is one of the largest datasets used in compressed video, containing a substantial **193,690** videos. Our approach also shows performance improvement on SSv2, illustrating its good scalability. \n\nW2:The design of some key components is not well justified and the figure 2(right) is confusing. For example: 1. Will the prompt join the multi-head self-attention (MHSA)? If so, why the frozen MHSA is able to leverage prompt feature from other modalities. 2. In the SCCP, are the input prompts directly from the previous SCCP other than MHSA? What’s the motivation and effectiveness evidence of keep using the SCCP-processed prompts in SCCP of different layers? 3. In L_2, there is a closed loop for the I-frames embedding. What is the connection to the previous layer. 4. Not sure what are summed together by using the adding function in L_1 and L_2.\n\nA2.1:Prompt will join the MHSA. Despite the frozen MHSA parameters, the capacity to tune both the input visual token and prompts remains intact. Furthermore, our SCCP module effectively functions as modality alignment. This is achieved through the simultaneous integration of improved motion cues into the I-frame token and the concurrent update of the prompt.\n\nA2.2: The SCCP module can dynamically update prompts within each layer to refine the model-tuning process. This strategy is also embraced by the VPT, where new prompts are inserted into each layer. We conducted a comparative experiment, revealing that our strategy yields superior performance in contrast to the alternative method that relies on prompts from the preceding MHSA layer.\n|Method|Acc.|\n|:-|:-:|\n|prompts from MHSA|83.7|\n|Ours|87.6|\n\nA2.3:The I-frame token is sourced from the output of the preceding layer's model output.\n\nA2.4:The adding function employed signifies the summation of $E_I$ from the SCCP output and  I-frame tokens from the backbone.\n\n\nW3:The experiments set-up in the ablation study is not clear enough. In Table 3, CPR without refinement and CPR are both activated in the last row, which is confusing. Also, the linear probe is used as the baseline, but some implementation details are missing, for example: the number of input frames, how prompts attend the self-attention, and the architecture of classifier. In Table 5, what is the implementation details of fully fine-tune?\n\nA3.1:Indeed, we will revise Table 3 to make it more clear. \n\nA3.2:In the linear probe configuration, an equivalent input to the prompt setup is used, encompassing 4 I-frames and 12 P-frames. A uniform embedding function is applied to encode all three modalities. The backbone is pre-trained ViT on the Kinetics-400 with raw videos, and the classification layer is a simple MLP layer that maps the latent features into classification categories. Subsequently, the concatenated tokens across modalities are input into the network. Similar to prior studies [15, 19], only the classification layer is involved in training within the linear probe framework. \n\nA3.3:In the full fine-tuning setup presented in Table 5, the same inputs utilized as in our approach are maintained, consisting of 4 I-frames and 12 P-frames. Distinctive embedding functions are applied to encode the various modalities. The backbone is either ViT or Swin under different pre-training manners. All network parameters are trainable within this configuration.\n\nW4:Ln 45 suggests the large parameter storage burden from previous methods. However, there is no related comparison in the following analysis. In addition to the trainable parameter, the throughput (samples/second), computational burden (GFLOPs), and memory cost (GB) are important metrics.\n\nA4:We make a comprehensive comparison with the raw video method, VideoMAE, in the table below. This comparison effectively demonstrates the superior efficiency of our approach in various aspects. Notably, our method excels in rapid inference by bypassing the need for video decoding. Furthermore, our utilization of compact compressed video inputs contributes to reduced GFLOPs, making our method computationally lighter. Moreover, our innovative prompt tuning framework which only involves training a small subset of the parameter, minimizes memory usage. This presents a significant advantage over the full fine-tuning method.\n|Method|Videos/Second|GFLOPs|Memory Cost (GB)|\n|:-|:-:|:-:|:-:|\n|VideoMAE (ViT-B)|0.40|1080.0|23.6|\n|CVPT (ViT-B)|3.80|772.2|13.4|\n\n\nW5:Ln 131 The resulting -> the resulting, There is no notation for g_4 to g_6 in Eq 7, Table 4, E_R -> P_R\n\nA5:Thank you for your valuable suggestion. We will revise the typo and rectify any potential errors.\n\nQ1:From the Ln124-125, what are the challenges, is there any examples? And is there any evidence to show the inconsistencies between upstream and downstream data?\n\nA1:The challenge under consideration encompasses not only divergences in tasks but also disparities in data modalities. This is evident in scenarios addressed in this paper – transitioning from pre-trained raw video large models to downstream tasks involving compressed video that includes additional motion vectors and residuals. Modality gaps also exist when transitioning from upstream RGB-based tracking models to downstream tasks involving RGB+infrared or RGB+depth-based tracking. Furthermore, this challenge extends to cases where upstream image large models are applied to fine-tuning tasks involving video and text modalities.\n\nL1:The scalability may be one of the limitations.\n\nA1:The progress in compressed video research has led to rapid advancements, despite the relatively smaller dataset scale in comparison to image datasets. Our approach has demonstrated its efficacy on SSv2 dataset which serves as a significant large benchmark. However, due to current constraints, the validation of our method on a larger dataset remains unachievable."}]

Author rebuttal:
We are appreciative of the valuable insights shared by the reviewers. In response, we have thoroughly addressed each comment, providing individualized responses to each reviewer. The supplied PDF includes visualizations corresponding to Weakness 1, as referenced by Reviewer Eouy.

Meta-review: This is a borderline submission. The reviewers found the problem relevant and well motivated, and the proposed solution simple and effective. The rebuttal resolved some of the issues raised, with one reviewer left unconvinced. My take is that the simplicity of the method will be interesting for the NeurIPS video understanding community and that the empirical results in this context are strong. Hence, I will recommend acceptance.
Decision: Accept (poster)