SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bing Li $^{1,2}$  Jiaxin Chen $^{2}$  Xiuguo Bao $^{3}$  Di Huang $^{1,2*}$
$^{1}$ SKLSDE, Beihang University, Beijing, China
$^{2}$ IRIP Lab, SCSE, Beihang University, Beijing, China
$^{3}$ CNCERT/CC, Beijing, China
{libingsy, jiaxinchen, dhuang} @ buaa.edu.cn, baoxiuguo@139.com
Compressed videos offer a compelling alternative to raw videos, showing the possibility to significantly reduce the on-line computational and storage cost. However, current approaches to compressed video processing generally follow the resource-consuming pre-training and fine-tuning paradigm, which does not fully take advantage of such properties, making them not favorable enough for widespread applications. Inspired by recent successes of prompt tuning techniques in computer vision, this paper presents the first attempt to build a prompt based representation learning framework, which enables effective and efficient adaptation of pre-trained raw video models to compressed video understanding tasks. To this end, we propose a novel prompt tuning approach, namely Compressed Video Prompt Tuning (CVPT), emphatically dealing with the challenging issue caused by the inconsistency between pre-training and downstream data modalities. Specifically, CVPT replaces the learnable prompts with compressed modalities (e.g. Motion Vectors and Residuals) by re-parameterizing them into conditional prompts followed by layer-wise refinement. The conditional prompts exhibit improved adaptability and generalizability to instances compared to conventional individual learnable ones, and the Residual prompts enhance the noisy motion cues in the Motion Vector prompts for further fusion with the visual cues from I-frames. Additionally, we design Selective Cross-modal Complementary Prompt (SCCP) blocks. After inserting them into the backbone, SCCP blocks leverage semantic relations across diverse levels and modalities to improve cross-modal interactions between prompts and input flows. Extensive evaluations on HMDB-51, UCF-101 and Something-Something v2 demonstrate that CVPT remarkably outperforms the state-of-the-art counterparts, delivering a much better balance between accuracy and efficiency.
In recent years, there has been a surge of interest in compressed video understanding [51, 36, 8, 46, 11, 19] due to its great potential in saving the computational and storage cost for on-line processing, which directly performs inference on compressed data and thus bypasses the resource-consuming decoding phase.
Raw videos consist of dense RGB frames conveying coherent and consistent content, while compressed videos are composed of sparsely decoded RGB frames with entire spatial information, i.e. Intra-frames (I-frames), and incompletely decoded RGB frames with free but coarse motion clues, i.e. Predicted frames (P-frames) or Bidirectionally predicted frames (B-frames). In this case, existing approaches to compressed video understanding generally follow the pre-training and fine-tuning paradigm, which initially pre-trains models on compressed videos from a large database (e.g. Kinetics-400 [3]) and fully fine-tunes them on downstream ones. According to different network structures,
Figure 1: Different types of approaches to representation learning in compressed videos. (a) and (b) are the two main variants of the existing pre-training and fine-tuning paradigm, where the former is based on multi-stream CNNs and the latter is based on ViT, both of which require pre-training models on compressed videos. (c) is a plain prompt tuning scheme which can be considered as a straightforward attempt to leverage frozen off-the-shelf models pre-trained on raw videos, but it does not well adapt to the given task with degraded performance. (d) CVPT advances (c) by designing conditional prompts for Motion Vectors and Residuals, delivering a novel paradigm which successfully learns powerful representations for compressed videos in a more effective and efficient manner.
they can be broadly classified into two categories. (i) [20, 1, 21, 19, 11] apply multi-branch CNNs for individual modalities, which are then fused to deliver final results, as depicted in Figure 1 (a). (ii) [5] employs a modified Vision Transformer (ViT) for all modalities, motivated by the advantage of ViT in reasoning tasks and the ability to handle multi-modalities in text-image [15, 37, 50] and multi-task domains [31, 18], as shown in Figure 1 (b). Despite their effectiveness, such a paradigm suffers two major drawbacks. Firstly, it requires pre-training networks on compressed videos, resulting in a high computational complexity in GPU-day in off-line mode. Several attempts are made to speed up pre-training by loading or partially loading off-the-shelf models built based on raw videos; however, it is still not favorable enough for wide popularization. Secondly, it relies on fully fine-tuning backbones, which incurs a large parameter storage burden, making it challenging for deployment in real-world applications. Furthermore, full fine-tuning tends to disrupt the knowledge learned by the pre-trained model, degrading the generalization performance [54]. The limitations aforementioned highlight the need for a more effective and efficient alternative to address these issues.
Inspired by the advancements of prompting techniques in Natural Language Processing (NLP), researchers from the field of Computer Vision (CV) have been discussing this representation learning paradigm. By freezing the pre-trained model and introducing a small number of learnable parameters on input images or videos, they create decent visual prompts [4, 23]. This paradigm enables efficient information mining from pre-trained models and eliminates the necessity for fully fine-tuning networks. On the other side, raw videos contain richer spatial and temporal clues than compressed ones, and if sufficient and important knowledge can be mined from the models pre-trained on raw videos, it should strengthen representation learning of compressed videos. Both the facts trigger us to investigate a prompt tuning paradigm to compressed video understanding with pre-trained raw video models as a more effective and efficient alternative to the pre-training and fine-tuning one. Although current prompt tuning approaches [32, 16, 24, 4, 9] have achieved successes in various tasks, they generally work in the case that pre-training and downstream data share the same modality and it remains problematic when they have different modalities, i.e. the pre-training data are in the raw video (dense RGB frames) modality and the downstream data are in the compressed video modality (I-frames, Motion Vectors and Residuals), as shown in Figure 1 (c).
In this paper, we present a novel paradigm to video understanding in the compressed domain, namely Compressed Video Prompt-Tuning (CVPT). As in Figure 1 (d), our approach freezes the whole pretrained raw video-based model and learns a few modal-specific visual prompts. It allows us to leverage the common prior knowledge in the feature and semantic spaces shared by the two types of videos, to facilitate representation learning of compressed videos. Specifically, VPT employs a conditional prompt framework with refinement, in which Motion Vectors and Residuals are re-parameterized into conditional prompts. These prompts are more adaptable to individual instances and alleviate the impact of freezing these modalities which can lead to performance degradation. Additionally, CVPT incorporates simple and lightweight modality-complementary prompt blocks, namely Selective Cross-modal Complementary Prompt (SCCP), into the frozen pre-trained model, capturing cross-modal interactions for performance improvement. Concretely, SCCP firstly embeds the I-frame
tokens as well as the conditional Motion Vector and Residual prompts into a low-dimensional feature space, in pursuit of computational efficiency. Based on attention maps induced by the Residual prompts, the Motion Vector prompts are subsequently refined via suppressing the noisy motions, which are further integrated into the I-frame tokens as complementary motion cues. As a consequence, the cross-modal interaction is significantly strengthened, remarkably facilitating model tuning for efficient representation learning of compressed videos.
The main contributions of this paper are summarized in four-fold:
- We propose CVPT, a novel visual prompt tuning framework, which enables pre-trained raw video models to adapt to compressed video understanding tasks. To the best of our knowledge, this is the first work to address this issue.
- We introduce modal-related conditional prompts to replace conventional learnable ones, proving more adaptable to individual instances and alleviates the inconsistency between pre-training and downstream data modalities.
- We design selective cross-modal complementary prompt blocks, and these blocks substantially strengthen cross-modal interactions and selectively complement dynamic cues in Motion Vectors to input flows.
- We conduct extensive experiments on three widely used benchmarks and achieve newly state-of-the-art results. Notably, our approach maintains the parameter-efficiency with less than  $1\%$  parameters being trainable.
2.1 Compressed Video Representation Learning
Early studies commonly adopt multi-stream CNNs and focus on dedicatedly designing the model structures for P/B-frames and fusion modules. The pioneering work by [49] introduces the use of CNNs to process compressed videos. They replace optical flows in the two-stream network [14] with Motion Vectors, resulting in a higher efficiency. CoViAR [45] extends a third stream to leverage all the modalities, including I-frames, Motion Vectors, and Residuals, completely bypassing video decoding. To improve the quality of dynamics, [2] applies image denoising techniques, while [40] employs GANs to generate Motion Vectors that approximate optical flows. [22] and [1] adopt a similar approach by feeding compressed modalities into a CNN model to mimic a raw video-based teacher. [22] presents Temporal Trilinear Pooling for fusing multiple modalities in lightweight models suitable for portable devices. Inspired by SlowFast [12, 30] propose the Slow-I-Fast-P model, where pseudo optical flows are estimated by a specific loss function. MM-ViT [5] is the first work using ViT on compressed videos, factorizing the self-attention across the space, time and modality dimensions.
The advent of self-supervised learning, particularly contrastive learning and Masked AutoEncoder (MAE), has led to the exploration of self-supervised pretext tasks for compressed videos. In this context, IMRNet [48] stands out as the first work to delve into self-supervised learning for compressed videos. It employs a three-stream network to individually encode I-frames, Motion Vectors, and Residuals, and introduces two pretext tasks that make use of motion statistics and internal GOP (Group of Pictures) structures in the compressed domain. Another notable approach is MVCGC [21], which delivers a cross-guidance contrastive learning framework with multi-views. MVCGC utilizes Motion Vectors as supervision signals for RGB frames and vice versa.
To sum up, the existing approaches in compressed video understanding involve the design of specialized modules or pretext tasks, which makes the pre-training and fine-tuning paradigm resource-consuming, not favorable enough to various downstream applications. In contrast, our work presents a prompt tuning based alternative, where necessary knowledge are mined from the off-the-shelf model pre-trained on raw videos through only a few modal-specific visual prompts, and is more effective and efficient.
2.2 Visual Prompt Tuning
Prompt tuning has emerged as a promising way to efficient representation learning in the field of NLP. By leveraging textual prompts, researchers have successfully reformulated downstream tasks
Figure 2: Framework of the proposed CVPT. It firstly embed the input compressed videos, consisting of I-frames, Motion Vectors and Residuals, into the I-frame tokens  $E_{\mathrm{I}}^{0}$ , the conditional Motion Vector prompts  $P_{\mathrm{MV}}^{0}$  and the Residual prompts  $P_{\mathrm{R}}^{0}$ , respectively. A frozen pre-trained Transformer with  $L$  layers is adopt for feature extraction. The Selective Cross-modal Complementary Prompt (SCCP) blocks are employed to perform conditional prompt learning as well as facilitate the interactions between the prompts and the I-frame tokens for efficient fine-tuning.
to resemble pre-training tasks that have already been solved [28, 33]. This approach offers several advantages over full fine-tuning, such as the high parameter efficiency and the reduced storage cost. It has also shown promising results in various CV applications. For instance, VPT [23] introduces learnable parameters to Transformer encoders and achieves superior performance compared to full fine-tuning on multiple recognition tasks. AdaptFormer [7] enhances ViT models with lightweight modules and surpasses the performance of fully fine-tuned models in action recognition benchmarks. Convpass [25] proposes convolutional bypasses for prompt learning in pre-trained ViT models. Some prompting attempts have also been made to deal with multi-modal tasks, such as visual and language. CoOp [53] fine-tunes CLIP using a continuous set of prompt vectors, while Co-CoOp [52] improves its generalization by conditioning prompts on image instances. [26] strengthens CLIP for video understanding tasks by adding lightweight Transformers on visual features and text prompts. MaPLe [27] optimizes the prompting mechanism in both the vision and language branches to improve the adaptability of CLIP.
Different from current prompt-learning approaches in NLP and CV that primarily apply in the case that pre-training and downstream data share the same modality, our work further addresses the challenge arising from the inconsistency between pre-training and downstream data modalities, i.e. the pre-training data are dense RGB frames while the downstream data are sparse I-frames and coarse Motion Vectors and Residuals.
In this section, we describe the details of the proposed Compressed Video Prompt Tuning (CVPT) approach. Briefly, CVPT employs a conditional prompt framework with refinement as shown in Figure 2, by proposing a Selective Cross-modal Complementary Prompt (SCCP), which are depicted in the rest part.
3.1 Conditional Prompt with Refinement
Prompt tuning [28, 33] is commonly utilized to efficiently fine-tune a model pre-trained on the upstream tasks to align with the downstream tasks. Existing prompt tuning approaches often assume
that the pre-trained task and the downstream task share the same data modality. However, when fine-tuning for compressed video-based applications, the data modality differs as the downstream tasks adopt the compressed videos, and the pre-trained tasks utilize the raw videos. A straightforward fine-tuning without considering the modality gap often severely deteriorates the performance, since the Motion Vectors and Residuals are unique to compressed videos and have distinct data distributions from the raw videos. To address this issue, existing works employ an extra feature extraction network that may negatively abate the inter-modal interactions; or modify the Transformer structure to process the multi-modal compressed video data, which however still requires pre-training and full fine-tuning.
In this paper, we propose an efficient prompt tuning approach for compassed video-based tasks. Motivated by [54], we re-parameterize the Motion Vectors and Residuals into prompts and enhance the cross-modal interactions with I-frames for prompt learning. Different from [54], we embed the Motion Vectors and Residuals into distinct types of prompts, and refine the Motion Vector prompts by leveraging the Residual counterparts, which are further integrated into the I-frame Motion Vectors as strengthened complements of motion cues.
Concretely, the input I-frames  $X_{\mathrm{I}} \in \mathbb{R}^{3 \times T_{\mathrm{I}} \times H \times W}$  are firstly encoded into the I-frame tokens  $E_{\mathrm{I}}^{0} \in \mathbb{R}^{N \times d}$  by a frozen patch embedding layer, where  $H, W, d$  represent the frame height, frame width and the encoding dimension, respectively. The Motion Vectors  $X_{\mathrm{MV}} \in \mathbb{R}^{2 \times T_{\mathrm{P}} \times H \times W}$  and the Residuals  $X_{\mathrm{R}} \in \mathbb{R}^{3 \times T_{\mathrm{P}} \times H \times W}$  are encoded by learnable patch embedding layers, generating the conditional Motion Vector prompts  $P_{\mathrm{MV}}^{0} \in \mathbb{R}^{N \times d}$  and the Residual prompts  $P_{\mathrm{R}}^{0} \in \mathbb{R}^{N \times d}$ , respectively.  $T_{\mathrm{I}} / T_{\mathrm{P}}$  denote the number of I/P-frames.
The I-frame tokens  $E_{\mathrm{I}}^{0}$  together with the conditional prompts  $X_{\mathrm{MV}}$  and  $X_{\mathrm{R}}$  are comprehensively integrated by the proposed Selective Cross-modal Complementary Prompt (SCCP) at the first layer denoted by  $\mathcal{M}_1(\cdot)$ , where the output are fed into the Transformer block as complementary motion cues to the I-frame tokens, as well as inputs to the SCCP block from the subsequent layer. The SCCP blocks are successively stacked by aligning with the original ViT layers. The overall forward propagation process for prompt tuning can be briefly formulated as follows:
$$
\left[ \boldsymbol {P} _ {\mathrm {M V}} ^ {l}, \boldsymbol {P} _ {\mathrm {R}} ^ {l}, \boldsymbol {E} _ {\mathrm {I}} ^ {l - 1} \right] = T _ {l} \left(\left[ \hat {\boldsymbol {P}} _ {\mathrm {M V} ^ {\prime}} ^ {l - 1}, \hat {\boldsymbol {P}} _ {\mathrm {R} ^ {\prime}} ^ {l - 1}, \boldsymbol {E} _ {\mathrm {I}} ^ {l - 1} + \hat {\boldsymbol {E}} _ {\mathrm {I} ^ {\prime}} ^ {l - 1} \right]\right), \quad l = 1, 2, \dots L; \tag {1}
$$
$$
\boldsymbol {y} = \operatorname {C L S} _ {-} \operatorname {H e a d} \left(\left[ \boldsymbol {P} _ {\mathrm {M V}} ^ {L}, \boldsymbol {P} _ {\mathrm {R}} ^ {L}, \boldsymbol {E} _ {\mathrm {I}} ^ {L} \right]\right). \tag {2}
$$
where  $L$  is the number of Transformer layers,  $T_{l}(\cdot)$  and  $\mathrm{CLS\_Head}(\cdot)$  refer to the Transformer block at the  $l$ -th layer and the classification head, respectively. Here,  $[\hat{P}_{\mathrm{MV}}^{l-1}, \hat{P}_{\mathrm{R}}^{l-1}, \hat{E}_{\mathrm{I}}^{l-1}]$  represents the output of the SCCP block at the  $l$ -th layer, which will be elaborated in Section 3.2.
3.2 Selective Cross-modal Complementary Prompt
Given the I-frame tokens  $E_{\mathrm{I}}^{0}$ , the conditional Motion Vector prompts  $P_{\mathrm{MV}}^{0}$ , the Residual prompts  $P_{\mathrm{R}}^{0}$  and a frozen pre-trained encoder  $\mathcal{G}_{RGB}$  containing  $L$  Transformer layers, SCCP aims to complement the motion cues from the conditional prompts to the I-frame branch. Generally, SCCP aggregates the tokens and prompts from multiple modalities of the previous layer to generate the new ones at the current layer, which is formulated as below:
$$
\left[ \boldsymbol {P} _ {\mathrm {M V} ^ {\prime}} ^ {l - 1}, \boldsymbol {P} _ {\mathrm {R} ^ {\prime}} ^ {l - 1}, \boldsymbol {E} _ {\mathrm {I} ^ {\prime}} ^ {l - 1} \right] = \mathcal {M} _ {l} \left(\left[ \boldsymbol {P} _ {\mathrm {M V}} ^ {l - 1}, \boldsymbol {P} _ {\mathrm {R}} ^ {l - 1}, \boldsymbol {E} _ {\mathrm {I}} ^ {l - 1} \right]\right), \qquad l = 1, 2, \dots L. \tag {3}
$$
It is worth noting that the Motion Vectors, which are block-wise motion cues estimated by matching, tend to be coarse and noisy compared to the optical flow counterparts displayed in Figure 3(a). As depicted in [39], motion information near object boundaries plays a crucial role in representation learning. Inspired by this insight, we generate an attention map to select refined motion cues from the Motion Vectors by leveraging the Residuals, based on the observation that the Residuals are generally well-aligned with the boundaries of moving objects and strongly correlated with Motion Vectors as observed in [40]. The refined motion cues are successively adopted as a informative complements to the RGB modality, thus boosting learning discriminative representations.
The detailed design of SCCP is illustrated in Figure 3(b). Specifically, at the  $l$ -th layer, the input of SCCP consists of the I-frame tokens  $E_{\mathrm{I}}^{l-1}$ , the conditional prompts  $P_{\mathrm{MV}}^{l-1}$  and  $P_{\mathrm{R}}^{l-1}$ . In order to obtain compact representations and reduce the overall size of parameters, we firstly embed the  $C$ -dimensional input into a  $(C / \alpha)$ -dimensional feature space as follows:
$$
\hat {\boldsymbol {E}} _ {\mathrm {I}} ^ {l - 1} = g _ {1} \left(\boldsymbol {E} _ {\mathrm {I}} ^ {l - 1}\right), \quad \hat {\boldsymbol {P}} _ {\mathrm {M V}} ^ {l - 1} = g _ {2} \left(\boldsymbol {P} _ {\mathrm {M V}} ^ {l - 1}\right), \quad \hat {\boldsymbol {P}} _ {\mathrm {R}} ^ {l - 1} = g _ {3} \left(\boldsymbol {P} _ {\mathrm {R}} ^ {l - 1}\right), \tag {4}
$$
Figure 3: (a) Illustration on the optical flows representing accurate motion cues, the coarse Motion Vectors and the Residuals. The red rectangles highlight the areas that Motion Vectors contain noises, which however can be refined by leveraging the Residuals. (b) Detailed architecture of the proposed Selective Cross-modal Complementary Prompt.
where  $\alpha$  is the reduction factor, the embedding functions  $g_{1}(\cdot), g_{2}(\cdot)$ , and  $g_{3}(\cdot)$  are simple  $1 \times 1$  convolutions. In the ViT-B backbone, the input feature dimension  $C$  is 768, and the reduction factor  $\alpha$  is set to 96 in all prompt blocks. As the scale of backbone grows,  $\alpha$  usually increases proportionally. In the case of Swin Transformer, as the feature dimension  $C$  becomes larger in deeper layers, we gradually increase it to maintain the same embedding feature size across different layers.
After feature embedding, we generate a spatial-temporal attention map  $\mathcal{A}$  by performing Softmax(·) on the embedded Residual prompts  $\hat{P}_{\mathrm{R}}^{l-1}$ . To strengthen the smoothness of the attention map, we introduce a learnable parameter. The attention map is subsequently applied to select representative motion cues from the embedded Motion Vector prompts  $\hat{P}_{\mathrm{MV}}^{l-1}$  as the following:
$$
\hat {\boldsymbol {P}} _ {\mathrm {M V}} ^ {l - 1} = \mathcal {A} \odot \hat {\boldsymbol {P}} _ {\mathrm {M V}} ^ {l - 1} = \operatorname {S o f t m a x} (\omega \times \hat {\boldsymbol {P}} _ {\mathrm {R}} ^ {l - 1}) \odot \hat {\boldsymbol {P}} _ {\mathrm {M V}} ^ {l - 1}. \tag {5}
$$
The refined prompts  $\hat{P}_{\mathrm{MV}}^{l-1}$  is further integrated into the intermediate embedding  $\hat{E}_{\mathrm{I}}^{l-1}$  by addition, in order to complement the motion cues as below
$$
\hat {\boldsymbol {E}} _ {\mathrm {I}} ^ {l - 1} := \hat {\boldsymbol {E}} _ {\mathrm {I}} ^ {l - 1} + \hat {\boldsymbol {P}} _ {\mathrm {M V}} ^ {l - 1}. \tag {6}
$$
After performing the above cross-modal interactions, the embedded tokens  $\hat{E}_{\mathrm{I}}^{l-1}$ , the embedded prompts  $\hat{P}_{\mathrm{MV}}^{l-1}$  and  $\hat{P}_{\mathrm{R}}^{l-1}$  are finally project back to the high-dimensional ones by using the following formulation:
$$
\hat {\boldsymbol {E}} _ {\mathrm {I} ^ {\prime}} ^ {l - 1} = g _ {4} \left(\hat {\boldsymbol {E}} _ {\mathrm {I}} ^ {l - 1}\right), \quad \hat {\boldsymbol {P}} _ {\mathrm {M V} ^ {\prime}} ^ {l - 1} = g _ {5} \left(\hat {\boldsymbol {P}} _ {\mathrm {M V}} ^ {l - 1}\right), \quad \hat {\boldsymbol {P}} _ {\mathrm {R} ^ {\prime}} ^ {l - 1} = g _ {6} \left(\hat {\boldsymbol {P}} _ {\mathrm {R}} ^ {l - 1}\right), \tag {7}
$$
where  $g_4(\cdot), g_5(\cdot)$ , and  $g_6(\cdot)$  are  $1 \times 1$  convolutions.
In order to comprehensively enhance the representation learning, the SCCP block is applied to each Transformer block, and is successively stacked, i.e. the conditional prompts  $P_{\mathrm{MV}}^{l-1}$  and  $P_{\mathrm{R}}^{l-1}$  are updated in each SCCP block and is fed into the next SCCP block as input.
4 Experimental Results and Analysis
4.1 Datasets and Evaluation Metric