RAP: Efficient Text-Video Retrieval with Sparse-and-Correlated Adapter (2024)

Abstract: Text-Video Retrieval (TVR) aims to align relevant video content with natural language queries. To date, most state-of-the-art TVR methods learn image-to-video transfer learning based on large-scale pre-trained visionlanguage models (e.g., CLIP). However, fully fine-tuning these pre-trained models for TVR incurs prohibitively expensive computation costs. To this end, we propose to conduct efficient text-video Retrieval with a sparse-andcorrelated AdaPter (RAP), i.e., fine-tuning the pre-trained model with a few parameterized layers. To accommodate the text-video scenario, we equip our RAP with two indispensable characteristics: temporal sparsity and correlation. Specifically, we propose a low-rank modulation module to refine the per-image features from the frozen CLIP backbone, which accentuates salient frames within the video features while alleviating temporal redundancy. Besides, we introduce an asynchronous self-attention mechanism that first selects the top responsive visual patches and augments the correlation modeling between them with learnable temporal and patch offsets. Extensive experiments on four TVR datasets demonstrate that RAP achieves superior or comparable performance compared to the fully fine-tuned counterpart and other parameter-efficient fine-tuning methods.

heading: 1 Introduction
text: Text-Video Retrieval (TVR) (Gabeur et al., 2020; Gorti et al., 2022; He et al., 2021a; Lei et al., 2021; Luo et al., 2022; Ma et al., 2022; Wang et al., 2022) is a pivotal task in the realm of multimodal research, which aims to find the most relevant video content within a repository in response to the text query, and vice versa. With the rapid progress in large-scale image-text pre-training (Jia et al., 2021; Radford et al., 2021; Yu et al., 2022; Yuan et al.,
* Equal contributions. † Corresponding author.
2021), current research focuses on how to transfer pre-trained image-text models (e.g., CLIP (Radford et al., 2021)) to the video-text domain. However, fully fine-tuning the video model is computationally expensive and may have the risk of overfitting.
To alleviate this dilemma, Parameter-Efficient Fine-Tuning (PEFT) stemmed from natural language processing (Houlsby et al., 2019; Lester et al., 2021; Zaken et al., 2022; Hu et al., 2021) has also aroused extensive research interest in the field of computer vision (Chen et al., 2022b,a) and cross-modal learning (Sung et al., 2022). Recently, some exploratory work (Zhang et al., 2023; Jiang et al., 2022; Diao et al., 2024) has also attempted to introduce PEFT into TVR. These methods, however, simply introduce existing PEFT algorithms
7160
(Houlsby et al., 2019; You et al., 2022; Karimi Mahabadi et al., 2021) without considering the inherent characteristics of video data.
To this end, we argue that an ideal PEFT method for VTR should be equipped with two characteristics: 1) Temporal Sparsity: As shown in Figure 1, the video data inherently contains lots of redundancies or repetitions in the temporal perspective. The visualized frame-by-frame embedded CLIP features are over-smooth, resulting in the loss of important details or nuances within the video data. In contrast, the video feature adapted from pre-trained CLIP should capture the most informative frames, allowing for a more sparse representation. 2) Temporal Correlation: The desired video adapter is supposed to incorporate the dependencies and relationships between consecutive frames, especially when dealing with actions or events that happen in several frames, as the features can encapsulate the evolving context over time. For example in Figure 1, the query sentence contains two entities including dog and bird. Given the query patch ( in frame #3), we visualize the similarity distribution within the other patches. As in this example, vanilla self-attention can only attend to the dog instance while the other bird instance is overlooked.
In the realm of video processing and analysis, the temporal dimension often contains redundancies due to the inherent correlation between adjacent frames. This redundancy can lead to inefficiencies in computational resources and storage when dealing with large-scale video data. Therefore, there is a need to extract meaningful and informative features while reducing temporal redundancy.
To alleviate these aforementioned issues, we
propose an efficient text-video Retrieval framework with sparse-and-correlated AdaPter (dubbed as RAP). Our proposed RAP not only streamlines the trainable parameters, enhancing efficiency in computational resources, but also tailors the architecture to adeptly capture and model the nuanced temporal characteristics of video data.
To achieve temporal sparsity, we propose a LowRank Modulation (LoRM) module to refine the pre-trained CLIP feature (Radford et al., 2021) on the principle of redundancy reduction and essential information extraction. This design stems from a simple hypothesis that the change in temporal weights resides on a low intrinsic rank (Zhang and Tao, 2012). Therefore, we introduce layer-wise low-rank scale parameters and shift parameters, which could be considered as variance and mean to modulate the CLIP feature. Specifically, both scale and shift parameters are instantiated by the multiplication of two low-rank trainable matrices. These parameters are input-independent and therefore more flexible. LoRM allows us to calibrate the video features to highlight salient frames and mitigate temporal redundancy.
For temporal correlation modeling, we replace vanilla self-attention with the proposed Asynchronous Self-Attention (ASA), which introduces temporal dynamics among video frames to capture temporal relationships. Since the attention computing in pre-trained CLIP is constrained within each frame feature, it is challenging to apply to the video domain due to the temporally dynamic nature of video frames. Previous methods employ either temporal Transformer (Jiang et al., 2022; Yang et al., 2022; Zhang et al., 2023) or 3D convolution networks (Yao et al., 2023; Liu et al., 2023) to encode temporal dependencies. Instead of introducing additional modules, we propose an asynchronous selfattention that only warps partial patch tokens in a parameterized way. Firstly, for each frame, we filter semantically significant patches via a parameterfree text-conditioned selection mechanism. Specifically, we compute the similarities between patch features and the corresponding sentence and select the patches with the highest responses. Secondly, each selected patch within the current frame is dynamically warped to attend to the temporally related patches in other frames. The proposed asynchronous self-attention empowers the flexibility in capturing correlations between video frames at the fine-grained patch level.
Overall, the main contributions of this work are:
• We propose RAP to adapt the pre-trained CLIP to efficient TVR, which not only reduces the tunable parameters but also generates temporally sparse and correlated video features.
• To alleviate the temporal redundancy, a low-rank modulation module is introduced to calibrate the frame-wise representation linearly.
• We propose an asynchronous self-attention that captures long-range dependencies with negligible computational overheads.
• Extensive experiments show that our RAP is on par with or even superior to previous PEFT methods and the fully fine-tuned counterpart.

heading: 2 Related Work
text: Text-Video Retrieval. TVR (Yu et al., 2018; Croitoru et al., 2021; Yang et al., 2021; Wang et al., 2021; Chen et al., 2020; Wang and Shi, 2023; Jin et al., 2022, 2023a,b; Liu et al., 2022) is a fundamental research topic in the video-language domain which aims to retrieval the relevant video/text based on the given text/video query. The pioneer works (Yu et al., 2018; Gabeur et al., 2020) rely on pre-extracted features from frozen video and text encoders. To facilitate the end-to-end training, ClipBERT (Lei et al., 2021) proposes a sparse sampling strategy for efficient text-video training. With the great success of large-scale image-text pretraining model CLIP (Radford et al., 2021), the majority of the state-of-the-art TVR methods (Luo et al., 2022; Ma et al., 2022; Wang et al., 2023; Hannan et al., 2023; Jin et al., 2022) focus on transferring the powerful CLIP encoder to the video-text domain by designing various cross-modal alignment strategies. As the first attempt, CLIP4Clip (Luo et al., 2022) employs mean-pooling or Transformer to aggregate video features and conduct coarse-grained (video-sentence level) contrastive alignment. Instead of using the text-agnostic aggregation manner, X-CLIP (Ma et al., 2022) proposes to aggregate video representations conditioned on the text’s attention weight and conduct the multi-grained contrastive learning at the frame-word, video-sentence, video-word and sentence-frame levels. For more comprehensive alignment, UCOFIA (Wang et al., 2023) unifies the coarse-grained and fine-grained alignment to capture both the high-level and lowlevel correspondence between text and video.
Most of the current TVR methods follow the fully fine-tuning paradigm. This scheme, however, is computation-intensive and may have the risk of
overfitting. Besides, additional temporal modeling models are required to bridge the image and video gap. In this paper, we propose RAP which conducts parameter-efficient fine-tuning for TVR which provides a more computationally efficient and potentially more robust approach. Besides, the tunable parameters in our RAP also bear the responsibility for temporal modeling, thus eliminating the need for external temporal modules. Parameter-Efficient Transfer Learning. PEFT (Houlsby et al., 2019; Hu et al., 2021; Lester et al., 2021; He et al., 2021b; Zaken et al., 2022; Sung et al., 2021) is firstly introduced in the NLP domain to reduce the number of trainable parameters while maintaining the comparable performance with the fully fine-tuning setting. Inheriting the merit from NLP, PEFT in computer vision (Jia et al., 2022; Bahng et al., 2022; Jie and Deng, 2022; Sung et al., 2022) also gained extensive research attention. VPT (Jia et al., 2022) follows the prompt tuning strategy by introducing the task-specific learnable prompts on the vision Transformer. To be more compatible with vision tasks, Convpass (Jie and Deng, 2022) introduces the inductive bias of convolutional layers by reconstructing the spatial structure of the token sequence via convolution operations. VL-Adapter (Sung et al., 2022) pioneeringly benchmarks different types of PEFT techniques including Adapter (Houlsby et al., 2019), Hyperformer (Mahabadi et al., 2021), and Compacter (Karimi Mahabadi et al., 2021) in the multitask setting.
There also exist several works (Yang et al., 2022; Pan et al., 2022; Lin et al., 2022; Li and Wang, 2023; Yao et al., 2023; Jiang et al., 2022; Zhang et al., 2023; Lu et al., 2023) focusing on the imageto-video transfer learning. Based on the pre-trained CLIP model, these methods either introduce temporal convolution (Pan et al., 2022) or Transformer (Lu et al., 2023) in sequential (Zhang et al., 2023; Jiang et al., 2022) or parallel (Yao et al., 2023) ways. However, they overlook the inherent temporal structure of video data while our RAP pinpoints two key issues in video feature modeling and generate more representative video features.

heading: 3 Method
text: Text-video retrieval aims to search for and retrieve relevant videos/texts based on textual/video queries by evaluating the similarity between the videosentence pairs. Our proposed RAP is devoted to
bridging the gap between the frozen CLIP feature and the dynamic video scenario by introducing negligible parameter overheads.
The schematic illustration of our RAP is illustrated in Figure 3. In Sec. 3.1, we first present the preliminaries of RAP including the video and text feature embedding. Then we describe the proposed low-rank modulation and the asynchronous self-attention in Sec. 3.2 and Sec. 3.3, respectively.

heading: 3.1 Feature Embedding
text: Video Embedding. We utilize the visual backbone (ViT (Dosovitskiy et al., 2020)) of CLIP as the video encoder. Given the video data, we follow ViT (Dosovitskiy et al., 2020) to process each frame independently. Specifically, each frame with shape H ×W is split into non-overlapping patches with shape P × P and then linearly projected into the embedding space. Such linear projection generates N = HW/P 2 patch features for each frame. Besides, a learnable [CLS] token is prepended to each frame patch feature sequence to represent the global frame representations. The positional embedding is also added to incorporate positional information explicitly. Through the above process, we obtain the tth frame feature x0t ∈ R(N+1)×Dv , t ∈ [1, T ], where Dv is visual feature dimension and T is the total frame number.
The residual structure with serially connected multi-head self-attention (MHSA) and multilayer perceptron (MLP) is applied to capture sequential dependencies and contextual relationships within each frame patch sequence. Repeating the above steps for each frame, we obtain the video embed-
ding at lth layer xl ∈ RT×(N+1)×Dv , l ∈ [1, L], where L denotes the layer number. Specifically, we decompose xl = [f l,pl], where f l ∈ RT×Dv represents the frame-wise features (i.e., [CLS] token feature) while pl ∈ RT×N×Dv is patch-wise representation at the lth layer. Text Embedding. For text embedding, we directly use the text encoder of CLIP to generate the textual representation. The text encoder is a Transformer (Vaswani et al., 2017) with the architecture modifications as described in (Radford et al., 2019). An [EOS] token is also appended to encode the global sentence feature. Concretely, we denote the sentence features at the lth layer as wl ∈ R1×Dt , where Dt is the text feature dimension.

heading: 3.2 Low-rank Modulation
text: In this section, we elaborate on the feature modulation for both video and text features. Since all the layers share the same modulation process, we omit the superscript of layer index l for brevity. Low-rank Modulation for Video. The frame-byframe encoded video features x cannot reflect the characteristics of the video data. The redundancy in the temporal dimension is a major feature that distinguishes videos from static images. To this end, we introduce low-rank scale parameters and shift parameters, which serve as the variance and mean values to modulate the pre-trained CLIP feature. These parameters are input-independent, rendering them comparatively lightweight in nature and hopefully more scalable. Specifically, the video scale parameter cv ∈ RT×Dv and video shift parameter
sv ∈ RT×Dv are decomposed as follows: cv = c
a · cb, sv = sa · sb, (1) where ca, sa ∈ RT×R, cb, sb ∈ RR×Dv are learnable parameters and we set rank R≪ min(T,Dv) to implement the low-rank requirement. The lowrank modulation is applied as follows.
u = cv ⊙ x+ sv, (2) where ⊙ denotes the element-wise multiplication with broadcast. During training, the vanilla feature x is extracted through frozen CLIP backbone and the learnable cv and sv help modify x to be of temporally low-rank. u ∈ RT×(N+1)×Dv is the modulated video feature. Modulation for Text. We also modulate the textual embedding w with parameters ct and st as follows.
z = ct ⊙w + st, (3) where ct, st ∈ R1×Dv are learnable parameters. We do not conduct modulation at the word level or use parameter low-rank decomposition since the textual data do not exhibit the sparsity characteristic.

heading: 3.3 Asynchronous Self-Attention
text: Let’s review the vanilla self-attention in the video encoder. For clarity, we take the tth frame of the input video for illustration. The corresponding modulated feature is denoted as ut ∈ RN×Dv , t ∈ [1, T ] (c.f . Equation (2)). Note that here we define ut as the patch-wise feature which does not contain the global [CLS] token features. We also omit the superscript of layer index l.
The vanilla self-attention first performs three different linear projections on the input feature ut to obtain the triplet of query, key, and value, i.e.,
qt = ut ·Wq, kt = ut ·Wk, vt = ut ·Wv, (4) where Wq,Wk,Wv ∈ RDv×Dv are frozen transformation weights. Then the scaled dot-product attention is computed to achieve the contextual information.
The vanilla self-attention only attends to the intra-frame correlation modeling, which leads to the modality gap between video and image. Instead of introducing an additional serial or parallel temporal modeling module (temporal Transformer (Liu et al., 2023; Yang et al., 2022) or 3D Convolution (Pan et al., 2022)), we propose a novel asynchronous self-attention which introduces patchwise temporal offset to model inter-frame relationship. Besides, to stabilize the training process, we propose a text-conditioned selection mechanism.
Text-conditioned Selection. Here we take the video-to-text retrieval as an example to illustrate this. For given frame-wise video feature f ∈ RT×Dv , we conduct mean pooling on the frame dimension to obtain the video-level features f ∈ R1×Dv . Then we select the most similar sentence w∗ ∈ W as follows.
w∗ = argmax w∈W
( Proj(f) ·w⊺ ) , (5)
where w ∈ R1×Dt is the candidate sentence features. Proj(·) is a linear projection layer to transform the visual dimension Dv to the textual dimension Dt.
Then, we compute the sentence-patch similarity and select the top K responded patches.
St = arg topk t∈[1,T ] (Proj(ut) ·w∗⊺) , (6)
where St is the filtered patch index set. Asynchronous Self-Attention. Then we only apply the proposed asynchronous self-attention on patches indexed by the set of St. Specifically, the query features are adapted as follows.
k̂ n
t , v̂ n t =
{ k n+γn t+δt ,v n+γn t+δt
, n ∈ St knt ,v n t , n /∈ St
(7)
where γ ∈ RN×1, δ ∈ RT×1 are layer-shared learnable parameters representing the offset distance in the patch and temporal dimension, respectively. kn+γnt+δt and v n+γn t+δt
denote the key and value features of the (n + γn)th patch in the (t + δt)th frame, respectively. k̂t, v̂t ∈ RN×Dv represents the adapted features. Finally, asynchronous selfattention is computed as follows.
Atten(qt, k̂t, v̂t) = softmax( qtk̂ ⊺ t√
Dv )v̂t, (8)
where qt is illustrated in Equation (4) while k̂t and v̂t are defined in Equation (7).

heading: 4 Experiments

heading: 4.1 Experimental Settings
text: Datasets. We validate the performance of our proposed RAP on four benchmarked datasets. 1) MSRVTT (Xu et al., 2016) contains 10,000 YouTube videos and each video is associated with 20 textual descriptions. We follow the 1k-A split (Yu et al., 2018) where 9,000 videos are used for training
and 1,000 videos for testing. 2) MSVD (Chen and Dolan, 2011) is composed of 1,970 videos. Following the official split, we used 1,200 videos for training and 670 videos for testing, respectively. 3) ActivityNet Captions (Krishna et al., 2017) covers 20,000 untrimmed videos of complex human activities with an average duration of two minutes. We report results on the “val1” split (10,009 training videos and 4,917 testing videos) as in (Gabeur et al., 2020). 4) DiDemo (Anne Hendricks et al., 2017) consists of 10,464 unedited, personal videos in diverse visual settings annotated with 40,543 text descriptions. We follow the training and evaluation protocol in (Luo et al., 2022).
Evaluation Metrics. Following the previous work (Luo et al., 2022), we evaluate the performance with standard retrieval metrics: recall at rank K (R@K, higher is better), median rank (MdR, lower is better) and mean rank (MnR, lower is better). R@K is defined as the percentage of samples for which the correct result is found in the top-K retrieved results. We set K to {1, 5, 10} in our experiments. MdR calculates the median of the groundtruth results in the ranking while MnR calculates the mean rank of all the correct results.
Implementation Details. We set the input frame length to 12, 64, 12, 64 and the caption token length to 32, 64, 32, 64 for MSR-VTT, DiDeMo, MSVD, and ActivityNet Captions, respectively. The pre-
trained CLIP (Radford et al., 2021) was adopted as the video and text encoders. BertAdam was used as the optimizer, with 0.1 proportion warmup cosine annealing, and a learning rate of 1e-4. All the models were trained for 5 epochs except on DiDeMo which was fine-tuned with 10 epochs. The temporal rank R and the number of selected tokens K were both set to 3. All experiments were carried out on 4 NVIDIA Tesla A100 GPUs.

heading: 4.2 Comparisons with State-of-the-Arts
text: The comparison results are summarized in Table 1 and Table 2. Specifically, we set three sets of comparison experiments: 1) Fine-tuning: We take the fully fine-tuned CLIP4clip (Luo et al., 2022) for comparisons. Besides, we also list the zero-shot performance of CLIP4clip, i.e., µ CLIP in Table 1, for comparisons; 2) Prompt-tuning: We compare our proposed RAP to prompt-tuning methods including CoOp (Zhou et al., 2022), VPT (Jia et al., 2022) and VoP (Huang et al., 2023). Since VPT is tailored for purely visual tasks, we experiment by fine-tuning or freezing the textual branch of CLIP, respectively; 3) Adapter: We conduct experiments with the state-of-the-art adapters including ST-Adapter (Pan et al., 2022), LoRA (Hu et al., 2021) and SSF (Lian et al., 2022). Notably, STAdapter is applied on the visual branch and the textual branch is either fine-tuned or freezed. For the experiments with CoOP, we insert 32 learnable
prompt tokens at the input of the textual encoder. The comparison results demonstrate the superior performance of our proposed RAP. For example, on the MSR-VTT dataset, our RAP surpasses the fully fine-tuned CLIP4clip by 2.2% (42.6 vs. 44.8) on R@1 with only 0.7% parameters (1.06 M vs. 151.28 M) using CLIP-ViT-B/32 backbone. Besides, we also achieve superior performance compared to current prompt-tuning and adapter-tuning methods. Although the parameters of our RAP are slightly higher than LoRA and SSF, considering the considerable performance improvement, our RAP strikes a better balance between parameters and performance.
Besides, to further probe the memory usage and computational complexity of the proposed model, we summarize the GPU memory usage during the training process and GFLOPs of the model in Table 3. For fair comparisons, we coequally set the number of input frames of each model to 12 frames and experiment with the ViT-B/32 backbone. We set up a lightweight RAP which only applies LoRM and ASA at the last four layers. As shown, compared to the fully fine-tuned Clip4clip, RAP_light remarkably reduces the tunable parameters, slightly lowers the memory footprint and boosts the performance. In brief, our RAP_light achieves the balance be-
tween computational overhead and performance, i.e., paying affordable overhead while obtaining considerable performance gains.

heading: 4.3 Ablations Study
text: We conduct all the ablation studies on the MSRVTT dataset with the ViT-B/32 backbone. The input frame number is set to 12. Component Ablations. We ablate the proposed low-rank modulation module and the asynchronous self-attention. The results are summarized in Table 4. We can conclude that both components are crucial to superior performance at the cost of negligible parameter overhead. For example, LoRM yields a 2.3% performance boost on R@1 with the cost of 0.42 M parameter (mode #1 vs. mode #3). Ablations on the low-rank decompose manner of LoRM. In Equation (1), we conduct the lowrank decomposition in the temporal dimension, and the modulation weights are with the dimension of RT×Dv , i.e., RT×Dv ← RT×R · RR×Dv . Here we ablate more decomposition options: i)
the spatial-temporal decomposition: The modulation is applied at the spatial-temporal dimension with the weight of RT×N×Dv , i.e., RT×N×Dv ← RT×N×R · RR×Dv , where T and N denote and frame number and the patch number within each frame, respectively. ii) the spatial-temporal-layer decomposition: We uniformly decompose all the modulation weights across all the layers. Specifically, the modulation weights are of the shape of RM×T×N×Dv , i.e., RM×T×N×Dv ← RM×R · RR×T×N×R · RR×Dv , where M denotes the inserted module number of all layers.
The comparison results are summarized in Table 5. From the comparison results, we observe that using temporal decomposition alone brings about the optimum performance. Additionally introducing decomposition on the spatial and layerwise dimension leads to the performance degrade. These results manifest our motivation that video data exhibits a substantial degree of redundancy in the temporal dimension. Ablations on the text-conditioned selection manners. To stabilize the training process of ASA, we propose a text-conditioned selection strategy to constrain the asynchronous attention computation within the selected top-related patch features (c.f . Sec. 3.3). For clarity, we denote this filter manner as text-top-K. Here we experiment with more visual token selection manners: i) random: randomly select K patch feature within each frame; ii) textbottom-K: For each patch token feature, we compute the sentence-patch similarity and select lowest K responded patches; iii) vision-top-K: Instead of using sentence features, we compute the simi-
larities between each patch feature and the [CLS] token feature of the frame. The filtered set is constituted by selecting the top K responsive patches; iv) vision-bottom-K: Similar to vision-top-K, we compute patch-wise similarities with [CLS] token and select lowest K responded patches; v) ∅: none of the selection strategies are used and all the patch features are wrapped. The comparison results of the above selection strategies are summarized in Table 6. We have the following findings. Firstly, not using the token selection strategy (i.e., ∅ in Table 6) causes substantial performance degradation, e.g., reaching only 41.4% on R@1. This is probably because warping each patch tokens wreaks havoc on the well-trained CLIP weights. Secondly, our proposed text-top-K policy outperforms the other ones on all five metrics. This demonstrates that selectively warping partial patch tokens in a parameterized way can better adapt the vanilla CLIP to the video scenario. Ablations on the warping manner of ASA. In Sec. 3.3, we predict the patch-wise warping distance in both the temporal and spatial dimensions. Here we ablate either of the two dimensions to see the difference. As shown in Table 7, restricting warping in either temporal or spatial dimension will lead to performance degradation, which demonstrates that free-form patch-wise warping is crucial to the final performance. Ablations on plug-and-play performance. Both the proposed LoRM and ASA modules serve as plug-and-play modules and can be compatible with versatile CLIP-based methods. To demonstrate this, we conduct experiments based on the more advanced CLIP-based method X-CLIP (Ma et al.,
2022). Specifically, we freeze the CLIP backbone and then insert LoRM within each Transformer layer and replace the vanilla self-attention with our proposed ASA. The comparison results in Table 8 show that our LoRM and ASA can consistently benefit the retrieval performance even with the more advanced X-CLIP baseline. Besides, compared to the fully finetuned counterpart, our proposed LoRM and ASA show great advantages in terms of tunable parameters. Ablations on hyper-parameters. We conduct ablation studies on the temporal rank R and selected token number K in Table 9. We set R = 3 and K = 3 to achieve the best retrieval performance.

heading: 5 Conclusions
text: In this work, we present RAP to efficiently transfer the pre-trained CLIP model to TVR. To accommodate the inherent video structure and the crossmodality setting, we introduce a low-rank modulation module to achieve the frame-wise sparse representation and an asynchronous self-attention module to enhance the cross-frame correlations. Extensive experiments illustrate that RAP achieves comparable or even better performance than previous arts and the fully fine-tuned counterpart.
Impact Statements
Ethics Statement. Our RAP aims to conduct parameter-efficient text-video retrieval through a temporally sparse and correlated adapter. The ethical issues may exist in the following two perspectives. Firstly, similar to many data-driven methods, there are concerns about the issue of data privacy, anonymization, and compliance with relevant data protection regulations. Secondly, the considerations related to potential bias in the dataset and the retrieval model, especially concerning sensitive topics, should be acknowledged. We are transparent about the ethical considerations in our research to uphold the integrity of the academic process and to ensure that this work aligns with ethical standards and norms in the field. Limitation. Despite the remarkable progress, our RAP still faces several limitations. Firstly, we use the text-conditioned selection to filter the most representative visual patches. Due to the semantic gap conveyed by the textual and visual signals, the alignment of complex concepts and contexts across different modalities should be conducted in a more fine-grained manner. Secondly, due to
the limitations of computing resources, we experiment with the backbone of ViT-B/32 and ViT-B/16. The salable experiments on ViT-L/14 and ViT-E/14 backbones are left for future work.

heading: Acknowledgements
text: This work was supported by the Natural Science Foundation of China (No.62202014, 62332002), Shenzhen Basic Research Program under Grant JCYJ20220813151736001

heading: A Appendix

heading: A. More Experiments
text: Video-to-text Performance: We supplement the video-to-text performance on the DiDeMo, MSVD and ActivityNet Captions datasets in Table 10. The experimental results consistently demonstrate the superiority of our RAP. For example on the MSVD dataset, RAP surpasses the fully fine-tuned method by 8.6% on R@1. Low-rank modulation on textual features: In Sec. 3.2, we apply the low-rank decomposition modulation in the visual branch, specifically in the temporal dimension. Here we apply the lowrank modulation on the textual branch to see the differences. Concretely, the modulation weights are with the shape of RW×Dt ← RW×Rt · RRt×Dt , where W denotes the total word length, Rt is the low-rank hyper-parameter and Dt is the textual feature dimension. We set Rt = 3.
The ablation results are shown in Table 11. As shown, applying the low-rank modulation on textual features causes performance degradation, which may be because word-level representations do not exhibit the same redundancy as frame-level visual features.
B. Visualization Results
We visualize the per-frame modulation weights with or without the low-rank decomposition. As shown in Figure 4, the modulation weights with decomposition demonstrate more salient distributions, which manifests the temporal sparsity characteristic of video data.
Besides, we visualize the effect of the asynchronous self-attention. We randomly select one patch feature ( ) in the frame and compute its similarity distribution with the patches in other frames. The results in Figure 5 show that the proposed asynchronous self-attention can adaptively attend to patch-of-interest, which leads to effective temporal correlation modeling.
Query: A man extinguishes a fire outside. w/ vanilla self-attention

heading: Future Work
text: The salable experiments on ViT-L/14 and ViT-E/14 backbones are left for future work