TempMe: Video Temporal Token Merging for Efficient Text-Video Retrieval
Authors: Leqi Shen
Tianxiang Hao
Tao He
Sicheng Zhao
Yifeng Zhang
pengzhang liu
Yongjun Bao
Guiguang Ding
Venue: ICLR 2025 Poster (2025)
Keywords: Text-Video Retrieval
Efficient Fine-tuning
Temporal Redundancy
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Most text-video retrieval methods utilize the text-image pre-trained models like CLIP as a backbone. These methods process each sampled frame independently by the image encoder, resulting in high computational overhead and limiting practical deployment. Addressing this, we focus on efficient text-video retrieval by tackling two key challenges: 1. From the perspective of trainable parameters, current parameter-efficient fine-tuning methods incur high inference costs; 2. From the perspective of model complexity, current token compression methods are mainly designed for images to reduce spatial redundancy but overlook temporal redundancy in consecutive frames of a video. To tackle these challenges, we propose Temporal Token Merging (TempMe), a parameter-efficient and training-inference efficient text-video retrieval architecture that minimizes trainable parameters and model complexity. Specifically, we introduce a progressive multi-granularity framework. By gradually combining neighboring clips, we reduce spatio-temporal redundancy and enhance temporal modeling across different frames, leading to improved efficiency and performance. Extensive experiments validate the superiority of our TempMe. Compared to previous parameter-efficient text-video retrieval methods, TempMe achieves superior performance with just 0.50M trainable parameters. It significantly reduces output tokens by 95% and GFLOPs by 51%, while achieving a 1.8X speedup and a 4.4% R-Sum improvement. With full fine-tuning, TempMe achieves a significant 7.9% R-Sum improvement, trains 1.57X faster, and utilizes 75.2% GPU memory usage. The code is available at https://github.com/LunarShen/TempMe.

Published as a conference paper at ICLR 2025
TEMPME: VIDEO TEMPORAL TOKEN MERGING FOR EFFICIENT TEXT- VIDEO RETRIEVAL
Leqi Shen Tianxiang Hao
Tao H
Sicheng Zhao
Yifeng Zhang
Pengzhang Liu
Yongjun Bao
Guiguang Ding 1.2
1 School of Software, Tsinghua University 2 BNRist, Tsinghua University
3 Hangzhou Zhuoxi Institute of Brain and Intelligence
4 JD.com GRG Banking Equipment Co., Ltd.
South China University of Technology
ABSTRACT
Most text-video retrieval methods utilize the text-image pre-trained models like CLIP as backbone. These methods process each sampled frame independently by the image encoder, resulting in high computational overhead and limiting practical deployment. Addressing this, we focus on efficient text-video retrieval by tackling two key challenges: 1. From the perspective of trainable parame- ters, current parameter-efficient fine-tuning methods incur high inference costs; 2. From the perspective of model complexity, current token compression meth- ods are mainly designed for images to reduce spatial redundancy but overlook temporal redundancy in consecutive frames of video. To tackle these chal- lenges, we propose Temporal Token Merging (TempMe), a parameter-effiiient and training-inference efficient text-video retrieval architecture that minimizes train- able parameters and model complexity. Specifically, we introduce a progressive multi- graniiatrtllrrtt framework. By gradually combining neighboring clips, we re- duce tertoral redundancy and enhance temporal modeling across different frames, leading to improved efficiency and performance. Extensive experiments validate the superiority of our TempMMe. Compared to previous parameter-efficient text-video retrieval methods, TempMe achieves superior performance with just 0.50M trainable parameters. It significantly reduces output tokens by 95% and GFLOPs by 51%, while achieving 1.8x speedup and 4.4% R-Sum improve- ment. With full fine-tuning, TempMe achieves significant 7.9% R-Sum im- provement, trains 1.57x faster, and utilizes 75.2% GPU memory usage. The code is available http::ttp//.m btttps://githnn com htam//lanarShee/Terret
INTRODUCTION
In the domain of video-language understanding. text-video retrieval a crucial task focused on matching videos that correspond to specific query texts vice Given the powerful capacity of large-scale text-in text-image pre- pre-trrreiiie training[Fat et al. (2024); Radford et al. (2021): Xu et al. (2024a), recent studies [uo al. (2022): Wang et al. (2023): y Pei
(2023): Jin et al. (2022) have increasingly focused on full fine-tuning of Radford et al. (202T for text-video retrieval. Specifically, these methods incorporate cumbersome modules extract video and text representations. However, the slow inference speed of these methods severely limits their real-world applications. For example, the training process of CLIP4Clip Luo al. (2022) with CLIP-ViT-B// r GB GPU memory usage and takes 6.5 hours. Therefore, we focus on efficient fine-tuning tixt-video retrieval Ju et al. (2022); ; Huang al.) (2023); Yang et al. (2024) lower computational expenses, which freeze the pre-trained backbone and introduce minimal trainable parameters to model video data.
Efficient adaptation of text-image pre-trained models for text-video retrieval remains challenging due to the inherent differences between image and video modalities. The shift from processing single image to handling multiple sampled frames dramatically raises the number of patch tokens fed into the model, complicating the extraction of video representations. As a result, this adaption
"qqual contribution. Emails: lunarshene gmailmmaallll com, beyondhix@@aa sernchhteggmmal... com "ooresponning authors. Emails: schzhao@gma com, dingggEts nnghua edu.ch

---

Published as a conference paper at ICLR 2025
Cat
Monkey
Background
Wall
time
(e) An example showcasing lemporal redundancy.
Image hmonder Layer
V
Te
Image Fnoodert I aper V5
Vg
204
R-Sum
202
VoP
LoRA
DiffRate
Image Enonder I aper V22
200
DGL
ToMe
158
GFLOPs &
Image Eucoder Layer
Image Encoder Layer
Adapter
200
154
Prompt
(b) Existing vixt-video retrioval methods.
Our TempMe.
Performance comparison.
Figure 1: (a) An example illustrates the large temporal redundancy between adjacent frames. Iden- tical subjects are highlighted the same color. (b) Current methods treat video input as a sequence of multiple sampled frames, causing high complexity due the large number of tokens. (c) In contrast, our TempMe reduces temporal redundancy by progressively merging redundant tokens in adjacent video clips. (d) With CLIP- ViT-B16 on MSRVTT, our TempMe reaches stte-of-the-as performance with minimal computational overhead. R-Sum is the sum of R @ 1, R@5, and R @@ 10.
suffers from high computational overhead, which makes practical deployment difficult. As shown in Figure Ib, existing parameter-efficient text-video retrieval methods Huang et al. (2023); Yang al. 2024); Jin al. 2024); Cao et (2024) treat video input sequence of multiple sampled frames, achieving competitive accuracy but at the cost of inference efficiency.
Motivated by these limitations, we argue that temporal redundancy plays a major role the model's high complexity when employing the pre-trained image encoder for video inputs. The sequentially sampled frames in video capture a continuous temporal progression, illustrating subject evolution, interaction, and transition over time. In Figure D the video shows a cat and a monkey fight- ing, showing the same subjects in adjacent frames against a repetitive background. This temporal progression introduces substantial redundancy due to repeated information in consecutive frames, contributing to the high complexity resulting from repeated model calculations.
To mitigate computational complexity, recently popular technique, token compression Rao et al. (2021): Kong et al. (2022); Chen et al. (2023a); [Bolya et al. (2022), might be a fascinating solution, which has achieved huge success in accelerating image models. Classically, token compression methods aim to merge redundant spatial tokens in the image to improve efficiency. However, they fail to resolve the unique challenges imposed by video modality. Specifically, they overlook the temporal redundancy across different frames, leading to inferior performance. Currently, there is no dedicated compression algorithm designed for text-video retrieval models, and the efficiency issue urgently needs to be addressed.
In conclusion, existing methods struggle to achieve efficient text-video retrieval: (1) Parameter- efficient methods still lead to high inference costs. (2) Token compression methods fail to address temporal redundancy. In this paper, we propose Temporal Token Merging (TempMe), which effec- tively integrates parameter-efficient tuning with token compression to overcome these limitations. Particularly, our method adopts a progressive multi-granularity framework, enabling the efficient merging of redundant temporal tokens, as visualized in Figure De. Videos can be viewed as aggrega- tions of clips with varying temporal granular levels, from individual images at the micro-level to the entire video the lecrl- Through gradually combining neighboring clips, the total number of tokens is dramatically decreased, while the features evolve from fine- fine-graing grained image-level features to holistic video-level features, leading lower complexity and better performance.
2

---

Published as a conference paper at ICLR 2025
To validate the advantages of our TempMe, extensive experiments conducted on four benchmark datasets, MSRVTT Xu et al. (2016), ActivityNet Krishna et al. (017), DiDeMo Anne Hendricks et al.] (2077) and LSMDC Rohrbach et al. (2013). Experimental results consistently demonstrate that our Me offers a leading balance between speed and accuracy, outperforming other efficient fine-tuning methods. Figure [ shows the performance comparison of CLIP- VIT-BB11 on MSR- VTT. Compared to VoP Huang et al. (2023) and DGL Yang et al. (024). TempMe significantly reduces output tokens by 95% and GFLOPs by 51%, while achieving 1.8x speedup and 4.4% R-Sum improvement. Moreover, TempMe can be seamlessly integrated with various parameter- efficient fine fine-tuning (see Appendix (C2) and full fine-tuning (see Table 55), which demonstrates its robust generalization capabilities. When fully fine-tuning with -LLP-VVT-B/16. TempMe achieves significant 7.9% R-Sum improvement, trains 1.57x faster, and utilizes 75.2% GPU memory usage.
Our major contributions are summarized as follows: (1) Our study reveals that temporal redundancy in video content causes excessive computational demands when adapting pre-trained text-image models for text-video retrieval. Existing parameter-efficient methods incur high inference costs, while current token compression methods fail to address temporal redundancy. (2) To overcome such limitations, we propose TempMe, a parameter- and inference-efficien text-video retrieval chitecture that merges redundant temporal tokens in adjacent video clips step by step, simplifying model complexity while extracting unified and discriminative video features. (3) Results four benchmarks reveal that TempMe outperforms current SOTA methods, including both parameter- efficient video-text retrieval and compression methods. Extensive experiments indicate that our TempMe effectively reduces model complexity while achieving superior performance.
2 RELATED WORKS
Tixt-Video Retrieval. Due to the great success of language-vision pre-training Wang al. (002b): Xue et al.] (2022); Chen et al. (202bb); Li et al. (2023); Fan et al. (2024); Radford et al. 20221 ; Xu et al. (2024a): Li et al. (2021) 2022) in various downstream cross-modal tasks Antol cet al. 2015 Xu et al. (2015 2015); Karpathy & Fei-Fei (2015); Shen et al. (2024), numerous studies Luo al. 2022 Gorti et al. (2022 g Liu et al. (2022): : Wang al. (2023 Guan et al. (2023): Jin et al.
(2023b): Pei et al. 2023) Jin et al. (2022); Shen et al. (2025) Lei et al. (2021) have achieved re- markable results by utilizing CLIP's s Radford et al. (2021 knowledge for text-video retrieval. These approaches typically employ CLIP's encoders as the backbone and use complex similarity calcula- tions. Specifically, CLIP4Clip Luo et al. (2022) introduces several video aggregation schemes to obtain video features. Cap4 Wu et al. (2023) generates associated captions to improve perfor- mance. HBI Jin et al. (203a)) formulates video and text as players in a cooperative game. However, these full fine-tuning methods with their customized modules require significant memory and com- putational resources. Therefore, our work aims to explore efficient fine-tuning methods to lower computational overhead.
Mainstream parameter-efffcient fine-tuning methods in the image domain, such as Prompt Khattak al. (2023): y Tia et al. al. (2022); /hou et al. (2022h a); HHo et al. (2024), LoRA Hu al. (2021 Zhang et al. [2022]] Xiong et al. (2024). and Adapter Houlsby et al. (2019): He et al. (2021); Chen et al.
(2022): ; Xu et al. (2024b); Hao et al. (2023b a), freeze pre-trained model parameters while adding extra tunable parameters to minimize storage demands. For parameter-efficien text-video retrieval, VoP Huang al. (2023) and DGL Yang al. (2024) introduce extra modules to generate prompt tokens and capture global video information. Despite their good performance with small tunable parameters, they face high inference costs due to the significant token number in videos.
Token Compression Token compression methods Ren et al. (2023); Liang et al. (2022) Wang
et al. 2022a):: [ong et al. (2023 : Rao et al. (202T): Kong et al. (2022): Chen et al. (2023a); Bolya et al. 2022); ; Marin et al. (202 aim to reduce computational burden by reducing tokens while
preserving the essential information.
Image token compression methods focus on pruning merging tokens in single image. DeCo Yao et al. (2024) adopts 2D Adaptive Pooling to downsample vision tokens at the spatial level for MLLMs. EVIT Liang et al. (2022) identifies the attentive tokens and fuses the inattentive tokens. DiffRate Chen et al. (2023a) performs token pruning and merging simultaneously. ToMe|Bolya et al.
2022) reduces tokens in transformer gradually by merging fixed number of tokens in each block.
3

---

Published as a conference paper at ICLR 2025
Tuzeed Farms
+ Finza Partass
Cross dlip Mentging entree clip Marqinng
+
Crces- clip Megging cllp Merging
Imzageo Envcodeer Luyver
Imaage Excoodee
Imagge Exocodar
Imrign Marging
Imngleeee Eloock
Clipidee Block
ClipoMo Block
LoRA
LoRA
LoRA
clips, with frame {v] (-
clips, each with 2 frames (v("
11-1
clip with frames {ul fn
Image-level
Video-levd
Figure 2: Overview of our proposed TempMe. We introduce Progressive Multi-Granule Granularity (PMG) framework consisting of both image merging and clip merging stages. In the image merging stage, ImgMe Block merges redundant spatial tokens within single frame. Following this, ClipMe Block progressively forms new clips from adjacent ones, facilitating video-level feature learning and re- ducing temporal redundancy by merging tokens across different frames.
Although ToMe has been applied to video classification by simultaneously processing all frame to- kens Spatiotemporal MAE Feichtenhofer et al. (2022), this implementation is incompatible with CLIP's image encoder for text video retrieval. When processing each sampled frame via CLIP's im- age encoder, image token compression methods effectively reduce spatial redundancy by merging similar tokens within single frame. However, they do not address the temporal redundancy across frames, which greatly contributes to computational overhead.
Video token compression methods Wang al. (2022a)] Ding et al. 2023 Ren et al. (2223) focus on pruning or merging tokens across frames. STA Ding et al. (2023) considers temporal redundancy and semantic importance, which tailored for video architectures where all frame tokens are processed jointly. TESTA Ren et al. (2023) introduces temporal aggregation for video-language pre- pre-training. We re-implement these video token compression methods on CLIP for text-video retrieval.
In this work, we focus on text-video retrieval using CLIP, where each sampled frame is processed as an independent token set. Existing token compression methods are limited pruning merging tokens within single token set for an image video, without addressing token compression across multiple sets or incorporating temporal fine-tuning. In contrast, we have explored a practical and feasible path to reach both superior performance and computational efficiency. By fruitfully inte- grating parameter-efficient fine-tuning and token compression techniques, we propose TempMe and reach state-of-the of-te-artt- performance. TempMe can progressively merge different frame token sets, and thus minimize apatio-tempont redundancy and enhance temporal modeling across frames.
3 METHODOLOGY
3.1 PRELIMINARIES
Text-video retrieval involves leveraging textual queries accurately retrieve related videos, well as utilizing video queries to find associated textual descriptions. The primary goal is to bridge the semantic gap between textual description and video content by learning similarity function s(t, v). Here, denotes a text, and represents a sequence of sampled frame ( where F the number of sampled frames in time. We employ LoRA Hu et al. (2021) to adapt CLIP Radford cet al. (202I) for text-video retrieval tasks. CLIP employs transformer that includes both multi-head self self-attention (MHSA) and feed-forwart network (FFN). The trainable matrices of LoRA can be merged, introducing no extra inference latency. Further details are provided in Appendix A
Notably, the classic method, VoP Huang et al. (2023). does not implement specific designs for the text modality. It merely introduces prompts within the text encoder. Similar to VoP, we apply LoRA to the text encoder, while primarily concentrating the video modality.
4

---

are aggregated into four Published as a conference paper at ICLR 2025
ClipMe Block
Intra-clip Merging
1 dbp
clip x (N x R
(NR,R,
1 X
Output clip
DOODA
4
Feed Forward
1. ssign tokems in single clip S Set
Connect each token in Set.A to its most simular token in
3. Select the N x (1 1 R2)
Merge connected tokens.
x (FNR,
RD
similar pairs.
Intra-clip Merging
Cross-clip Merging
Atbention
LoRA
1 (FM x R
co
x (/N x Rc)
Cress-clip Merging
Impott
clips
time
Assign tokens 2. Connect each token most
3. Select the top x Rc]
4.Add trainable clip povitionall emboddings
Mergee und form new clip
clims to SatA Set similar token in Set B. most similar pairs.
FXNXD
Figure 3: ClipMe Block. Given an input of f clips fXNND the cross-clip merging step merges tokens from all clips to form new clip R RIXINR.X Subseguently, the intra-clip merging step merges tokens within this newly formed clip, producing R RIx/NR,RRXD If the input contains only one clip, the cross-clip merging is skipped.
3.2 TEMPORAL TOKEN MERGING
For efficient text-video retrieval, we freeze the pre-trained CLIP and merely train LoRA in both the image and text encoders. this subsection, focus on merging temporal tokens across multiple sampled frames to minimize temporal redundancy and extract more comprehensive video features.
Progressive Multi-Gramularity Framework. In text-video retrieval, the significant impact of temporal redundancy on model complexity has not been adequately explored. To mitigate tem- poral redundancy, we proposed the Progressive Multi-Granularity (PMG) framework that efficiently merges similar temporal tokens across different frames. In our PMG framework (see Figure 2 we introduce two-stage merging approach: image merging and clip merging. Initially, ImgMe block independently encodes each single frame, merging similar tokens within the same frame and lever- aging pre-trained image knowledge to capture low-level spatial details. Next, ClipMe block (see Figure 3) is proposed to aggregate short-frame clips into extended-frame clips, merging temporal tokens across different frames and learning unified and discriminative video features.
discrete video clip lasting only frame //1 Concatenating these consecutive clips yields an extended clip with increased temporal length.
Specifically, each sampled frame I can be regarded
In the toy example presented in Figure
8 individual frames
f=22 clips, each containing with two frames 2
t=1" These clips are then aggregated to form com-
As clips are progressively aggregated, we merge a significant number of similar tokens that arise from overlapping information across frames. Si- multaneously, comprehensive video information is captured through the attention mechanism that processes tokens across different frames.
plete video clip with all 8 frames
o s=1"
ImgMe Block. Inspired by ToMe Bolya et al. (0022). the image merging step merges tokens to reduce by r per layer, which is employed between the MHSA and FFN branches each ImgMe Block. Following Bipartite Soft Matching Bolya al. (2022), sets and B by alternating. For each token set A,, we find its most similar token in Set using cosine similarity. The top most similar pairs then merged by averaging their features. Given
first partition the tokens into two
5

---

Published as a conference paper at ICLR 2025
Table 1: Complexity comparisons. We report the R-Sum metric for 12 sampled frames on MSRVTT and 64 sampled frames on ActivityNet, respectively. The best metric is highlighted in bold.
Backbone
CLIP B/3
CLIP VIT
* Feames
12
64
12 Tokens
Methed
GFLOPs
Tekens
R Sum
GFLORSs
e Tokens
R8//RRSm
GFLOPS
R L/R Sum
LoRA
53.0:100%)]
12x.0 50 (100%)
43.7/192.0
276.7 (100%)
64
50 (100%)
38.2/191.5
211.3 (100%)
12 197 (100%)
47.3/201.4
DimRate
36.6 6996
12 x2xx20 26 4298)
41.5/189.9
199.1 69%)
64
20 40%B)
38.1/18877
138.5 66%)
12x 49 25%3
47.3 47.3/220.4
ToMe
40.2; 76461
12 26 52%)
42.9/191.
208.5 75%8)
64
25 52%6
38.6/189.6
146.4 68%)
12
771 2998)
46.2/200.6
TompMe
34,8 68%)
97 16%)
46.1/198.6
180.3 68%)
x 500
16%)
44.9/28..6
121.41 57%)
1xx 1277
5%)
49.0/206.7
an input IIINXXD the image merging stage with K ImgMe Block outputs RIx(N-rK)DD, where N denotes the number of tokens, and D denotes the dimension of the token feature.
To prevent loss of image information, is set to a low value: 2 for CLIP- VIT-B/22 and r = 10 However, the model complexity remains high due to the processing of multiple for sampled frames. Therefore, we aim to merge temporal tokens to further reduce model complexity.
Clip.M Block. Instead of ImgMe Block which merges tokens in each single frame, we propose ClipMe Block to process multi-frame clips as shown in Figure 3 In ClipMe Block, we introduce two novel steps: cross-clip merging and intra-clip merging. During cross-clip merging, adjacent clips are aggregated. which significantly reduces the number of temporal tokens and generates a new clip. The intra-clip merging step further compresses the tokens within the newly formed clip.
In cross-clip merging, we aim to combine tokens with their most similar tokens from the same sub- ject, whether these matches occur within single frame or span across multiple frames. Initially, all tokens from the input f clips R/x x X are alternately divided into Set A or Set B. This alternating assignment scheme is based on the observation that the adjacent tokens often depict the same subject, thus their features tend to be similar. Moreover, this assignment ensures that tokens representing the same subject across various frames are distributed between both sets. Considering the high tempo- ral redundancy observed in videos, we select a large subset, fN x (1 Re), of the most similar token pairs for merging, where Rg denotes the ratio of kept tokens in cross-clip merging. Before merging, the trainable clip positional embeddings are added to facilitate temporal understanding in MHSA. Subsequently, these merged tokens, along with the unmerged tokens, constitute a new clip [xx((fNX.)xx)) which is then fed into MHSA.
During intra-clip merging, the tokens in the newly formed clip, improved by MHSA for better tem- poral and spatial details, are further reduced. In contrast to the image merging step that merges small number of tokens in an individual frame, the intra-clip merging step merges a larger propor- tion of tokens within a clip containing multiple frames, compressing R R X into [Ix(NNxRixxD, where R2 denotes the ratio of kept tokens in intra-clip merging.
3.3 COMPLEXITY ANALYSIS.
To verify the efficiency of our proposed TempMe, we perform detailed complexity analysis in Table [ Following common protocols Luo et al. (2022); Yang et al. (2024), the frame length is set to 12 for MSR-VTT and LSMDC, and to 64 for ActivityNet and DiDeMo. The complexity comparison between ToMe and our TempMe for these frame lengths is shown in Table Compared to ToMe, our TempMe reduces tokens by more than 30% for both CLIP-ViT- B/32 and CLIP- ViT- B/16, effectively decreasing model complexity while significantly surpassing accuracy. Particularly, with a 12-frame length and CLIP-VTT-B/16 backbone on MSRVTT, TempMe outputs only 5% of the input tokens, reaches 57% GFL.Os,, and achieves 5.3% R-Sum gain.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Following common practice, we perform experiments on four widely used bench- marks for text-video retrieval: MSRVTT Xu et al. (2016), ActivityNet Krishna et al. (2017),
Datasets.
DiDeMo Anne Hendricks et al. (2017), and LSMDC Rohrbach et al. (2015). Detailed descriptions
of these datasets provided in Appendix B

---

Published as a conference paper at ICLR 2025
Table Comparisons in the text-to-video task MSRVTT. We evaluate GFLOPs only for the video backbone during inference. A detailed comparison of memory usage can be found in Appendix C.7
Methods
Parums (M)
GFLOPs
R@lt
R@st
R@ 101
R-sumt MnR.
CLIP- ViT-B/32
Fine-tuning
CLIP4Clip
123,54
43.1
70.4
80.8
194.3
16.2
Prompt
0.08
58.2
40.4
66.3
77.3
184.0
16.7
Adapter
0.26
53.1
41.9
69.9
78.7
190.2
14.9
LoRA
0.49
53.0
43.7
68.9
80.4
193.0
16.0
Enrameter-Fffiiien
PLEVU
6.35
36.7
64.6
76.8
178.1
Vop++0
14.10
44.6
69.9
80.3
194.8
16.3
DGL
0.83
44.7
70.5
79.2
194.4
16.2
DGL
9.57
45.8
79.4
194.5
16.3
EVIT
0.49
37.2
41.4
69.0
78.1
188.5
16.8
Parameter-Eiffcicet
DiffRate
0.49
36.8
41.5
68.6
79.8
189.9
16.3
STA
0.49
35.7
42.6
69.5
78.8
190.9
17.0
&
Inference-Efficien
ToMe
0.49
40.2
42.9
68.3
80.2
191.4
16.2
TESTA
0.59
43.7
79.4
16.8
TempMe
0.50
34.8
46.1
71.8
80.7
198.6
14.8
CLIP CLIP-ViT-B/IK
MV-Adapter
3.6
>210
46.0
72.0
82.1
200.1
RAP
1.06
>210
46.5
73.9
82.0
202.4
12.1
Purameter-Efficceen
VoP
14.10
246.2
47.7
72.4
82.2
202.3
12.0
DGL
0.83
48.3
71.8
80.6
200.7
13.4
Param&Iniee Efficien
TempMe
0.50
49.0
74.4
83.3
206.7
11.9
Table 3: Comparisons in text-to-video task on ActivityNet, DiDeMo, and LSMDC with CLIP-
1/3-B/32. CLIP4Clip is classic full fine-tuning method.
Methods
NetivityNet
DiDeMo
LSMDC
RØ11
RØ5T
R
MnR.
R
RØ5T
RØ T0T
MnR
Rw
RwsT
RØ 1O7
MnR
CLIP4Clip
40.5
72.4
7,4
43.4
70.2
80.6
17.5
20.7
38.9
47.2
65.3
Prompt
36.0
79.4
10.2
39.4
67.6
78.3
20.6
19.1
38.2
46.1
67.5
Adapter
37,8
69.0
81.9
41.7
68.0
79.3
19.9
21.3
38.7
48.3
60.7
LoRA
38.7
70.6
82.2
8.7
41.6
68.8
78.6
20.8
20.7
39.5
47.4
68.3
PLEVU
64.8
13.4
29.5
40.3
Vop
35.1
63.7
77.6
11.4
46.4
71.9
81.5
13.6
21.1
40.9
49.6
60.1
DGILE
38.6
69.2
81.6
9.0
21.4
39.4
48.4
64.3
DifRRate
38.0
69.1
81.6
40.8
67.8
77.8
20.6
20.2
37.4
47.6
70.0
ToMe
38.6
81.2
40.5
68.6
77.8
20.9
20.5
38.8
46,8
67.8
TempMe
44.9
75.2
85.5
6.8
48.0
72.4
81.8
13.7
23.5
41.7
51.8
53.5
Metrics. We evaluate the performance using metrics such Recall at K (R@ R@5, and R @ 10), the sum of these recalls (R-Sum), and Mean Rank (MnR). higher recall score 1 indicates better performance, while lower MnR score J. denotes better performance. We evaluate GFL.Oss (Giga Floating-PPiin Operations per Second) and throughput only for the video backbone during inference. All throughputs measured on a 1000.
Implementation Details. Following previous works Luo et al. (202); Ju al. (202)): Huang et al. (2023); Yang et al.] (2024). we use the pre-trained CLIP as the backbone. We employ the AdamW optimizer Loshchilov Hutter (2016) with a batch size of 128. The initial learning rate is set to 6e-4 with cosine learning rate schedule Goyal et al. (2017) for 5 epochs. The dimension of LoRA is set to 8 in experiments. The ImgMe Block employs r = for CLIP-ViT-B/2 and = 10 for CLIP-VTT-B/1. The ClipMe Block employs RG 70%, R2 90% for ViT-B/32 and Rc = 60%, R2 80% for CLIP- For short video retrieval datasets like MSRVTT and LSMDC, the max word and frame lengths set to 32 and 12. 12 frames merged into 6 clips at layer 9. These clips are then merged into clips at layer 10, and finally into a single
clip layer 11. The cross-clip merging step is skipped in the last layer. This process is denoted as 12 2, 10, 3 11, 1. For long video retrieval datasets like ActivityNet and DiDeMo, the max word and frame lengths are set to 64 and 64. We employ 64 2, 16 10, 11, 1. Unless noted otherwise, CLIP-ViT-B/32 is employed as the backbone.
Compared Baselines. We compare TempMe with several strong baselines: (1) CLIP4Clip Luo et al. (2022), a classic full fine-tuning method. For a fair comparison, we only consider the parameter-free type that does not introduce additional modules. (2) Prompt Khattak

---

Published as a conference paper at ICLR 2025
Table 4: Computational overhead comparisons of the CLIP-ViT-BI/ V backbone. We evaluate the text-to- tovieo accuracy metrics on MSRVTT.
Table 5: Generalization analysis of full fine- tuning on MSRVTT with CLIP-ViT-B/16. The accuracy metrics are evaluated in the text-to- video task. Each experiment was conducted five
Methods
videos/s
GFLOIs
Tokens
R I/R Sum!
Prompt
29.7
216.8
2369
44.3/194.2
times with different random seeds. denotes our
Adapter
29.4
211.7
44.9/196.8
LoRA
30.6
211.3
2364
47.3/201 4
own re-implementation.
Vop*+c
DGL
25.0
3.3
251.2
246.2
2416
2368
47.7/202.3
48,3/200.7
Methods
GELDOSs
Speed
Train
Train Memory
Infer Memory
Ro: 1/R-Sum]
DiftRate
40.6
138.5
588
47.3/202.4
CLIPAClip
211.3
1,00x
70.1GB
8666BB
47,3/202.3
ToMe
40.8
144.4
924
46,2/200.6
TTMee
144.4
1.38%
61
8.5GGB
47.2/2016
TempMe
45.1
121.4
49.0/206.7
ToopMMc
52.76
8.5GB
5019/210.2
Table 6: Application in video foundation models for text- video retrieval. We evaluate full fine-tuning on MSRVTT
Application in video foun- dation models for text-video QA. We evaluate full fine-tuning on MSR-QA with UMT-B/16-25M.
Table 7:
for the text-to-video task with
Train Time
Methods
GFLOPs
R@ 1
R-Sumt
Methods
GELOPS
Time
Accuracy
UMT
304.3
9.5h
51.0
212.3
UMT
304.3
6.3h
44.9
UMT4Clip
210.1
4.8h
48.8
206.7
UMT4Clip
210.1
UMT4Clip+TempMe
111.5
3.5h
51.1
209.2
UMT4Clip+TempMe
111.5
2.3h
44.6
al. 2023) LoRA Hu et al. (2011) and Adapter Houlsby al. (2019), which mainstream
parameter-efficien fine-tuning methods. (3) PLEVU Ju et al. [ (2022), VoP Huang et al. (2023), DGL Yang al. (2024). MV-Adapter Jin el. (2024), and RAP Cao et al. (2024), which are meth- ods tailored for parameter-efficien text-video retrieval. (4) EVIT
et al. (2022). DiffRate al. (222a), and ToMe Bolya al. (2022). which are token compression methods for image mod-
els. (5) STA Ding et al. (2023) and TESTA Ren et al. (2023), which are video token compression
methods. For fairness, token compression methods apply LoRA like our TempMe.
4.2 COMPARISONS WITH STATE-OF-THEE-ME METHODS
Efficient Fine-tuning. In Table 2 we compare the 12v performance MSRVTT. Our TempMe achieves significant improvements over previous methods, with a 3.8% R-Sum increase using ViT- B/32 and 4.3% R-Sum increase using ViT-B/16, while maintaining minimal GFLOPs. Table shows the 12v results on ActivityNet, DiDeMo, and LSMDC, demonstrating that TempMe consis- tently outperforms state-of-the-ee methods. Detailed v2t results are provided in Appendix C.I
Computational Overhead. In Table accuracy metries evaluated MSRVTT with CLIP- VIT-B/16. We also report the model throughput and complexity during the inference phase. During this phase, the tunable weights can be merged in LoRA. DiffRate, ToMe, and TempMe (see Sec- tion 3.1. For fairness, prompt generation VoP and DGL. is omitted when evaluating throughput and complexity. Compared LoRA, ToMe reduces tokens per frame 39%, with minor perfor- mance degradation. However, our TempMe further reduces temporal tokens across frames to just 5%, while also improving the R-Sum by 5.3%. Unlike VoP and DGL, which compromise efficiency for performance, our TempMe achieves a 1.8x speedup over VoP and 13.7x x speedup over DGL, while reducing GFLOPs by 51% and improving R-Sum by 4.4%.
4.3 GENERALIZATION ANALYSIS
Full Fine-tuning. Table shows that our TempMe can be applied to full fine-tuning. The image encoder must process each sampled frame individually, resulting in significant GPU memory usage and training times. ToMe reduces spatial redundancy boost model speed but suffers from a minor performance decrease. In contrast, our TempMe not only reduces temporal redundancy but also learns video-level features, with a 1.57 speedup and a 7.9% R-Sum gain.
Application in Video Foundation Models. We extend our method to video foundation methods, such as LM et al. (2023). TempMe is built upon the text-image CLIP model, which processes each sampled frame individually. This contrasts with UMT, which processes all frame tokens simul-
8

---

Published as a conference paper at ICLR 2025
Table 8: Ablation on each component (2v on MSRVTT with CLIP-ViT-BAA CLIP-straight denotes the zero-shot performance.
Table 9: Ablation on each function in 12v on MSRVTT with Modeling is shown in Figure 6 of Appendix.
Temporal
Mothods
GFLOPs
R@I1
R-Sum
MnR.
Methods
GFLOP:
R
R R-Sum
MnR.
CLIP Straight
30.8
147.9
41.8
LoRA
43.7
193.0
16.0
*LoKA
55.0
43.7
1930
16.0
Temporal Mideling
54.3
46.3
199.7
14,8
+ligMe
40.2
42.9
191.4
16.2
Token Reduction
41.6
16.6
+ClipMe
34,8
46.1
198.6
14,8
TempMe
34.8
46.1
198.6
14.8
taneously. To align with our method, we introduce a new baseline UMT4Clip. where UMT handles frames separately. These differences in token processing strategies inevitably lead to a decline in performance. Following the practice of UMT, we conduct fully fine-tuning experiments on both text-video retrieval and video QA tasks. In Table 6 our TempMe achieves performance close to UMT while requiring only one-third of the GFLOPs and training time. Morcover, we also extend our method to video QA in 7 These results demonstrate the generalizability of our method.
More Generalization Analysis. To further validate the generalization capability, we integrate our TempMe with other parameter-efffiient fine-tuning methods such as Prompt and Adapter in Ap- pendix C.2 Moreover, we evaluate our TempMe on other backbones. These detailed experimental results provided in Appendix C.3
4.4 ABLATION STUDY
We conduct an analysis of TempMe from structural and functional perspectives, respectively.
Ablation on Each Component. TempMe is architecturally divided into the ImgMe and ClipMe blocks. Table g illustrate the impact of each component. CLIP-straight denotes the zero-shot per- formance of CLIP, and LoRA is employed as the baseline for efficient fine-tuning. The ImgMe block reduces spatial redundancy in each frame. The ClipMe block aggregates short-frams clips into extended-frame clips, facilitating temporal learning while minimizing temporal redundancy. Finally, our TempMe outperforms LoRA by 5.6% R-Sum and reduces 18.2 GFLOPs.
Ablation on Each Function. From a functional perspective, TempMe can be categorized into Temporal Modeling and Token Reduction, aimed at improving accuracy and efficiency, respectively. Table demonstrates the impact of each function. (1) The accuracy improvements are attributed to Temporal Modeling, which aggregates clips progressively to enhance spatio-temporal learning. In this framework, the attention modules of the early layers are applied to intra-frame tokens in the spatial domain. In the later layers, they operate on tokens across frames in the spatio-temporal do- main. (2) The efficiency improvements arise from Token Reduction, which reduces redundancy in the entire framework. In the early layers, it slightly reduces intra-frame tokens to decrease spatial redundancy. In the later layers, it significantly reduces tokens among frames to address large tempo- ral redundancy. (3) Without the support of Temporal Modeling, Token Reduction alone significantly reduces tokens in the spatial domain, leading to a substantial 4.2% decrease in R-sum. However, due to the considerable temporal redundancy, the combination of Temporal Modeling and Token Reduction (TempMe) significantly reduces complexity while achieving high performance.
Ablation on Merging Strategy. Table 10 validates the effectiveness of the merging strategy uti- lized in our PMG framework. (1) Progressive Merging, In A1, we observe that merging all clips once leads high model complexity, due the quadratic complexity of MHSA with the surge in token numbers. Alternatively, the progressive merging of clips benefits video-level information. Our AO with lower complexity achieves better performance. (2) Holistic Merging. We conduct experiments in A2 to evaluate the effectiveness of merging all clips. A2 shows that a notable perfor- mance decline when frames are not fully merged (e.g.. 12 frames into or frames). These findings indicate that encoding all frame information simultaneously in MHSA is crucial for holistic video understanding. (3) Continuous Merging. Experiments on merging with various gaps are conducted in A3. Larger gaps trigger earlier start of the clip merging process, which leads to a higher to- ken reduction but also a drop in performance. In conclusion, we adopt progressive, holistic, and continuous merging (AO) as our final strategy.
9

---

Published as a conference paper at ICLR 2025
Table 10: Ablation analysis of different merging strategies on MSRVTT with CLIP-VIT-B/32. We adopt progressive, holistic, and continuous merging (A0) as our final strategy.
Text-to- Video
Video-to Text
Methods
GFLOPs
R@
R
RT
MnR.
RwIT
R@ST
Rw Rw TOT
MnR.
A0
12
4 6
14
11
34.8
46.1
71.8
80.7
14.8
45.6
72.4
81.2
10.2
12
4 4
35.4
45.7
71.2
80.5
14.8
45.0
70.8
81.7
11.1
12 9
1
37.0
44.7
69.3
80.8
15.8
44.6
70.7
80.4
11.2
12 8 6
_
35.4
43.6
70.2
79.8
15.9
43.5
72.1
81.0
11,4
12 94
36.8
43,9
70.5
81.2
15.6
43.8
71.6
81.5
11.2
12 5 6
9
41
31.9
44.4
71.2
80.5
15.7
43.8
72.2
81.7
11.4
A3
12 4
6
10
25.8
42.5
69.5
79.2
15.5
42.6
70.3
80.4
11.3
12 4
6
31
18.1
40.8
68.1
78.6
15.8
40.6
67.8
79.2
12.5
Input
ToMe
TempMe
Figure 4:
Qualitative comparisons on MSRVTT with CLIP-ViT-B/66. Patches that share the same inner and border color are merged. TempMe merges tokens of similar elements across frames.
More Ablation Analysis. We further conduct a series of ablation studies. First, since the trainable parameters of TempMe using CLIP-VTT-B/2 include LoRA (~0.49M) and clip positional embed- dings (~0.01M), we perform ablation on the clip positional embeddings of the ClipMe block in Appendix C.4 Next, we provide detailed analysis of the intra-clip and cross-clip merging steps within the ClipMe block in Appendix C.5) Furthermore, given that the number of sampled frames affects the model complexity, we conduct an ablation study on frame sampling in Appendix [C.6 Finally, we report memory usage during training and inference in Appendix
4.5 QUALITATIVE RESULTS
Figure 4 presents visualization results of ToMe and our TempMe. ToMe only merges spatial tokens within individual frames, leaving a large number of redundant tokens across frames, contributing to high computational overhead. In contrast, TempMe merges tokens of similar elements across contiguous frames. For instance, TempMe merges similar body regions of Patrick Star across frames, such the head, upper body, and lower body. These visualization results show that our TempMe effectively reduces temporal redundancy. More qualitative results are provided in Appendix C.8
5 CONCLUSION
We explore temporal redundancy for efficient text-video retrieval. A parameter- and inference- efficient text-video retrieval method, Temporal Token Merging, is proposed to reduces model com- plexity while simultaneously extracting comprehensive video features. We introduce innovative progressive multi cuatiigranularityy framework to merge redundant tokens in consecutive clips and encode information from various frames. Extensive experiments validate the superiority and generalizability of TempMe. Further discussion on the limitations and broader impacts is provided Appendix D In the future, aim extend the application of proposed method other video tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the Key R & D Program of Xinjiang, China (2022B01006), Zhe- jiang Provincial Natural Science Foundation of China (No. LDT23F01013F01), the National Natural Science Foundation of China (No. 62441614), CCF-DiDi GAIA Collaborative Re- search Funds, China Postdoctoral Science Foundation (2024M750565), Guangdong S & Pro- gram (2024B0101040008) and the Key Realm Research and Development Program of Guangzhou (No.2024B010007)
REFERENCES
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Rus- sell. Localizing moments in video with natural language. In Proceedings of the International Conference on Computer Vision, 2017.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the International Conference on Computer Vision, 2015.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and im- age encoder for end-to-end retrieval. In Proceedings of the International Conference on Computer Vision, 2021.
Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffmann. Token merging: Your vit but faster. In Proceedings of the International Conference on Learning Representations, 2022.
Meng Cao, Haoran Tang, Jinfa Huang, Peng Jin, Can Zhang, Ruyang Liu, Long Chen, Xiaodan Liang, Li Yuan, and Ge Li. RAP: Efficient text-video retrieval with ppars-and-correeltte adapter. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Com- putational Linguistics ACL 2024, pp. 7160-7174, 2024.
Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo. Diffrate: Differentiable compression rate for efficient vision transformers. In Proceedings of the International Conference on Computer Vision, 2023a.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 2022.
Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: omni-modality foundation model and dataset. Advances in Neural
A ission-audio-subiite Information Processing Systems, 36:72842-72866, 2023b.
Shuangrui Ding, Peisen Zhao, Xiaopeng Zhang, Rui Qian, Hongkai Xiong, and Qi Tian. Prune opatio-temporal tokens by semantic-aware temporal accumulation. In Proceedings of the Interna- tional Conference on Computer Vision, pp. 16945-16956, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Mathhis Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 2rXiv:2010.11929, 2020.
Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Systems, 2024.
Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal learners. Advances in Neural Information Processing Systems, 2022.
Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In Proceedings of the European Conference on Computer Vision, 2020.
11

---

Published as a conference paper at ICLR 2025
Satya Krishna Gorti, Noël Vouitsis, Junwei Ma, Keyvan Golestan, Maksims Volkovs, Animesh Garg, and Guangwei Yu. X-pool: Cross-modal language-video attention for text-video retrieval. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2022.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: 02677, 2017.
Peiyan Guan, Renjing Pei, Bin Shao, Jianzhuang Liu, Weimian Li, Jiaxi Gu, Hang Xu, Songcen Xu, Youliang Yan, and Edmund Y Lam. Pidro: Parallel isomeric attention with dynamic routing for text-video retrieval. In Proceedings of the International Conference on Computer Vision, 2023.
Tianxiang Hao, Hui Chen, Yuchen Guo, and Guiguang Ding. Consolidator: Mergable adapter with group connections for vision transformer. In International Conference on Learning Representa- tions, 2023a.
Tianxiang Hao, Mengyao Lyu, Hui Chen, Sicheng Zhao, Xiaohan Ding, Jungong Han, and Guiguang Ding. Towards efficient vision-language tuning: More information density, more gen- eralizability, arXiv preprini arXiv:2312. 10813, 2023b.
Tianxiang Hao, Xiaohan Ding, Juexiao Feng, Yuhong Yang, Hui Chen, and Guiguang Ding. Quan- tized prompt for efficient generalization of vision-language models. In Proceedings of the Euro- pean Conference on Computer Vision, 2024.
Junxian He, Chunting Zhou, Xuezhe Ma. Taylor Kerr-Kiikaatrick, and Graham Neubig. Towards unified view of parameter-efficient transfer learning. In Proceedings of the International Confer- ence on Learning Representations, 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efffcient transfer learning for nlp. In Proceedings of the International Conference on Machine Learning, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
arXiv:2106.09685 2021.
Siteng Huang, Biao Gong, Yulin Pan, Jianwen Jiang, Yiliang Lv, Yuyuan Li, and Donglin Wang. Vop: Text-video co-operative prompt tuning for cross-modal retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Proceedings of the European Conference on Computer Vision, 2022.
Peng Jin, Jinfa Huang, Fenglin Liu, Xian Wu, Shen Ge, Guoli Song, David Clifton, and Jie Chen. pnnectation-maximizee
contrastive learning for compact aideo-and-languaaa representations. Advances in Neural Information Processing Systems, 2022.
Peng Jin, Jinfa Huang, Pengfei Xiong, Shangxuan Tian, Chang Liu, Xiangyang Ji, Li Yuan, and Jie Chen. Video-text game players: Hierarchical banzhaf interaction for cross-modal rep- resentation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023a.
Peng Jin, Hao Li, Zesen Cheng, Kehan Li, Xiangyang Ji, Chang Liu, Li Yuan, and Jie Chen. Diffu- sionret: Generative text-video retrieval with diffusion model. Proceedings of the International Conference Computer Vision, 2023b.
Xiaojie Jin, Bowen Zhang, Weibo Gong, Kai Xu, Xueqing Deng, Peng Wang, Zhao Zhang, Xiaohui Shen, and Jiashi Feng. Mv-adapter: Multimodal video transfer learning for video text retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 27 144 27153, 2024.
12

---

Published as a conference paper at ICLR 2025
Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual- mod- els for efficient video understanding. In Proceedings of the European Conference on Computer Vision, 2022.
Andrej
Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, al. Spvit: Enabling faster vision transformers via latency-awar soft token pruning. In Proceedings of the European Conference on Computer Vision, 2022.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the International Conference on Computer Vision, 2017.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for iide-aand-langusee learning via sparse sampling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafig Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems, 34:9694-9700 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In Proceedings of the Interna- tional Conference on Machine Learning, pp. 12888 12900. PMLR, 2022.
Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training- efficient video foundation models. In Proceedings of the International Conference on Computer Vision, pp. 19948-19960. 2023.
Liang, C Ge, Z Tong. Y Song, P Xie, et al. Not all patches are what you need: Expediting vision transformers via token reorganizations, In Proceedings of the International Conference Learning Representations, 2022.
Yuqi Liu, Pengfei Xiong, Luhui Xu, Shengming Cao, and Qin Jin. Ts2-net: Token shift and selection transformer for text-video retrieval. Proceedings of the European Conference on Computer Vision, 2022.
Sifan Long, Zhen Zhao, Jimin Pi, Shengsheng Wang, and Jingdong Wang. Beyond attentive tokens: Incorporating token importance and diversity for efficient vision transformers. In Proceedings of the IÉEE Conference on Computer Vision and Pattern Recognition, pp. 10334-10343, 2023.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In Pro- ceedings of the International Conference on Learning Representations, 2016.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 2022.
Dmitrii Marin, Han-Hao Rick Chang, Anurag Ranjan, Anish Prabhu, Mohammad Rastegari, and Oncel Tuzel. Token pooling in vision transformers. arXiv preprint 1rX0v:2110003800, 2021.
Antoine Miech, Dimitri Zhukov, Jean- Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 100m Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the International Conference on Computer Vision, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 807.03748 2018.
13

---

Published as a conference paper at ICLR 2025
Renjing Pei, Jianzhuang Liu, Weimian Li, Bin Shao, Songcen Xu, Peng Dai, Juwei Lu, and Youliang Yan. Clipping: Distilling clip-based models with a student base for video- Languaa retrieval. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning, 2021.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in Neural Information Processing Systems, 2021.
Shuhuai Ren, Sishuo Chen, Shicheng Li, Xu Sun, and Lu Hou. TESTA: Temporal-spatial token aggregation for long-form video-language understanding. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 932 947, 2023.
Anna Rohrbach, Mareus Rohrbach, and Bernt Schiele. The long-short story of movie description. In Pattern Recognition, 2015.
Leqi Shen, Sicheng Zhao, Yifeng Zhang, Hui Chen, Jundong Zhou, Pengzhang Liu, Yongjun Bao, and Guiguang Ding. Multi-labod learning with block diagonal labels. In Proceedings of the ACM international conference on Multimedia, 2024.
Leqi Shen, Sicheng Zhao, Yifeng Zhang, Pengzhang Liu, Yongjun Bao, and Guiguang Ding. Spatio- temporal attention for text-video retrieval. ACM Transactions on Multimedia Computing. Com- munications and Applications, 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances Neural Informa- tion Processing Systems, 2017.
Junke Wang, Xitong Yang, Hengduo Li, Li Liu, Zuxuan Wa, and Yu-Gang Jiang. Efficient video transformers with tpatial-temporal token selection. In Proceedings of the European Conference on Computer Vision, pp. 69-86. Springer, 2022a.
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:22 2022b.
Ziyang Wang, Yi-Lin Sung, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Unified coarse-to-fine alignment for video-text retrieval. In Proceedings of the International Conference on Computer Vision, 2023.
Wenhao Wu, Haipeng Luo, Bo Fang, Jingdong Wang, and Wanli Ouyang. Cap4video: What can auxiliary captions do for text-video retrieval? In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, 2023.
Yizhe Xiong, Hui Chen, Tianxiang Hao, Zijia Lin, Jungong Han, Yuesong Zhang, Guoxin Wang, Yongjun Bao, and Guiguang Ding. Pyra: Parallel yielding re-activation for training-inference efficient task adaptation. In Proceedings of the European Conference on Computer Vision, 2024.
Hu Xu, Saining Xie, Xiaoqing Tan, Pa-Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. In Pro- ceedings of the International Conference on Learning Representations, 2024a.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the International Conference on Machine Learning, 2015.
14

---

Published as a conference paper at ICLR 2025
Xinhao Xu, Hui Chen, Zijia Lin, Jungong Han, Lixing Gong, Guoxin Wang, Yongjun Bao, and Guiguang Ding. Tad: A plug-and-pla and-play task-aware decoding method to better adapt llms on down- stream tasks. In Proceedings of the International Joint Conference Artificial Intelligence, 2024b.
Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip- vip: Adapting pre- pre-trained image-text model video-language alignment. In Proceedings of the International Conference on Learning Representations, 2022.
Xiangpeng Yang, Linchao Zhu, Xiaohan Wang, and Yi Yang. Dgl: Dynamic global-local prompt tuning for text-video retrieval. Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
Linli Yao, Lei Li, Shuhuai Ren, Lean Wang. Yuanxin Liu, Xu Sun, and Lu Hou. Deco: Decou- pling token compression from semantic abstraction in multimodal large language models. arXiv preprint 2rXiv:2405.20983 2024.
Bowen Zhang. Hexiang Hu, and Fei Sha. Cross-modal and hierarchical modeling of video and text. In Proceedings of the European Conference on Computer Vision, 2018.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. arXiv:2206.0467 2022.
Neural prompt search. arXiv preprint
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 2022b.
15

---

Published as a conference paper at ICLR 2025
A PRELIMINARIES
CLIP for Text- Video Retrieval. Following the approach of previous works Luo et al. (2022), : Ju et al. (2022): Huang al. 2023) Yang et al. (2024), we utilize the pre-trained text-image CLIP model Radford et al. (2021) as the backbone, which exhibits strong performance in downstream tasks. The full CLIP is composed of both a text encoder and image encoder, employing the Transformer architecture Vaswani et al. (2017); Dosovitskiy et al.] (2020) which consists of alter- nating blocks of multi-head self-attention (MHSA) and feed-forward network (FFN). The attention module in MHSA:
Attention (x) softmax (xQ) (xV),
vd
where Dxd, K RRd, and V E DDxd three projection matrices.
Given an input text t, the text encoder first tokenizes and transforms the text description into word tokens. Then, these tokens processed through 12 text encoder layers extract the final text feature t. For an input video v, each sampled frame [ = 1, 1.... F) is separately processed by the image encoder shown in Figure Db. The image 7 is divided into fixed-size patches and projected into patch tokens, which are inputted into 12 image encoder layers to extract the frame feature f.
The final video feature is obtained by averaging all frame features (f)),
Finally, the cross-modal contrastive loss Oord al. (2088) is applied to jointly optimize in both text-to-video and video-to-text directions:
B exp(s((,,us)7T)
1
L120
log
(2)
B
exp(s(ti,
i=1
1
exp(s((;,4:)/7)
Lez
log
(3)
B
E exp(s(t;,, w/)T)
L (L12v + 2/2
TV where B the batch size, is the temperature hyper-parameters. and s(t, [ [ the cosine similarity. This loss function maximizes the similarity of corresponding text-video pairs.
LoRA. LoRA Hu et al. 2021) is a parameter-efficient method to adapt CLIP with minimal train- ing. LoRA injects trainable rank decomposition matrices into the attention module of each layer. For the pre-trained matrices W E Dxd in Q, K, and V of Eq. (1. LoRA optimize their rank- decomposed changes, AW Wdoww where Wdown E R DxT Wap E grxdd and the rank < min(D (ann(Ddd d). For z xW, the forward pass is modified to zW + zAW. The trainable matrices AW can be merged with the frozen weights W, introducing no extra inference latency.
B ADDITIONAL EXPERIMENTAL SETTINGS
Datasets. We evaluate on four benchmark datasets: MSR-VTT Xu al. (006)), Activity Kr- ishna et al. (2017), DiDeMo Anne Hendricks et al. 2017). and LSMDC Rohrbach et al. (2015). (1) MSR-VTT Xu et al. (0016) is comprised of 10,000 Tube videos, each paired with 20 text descriptions. Following the data split in Gabeur et al. 2020); Miech et al. (2019), we train models on 9000 vaain videos with the corresponding captions and test on the IK-A test set with 1000 video-text pairs. (2) ActivityNet Krishna et al. (2017) contains 20,000 You Tou'e videos. We eval- uate models on the 'vall' split, comprising 10,009 videos for training and 4,917 for testing. We follow the setting from Gabeur et al. (2020); Zhang et al. (0018). where all sentence descriptions for a video are concatenated into a single query. (3) DiDeMo Anne Hendricks et al. (2017) contains 10,000 videos annotated with 40,000 text descriptions. There 8,395 videos in the train set and 1,004 videos in the test set. Following the setting in Bain et al. (202T); Lei et al. (202I), we con- catenate all the descriptions of video form a paragraph and evaluate models for paragraph-video
16

---

Published as a conference paper at ICLR 2025
Table 11: Comparisons in the video-to-text task MSRVTT.
Methods
Params (M)
GFLOPs
R@ 11
R@
RØ 101
R-sumt
MnR
CLIP- LLTP-VIT-B/32
Full Fine-tuning
CLIP4Clip
123,54
53.0
43.1
70.5
81.2
194,8
12.4
Prompt
0.08
58.2
42.2
69.7
79.2
191.1
12.4
Adapter
0.26
53.1
43.6
69.9
80.1
193.6
11.5
Parameter- Elfficien
LoRA
0,49
82.2
195.4
DGL
80.6
192.7
Vop
14.10
70.7
80.6
195.8
EVIT
0.49
37.2
43.5
80.2
193.1
DifiRaee
0.49
36.8
43.6
70.1
81.2
194.9
12.2
Parameter-
Parameter-Efficien Elficient
STA
0.49
35.7
41.1
69.7
81.1
191.9
12.7
ToMe
0.49
40.2
42.5
69.1
80.6
192.2
12.5
Inferenco-Effhieett
TESTA
0.59
40.6
43.0
70.4
80.5
193.9
12.5
TempMe
0.50
34.8
45.6
72.4
81.2
199.2
10.2
CLIP-VIT-B/IE
MV-Adapter
3.6
>-10
45.6
74.0
83.8
203.4
Purameter- Erftitte-Effficeent
RAP
1.06
>-20
45.3
76.4
84.8
206.5
9.1
DGLE 0.83
251.2
45.7
74.0
82.9
202.6
10.9
Infam&IIfcr-Effiitte
TempMe
0.50
47.6
85.4
208.3
Table 12: Comparisons in the vidco-to-tex task on ActivityNet, DiDeMo, and LSMDC with CLIP- ViT-B/32. CLIP4Clip is a classic full fine-tuning method.
ActivityNet R@St
DiDeMo
LSMDC
Methods
R@1-
R@101
MoR
R@ 11
R@51
R@ 101
MnR,
R@11
R@51
R@ 101
MnR
CLIPCClip
42.5
74.1
6.6
42.5
70.6
20.6
39.4
56.7
Prompt
38.4
68.8
81.1
9.3
39.6
68.2
78.5
12.4
19.9
37.6
46.6
58.2
Adapter
40.0
71.0
82.9
7.7
42.7
70.4
79.4
12.3
20.5
38.7
48.7
53.6
LoRA
40.0
71.4
82.9
7.9
41.2
69.0
79.2
12.5
20.4
37.3
47.1
59.7
VopF-c
35.6
65.9
77.8
10.4
44,4
71.8
81.8
9.5
22.3
40.3
50.7
51.1
DiflRate
39.1
69.7
82.3
8.4
40.4
67.5
79.0
12.6
20.0
37.8
46.9
61.6
ToMe
39.7
70.2
82.4
8.3
40.1
68.5
12.9
20.2
37.4
60.2
TempMe
45.3
74.7
86.2
6.4
48.4
75.4
83.6
9.1
22.2
41.5
51.5
48.0
2015) a movie clip dataset containing 118081 videos each paired with single text description. 101,079 videos used for training. 7,408 and 1,000 videos are used for validation and testing, respectively. The results of the test set are reported.
(Rorhaah retrieval. (4) LSMDC Rohrbach et al.
C ADDITIONAL EXPERIMENTAL RESULTS
C.I COMPARISONS IN THE VIDEO-TO-TEX TASK
In Table TI we compare the v2t performance on MSRYTT. In Table [2] we compare the v2t perfor- mance on ActivityNet, DiDeMo, and LSMDC. Consistent with the t2v comparisons detailed in the man paper, our TempMe achieves stae-of-the-an performance with minimal computational costs.
C.2 GENERALIZATION ANALYSIS OF DIFFERENT PARAMETER- PARAMETER-EPRICINTT METHODS
In our main paper, our TempMe employs LoRA for parameter-efficient adaption. Moreover, our TempMe is compatible with other parameter-efficient fine-tuning methods, such as Prompt and Adapter, shown in Tabel 13 We apply the same hyper-parameter (see Section 4.1) for both Prompt and Adapter. The extra ~0.01 tunable parameters of TempMe are introduced by the tunable clip positional embeddings (see Figure 3. These results indicate that our TempMe can effectively integrate with various parameter-efficien methods.
C.3 GENERALIZATION ANALYSIS OF DIFFERENT BACKBONES
To further assess the generalization capability of TempMe, we employed additional text-image pre- trained models the backbone. In Table 14 utilize VTT-B/16 the backbone architecture,
17

---

Published as a conference paper at ICLR 2025
Table 13: Generalization analysis of different parametee--fffcieet fine-tuning methods on MSRVTT with CLIP-ViT-32.
Params (M)
Text-to- Video
Video-to-1e Texeo-to-Text
Methods
R@11
R@st
RØ I01
MnR.
R@I
R@51
R@ 101
MnR.
Prompt
0.08
40.4
66.3
77.3
16.7
42.2
69.7
79.2
12.4
+ToMe
0.08
39.9
67.2
78.0
17.0
41.0
69.7
78.7
13.4
+TempMe
0.09
44.4
70.0
80.5
15.1
44.7
72.7
82.0
11.3
Adapter
0.26
41.9
69.9
78.7
14.9
43.6
69.9
80.1
11.5
+ToMe
0.26
41.3
68.7
79.7
15.1
42.4
71.1
79.5
12.0
+TempMe
0.28
45.7
71.2
80.1
14.1
44.6
72.1
81.1
11.2
LoRA
0.49
43.7
68.9
80.4
16.0
43.0
70.2
82.2
12.0
+ToMe
0.49
42.9
68.3
80.2
16.2
42.5
69.1
80.6
12.5
+TempMe
0.50
46.1
71.8
80.7
14.8
45.6
72.4
81.2
10.2
Table 14: Generalization analysis of different backbones on MSRVTT.
Text-to Viddeo
Video-to-Ted
Backhone
Methods
GFLOPs
R@
R@ST
R@ 10T
MnR
R@
K@sT
R0
MnR]
LoRA
211.3
44.5
71.4
79.7
14.4
43,9
71.2
82.9
LaCLIP
ToMe
144.4
43.4
70.9
80.7
16.0
43.9
71.2
82.8
11.5
TempMe
121.4
47.2
70.8
81.4
14.2
46.5
73.1
82.7
11.1
LoRA
211.3
45.3
71.2
80.6
13.8
46.5
73.7
82.2
11.5
MetaCLIP
ToMe
144.4
44.1
70.8
80.6
14.8
44.6
73.5
82.3
11.7
TempMe
121.4
48.2
72.6
82.0
13.0
47.2
73.9
83.0
10.0
Table 15: Ablation on clip positional embed- dings in t2v MSRVTT with CPE indicates Clip Positional Embeddings. The parameters of trainable CPE approxi- mately 0.01M.
Table 16: Ablation on frame sampling in 12v on MSRVTT with ELIP-ViT-B/SE F = N indi- cates the sampling of frames from a video.
Methods
GFL.Ops
R@
R-Sumt
MnR
LoRA
42.1
Methods TempMe
#Params (M)
R@11
R-Sum!
MnR.
TampMe F =8
25.0
44.4
195.6
15.6
46.1
198.6
w/o CPE
0.49
45.7
197.6
15.5
TempMe F 12
34,8
46.1
198.6
14,8
maintaining the same syper-parameters for both LaCLIP Fan et al. (2024) and MetaCLIP et al. (024a) used in CLIP. Due utilizing the same Transformer architecture CLIP, the model complexity is consistent with that of CLIP. These results indicate that our TempMe can effectively integrate with various text-image pre-trained backbones.
C.4 ABLATION ON CLIP POSITIONAL EMBEDDINGS
The trainable parameters of TempMe using CLIP- ViT-B/32 include LoRA (~00499M) and clip posi- tional embeddings (~~0.01M), Clip positional embeddings are only added the cross-clip merging step. Specifically, clip positional embeddings are added just before the tokens merged. When tokens from different clips are merged, their respective clip positional embeddings are also merged, effectively preserving the temporal information. Table [5 presents ablation study of clip posi- tional embeddings, which improves 0.4% RI and 1.0% R-Sum.
C.5 HYPER-PARAMEN PARAMETERS ANALYSIS OF CLIPME BLOCK
We conduct controlled experiments to identify the effect of hyper-parameters in our proposed ClipMe Block. (1) Start Layer of Clip Merging, Figure 5a shows the impact of Startalip- Merging at earlier layers leads to noticeable performance degradation. This is likely because early layers in the CLIP image encoder capture low-level features, which critical for frame-specifie details but inadequate for modeling temporal relationships. contrast, merging in later layers takes advantage of more semantic features that are better suited for learning spatio-temporal relationships. Larger
18

---

Published as a conference paper at ICLR 2025
66.5
46.5
45
44
45.5
45.5
43
GF GFLOPs
65.0
GGLOPS
45.0
GFLOPs
02-
R87
64.5
64.5
44.0
33
63.5
30
30
43.0
50
80
70
80
90
160
43.0
50
60
70
8D
100
52
(a) Startelip
(b)
Rc
(c)
R4
Figure 5:
analysis for text-to-video results on MSR-VTT with VIT-B/32. Each hyper-parameter evaluated while keeping all other paper-parameters fixed.
Table 17: Memory usage comparisons in the text-to-video task on MSRVTT with CLIP-ViT-B/22.
Methods
GFLOPs
Train Memory (MB)
Infer Memory (MB)
R@D
R-Sum
Prompl
58.2
16103
4941
40.3
184.0
Adapter
53.1
15291
4785
41.9
190.2
LoRA
53.0
16047
4783
43.7
193.0
VopPiC
58.0
18913
6275
44.6
194,8
DGLL
67.4
18863
5203
44.7
194.4
ToMe
40.2
13701
4737
42.9
191.4
TempMe
34.8
12381
4735
46.1
198.6
TempMe (F=18)
51.5
16410
5879
47.9
200.9
Startdip enhances performance until it reaches 9, which the most efficient value for maintaining
high performance with lower complexity. (2) Proportion of Remain Tokens. Re and Ry denote the proportions of remaining tokens for cross-clip and intra-clip merging, respectively. R. = 50% means that half the tokens merged. R. 100% implies that the merging process skipped. We seek a smaller R. that does not compromise accuracy. Figure (5] 5c reveal that the optimal trade-off between performance and complexity is achieved at Rc 70%, R₁ 90%.
C.6 ABLATION ON FRAME SAMPLING
Following standard protocol previous works, we sample 12 frames per video on MSRVTT. Reducing the number of sampled frames would decrease computational complexity. We conduct experiments with 8 frames per video in Table [6 Although the reduction from 12 to frames decreased the GFLOPs by 33%, it also caused 1.4% decrease in R-sum. Despite this, our TempME remains effective even when only 8 frames are sampled, indicating its robustness.
C.7 ABLATION ON MEMORY USAGE
In Table 17 we present the memory usage during training and inference for CLIP-V For all experiments, memory usage during training and inference measured on 4 A100 GPUs, each processing a batch size of 32.
(1) Although our TempMe achieves lower GFLOPs, it exhibits comparable inference memory us- age to ToMe. We speculate that this is partially influenced by the quadratic complexity of attention mechanisms, particularly the maximum token length processed (denoted as Atn Capacity). Specif- ically, ToMe's Attn Capacity is 50 layer 1, while TempMe's 's Attn Capacity is 118 at layer I.. Additionally, a similar trend memory usage is observed full fine-tuning with CLIP- ViT- in Table 5 Despite comparable inference memory usage, TempMe achieves significant perfor- mance improvement, with a 3.2% increase in R@ and 7.2% increase in R-Sum, demonstrating its overall effectiveness.
(2) Compared to previous PEFT TVR methods such as VoP and DGL, our TempMe requires signifi- cantly less memory during training and inference, validating the efficiency of our memory optimiza-
19

---

Published as a conference paper at ICLR 2025
tion. Specifically, Our TempMe not only improves 1.4% R @ but also achieves greater efficiency in terms of both model complexity and memory usage.
(3) Increasing the imput frames to 18 significantly improves performance by leveraging ral information. However, this comes at the cost higher GFLOPs and memory usage, as each GPU needs to process 32x18 frames. Compared previous PEFT TVR methods like VoP and DGL., TempMe with 18 frames achieves a significant improvement of 3.2% @@ and 6. R-sum under comparable inference memory usage. These results highlight the effectiveness of our TempMe in utilizing memory resources efficiently while achieving superior performance.
tempo-
C.8 ADDITIONAL VISUALIZATION RESULTS
In Figure 7 we present more visualization results. Regions sharing the same color in different vaguely identify
frames are merged together. Through the boundaries of the merged regions, the objects they depict. The merged regions in TempMe are larger than those in ToMe. Moreover, although TempMe merges greater number of tokens than ToMe, TempMe consistently outperforms ToMe in accuracy, confirming its advantage over ToMe.
D ADDITIONAL DISCUSSIONS
For efficient text-video retrieval, we propose TempME, a novel method to reduce temporal redun- dancy when utilizing the pre-trained image encoder for video feature extraction. Our research em- ploys publicly available datasets, which are restricted to non-commercial applications.
D.1 LIMITATIONS
At present, we concentrate on accelerating video processing. However, there is potential for further improving the text encoder. Therefore, we suggest that an efficient method for both text and image encoders could be a valuable direction for further research.
When adapting method to video foundation models (see Section 4.3 of our main paper), the transfer only feasible when the video encoder can treated as an image encoder. Specifically, TempMe is successfully extended UMT Li et al. (2023). However, it faces challenges when applicd to specialized video architectures, particularly CLIP-ViP ViP et al. (2022) which uses video proxies to process all tokens simultaneously, making it less compatible with our TempMe.
D.2 BROADER IMPACTS
The efficient text-video retrieval task focuses on efficiently transferring foundation models like CLIP. Previous studies Huang al. (2023); Yang al. (2024): ;; [Ji et al. (2024): Cao al. (2024) focus on freezing the backbone and training only minimal tunable parameters. However, these studies often neglect the importance of model complexity and inference efficiency. Our research encourages the community to consider both model storage and speed during transfer.
Our proposed method increases the efficiency of retrieving relevant video content based on textual queries, leading to substantial savings in time and resources. In era where video content dom- inates digital media, efficiently locating relevant videos is invaluable. Nonetheless, it also raises concerns about potential misuse surveillance and monitoring, it simplifies the extraction and analysis of video content from various sources.
20

---

Published as a conference paper at ICLR 2025
Frozen
Altention
Fezd Forward
Tuned params
e LoRA
Tokens in a frame
x
Image-level
8XNXD
(a) The LoRA block.
(b) The LoRA baseline.
Image-level
Temp Medeling.
Figure 6: toy example with sampled frames as input. (a) LoRA injects trainable weights into the attention module of each layer. (b) LoRA processes each frame individually. (c) Our Temporal Modeling aggregates clips progressively to enhance spatio-temporal learning. In this framework, the self-attention modules of the early layers is applied to intra-frame tokens in the spatial domain, targeting image-level details. In the later layers, they operate on tokens across frames in the spatio- temporal domain, capturing video-level information. The PMG framework, shown in Figure 2 of the main paper, has Temporal Modeling as a core function.
21

---

Published as a conference paper at ICLR 2025
Toput
ToMe
TempMe
a
Toput
ToMe
TempMe
(b)
Toput
ToMe
TempMe
(c)
Input
ToMe
TempMe
(d)
Figure 7: Qualitative comparisons MSRVTT with Patches that share the same inner and border color are merged. TempMe merges tokens of similar elements across frames.
22


Reviews:
[{"review_id": "HUogMAe7zJ", "reviewer": "Reviewer_AYTX", "summary": "To enhance the efficiency of text-video retrieval, this paper introduces Temporal Token Merging (TempMe), a parameter- and inference-efficient architecture aimed at reducing spatial-temporal redundancy. The framework primarily consists of ImageMe Blocks for image merging and ClipMe Blocks for clip merging, achieving a progressive multi-granularity approach. Extensive experiments validate the superiority of the proposed method.", "questions": "The authors should clarify how this work distinguishes itself from existing methods for token pruning or merging.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1, The paper addresses the issue of spatial-temporal redundancy in videos for text-video retrieval and introduces an efficient method that achieves faster training times and inference.\n2, Extensive experiments and analyses of TempMe demonstrate its efficiency, effectiveness, and generalization capabilities.\n3, The ClipMe Block mainly involves two steps, \"Intra-clip Merging\" and \"Cross-clip Merging\", each employing distinct methods for token grouping. This design effectively aids in information merging across clips.", "weaknesses": "1, Although the paper addresses spatial-temporal redundancy in text-video retrieval, there is already substantial work on token merging and pruning in video processing. This overlap may affect the perceived uniqueness of the proposed approach.\n2, A few symbols lack adequate definitions, which may hinder readability. For instance, \"R_c\" in Section 3.2, while defined in Figure 3 as the ratio of kept tokens, should also be described in the text when first introduced for clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3k8buoZhNY", "reviewer": "Reviewer_A4xM", "summary": "This work proposes TempMe, a temporal token merging method for T2V retrieval, designed to address two main issues: (1) high inference costs in current PEFT methods, and (2) the tendency of token merging methods to overlook temporal redundancy. TempMe introduces a PMG framework, which first merges tokens within individual images and then progressively merges across video clips. Experimental results demonstrate TEMP’s effectiveness, showing improvements in GFLOPS, training time, and retrieval performance.", "questions": "If the memory usage of TempMe proves to be more efficient compared to previous PEFT or full finetuning methods, I would consider increasing my score.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. **Efficient Training/Inference Acceleration**: TempMe achieves significant improvements in training efficiency by implementing image merging followed by progressive video clip merging. This approach leads to substantial reductions in GFLOPS and training time compared to previous methods.\n\n2. **Comprehensive Ablation Studies**: Extensive ablation experiments validate the effectiveness of clip merging at different layers and intervals, as well as its applicability on stronger video-pretrained backbones like UMT and MetaCLIP. Qualitative results further show that TEMP effectively merges objects and backgrounds.\n\n3. **Cross-Clip Merging**: TempMe introduces a relatively novel approach by incorporating cross-clip (inter-clip) merging, expanding beyond traditional intra-clip merging to capture cross-clip information, which enhances the learning of temporal relations across clips.", "weaknesses": "1. **Limited Improvement in R@1 for T2V Retrieval**: The R@1 improvement over previous methods is relatively small, especially given the advances in MLLM models. In 2023, T2V retrieval methods like HBI and Cap4video on CLIP-ViT-32/16, R@1 have reached around 48/50 (e.g., [1][2]). I suspect the limited improvement may stem from TempMe still focusing on video merging within the encoder, without further optimization after obtaining the clip-level representation.\n\n2. **Lack of Memory Usage Comparison**: Efficiency is not solely about GFLOPS—memory usage for training and inference is also crucial. While TempMe provides training acceleration, memory optimization would be even more beneficial. In Table 5, the authors report memory usage for CLIP-ViT-16, but it would be helpful to see detailed comparisons for CLIP-ViT-32 as well, particularly relative to previous methods.\n\n3. **Limited Novelty**: The intra-clip merging process is highly similar to the original token merging approach [3], which limits the novelty of this method. In fact, the flowchart for intra-clip merging in Figure 3 closely resembles the diagram in Token Merging (ToMe), making it difficult to discern substantial differences. Essentially, the only distinct contribution is the cross-clip merging mechanism.\n\n4. **Efficiency and Early Layer Merging**: Performing token merging in earlier layers of CLIP could theoretically reduce both GFLOPS and memory usage, though it might harm performance metrics like R@1. TempMe, however, only applies merging in the last three layers with a 12-6-3-1 merging pattern. A potential improvement could be to increase the input frames and match GFLOPS with previous PEFT methods, potentially enhancing R@1 without excessive memory usage. Additionally, it would be beneficial to provide memory usage details for CLIP-ViT-32 to understand whether further optimization is feasible.\n\n5. **Metric Suggestion**: Presenting an overall sum@R can be unclear for readers. R@1 is generally a more meaningful metric than sum@R and would offer a clearer understanding of the model's performance.\n\n[1] Jin, P. et al., \"Video-Text As Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learning,\" CVPR, 2023.  \n[2] Wu, W. et al., \"Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?\" CVPR, 2023.  \n[3] Bolya, D. et al., \"Token Merging: Your ViT but Faster,\" ICLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9JjbBmaqjO", "reviewer": "Reviewer_JLDW", "summary": "This paper tackles the text-video retrieval task and specifically propose a parameter efficient method for this task. The proposed framework merges redundant tokens across adjacent video clips. The method has a good trade-off between inference and parameter efficiency and accuracy achieving state of the art results when compared to other parameter efficient methods.", "questions": "Is any part of the method that is specifically designed for text-video retrieval or could it be applied for other text-video tasks as well?\n[1] shares a similar idea, can you please summarize the similarities and differences between the proposed method and [1]?\n\n\n[1] Yao, Linli, et al. \"DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models.\" arXiv preprint arXiv:2405.20985 (2024).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper addresses an important task, namely text-video retrieval and proposes a parameter-efficient method for this. One strength of the paper is that the proposed method was extended to video foundation methods such as UMT and various backbones showcasing the extensibility of the method. Also, the method is based on a well-known fact, namely the video has a lot of redundant information from frame to frame and the method is built upon that observation and compresses the redundant information. Additionally the method achieves good results both in term of performance and inference efficiency.", "weaknesses": "- Abstract: the abstract is hard to follow and is not clear if the method addressed inference speed-ups or training time speed-ups or both.\n- overall the quality of the writing can be improved because it's not straightforward to follow the paper, for example in the introduction the transitions between paragraphs are abrupt.\n- the choice of sampling 12 frames for MSRVTT and 64 frames for ActivityNet seems arbitrary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)