TempMe: Video Temporal Token Merging for Efficient Text-Video Retrieval

## Abstract

Abstract Most text-video retrieval methods utilize the text-image pre-trained CLIP as a backbone, incorporating complex modules that result in high computational overhead. As a result, many studies focus on efficient fine-tuning. The primary challenge in efficient adaption arises from the inherent differences between image and video modalities.
Each sampled video frame must be processed by the image encoder independently, which increases complexity and complicates practical deployment.
Although existing efficient methods fine-tune with small trainable parameters, they still incur high inference costs due to the large token number. In this work, we argue that temporal redundancy significantly contributes to the model’s high complexity due to the repeated information in consecutive frames. Existing token compression methods for image models fail to solve the unique challenges, as they overlook temporal redundancy across frames. To tackle these problems, we propose Temporal Token Merging (TempMe) to reduce temporal redundancy. Specifically, we introduce a progressive multi-granularity framework. By gradually combining neighboring clips, we merge temporal tokens across different frames and learn video-level features, leading to lower complexity and better performance. Extensive experiments validate the superiority of our TempMe. Compared to previous efficient text-video retrieval methods, TempMe significantly reduces output tokens by 𝟗𝟓 % percent 95 \mathbf{95\%} bold_95 % and GFLOPs by 𝟓𝟏 % percent 51 \mathbf{51\%} bold_51 % , while achieving a 1.8 × \mathbf{1.8\times} bold_1.8 × speedup and a 4.4 % percent 4.4 \mathbf{4.4\%} bold_4.4 % R-Sum improvement. Additionally, TempMe exhibits robust generalization capabilities by integrating effectively with both efficient and full fine-tuning methods. With full fine-tuning, TempMe achieves a significant 7.9 % percent 7.9 \mathbf{7.9\%} bold_7.9 % R-Sum improvement, trains 1.57 × \mathbf{1.57\times} bold_1.57 × faster, and utilizes 75.2 % percent 75.2 \mathbf{75.2\%} bold_75.2 % GPU memory usage. Our code will be released.

## 1 Introduction

In the domain of video-language understanding, text-video retrieval is a crucial task focused on matching videos that correspond to specific query texts or vice versa. Given the powerful representation capacity of large-scale text-image pre-training [, , ], recent studies [, , , ] have increasingly focused on full fine-tuning of large foundational models for text-video retrieval. Specifically, leveraging the pre-trained CLIP [] as the backbone, present full fine-tuning approaches incorporate cumbersome modules to extract video and text representations. However, the slow inference speed of these approaches severely limits their real-world applications.
For example, the training process of CLIP4Clip [] with ViT-B/16 requires 70.1GB GPU memory usage and takes 6.5 hours.
Therefore, we focus on efficient fine-tuning text-video retrieval [, , ] to lower computational expenses.

There are substantial challenges in efficiently adapting a text-image pre-trained model for text-video retrieval, mainly due to the inherent differences between image and video modalities. Since videos consist of numerous frame images, each sampled frame in a video needs to pass through the pre-trained image encoder, leading to high complexity. The shift from processing a single image to handling multiple sampled frames dramatically raises the number of patch tokens fed into the model, complicating the extraction of video representations. As a result, this adaption suffers from high computational overhead, which makes practical deployment difficult.
As shown in Figure 1b, existing efficient text-video retrieval methods like VoP [] and DGL [] freeze the pre-trained backbone and introduce trainable prompts to model video data. They treat video input as a sequence of multiple sampled frames, acquiring considerable accuracy at the expense of inference efficiency.

Figure: Figure 1: (a) An example illustrates the large temporal redundancy between adjacent frames. Identical subjects are highlighted in the same color. (b) Current methods treat video input as a sequence of multiple image frames, causing high complexity due to the large number of tokens. (c) In contrast, our TempMe reduces temporal redundancy by progressively merging redundant tokens in adjacent video clips. (d) With ViT-B/16 as the backbone on MSRVTT, our TempMe reaches state-of-the-art performance with minimal computational overhead. R-Sum is the sum of R@1, R@5, and R@10.
Refer to caption: x1.png

When employing the pre-trained image encoder for video inputs, we argue that temporal redundancy plays a major role in the model’s high complexity. The sequential sampled frames within a video capture a continuous temporal progression, illustrating subject evolution, interaction, and transition over time.
As shown in Figure 1a, the video depicts a cat and a monkey in a fight, showing the same subjects across adjacent frames while maintaining a repetitive background. This temporal progression introduces substantial redundancy due to repeated information present in consecutive frames, contributing to the high complexity caused by repeated model calculations.

A recently popular technique, token compression [, , , ], might be a fascinating solution, which has achieved huge success in accelerating image models. Classically, token compression methods aim to eliminate spatial redundancy and merge similar tokens in the image to improve efficiency. However, they fail to resolve the unique challenges imposed by video modality. Specifically, they overlook the temporal redundancy across different frames, leading to inferior performance. Currently, there is no dedicated compression algorithm designed for text-video retrieval models, and the efficiency issue urgently needs to be addressed.

In this paper, we introduce Temporal Token Merging (TempMe) for efficient text-video retrieval, a novel method designed to minimize inherent temporal redundancy in video content. Our proposed TempMe effectively merges a large number of similar tokens across frames, significantly reducing computational overhead. Particularly, our method adopts a progressive multi-granularity framework, enabling the efficient merging of redundant tokens, as visualized in Figure 1c. Videos can be viewed as aggregations of clips with varying temporal granular levels, from individual images at the micro-level to the entire video at the macro-level. Through gradually combining neighboring clips, the total number of tokens is dramatically decreased, while the features evolve from fine-grained image-level features to holistic video-level features, leading to lower complexity and better performance.

To validate the advantages of our TempMe, extensive experiments are conducted on four benchmark datasets, MSRVTT [], ActivityNet [], DiDeMo [], and LSMDC []. Experimental results consistently demonstrate that our TempMe offers a leading balance between speed and accuracy, outperforming other efficient fine-tuning methods. Figure 1d shows the performance comparison of ViT-B/16 on MSR-VTT.
Compared to VoP [] and DGL [], TempMe significantly reduces output tokens by $\mathbf{95\%}$ and GFLOPs by $\mathbf{51\%}$, while achieving a $\mathbf{1.8\times}$ speedup and a $\mathbf{4.4\%}$ R-Sum improvement.
Moreover, TempMe can be seamlessly integrated with various efficient fine-tuning (see Appendix C) and full fine-tuning (see Table 5), which demonstrates its robust generalization capabilities. When fully fine-tuning with ViT-B/16, TempMe achieves a significant $\mathbf{7.9\%}$ R-Sum improvement, trains $\mathbf{1.57\times}$ faster, and utilizes $\mathbf{75.2\%}$ GPU memory usage.

Our major contributions are summarized as follows:
(1) Our study reveals that temporal redundancy in video content causes excessive computational demands when adapting pre-trained text-image models for text-video retrieval. The image encoder redundantly processes overlapping information across frames, resulting in increased model complexity.
(2) To overcome such limitations, we propose TempMe for efficient text-video retrieval to reduce temporal redundancy. A progressive multi-granularity framework is proposed to merge redundant tokens in adjacent video clips step by step, simplifying model complexity while extracting unified and discriminative video features.
(3) Experimental evaluation on four benchmark datasets demonstrates that our TempMe outperforms current state-of-the-art methods. Extensive experiments indicate that our TempMe effectively reduces model complexity while achieving superior performance.

## 2 Related Works

#### Text-Video Retrieval.

Due to the great success of language-vision pre-training [, , , , , , , , ] in various downstream cross-modal tasks [, , , , ],
numerous studies [, , , , , , , , , ] have achieved remarkable results by utilizing CLIP’s [] knowledge for text-video retrieval. These approaches typically employ CLIP’s encoders as the backbone and use complex similarity calculations.
Specifically, CLIP4Clip [] introduces several video aggregation schemes to obtain video features.
However, these full fine-tuning methods with their customized modules require significant memory and computational resources. Therefore, our work aims to explore efficient fine-tuning methods to lower computational overhead.

Mainstream parameter-efficient fine-tuning approaches in the image domain, such as Prompt [, , , ], LoRA [, ], and Adapter [, , , ], freeze pre-trained model parameters while adding extra tunable parameters to minimize storage demands.
For parameter-efficient text-video retrieval, VoP [] and DGL [] introduce extra modules to generate prompt tokens and capture global video information. Despite their good performance with small tunable parameters, these methods face high inference costs due to the significant token number in videos.

#### Token Compression

Token compression methods [, , , , , , , ] aim to reduce computational burden by reducing tokens while preserving the essential information.
Some works [, , ] focus on dynamically pruning non-essential tokens, leading to varying token numbers across images and thus impacting speed efficiency.
Alternatively, some works [, , ] focus on merging tokens.
ToMe [] reduces tokens in a transformer gradually by merging a fixed number of tokens in each block.
DiffRate [] performs token pruning and merging simultaneously.
DiffRate and ToMe can be combined with parameter-efficient fine-tuning methods to reduce tokens in text-video retrieval.
Although ToMe has been applied to video classification by simultaneously processing all frame tokens in Spatiotemporal MAE [], this implementation is incompatible with CLIP’s image encoder for text-video retrieval.
When processing each frame via CLIP’s image encoder, ToMe effectively reduces spatial redundancy by merging similar tokens within a single frame. However, it does not address the temporal redundancy across frames, which greatly contributes to computational overhead.

Inspired by ToMe, we merge similar tokens within individual frames during the first image merging stage. Then, we propose an innovative clip merging stage for cross-clip and intra-clip merging, which significantly reduces temporal tokens and enhances the learning of video-level features.

## 3 Methodology

In this work, we concentrate on efficiently adapting the pre-trained text-image model to text-video retrieval.
Firstly, preliminaries are presented in Section 3.1.
Subsequently, the implementation details of our TempMe are elaborated in Section 3.2.
Finally, complexity analysis is discussed in Section 3.3

Figure: Figure 2: Overview of our proposed TempMe. We introduce a Progressive Multi-Granularity (PMG) framework consisting of both image merging and clip merging stages. In the image merging stage, ImgMe Block merges redundant spatial tokens within a single frame. Following this, ClipMe Block progressively forms new clips from adjacent ones, facilitating video-level feature learning and reducing temporal redundancy by merging tokens across different frames.
Refer to caption: x2.png

### 3.1 Preliminaries

The task of text-video retrieval involves leveraging textual queries to accurately retrieve related videos, as well as utilizing video queries to find associated textual descriptions. The primary goal of text-video retrieval is to bridge the semantic gap between textual description and video content by learning a similarity function $s(t,v)$. Here, $t$ denotes a text, and $v$ represents a sequence of frame images $\left\{I_{i}\right\}_{i=1}^{F}$, where $F$ is the number of sampled frames in time.
We employ LoRA [] to adapt CLIP [] for text-video retrieval tasks. The trainable matrices of LoRA can be merged, introducing no extra inference latency.
Further details are provided in Appendix A.

### 3.2 Temporal Token Merging

For efficient text-video retrieval, we freeze the pre-trained CLIP and merely train LoRA in both the image and text encoders.
In this subsection, we focus on merging temporal tokens across multiple frames to minimize temporal redundancy and extract more comprehensive video features.

Figure: Figure 3: ClipMe Block. Given an input of $f$ clips $\mathbb{R}^{f\times N\times D}$, the cross-clip merging step merges tokens from all clips to form a new clip $\mathbb{R}^{1\times fNR_{c}\times D}$. Subsequently, the intra-clip merging step merges tokens within this newly formed clip, producing $\mathbb{R}^{1\times fNR_{c}R_{I}\times D}$. If the input contains only one clip, the cross-clip merging is skipped.
Refer to caption: x3.png

#### Progressive Multi-Granularity Framework.

In text-video retrieval, the significant impact of temporal redundancy on model complexity has not been adequately explored. To mitigate temporal redundancy, we proposed the Progressive Multi-Granularity (PMG) framework that efficiently consolidates similar temporal tokens across different frames.
In our PMG framework (see Figure 2), we introduce a two-stage merging approach: image merging and clip merging. Initially, ImgMe block independently encodes each single frame, merging similar tokens within the same frame and leveraging pre-trained image knowledge to capture low-level spatial details.
Next, ClipMe block (see Figure 3) is proposed to aggregate short-frame clips into extended-frame clips, merging temporal tokens across different frames and learning unified and discriminative video features.

Specifically, each frame image $I$ can be regarded as a discrete video clip lasting only one frame $v^{f=1}$. Concatenating these consecutive clips yields an extended clip with increased temporal length. In the toy example presented in Figure 2, 8 individual frames $\left\{v^{f=1}_{i}\right\}_{i=1}^{8}$ are aggregated into four clips, each containing with two frames $\left\{v^{f=2}_{i}\right\}_{i=1}^{4}$. These clips are then aggregated to form a complete video clip with all 8 frames $\left\{v^{f=8}_{i}\right\}_{i=1}^{1}$.
As clips are progressively aggregated, we merge a significant number of similar tokens that arise from overlapping information across frames. Simultaneously, comprehensive video information is captured through the attention mechanism that processes tokens across different frames.

#### ImgMe Block.

Inspired by ToMe [], the image merging step merges tokens to reduce by $r$ per layer, which is employed between the MHSA and MLP branches of each ImgMe Block. Following Bipartite Soft Matching [], we first partition the tokens into two sets $\mathbb{A}$ and $\mathbb{B}$ by alternating. For each token in set $\mathbb{A}$, we find its most similar token in Set $\mathbb{B}$ using cosine similarity. The top $r$ most similar pairs are then merged by averaging their features.
Given an input $\mathbb{R}^{1\times N\times D}$, the image merging stage with $K$ ImgMe Block outputs $\mathbb{R}^{1\times(N-rK)\times D}$, where $N$ denotes the number of tokens, and $D$ denotes the dimension of the token feature.

To prevent loss of image information, $r$ is set to a low value: $r=2$ for ViT-B/32 and $r=10$ for ViT-B/16. Nevertheless, the model complexity remains high because of the processing of multiple frames. Therefore, we aim to reduce temporal tokens to further mitigate the model complexity.

#### ClipMe Block.

Instead of ImgMe Block which compresses tokens within each single frame, we propose ClipMe Block to process multi-frame clips as illustrated in Figure 3. In ClipMe Block, we introduce two novel steps: cross-clip merging and intra-clip merging. During cross-clip merging, adjacent clips are aggregated, which significantly reduces the number of temporal tokens and generates a new clip. The intra-clip merging step further compresses the tokens within the newly formed clip.

In cross-clip merging, we aim to combine tokens with their most similar tokens from the same subject, whether these matches occur within a single frame or span across multiple frames.
Initially, all tokens from the input $f$ clips $\mathbb{R}^{f\times N\times D}$ are alternately divided into Set $\mathbb{A}$ or Set $\mathbb{B}$. This alternating assignment scheme is based on the observation that the adjacent tokens often depict the same subject, thus their features tend to be similar. Moreover, this assignment ensures that tokens representing the same subject across various frames are distributed between both sets. Considering the high temporal redundancy observed in videos, we select a large subset, $fN\times(1-R_{c})$, of the most similar token pairs for merging. Before merging, the trainable clip positional embeddings are added to facilitate temporal understanding in MSHA.
Subsequently, these merged tokens, along with the unmerged tokens, constitute a new clip $\mathbb{R}^{1\times(fN\times R_{c})\times D}$, which is then fed into MHSA.

During intra-clip merging, the tokens in the newly formed clip, improved by MHSA for better temporal and spatial details, are further reduced. In contrast to the image merging step that merges a small number $r$ of tokens within an individual frame, the intra-clip merging step merges a larger proportion of tokens within a clip containing multiple frames, compressing $\mathbb{R}^{1\times N\times D}$ into $\mathbb{R}^{1\times(N\times R_{I})\times D}$.

**Table 1: Complexity comparisons. We report the R-Sum metric for 12 sampled frames on MSRVTT and 64 sampled frames on ActivityNet, respectively. The best metric of all methods is highlighted in bold.**
| Backbone | CLIP-ViT-B/32 | CLIP-ViT-B/16 |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| # Frames | 12 | 64 | 12 |  |  |  |  |  |  |
| Method | GFLOPs | # Tokens | R-Sum | GFLOPs | # Tokens | R-Sum | GFLOPs | # Tokens | R-Sum |
| LoRA [] | 53.0 (100%) | $12\times 50$ (100%) | 193.0 | 276.7 (100%) | $64\times~{}~{}50$ (100%) | 191.5 | 211.3 (100%) | $12\times 197$ (100%) | 201.4 |
| DiffRate [] | 36.8 (  69%) | $12\times 20$ (  40%) | 189.9 | 190.1 (  69%) | $64\times~{}~{}20$ (  40%) | 188.7 | 138.5 (  66%) | $12\times~{}~{}49$ (  25%) | 202.4 |
| ToMe [] | 40.2 (  76%) | $12\times 26$ (  52%) | 191.4 | 208.5 (  75%) | $64\times~{}~{}26$ (  52%) | 189.6 | 144.4 (  68%) | $12\times~{}~{}77$ (  39%) | 200.6 |
| TempMe | 34.8 (  65%) | $\mathbf{1\times 97}$ (  16%) | 198.6 | 180.3 (  65%) | $\mathbf{1\times 500}$ (  16%) | 205.6 | 121.4 (  57%) | $\mathbf{1\times 127}$ (    5%) | 206.7 |

### 3.3 Complexity Analysis.

To verify the efficiency of our proposed TempMe, we perform a detailed complexity analysis in Table 1.
Following common protocols [, ], the frame length is set to 12 for MSR-VTT [] and LSMDC [], and to 64 for ActivityNet [] and DiDeMo []. The complexity comparison between ToMe and our TempMe for these frame lengths is shown in Table 1. Compared to ToMe, our TempMe reduces tokens by more than 30% for both ViT-B/32 and ViT-B/16, effectively decreasing model complexity while significantly surpassing accuracy. Particularly, with a 12-frame length and ViT-B/16 backbone on MSRVTT, TempMe outputs only 5% of the input tokens, reaches 57% GFLOPs, and achieves a 5.3% R-Sum gain.

## 4 Experiments

**Table 2: Comparisons in the text-to-video task on MSRVTT.**
| Methods | # Params (M) | GFLOPs | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | R-sum$\uparrow$ | MdR$\downarrow$ |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP-ViT-B/32 |  |  |  |  |  |  |  |  |
| Full Fine-tuning | CLIP4Clip [] | 123.54 | 53.0 | 43.1 | 70.4 | 80.8 | 194.3 | 16.2 |
| Parameter-Efficient | Prompt [] | 0.08 | 58.2 | 40.4 | 66.3 | 77.3 | 184.0 | 16.7 |
| Adapter [] | 0.26 | 53.1 | 41.9 | 69.9 | 78.7 | 190.2 | 14.9 |  |
| LoRA [] | 0.49 | 53.0 | 43.7 | 68.9 | 80.4 | 193.0 | 16.0 |  |
| PLEVU [] | 6.35 | - | 36.7 | 64.6 | 76.8 | 178.1 | - |  |
| VoP [] | 14.10 | 58.0 | 44.7 | 70.5 | 79.2 | 194.4 | 16.2 |  |
| DGL [] | 0.83 | 67.4 | 44.6 | 69.9 | 80.3 | 194.8 | 16.3 |  |
| Parameter-Efficient<br>&<br>Inference-Efficient | EVIT [] | 0.49 | 37.2 | 41.4 | 69.0 | 78.1 | 188.5 | 16.8 |
| DiffRate [] | 0.49 | 36.8 | 41.5 | 68.6 | 79.8 | 189.9 | 16.3 |  |
| STA [] | 0.49 | 35.7 | 42.6 | 69.5 | 78.8 | 190.9 | 17.0 |  |
| ToMe [] | 0.49 | 40.2 | 42.9 | 68.3 | 80.2 | 191.4 | 16.2 |  |
| TESTA [] | 0.59 | 40.6 | 43.7 | 69.0 | 79.4 | 192.1 | 16.8 |  |
| TempMe | 0.50 | 34.8 | 46.1 | 71.8 | 80.7 | 198.6 | 14.8 |  |
| CLIP-ViT-B/16 |  |  |  |  |  |  |  |  |
| Parameter-Efficient | MV-Adapter [] | 3.6 | >210 | 46.2 | 73.2 | 82.7 | 202.1 | - |
| RAP [] | 1.06 | >210 | 46.5 | 73.9 | 82.0 | 202.4 | 12.1 |  |
| VoP [] | 14.10 | 246.2 | 47.7 | 72.4 | 82.2 | 202.3 | 12.0 |  |
| DGL [] | 0.83 | 251.2 | 48.3 | 71.8 | 80.6 | 200.7 | 13.4 |  |
| Param&Infer-Efficient | TempMe | 0.50 | 121.4 | 49.0 | 74.4 | 83.3 | 206.7 | 11.9 |

**Table 3: Comparisons in the text-to-video task on ActivityNet, DiDeMo, and LSMDC with CLIP-VIT-B/32. CLIP4Clip is a classic full fine-tuning method.**
| Methods | ActivityNet | DiDeMo | LSMDC |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MnR$\downarrow$ | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MnR$\downarrow$ | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | MnR$\downarrow$ |  |
| CLIP4Clip | 40.5 | 72.4 | - | 7.4 | 43.4 | 70.2 | 80.6 | 17.5 | 20.7 | 38.9 | 47.2 | 65.3 |
| Prompt | 36.0 | 67.0 | 79.4 | 10.2 | 39.4 | 67.6 | 78.3 | 20.6 | 19.1 | 38.2 | 46.1 | 67.5 |
| Adapter | 37.8 | 69.0 | 81.9 | 8.7 | 41.7 | 68.0 | 79.3 | 19.9 | 21.3 | 38.7 | 48.3 | 60.7 |
| LoRA | 38.7 | 70.6 | 82.2 | 8.7 | 41.6 | 68.8 | 78.6 | 20.8 | 20.7 | 39.5 | 47.4 | 68.3 |
| PLEVU | - | - | - | - | 36.1 | 64.8 | - | - | 13.4 | 29.5 | 40.3 | - |
| VoP | 35.1 | 63.7 | 77.6 | 11.4 | 46.4 | 71.9 | 81.5 | 13.6 | 21.1 | 40.9 | 49.6 | 60.1 |
| DGL | 38.6 | 69.2 | 81.6 | 9.0 | - | - | - | - | 21.4 | 39.4 | 48.4 | 64.3 |
| DiffRate | 38.0 | 69.1 | 81.6 | 9.2 | 40.8 | 67.8 | 77.8 | 20.6 | 20.2 | 37.4 | 47.6 | 70.0 |
| ToMe | 38.6 | 69.8 | 81.2 | 9.1 | 40.5 | 68.6 | 77.8 | 20.9 | 20.5 | 38.8 | 46.8 | 67.8 |
| TempMe | 44.9 | 75.2 | 85.5 | 6.8 | 48.0 | 72.4 | 81.8 | 13.7 | 23.5 | 41.7 | 51.8 | 53.5 |

### 4.1 Experimental Settings

#### Datasets.

Following common practice, we perform experiments on four widely used benchmarks for text-video retrieval: MSRVTT [], ActivityNet [], DiDeMo [], and LSMDC []. Detailed descriptions of these datasets are provided in Appendix B.

#### Metrics.

We evaluate the performance using metrics such as Recall at K (R@1, R@5, and R@10), the sum of these recalls (R-Sum), and Mean Rank (MnR). A higher recall score $\uparrow$ indicates better performance, while a lower MnR score $\downarrow$ denotes better performance. We evaluate GFLOPs (Giga Floating-Point Operations per Second) and throughput only for the video backbone during inference. All throughputs are measured on a A100.

#### Implementation Details.

Following previous works [, , , ], we use the pre-trained CLIP as the backbone. We employ the AdamW optimizer [] with a batch size of 128. For short video retrieval datasets like MSRVTT and LSMDC, the max word and frame lengths are set to 32 and 12. For long video retrieval datasets like ActivityNet and DiDeMo, the max word and frame lengths are set to 64 and 64. Unless noted otherwise, ViT-B/32 is employed as the backbone. More details are provided in Appendix B.

#### Compared Baselines.

We compare our TempMe with several strong baselines:
(1) CLIP4Clip, a classic full fine-tuning method. For a fair comparison, we only consider the parameter-free type that does not introduce additional modules.
(2) Prompt [], LoRA [], and Adapter [], which are mainstream parameter-efficient fine-tuning methods.
(3) PLEVU [], VoP [], DGL [], MV-Adapter [], and RAP [], which are methods tailored for parameter-efficient text-video retrieval.
(4) EVIT [], DiffRate [], and ToMe [], which are token compression methods for image models.
(5) STA [] and TESTA [], which are video token compression methods.
For fairness, token compression methods apply LoRA like our TempMe (see Appendix A).

**Table 4: Computational overhead comparisons of the CLIP-ViT-B/16 backbone. We evaluate the text-to-video accuracy metrics on MSRVTT.**
| Methods | videos/s | GFLOPs | # Tokens | R-Sum$\uparrow$ |
| --- | --- | --- | --- | --- |
| Prompt | 29.7 | 216.8 | 2369 | 194.2 |
| Adapter | 29.4 | 211.7 | 2364 | 196.8 |
| LoRA | 30.6 | 211.3 | 2364 | 201.4 |
| VoP | 25.0 | 246.2 | 2368 | 202.3 |
| DGL | 3.3 | 251.2 | 2416 | 200.7 |
| DiffRate | 40.6 | 138.5 | 588 | 202.4 |
| ToMe | 40.8 | 144.4 | 924 | 200.6 |
| TempMe | 45.1 | 121.4 | 127 | 206.7 |

### 4.2 Comparisons with State-of-the-art Methods

#### Efficient Fine-tuning.

In Table 2, we compare the t2v performance on MSRVTT. Our TempMe achieves significant improvements over previous methods, with a $\textbf{3.8}\%$ R-Sum increase in t2v using ViT-B/32 and a $\textbf{4.3}\%$ R-Sum increase in t2v using ViT-B/16, while maintaining minimal GFLOPs.
Table 3 shows the t2v results on ActivityNet, DiDeMo, and LSMDC, demonstrating that TempMe consistently outperforms state-of-the-art methods. Detailed v2t results are provided in Appendix C.

#### Computational Overhead.

In Table 5, accuracy metrics are evaluated on MSRVTT with ViT-B/16. We also report the model throughput and complexity during the inference phase. During this phase, the tunable weights can be merged in LoRA, DiffRate, ToMe, and TempMe (see Section 3.1). For fairness, prompt generation in VoP and DGL is omitted when evaluating throughput and complexity.
Compared to LoRA, ToMe reduces tokens per frame to $\textbf{39}\%$, with a minor performance degradation. However, our TempMe further reduces temporal tokens across frames to just $\textbf{5}\%$, while also improving the R-Sum by $\textbf{5.3}\%$.
Unlike VoP and DGL, which compromise efficiency for performance, our TempMe achieves a $\textbf{1.8}\times$ speedup over VoP and a $\textbf{13.7}\times$ speedup over DGL, with the highest 206.7 R-Sum.

### 4.3 Generalization Analysis

#### Full Fine-tuning.

Table 5 shows that our TempMe can be applied to full fine-tuning. The image encoder must process each sampled video frame individually, resulting in significant GPU memory usage and training times.
ToMe reduces spatial redundancy to boost model speed but suffers from a minor performance decrease.
In contrast, our TempMe not only reduces temporal redundancy but also learns video-level features, with a $\textbf{1.57}\times$ speedup and a $\textbf{7.9}\%$ R-Sum gain.

**Table 6: Application in video foundation models for text-video retrieval. We conduct a evaluation of full fine-tuning on MSRVTT for the text-to-video task with UMT-B/16-25M.**
| Methods | GFLOPs | Train<br>Time | R@1$\uparrow$ | R-Sum$\uparrow$ |
| --- | --- | --- | --- | --- |
| UMT | 304.3 | 9.5h | 51.0 | 212.3 |
| UMT4Clip | 210.1 | 4.8h | 48.8 | 206.7 |
| UMT4Clip+TempMe | 111.5 | 3.5h | 51.1 | 209.2 |

#### Application in Video Foundation Models.

We extend our method to video foundation methods, such as UMT []. TempMe is built upon the text-image CLIP model, which processes each frame individually. This contrasts with UMT, which processes all frame tokens simultaneously. To align with our method, we introduce a new baseline UMT4Clip, where UMT handles frames separately. These differences in token processing strategies inevitably lead to a decline in performance. Following the practice of UMT, we conduct fully fine-tuning experiments on both text-video retrieval and video QA tasks. In Table 7, our TempMe achieves performance close to UMT while requiring only one-third of the GFLOPs and training time. Moreover, we also extend our method to video QA in Table 7. These results demonstrate the superiority and generalizability of our method.

#### More Generalization Analysis.

To further validate the generalization capability, we integrate our TempMe with other efficient fine-tuning methods such as Prompt and Adapter. Moreover, we evaluate our TempMe on other backbones. These detailed experimental results are provided in Appendix C.

**Table 8: Ablation analysis of individual components. We evaluate in the text-to-video task on MSRVTT with CLIP-ViT-B/32.**
| Methods | #Params | GFLOPs (M) | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | R-Sum$\uparrow$ | MdR$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP-Straight | - | 53.0 | 30.8 | 53.8 | 63.3 | 147.9 | 41.8 |
| +LoRA | 0.49 | 53.0 | 43.7 | 68.9 | 80.4 | 193.0 | 16.0 |
| +ImgMe | 0.49 | 40.2 | 42.9 | 68.3 | 80.2 | 191.4 | 16.2 |
| +ClipMe | 0.50 | 34.8 | 46.1 | 71.8 | 80.7 | 198.6 | 14.8 |

**Table 9: Ablation analysis of individual components. We evaluate in the text-to-video task on MSRVTT with CLIP-ViT-B/32.**
| Methods | #Params | GFLOPs (M) | R@1$\uparrow$ | R@5$\uparrow$ | R@10$\uparrow$ | R-Sum$\uparrow$ | MdR$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| LoRA | 0.49 | 53.0 | 43.7 | 68.9 | 80.4 | 193.0 | 16.0 |
| Temporal Modeling | 0.50 | 54.3 | 46.3 | 72.1 | 81.3 | 199.7 | 14.8 |
| Token Reduction | 0.49 | 34.7 | 41.6 | 67.9 | 79.3 | 188.8 | 16.6 |
| TempMe | 0.50 | 34.8 | 46.1 | 71.8 | 80.7 | 198.6 | 14.8 |

### 4.4 Ablation Study

We conduct an in-depth analysis of TempMe from structural and functional perspectives, respectively.

#### Ablation on Each Component.

TempMe is architecturally divided into the ImgMe and ClipMe blocks. Table 8 illustrate the impact of each component. CLIP-straight denotes the zero-shot performance of CLIP, and LoRA is employed as the baseline for efficient fine-tuning. The ImgMe block reduces spatial redundancy in each frame. The ClipMe block aggregates short-frame clips into extended-frame clips, facilitating temporal learning while minimizing temporal redundancy. Finally, our TempMe outperforms LoRA by 5.6% R-Sum and reduces 18.2 GFLOPs.

#### Ablation on Each Function.

From a functional perspective, TempMe can be categorized into Temporal Modeling and Token Reduction, aimed at improving accuracy and efficiency, respectively. Table 9 demonstrates the impact of each function. (1) The accuracy improvements are attributed to Temporal Modeling, which aggregates clips progressively to enhance spatio-temporal learning. In this framework, the self-attention modules of the early layers is applied to intra-frame tokens in the spatial domain. In the later layers, they operate on tokens across frames in the spatio-temporal domain. (2) The efficiency improvements arise from Token Reduction, which reduces redundancy in the entire framework. In the early layers, it slightly reduces intra-frame tokens to decrease spatial redundancy. In the later layers, it significantly reduces tokens among frames to address large temporal redundancy. (3) Without the support of Temporal Modeling, Token Reduction alone significantly reduces tokens in the spatial domain, leading to a substantial 4.2% decrease in R-sum. However, due to the considerable temporal redundancy, the combination of Temporal Modeling and Token Reduction (TempMe) significantly reduces complexity while achieving high performance.

#### More Ablation Analysis.

For a more comprehensive evaluation of our method, we conduct a series of ablation studies, as detailed in Appendix C. First, since the trainable parameters of TempMe using ViT-B/32 include LoRA (0.49M) and clip positional embeddings (0.01M), we perform ablation on the clip positional embeddings of the ClipMe block. Next, we thoroughly conduct an exhaustive analysis of the merging strategy. Following this, we provide a detailed analysis of the intra-clip and cross-clip merging steps within the ClipMe block. Furthermore, given that the number of sampled frames affects the model complexity, we conduct an ablation study on frame sampling.

Figure: Figure 4: Qualitative comparisons on MSRVTT with ViT-B/16. Patches that share the same inner and border color are merged together. TempMe merges tokens of similar elements across frames.
Refer to caption: x4.png

### 4.5 Qualitative Results

Figure 4 presents visualization results of ToMe and our TempMe.
ToMe only merges spatial tokens within individual frames, leaving a large number of redundant tokens across frames, contributing to high computational overhead.
In contrast, TempMe merges tokens of similar elements across contiguous frames.
For instance, TempMe merges similar body regions of Patrick Star across frames, such as the head, upper body, and lower body.
These visualization results show that our TempMe effectively reduces temporal redundancy.
More qualitative results are provided in Appendix C.

## 5 Conclusion

We explore temporal redundancy for efficient text-video retrieval. A novel method, Temporal Token Merging, is proposed to reduce temporal redundancy.
We introduce an innovative progressive multi-granularity framework to merge redundant tokens in consecutive clips and encode information from various frames.
This strategy effectively reduces model complexity while simultaneously extracting comprehensive video features.
Extensive experiments validate the superiority and generalizability of TempMe.
Further discussion on the limitations and broader impacts is provided in Appendix D.
In the future, we aim to extend the application of our proposed method to other video tasks.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: The authors should clarify how this work distinguishes itself from existing methods for token pruning or merging.
soundness: 3
strengths: 1, The paper addresses the issue of spatial-temporal redundancy in videos for text-video retrieval and introduces an efficient method that achieves faster training times and inference.
2, Extensive experiments and analyses of TempMe demonstrate its efficiency, effectiveness, and generalization capabilities.
3, The ClipMe Block mainly involves two steps, "Intra-clip Merging" and "Cross-clip Merging", each employing distinct methods for token grouping. This design effectively aids in information merging across clips.
summary: To enhance the efficiency of text-video retrieval, this paper introduces Temporal Token Merging (TempMe), a parameter- and inference-efficient architecture aimed at reducing spatial-temporal redundancy. The framework primarily consists of ImageMe Blocks for image merging and ClipMe Blocks for clip merging, achieving a progressive multi-granularity approach. Extensive experiments validate the superiority of the proposed method.
weaknesses: 1, Although the paper addresses spatial-temporal redundancy in text-video retrieval, there is already substantial work on token merging and pruning in video processing. This overlap may affect the perceived uniqueness of the proposed approach.
2, A few symbols lack adequate definitions, which may hinder readability. For instance, "R_c" in Section 3.2, while defined in Figure 3 as the ratio of kept tokens, should also be described in the text when first introduced for clarity.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: If the memory usage of TempMe proves to be more efficient compared to previous PEFT or full finetuning methods, I would consider increasing my score.
soundness: 3
strengths: 1. **Efficient Training/Inference Acceleration**: TempMe achieves significant improvements in training efficiency by implementing image merging followed by progressive video clip merging. This approach leads to substantial reductions in GFLOPS and training time compared to previous methods.

2. **Comprehensive Ablation Studies**: Extensive ablation experiments validate the effectiveness of clip merging at different layers and intervals, as well as its applicability on stronger video-pretrained backbones like UMT and MetaCLIP. Qualitative results further show that TEMP effectively merges objects and backgrounds.

3. **Cross-Clip Merging**: TempMe introduces a relatively novel approach by incorporating cross-clip (inter-clip) merging, expanding beyond traditional intra-clip merging to capture cross-clip information, which enhances the learning of temporal relations across clips.
summary: This work proposes TempMe, a temporal token merging method for T2V retrieval, designed to address two main issues: (1) high inference costs in current PEFT methods, and (2) the tendency of token merging methods to overlook temporal redundancy. TempMe introduces a PMG framework, which first merges tokens within individual images and then progressively merges across video clips. Experimental results demonstrate TEMP’s effectiveness, showing improvements in GFLOPS, training time, and retrieval performance.
weaknesses: 1. **Limited Improvement in R@1 for T2V Retrieval**: The R@1 improvement over previous methods is relatively small, especially given the advances in MLLM models. In 2023, T2V retrieval methods like HBI and Cap4video on CLIP-ViT-32/16, R@1 have reached around 48/50 (e.g., [1][2]). I suspect the limited improvement may stem from TempMe still focusing on video merging within the encoder, without further optimization after obtaining the clip-level representation.

2. **Lack of Memory Usage Comparison**: Efficiency is not solely about GFLOPS—memory usage for training and inference is also crucial. While TempMe provides training acceleration, memory optimization would be even more beneficial. In Table 5, the authors report memory usage for CLIP-ViT-16, but it would be helpful to see detailed comparisons for CLIP-ViT-32 as well, particularly relative to previous methods.

3. **Limited Novelty**: The intra-clip merging process is highly similar to the original token merging approach [3], which limits the novelty of this method. In fact, the flowchart for intra-clip merging in Figure 3 closely resembles the diagram in Token Merging (ToMe), making it difficult to discern substantial differences. Essentially, the only distinct contribution is the cross-clip merging mechanism.

4. **Efficiency and Early Layer Merging**: Performing token merging in earlier layers of CLIP could theoretically reduce both GFLOPS and memory usage, though it might harm performance metrics like R@1. TempMe, however, only applies merging in the last three layers with a 12-6-3-1 merging pattern. A potential improvement could be to increase the input frames and match GFLOPS with previous PEFT methods, potentially enhancing R@1 without excessive memory usage. Additionally, it would be beneficial to provide memory usage details for CLIP-ViT-32 to understand whether further optimization is feasible.

5. **Metric Suggestion**: Presenting an overall sum@R can be unclear for readers. R@1 is generally a more meaningful metric than sum@R and would offer a clearer understanding of the model's performance.

[1] Jin, P. et al., "Video-Text As Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learning," CVPR, 2023.  
[2] Wu, W. et al., "Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?" CVPR, 2023.  
[3] Bolya, D. et al., "Token Merging: Your ViT but Faster," ICLR, 2023.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: Is any part of the method that is specifically designed for text-video retrieval or could it be applied for other text-video tasks as well?
[1] shares a similar idea, can you please summarize the similarities and differences between the proposed method and [1]?


[1] Yao, Linli, et al. "DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models." arXiv preprint arXiv:2405.20985 (2024).
soundness: 3
strengths: The paper addresses an important task, namely text-video retrieval and proposes a parameter-efficient method for this. One strength of the paper is that the proposed method was extended to video foundation methods such as UMT and various backbones showcasing the extensibility of the method. Also, the method is based on a well-known fact, namely the video has a lot of redundant information from frame to frame and the method is built upon that observation and compresses the redundant information. Additionally the method achieves good results both in term of performance and inference efficiency.
summary: This paper tackles the text-video retrieval task and specifically propose a parameter efficient method for this task. The proposed framework merges redundant tokens across adjacent video clips. The method has a good trade-off between inference and parameter efficiency and accuracy achieving state of the art results when compared to other parameter efficient methods.
weaknesses: - Abstract: the abstract is hard to follow and is not clear if the method addressed inference speed-ups or training time speed-ups or both.
- overall the quality of the writing can be improved because it's not straightforward to follow the paper, for example in the introduction the transitions between paragraphs are abrupt.
- the choice of sampling 12 frames for MSRVTT and 64 frames for ActivityNet seems arbitrary.

Meta-review: This paper proposed a new temporal token merging method for text-video retrieval to educe temporal redundancy and training/inference cost. The motivation is reasonable and the experiments are comprehensive, demonstrating efficiency, effectiveness, and generalization capabilities.  After rebuttal, the main weakness of this paper is that the level of novelty is not sufficient to warrant a score of 8. Since all reviewers leaned to accept this paper, I recommend accept to it.

Decision: Accept (Poster)