TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding
Authors: Shuhuai Ren
Sishuo Chen
Shicheng Li
Xu Sun
Lu Hou
Venue: EMNLP 2023 Findings (2023)
Keywords: Video-Language Understanding
Token Aggregation
Long-form Video Understanding

Abstract: Large-scale video-language pre-training has made remarkable strides in advancing video-language understanding tasks. However, the heavy computational burden of video encoding remains a formidable efficiency bottleneck, particularly for long-form videos. These videos contain massive visual tokens due to their inherent 3D properties and spatiotemporal redundancy, making it challenging to capture complex temporal and spatial relationships. To tackle this issue, we propose an efficient method called TEmporal-Spatial Token Aggregation (TESTA). TESTA condenses video semantics by adaptively aggregating similar frames, as well as similar patches within each frame. TESTA can reduce the number of visual tokens by 75% and thus accelerate video encoding. Building upon TESTA, we introduce a pre-trained video-language model equipped with a divided space-time token aggregation module in each video encoder block. We evaluate our model on five datasets for paragraph-to-video retrieval and long-form VideoQA tasks. Experimental results show that TESTA improves computing efficiency by 1.7 times, and achieves significant performance gains from its scalability in processing longer input frames, e.g., +13.7 R@1 on QuerYD and +6.5 R@1 on Condensed Movie.

TESTA: Temporal-Spatial Token A Aggregation for Long-form Video-Language Understanding
Shuhuai Ren', Sishuo Chen Shicheng Lit, Xu Sun', Lu Hou
NNtiooall Key Laboratory for Multimedia Information Processing,
School of Computer Science, Peking University
CCenter for Data Science, Peking University "Huawei Noah's Ark Lab shuhuai pwu. enu.cn (lisc99, chensishuo, xusunJēpku.. edu. cn houlu3@huawei. com
Abstract
et al., 2021; Tong al., 2022). In fact, the number of visual tokens processed by Transformer- based models (Fu et al., 2021; Cheng et al., 2022: Ye et al., 2022; Li et al., 2021a; Wang et al., 2022b) can be over 150x more than text tokens. This poses an efficiency bottleneck for video-language understanding, especially for long-form videos last- ing more than 30 seconds (Wu and Krähenbühl, 2021; Sun et al., 2022).
Large-scale video-language pre-training has made remarkable strides in advancing video- language understanding tasks. However, the heavy computational burden of video encod- ing remains formidable efficiency bottle- neck, particularly for long-form videos. These videos contain massive visual tokens due their inherent 3D properties and spatiotem- poral redundancy, making it challenging to capture complex temporal and spatial rela- tionships. To tackle this issue, we propose an efficient method called TEmporal- Spatial Token Aggregation (TESTA). TESTA con- denses video semantics by adaptively aggregat- ing similar frames, as well similar patches within each frame. TESTA can reduce the num- ber visual tokens by 75% and thus acceler- ate video encoding. Building upon TESTA, we introduce pre- pre-trained video-language model equipped with a divided space-time to- ken aggregation module in each video encoder block. We evaluate our model on five datasets for paragraph-to-vine retrieval and long-form VideoQA tasks. Experimental results show that TESTA improves computing efficiency by 1.7 times, and achieves significant performance gains from its scalability in processing longer input frames, e.g., 13.7 R@ on QuerYD and +6.55 R@ on Condensed Movie.!
To encode long videos within limited computing budgets, previous approaches can be broadly cate- gorized into two types: (1) Sparse Sampling (Lei et al., 2021; Sun et al., 2022; Lei et al., 2022). This method reduces the number of visual tokens by sampling very few frames from the raw video. However, sparse sampling sacrifices rich temporal dynamics and storyline information, which limits model performance. (2) Offline Encoding (Luo et al., 2021; Bain et al., 2022). allows process- ing more frames within the same computation bud- gets by constraining the interaction between vi- sual tokens. It first uses off-the-shelf image encoder (Dosovitskiy et al., 2020; Radford et al., 2021) encode each frame independently, then uses a temporal module aggregate all the frame features. However, the frame features encoded offline may not be well adapted to downstream tasks in various domains. Additionally, the post- aggregation mechanism also prohibits the full fu- sion of frame features (Cheng et al., 2022). Con- sidering that both sufficient input frames and full temporal-spatial modeling in an end-to-end manner are pivotal for optimal performance. nat- ural question arises: Are there better approaches achieve efficient video coding without compromis- ing on either of these aspects?
1 Introduction
Video-language modeling aims to learn seman- tie alignment between video and language in joint representation space (Xu et al., 2021; Lei al., 2021) facilitate downstream tasks includ- ing text-video retrieval, video question answering (VideoQA), and video captioning. Unlike text, which can be represented concisely as a sequence of words with dense semantics, video input consists of much longer sequences due to its 3D properties and the redundancy in space-time information (He
For example, the QuerYD dataset, long-form video with sampled frames al resolution 224 224 pixels generates around 19K visual tokens after patchification, while the corresponding caption contains only 128 text tokens.
'Our cnde is available at httpps://gitee com/ RenShuhuai- Aww.huuui--nnn/ESSTT..
For instance, sample 4 frames from more than 5.4K frames for ActivityNet Captions dataset (Krishna et al., 2017).

---

Caption: flay steps grass
boy porch. reaches walksoer
Video Token:
1x18816 cub
Asgrepation Method
alobals di toturs Complestty:
TESTA (Ours)
pocudo
patch toben 116x15 patchi
Aszragation Mithhed:
in space Complessty:: 11
Input 96 frames
Patchfled:
52 frames
8 frames
Input video
196 patches
101 patches
11 patches
Temporal Aggregation
Spatial Aggregation
Figure 1: Two blocks on the left compare ToMe (Bolya et al., 2022) and our TESTA on three aspects: video token definition, aggregation method, and computation complexity. The block on the right illustrates TESTA's 's divided temporal aggregation (left) and spatial aggregation (right). Patches sharing the inner and border colors are merged together. Our aggregation gradually reduces the number of frames and patches by averaging their features during the forward process video encoding.
In this paper, we propose an efficient method named TEmporal-Spatil Token Aggregation (TESTA) inspired by Token Merging (ToMe) (Bolya et al., 2022). Specifically, TESTA samples input frames densely, but pro- gressively aggregates similar visual tokens during video encoding to reduce the token number and computational overhead. As shown in Fig. 1, our aggregation operates separately in temporal and spatial dimensions, allowing for the merging of similar frames well similar patches within each frame. This reduces ToMe's complexity from (( ((THW))) 2) O + ( 16) )2), making more efficient for encoding longer videos. After aggregation, around 75% visual tokens can be reduced and thus the video encoding is accelerated. To achieve this, we use the bipartite matching algorithm. Specifically, we select set of tokens and then find their most similar counterparts from the remaining set. Finally, aggregate the features of these pairs through mean pooling. This aggregation-based mechanism has three advantages: First, it does not incorporate addi- tional parameters and amenable to parallelism, which significantly improves the training and inference efficiency; Second, our method (1) adaptively condenses video semantics rather than directly discarding input information, (2) retains full end-to-end spatiotemporal fusion, which both ensure the performance. Third, compared convolution based feature down-sampling methods (Liu et al., 2021; Li et al., 2021c), our aggregation trajectory can be casily tracked and
recovered. The aggregated tokens often correspond to higher-level semantics (c.g.. objects, scenes, and events), making them more interpretable and even grounded in language.
Building upon TESTA, we design a pre-trained video-language model with temporal and spa- tial token aggregation module in each video en- coder block. We evaluate our model paragraph- to-video retrieval and long-form VideoQA tasks. When using an equal number of input frames, our model improves computing efficiency by 1.7 times while maintaining comparable performance. When accessing more frames, our model exhibits strong scalability and achieves significant performance gains compared to previous state-of-the-art meth- ods (e.g.. +13.7 13.77M R@ I on QuerYD and +6.5 R @ Condensed Movie).
Related Work
Video-Language Pre-trained Models. Benefit- ting from large-scale video-text datasets (Bain al., 2021; Xue et al., 2021) and advances in Trans- former model design (Gorti et al., 2022: Ren et al., 2021; Fu et al., 2021; Zellers et al., 2021; Wang et al., 2022a), pre-trained Video-Language Mod- els (VidLMs) (Chen et al., 2022; Sun et al., 2022; Cheng et al., 2022) have demonstrated impres- sive performance in video-language understanding tasks. VidLMs typically comprise video encoder and a text encoder, which encode video-text pairs into shared feature space learn the semantic alignment between video and language. Addition- ally, a text decoder can be added after the video

---

Lue
LyTM
CA
Mlrre the two tokens the then form
Video Encoder Block
Text Encoder Layer
Video grounded Text Encoder Layer
Video Text Decoder Layer
Feed forward
Feed Forward
Feed Forward
Feed Forward
Spatial Aggregation
Spatial Attention
(T x x DA
Attention
Attention
Temporal Aggregation mM
For each toson Set Ain its most siilaaten takenss from See
Self Attention
shaned
Self Attention
Causal Attention
Temperal
Mx
[CLS]
Linear Projection
*[CLS]+
+[SEP)"
*[ENC]+
+[SEP)
"[DEC]+
Assign talkans Set A Set
Aggregation Algorithm
Child walks walkway.
Architecture of our pre-trained model and token aggregation algorithm of TESTA. We record the size of the input and output features red. The circles the left panel denote either patch tokens or frame tokens.
Figure 2:
encoder for tasks such video captioning and VideoQA (Yan et al., 2022; Zhang al., 2020),
et al., 2023; Cao et al., 2023; Xu et al., 2022; Ryoo et al., 2021; Marin al., 2021). In contrast, we propose unified mechanism simultaneously ag- gregate frames and patches. Our method gradually aggregates features during video encoding, improv- ing efficiency while ensuring sufficient interaction between features in both space and time.
Efficient Video Transformer. A Transformer- based video encoder typically pachifies each video into massive visual tokens, which will cause pro- hibitive computation costs for full self-attention with quadratic computational complexity. There- fore, research on efficient video Transformers has always been active. Representative work like TimeSFormer (Bertasius al., 2021) and ViVIT (Arnab al., 2021) propose to factorize the spatial and temporal dimensions of the input, then separately apply spatial and temporal attention. Video Swin Transformer (Liu et al., 2021) ) keeps the joint temporal-spatial attention but restricts it within local 3D window. Orthogonal to the ad- vances of efficient Transformer architectures, our TESTA aggregates token features from the spatial and temporal dimensions, which reduces the size of input features for each Transformer block and can further boost the efficiency video encoding.
Method
In this section, we first introduce our video- language pre-trained model and its architecture 3.1. To improve the efficiency of encoding long- form videos, we propose a novel temporal-spatial token aggregation mechanism ($ 3.2). Finally, we present the pre-training objectives in 3.3.
3.1 Model Architecture
Inspired by prevalent VidLMs (Li et al., 2022, 2021b), our model consists of three encoders and one decoder for video-language representation learning. Figure shows the model architecture.
Feature Aggregation Video Transformers. Existing feature aggregation methods can be broadly categorized into two branches. Temporally, frame features can be encoded by pre-trained im- age encoder and aggregated using self-attention, joint-attention, or mean pooling for post-temporal modeling purposes (Bain al., 2022; Luo al., 2021). Spatially, previous work explored merging similar patches in the image or aggregating tokens into additional proxy tokens (Bolya et al., 2022; Shi
Text Encoder. The text encoder is uni-modal encoder similar BERT (Devlin al., 2019). A [CLS] token is prepended the beginning of the input text to represent its global feature.
Video- gedeo-groouned Text Encoder. This is cross- modal encoder. Compared the uni-modal text encoder, we add cross-modal module to each en- coder layer to enable information flow from video to language. We insert an [ENC] token before the

---

input text condense the cross- moosl information from both video and language.
feature after temporal and spatial token aggrega- tion is V E - smaller size and reducing the computing burden for subsequent blocks. After the forward process with M encoder blocks, the final number of visual tokens is reduced to (T MT) x (L MM)..
resulting in a
Video-grounded Text Decoder. This is cross- modal decoder with causal self- atteetttenton for auto- regressive text generation.
Video Encoder. This is uni-modal encoder. Given a raw video, the visual input V E RT*xxWxx3 is a sequence of T RGB frames of size H x W sampled from the video. Each frame is split into L non-overlapping patches* following ViT (Dosovitskiy et al., 2020). To represent the global video feature, an additional [CLS] token is also used. Our video encoder is similar to TimeS- Former (Bertasius et al., 2021) with the Divided Space-Time Attention. Specifically, each video en- coder block captures the temporal relations across frames using Temporal Attention and fuses the spa- tial information of objects, scenes, etc., within each frame using Spatial Attention. In contrast to TimeS- Former, we improve the efficiency of video encod- ing by equipping each video encoder block with Temporal Aggregation Module and Spatial Aggre- gation Module, which we will introduce in 3.2.
3.2.1 Objects for Aggregation
Our video encoder based TESTA involves two types of tokens for aggregation: patch tokens and frame tokens. Recall that each frame divided into a sequence of patches, which are treated as patch tokens. To ensure formally unified aggregation algorithm, we define frame tokens as pseudo to- kens to represent each frame by averaging all the patch tokens within it. When merging two frame to- kens, the corresponding I. patches [p() P/ in frame-1 and L patches [p] ......P. frame-2 are merged, resulting in patches [p] (1&2) , p( As our aggregation strategy agnostic the token type, we refer to both patch tokens and frame tokens as "tokens" throughout the rest of the paper, without loss of generality.
(2) in
3.2.2 Aggregation Strategy
3.2 Temporal-Spatial Token Aggregation
Recall that given sequence of N tokens, our target is to reduce R. tokens after each aggregation opera- tion.5 5 To achieve this, we can greedily merge two tokens with the highest similarity and then repeat R times, or merge N tokens into N R clusters us- ing clustering algorithms such as k-means (Lloyd, 1982). However, these iteration- baasd methods are not suited for parallelism and can slow down en- coding speed (Bolya et al., 2022). Therefore, we resort to the bipartite matching method. We first partition the N tokens into two disjoint sets A and B with R and N R tokens, respectively. The R tokens the set are selected elaborately as the tokens to be reduced. For each token in the set A. we find its most similar token from the set B. then merge them by averaging their features. As a result, the remaining N - R tokens in the set B form a new sequence as the output.
Videos have heavy spatiotemporal redundancy (He et al., 2021; Tong et al., 2022). On one hand, some activitics (e.g., conversations) can persist across multiple frames with little visual variations. On the other hand, some scenes like background of- ten contain numerous indistinguishable patches in each frame. Aggregating these similar frames and patches can simplify video feature representation and accelerate video encoding.
Accordingly, we introduce a Temporal Aggre- gation Module (TAM) and a Spatial Aggregation Module (SAM), i.c., the yellow modules in Fig- ure 2. After each aggregation, TAM reduces Ry frames while SAM reduce Rts patches, where Rt and Rs are hyper-parameters control the trade- off's between performance and efficiency. TAM and SAM are incorporated into each block of the video encoder, aggregating tokens progressively to re- duce their number. For the i-th Transformer block, let V E X represents the input video fea- ture, where T, L,, D denote the number of frames, the number of patches per frame, and the dimension of the token feature, respectively. The output video
For similarity calculation, we utilize the atten- tion keys (K) of tokens as features and measure their similarity using cosine similarity. The atten- tion keys contain summarized information intended for use in QKV self-attention, yielding accurate similarity measures (Bolya et al., 2022).
'The size of each patch is P x P, and the 1. patches span the entire frame (L = HW/P2).
For temporal aggregation, N = T' and R = Rr, for spatial aggregation, N = L. and R = Rs-

---

In practice, we introduce two aggregation al- gorithms, i... importance based aggregation and geometry-based aggregation.
Video Token Definition. ToMe uses joint space- time tokens (2 x 16 x 16 cubes), while our TESTA defines frame tokens (representing entire frames) and patch tokens (16 x 16 2D patches) for decou- pled aggregation. This tailored token design is more efficient for modeling long-form videos.
Importance-based Aggregation. In this algo- rithm, we pick out the least important R tokens into the set for aggregation, so as to minimize the negative effects token reduction. The impor- tance of the token i; is measured by the following score function S,, which is defined as the product of the attention it receives from the other tokens クj-LifiA
A ggregattion Method. ToMe performs global ag- gregation over all tokens, resulting in a complexity of ( H 16 2)). This becomes impractical for long more-fommoo video and causes out-of- issues beyond 16 frames. In contrast, TESTA uses divided aggregation in time and space, reducing complexity to O + ( W T6 )2). This allows efficient encod- ing of much longer videos (more than 128 frames under the same computation quota). The divided scheme also better captures spatial and temporal semantics, resulting in improved performance on long-form video understanding tasks (to be shown in $ 4.7).
N = J=1
Σ
softmax
QK vd
(1)
where Aji is the attention score from token Tj to Ty Q and K represent Queries and Keys in self- attention, respectively.
Geometry-based Aggregation. In practice, we notice that adjacent tokens have a larger similarity and should be merged. However, these adjacent tokens also have similar importance scores and thus are prone be grouped into the same set in importance-based strategy, which hinders their ag- gregation. To address this issue, we partition the N tokens in an alternative way inspired by Bolya et al. (2022), thus assigning adjacent tokens to different sets A and B. As shown in the left panel in Figure 2, for each token (*( in the set A, we find its most
3.3 Pre-training Objectives
We use the following three classic pre-training ob- jectives, i.e., video-text contrastive loss, video-text matching loss, and captioning loss. Please refer Appendix for more details.
4 Experiments
4.1 Implementation Details
To pre-train our TESTA model, we start by initializing it with the BLIP (12-layer ViT-B/16) checkpoint (Li et al., 2022), with the exception of the temporal attention, which is copied from the spatial attention weights. We use around 5M image-text and video-text pairs from two datasets for pre-training. See Appendix for more details.
(B) similar token z from the set B construct a pair
(x) .
(B)) and record their similarity. After that, we select R pairs with the greatest similarity and merge the two tokens in the top-R pairs. Finally, we concatenate the tokens in the two sets back into one sequence as the output.
For downstream fine-tuning, we uniformly sam- ple either 32 or 96 frames, each with a resolution of 224 x 224 pixels (196 patches per frame with a patch size of 16). To achieve approximately 50% reduction in computation cost, we employ differ- ent hyper-parameters for aggregation. Specifically, for 96-frame inputs, set RT to 4 and Rs 8, while for 32-frame inputs, Rr is 1 and Rs is 12. We use geometry- based aggregation by default since achieves better performance. Please refer to Appendix B for more fine-tuning details.
The above aggregation algorithms are parameter- free, and can be easily plugged into a Transformer- based video encoder. We conduct our aggregation during both training and testing. Although the to- ken similarity calculation brings additional com- puting overhead, it is negligible compared to the efficiency gained by reducing token numbers.
3.2.3 Novelty over Token Merging
Our work is inspired by Token Merging (ToMe) (Bolya al., 2022), which also proposes reduce video tokens by merging similar ones. However, we differentiate ourselves from ToMe in two significant ways:
4.2 Downstream Task Setups
We finetune and evaluate TESTA on two down- stream tasks of pirigra retrieval and

---

QuerYD
Condeused Movic
Method
#PT Data
#raame
GFLOPs 4
R@IT
R@51
R@
R@l1
R@51
101
MoEE (Miech et al., 2018)
11.6
30.2
43.2
7.8
13.4
TeachText (Croitoru et sl., 2021)
14.4
37.7
50.9
12.1
27.4
37.5
Frozen el 2021)
5M
LF-VILA (Sun 2022)
298
32.5
41.8
VINDLU (Cheng 2022)
25M
745
67.8
86.3
81.8
18.4
36.4
44.3
TESTA (Ours)
5M
420
77.0
91.3
94.6
21.5
42.4
50.7
TESTA w/o agg.
5M
786
79.7
92.6
95.5
45.4
54.8
TESTA (Ours)
1381
83.4
46.5
55.1
TESTA agg.
5M
2383
84.2
93.8
95.1
46.8
56.0
Table Paragraph-to to- video retrieval performance @Recall@kk) k) on QuerYD and Condensed Movie. #PT Data refers to the number of video-text pairs used for pre-training. f indicates the results of our re-implementation. TESTA w/o agg. denotes fine-tuning our pre-trained model without activating the token aggregation modules, resulting in reduction in token number. This serves as upper bound for TESTA's performance.
DiDeMo
ActivityNot Caption
Method
#PT Data
#Trame
GFLOPs 4
R@5
R@
R@I1
RØ5
R010f
TeachText (Croitoru al., 2021)
21.6
48.6
62.9
23.5
57.2
ClipBERT' (Lei et 2021)
0.2M
2
20.4
48.0
60.8
21.3
49.0
63.5
Frozen (Bain et al., 2021)
5M
4
178
31.0
72.4
LF-VILA (Sun ul., 2022)
298
35.0
65.4
ALPRO
2021a)
5M
197
35.9
67.5
78.8
BridgeFormer (Go al., 2022)
5M
37.0
62.2
73.9
Singularity (Lei al., 2022)
SM
32
589
47.4
75.2
84.0
43,0
70.6
81.3
HITxA (Ye al., 2022)
5M
12
98
51.8
79.1
85.3
45,1
73.5
84.2
VINDLU (Cheng el al., 2022)
SM
4
93
$4.6
81.3
89.0
51.1
79.2
88.4
All-in-one (Wang et al., 2022a)
138M
3
62
32.7
61.4
73.5
22.4
53.7
67.7
Clip4Clip (Lub et 2021)
400M
282
43.4
70.2
80.6
40.5
72.4
X-CLIP (Mael al., 2022)
400M
1086
47.8
79.3
46.2
75.5
CLIP-ViP al., 2022)
100M
212
50.5
87.1
81.4
90.0
TESTA (Ours)
5M
32
420
57.7
83.3
89.4
51.7
79.1
87.6
TESTA (Ours)
5M
96
1381
59.2
83.5
89.8
53.7
79.9
88.9
Table 2: Paragraph to- video retrieval performance on DiDeMo and ActivityNet Caption. We gray out methods that use significantly more pre-training data a fair comparison. The other notations are the same as those on Table 1.
Method
#PT Data
Accuracy (%)
4.3 Paragraph-to- Videeo
Video Retrieval
VI-IILA (Sun al., 2022)
8M
39,9
Table demonstrates the performance TESTA on two challenging and under-explored paragraph- to-video retrieval datasets, QuerYD and Condensed Movie, which involve videos with lengthy dura- tions (over 200 seconds on average). For 32-frame video inputs, TESTA achieves Recall @ of 77.0 on Qu and 21.5 Condensed Movie, sur- passing previous SOTA methods by 7.3 and 3.1, respectively. In terms of computational complex- ity, TESTA exhibits a significantly lower GFLOPs of 120 compared to Frozen (Bain et al., 2021) and VINDLU (Cheng et al., 2022). While LF- VILA (Sun et al., 2022) operates with even fewer GFLOPs (298), it necessitates feature aggregation within fixed local window, which can potentially undermine semantic integrity after concentration.
Singularity al., 2022)
41.8
VIOLET (Fu al., 2021)
183M
38.9
JustAsk (Yang et al., 2020)
69M
38.9
MERLOT (Zellers et al., 2021)
180M
41.4
TESTA (Ours)
5M
45.0
Table Accuracy (%) on ActivityNet-OR
long-form VideoQA. For paragraph-to-ve viviidooo re- trieval, we use four datasets: DiDeMo (Hendricks et al., 2017), QuerYD (Onceseu et al., 2020), Ac- tivityNet Captions (Krishna al., 2017), and Con- densed Movie (Bain et al., 2020). For long-form VideoQA, we use ActivityNet-OA (Yu et al., 2019). The details these datasets are shown in Ap- pendix C.
In contrast, our model enables the adaptive merg- ing of features with high similarity in the global scope, resulting in improved performance (+7.6

---

QuerYD
DiDeMo
ActivityNet Caption
Method
#PT Data
GFLOPs
R@IT
R@ST
10
R@1T
R@57
R8
R@IT
R@5t
R0101
ClipsCClip [Lao et 2021)
400M
282
50.0
74.5
83.3
43.6
71.3
79.0
25.0
51.6
65.7
BLIP (1. et al., 2022)
129M
707
50.7
67.6
71.5
60.9
84,9
91.0
34.2
60.0
70.7
TESTA (Ours)
5M
786
64.4
82,9
86.9
64,9
88.7
91.8
37.1
63.7
75.4
Table 4: Zero-shot evaluation (32 frames) on paragraph-to-video retrieval performance.
TESTA
R@ 1
R@51 1
R@ 101
Avg. 1
GFLOPs
Memory (GB) L
No Aggregation
84.2
93.8
95.1
91.0
2382.5
19.2
(1) Token Aggregation v.s. Token Pruning (w/o training for botk)
Token Pruning (Rr = Rs
71.0
86.1
90.6
82.6
1380.9
12.6
Token Aggregation (Ry 4. Rg
79.2
91.8
95.3
88.8
1381.4
12.6
(2) Aggregation Strategy
Impottance-basee Aggregation
80.2
91.7
94.6
88.9
1380.9
Geometry- boaoeety-bbaeddde Aggregation
83.4
93.8
95.3
90.8
13814
12.6
(3) Aggregation dimension
Only temporal (Rr =
79.5
92.9
95.4
89.3
1303.9
Only spatial (Rg 14)
81.4
93.3
95.1
89.9
1364.0
11.9
Both temporal and spatial (Rt = 4, Rs 8)
83.4
93.8
95.3
90.8
1381.4
12.6
Table 5: Ablation study on (I) token reduction method, (2) aggregation strategy, and (3) aggregation dimension.
The results are reported on QuerYD with frames. Avg. represents average recall across 1, R@5, R@ 10.
R@ @ on average compared to LF-VILA).
4.4 Long-Form Video Question-Answent
Table showcases the performance of TESTA on ActivityNet-QU QA (using 96-frame). The accuracy of TESTA is 45.0%, which is 3.2% higher than the previous SOTA, Singularity (Lei et al., 2022). This demonstrates that our method eliminates redundant information while integrating crucial visual cues to accurately answer the posed questions.
Given the importance of incorporating more in- put frames for long vidco understanding tasks, we finetune TESTA using 96-frame inputs and further promote R to 83.4 QuerYD and 24.9 on Con- densed Movie. This exhibits strong scalability of our model (see Appendix D for a detailed analy- sis). Additionally, we report the results of TESTA without token aggregation, which serves as an up- per bound for TESTA's performance. Although preserving full visual tokens yields higher recall, it requires 1.8 times more GLFOPs compared to TESTA. As the number of input frames increases from 32 96, the GFLOPs of TESTA w/o agg. exceed 2300, but the performance gain diminishes (only +0.8 R @@ 1 on QuerYD). This indicates the superiority our method in aggregating redundant tokens in long sequence inputs.
4.5 Zero-shot Generalizability
In Table of pre-trained CLIP4clip, BLIP, and TESTA on three datasets (32 frames). Although our TESTA is initialized by the BLIP checkpoint, it consistently outperforms BLIP (as well as CLIP4clip) after our pre-training, achieving average improvements of + +1..1, +2.9, and +-3.8 on QuerYD, DiDeMo, and ActivityNet respectively. This indicates sub- stantial gains long-form video datasets not solely due the strong BLIP checkpoint, but also owing to our temporal modeling and pre-training on video data.
show the zero-shot performance
Table 2 demonstrates model performance on DiDeMo and ActivityNet Caption, which consist of shorter videos (~000 seconds on average) and are considered less challenging. For 32-frame inputs, TESTA with 5M pre-training data achieves 57.7 R@1 I on DiDeMo, which even surpasses the models pre-trained with over 100M data. By increasing the number of frames to 96, TESTA achieves R@1 of 59.2 on DiDeMo and 53.7 on ActivityNet, outperforming previous SOTA methods by 2.7 and 2.6, respectively.
4.6 Ablation Study
We perform an extensive ablation study and analy- sis on various crucial components in our aggrega- tion algorithm to examine their impacts.
Token Aggregation v.s. Token Pruning. We first compare the performance and efficiency

---

Frames
Agg- tokens
Caption: We see baseball park for the first time. The first time the baseball stadium shown.
Visualization of our temporal and spatial aggregation. Frames that are enclosed within the same red rectangle, as well as patches that share the same inner and border color, merged together.
Figure 3:
token aggregation and token pruning (Rao et al., 2021). Regarding pruning, we calculate the im- portance score (Eq. (1)) for each token and prune the least important R tokens following previous methods (Goyal et al., 2020). We finetune our pre-trained model QuerYD without token aggre- gation, then apply token aggregation and pruning in an fff-the--helll manner for test evaluation. The results are presented the first block Table In comparison to the vanilla model (no aggregation), both pruning and aggregation decrease computa- tion costs, with only 58% GFLOPs and 66% GPU memory. However, the performance degradation of our token aggregation is much smaller than that of pruning (-2.2 v.s. -8.4 call), suggesting that aggregation better preserves the valuable visual semantics within videos.
a single dimension leads to excessive dilution information, while the information in other dimen- sions becomes overly redundant. This imbalance hurts the performance of the model. Therefore, our approach, with incorporates both temporal and spatial aggregation, achieves the best outcomes.
Additionally, Appendix discusses the impact of the number of reduced tokens Ry and Rs. Ap- pendix analyzes the properties of aggregated to- kens by probing their similarity.
0.03
0.03
0.49
8.4 in terms of average re-
0:06
0.15
0.06
0.04
0.12
Ablation on the Aggregation Strategy. To in-
0.06
vestigate the effectiveness of different aggregation strategies, we report the performance of TESTA using importance-based and geometry-based aggre- gation methods. The results in the middle block of Table 5 show that the simplest geometry-based aggregation method achieves the best Recall @ I of 83.4, outperforming the other method by 3.2. This confirms our hypothesis that adjacent tokens ex- hibit greater similarity and should be assigned to separate sets for aggregation.
S1: A girl in a pirate hat walks towards the camera.
0.03
0.05
0.14
0.04
0.07
0.29
0.08
$2: A dog walks up to a little girl then walks away.
Figure 4: Text grounding visualization. Fi denotes the tt frame in the video and Si denotes the 7th sentence caption. We calculate the similarity between the phrase query (in orange) and each region formed by our aggregation, then record the value in the region. The phrase queries can grounded to their corresponding aggregated regions, achieving the highest similarity.
Ablation on the Aggregation Dimension. We compare the performance of three aggregation methods: (1) temporal only, (2) spatial only, and (3) both temporal and spatial. To ensure roughly equal computational overhead, we adjust Rş and Ry accordingly. The results in the bottom block of Table 5 show that performing token aggregation on

---

Method
GFLOPs 4 R@
R@51 f
R@ f
kens (Ren et al., 2023b,a). Given the phrase query in the caption, it achieves the highest similarity of its oracle region formed by our aggregation, facili- tating fine- reee-grrind alignment between phrases and regions.
ToMe
252
59.9
82.2
88.6
TESTA
228
62.4
85.6
91.1
ToMe w/o agg.
450
66.1
86.4
90.4
TESTA w/o agg.
392
75.0
91.1
93.8
Table 6: Paragraph-to-video retrieval performance @Recall@k) on QuerYD (16 frames). w/o agg. de- notes fine-tuning without token aggregation; the only distinction lies in the attention mechanism. where ToMe employs global attention, while TESTA utilizes separate apatial-tempon attention.
Conclusion
In this paper, we present TESTA, an efficient method for long-form video-language understand- ing. By aggregating similar frames and patches, TESTA effectively condenses video semantics and accelerates video encoding. Experimental results on paragraph-to-viee retrieval and VideoQA tasks demonstrate that TESTA outperforms previous SOTA methods by considerable margin.
4.7
Comparison to Token Merging
We directly compare the performance of ToMe (Bolya et al., 2022) and TESTA by ini- tializing both models from the BLIP pre-trained checkpoint and fine-tuning them on QuerYD. As we noted in $ 3.2.3, due the extremely high computational complexity of ToMe's global attention, increasing the number of input frames can lead to out-of-memory issues without token aggregation (w/o agg.). Therefore, we limit the number of input frames to 16. Besides, We set the hyperparameter R (number of reduced tokens) ensure matched GFLOPs. Specifically, for ToMe, R 197, while for TESTA, RT and Rs = 2. The results in Table illustrate TESTA's efficiency and effectiveness for long-form video understanding, which can be attributed to our tai- lored design for divided spatial-temporal modeling, In comparison to ToMe, our approach achieves higher recall with fewer GFLOPs, regardless of whether token aggregation is applied.
Limitations
To facilitate future research, analyze the lim- itations and possible solutions in work. (1) Due to limited computing resources, we do not use long-form video pre-training datasets such as HD- VILA (Xue et al., 2021) or incorporate TESTA pre-training. We believe long video pre-training with TESTA could greatly improve pre-training ef- ficiency and obtain a video-language model with better performance. (2) For aggregation efficiency, only use video-side features to merge visual tokens. We believe that leveraging text signals for aggregation could make the final encoded features more suitable for downstream tasks. (3) Our model training only uses coarse objectives such as VTC, VTM, and CAP (Eq. (2)-(4) on video-text pairs. Considering TESTA can aggregate tokens into ob- jects, scenes, events, etc., training with fine-grained alignment functions (Ren et al., 2021; Wang et al., 2022c) could help some tasks like action local- ization and video object detection (Zhukov et al., 2019; 9; Real et al., 2017), on which we will perform more explorations in future work.
4.8 Visualization
Figure provides visualization of temporal and spatial aggregation on the DiDeMo dataset. TESTA effectively aggregates tokens with highly-similar semanties, demonstrating its strong interpretability. From temporal perspective, TESTA aggregates a sequence of frames captured during continuous lens movement (first frames). also condenses similar frames of athletes waiting for the game (last 3 frames). From spatial perspective, TESTA merges the patches belonging to the same scenes (e.g., sky, baseball park) and the same objects (e.g., billboard, back of the audience's head). More ex- amples can be found in Appendix G.
Acknowledgements
We thank all the anonymous reviewers for their constructive comments, and Rundong Gao and Lei Li for their valuable suggestions in preparing the manuscript. This work is supported in part by a Huawei Research Grant and National Natural Sci- ence Foundation of China (No. 62176002). Xu Sun is the corresponding author of this paper.
In Figure 4, we further show that TESTA enables grounding of language to the aggregated visual to-

---

References
Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020. An image is worth 16x 16 words: Transformers for image recognition scale. ArXiv, abs/2010.11929,
Anurag Amab. Mostafa Dehghani. Georg Heigold. Chen Sun, Mario Lucic, and Cordelia Schmid. 2021. Vivit: A video vision transformer. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 68 6816-6826.
Max Bain, Arsha Nagrani, A. Brown, and Andrew Zisserman. 2020. Condensed movies: Story based retrieval with contextual embeddings. ArXiv, abs/2005.04206.
Tsu-Jui Fu, Linjie Li. Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. 2021. Violet End-to-end video-language trans- formers with masked visual-token modeling. ArXiv,
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. 2021. Frozen in time: A joint video and image encoder for end-to-end retrieval. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1708-1718.
Yuying Ge. Yixiao Ge, Xihui Liu, Dian Li, Ying Shan, Xiaohu Qie, and Ping Luo. 2022. Bridging video- text retrieval with multiple choice questions. 2022 IEFF/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 16146-16155.
Max Bain, Arsha Nagrani, Güi Varol, and Andrew Zis- serman. 2022. A hitphhitchhike''s guide to long video retrieval. ArXiv, abs/2205, 005588.
Satya Krishna Gorti, Noël Vouitsis, Junwei Ma, Key- van Golestan, Maksims Volkovs, Animesh Garg. and Guangwei Yu. 2022. X-pool: Cross-modal language-video attention for text-video retrieval. 2022 IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 4996-5005.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. 2021. Is space-time attention all you need for video understanding? In International Conference on Ma- chine Learning.
Daniel Bolya, Cheng- Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. 2022. Token merging: Your vit but faster. ArXiv, abs/2210.09461.
Saurabh Goyal, Anamitra R. Choudhury, Saurabh Raje, Venkatesan T. Chakaravarthy, Yogish Sabharwal, and Ashish Verma. 2020. Power-bert. Accelerating bert inference via progressive word-vector elimination. In International Conference Machine Learning.
Qingqing Cao, Bhargavi Paranjape, and Hanna Ha- jishirzi. 2023. Pumer: Pruning and merging to- kens for efficient vision language models. ArXiv, abs/2305.17530.
Kaiming Hc, Xinlei Chen, Saining Xie, Yanghao Li, Pi- otr Doll' ar, and Ross B. Girshick. 2021. Masked autoencoders are scalable vision learners. 2022 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 15979-15988.
Soravit Changpinyo, Piyush Kumar Sharma, Nan Ding, and Radu Soricut. 2021. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long- tail visual concepts. 2021 IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3557-3567.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsu- pervised visual representation learning. In Proceed- ings of the IEEE/CVF conference computer vision and pattern recognition, pages 9729-9738.
Dongsheng Chen, Chaofan Tao, Lu Hou, Lifeng Shang, Xin Jiang, and Qun Liu. 2022. Litevl: Efficient video- language learning with enhanced modeling. In Conference on Empirical Methods in Natural Language Processing.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan C. Russell. 2017. Localizing moments in video with natural language. 2017 IEEE International Conference on Computer Vision (ICCV), pages 5804 5813.
Feng Cheng, Xizi Wang, Jie Lei, David J. Crandall, Mohit Bansal, and Gedas Bertasius. 2022. Vindlu: A recipe for effective vidco-and-language pretraining. ArXiv, abs/2212.0505.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Toana Croitoru, Simion-Vias Bogolin, Yang Liu, Samuel Albanie, Marius Leordeanu, Hailin Jin, and Andrew Zisserman. 2021. Teachtext: Crossmodal generalized distillation for text-video retrieval. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11563-11573.
Ranjay Krishna, Kenji Hata, Frederic Ren, Fei-Fei and Juan Carlos Niebles. 2017. captioning events in videos. 2017 IEEE International Confer- ence on Computer Vision (ICCV), pages 706-715.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. ArXiv, abs/18 18s//110 10.04805.
Jie Lei, Tamara L. Berg, and Mohit Bansal, 2022. Re- vealing single frame bias for video-and-land lnnguage learning. ArXiv, abs/2206.03428.

---

Jie Lei. Linjie Li, Luowei Zhou, Zhe Gan, Tamara Berg, Mohit Bansal, and Jingjing Liu. 2021. Less is more: Cliphert for video-and-language learning via sparse sampling. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
audio narrations. ICASSP 2021 2021 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2265-2269.
Rui Qian, Tianjian Meng. Boqing Gong, Ming- Lang-Hsuan Yang, Huisheng Wang, Serge Belongie, and Yin Cui. 2021. Spatiotemporal contrastive video representa- tion learning. In Proceedings of the IEEI/CV Con- ference on Computer Vision and Pattern Recognition, pages 6964-6974.
pages 7327-7337.
Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven C. H. Hoi. 2021a. Align and prompt: saleo-and-language pre-training with entity prompts. 2022 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4943- 4953.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Πlya Sutskever, 2021. Learn- ing transferable visual models from natural language supervision. In International Conference Machine Learning.
Jumnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022. Blip: Bootstrapping language- image pre- training for unified vision-language understanding and generation. In International Conference Ma- chine Learning.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. 2021. Dynamievit: Ef- ficient vision transformers with dynamic token sparsi- fication. In Neural Information Processing Systems.
Jumnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven C. H. Hoi. 2021b. Align before fuse; Vision and language representation learning with momentum distillation. ArXiv, 00w/2107.076511
Esteban Real, Jonathon Shlens, Stefano Mazzocchi. Xin Pan, and Vincent Vanhoucke. 2017. Youtube- boundingboxes: A large high- prccppreciioon human- annotated data set for object detection video. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7464-7473.
Yanghao Li, Chaoxia Wu, Haoqi Fan, Karttikeya Man- galam. Bo Xiong, Jitendra Malik, and Christoph Fe- ichtenhofer. 2021c. Mvitv2: Improved multiscale vision transformers for classification and detection. 2022 IEEE/CVF Conference on Computer Vision and Pullern Recognition (CVPR), pages 4794 4804.
Shuhuai Ren, Lei Li, Xuancheng Ren, Guangxiang Zhao, and Xu Sun. 2023a. Delving into the openness of CLIP. In Findings of the Association for Com- putational Linguistics: ACI. 2023. Association for Computational Linguistics.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. 2021. Vidco swin trans- former. 2022 IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 3192- 3201.
Shuhuai Ren, Junyang Lin, Guangxiang Zhao, Rui Men, An Yang. Jingren Zhou, Xu Sun, and Hongxia Yang. 2021. Learning relation alignment for calibrated cross-modal retrieval. In Annual Meeting of the As- sociation for Computational Linguistics.
Stuart Lloyd. 1982. Least squares quantization in pem. IEEE Trans. Inf. Theory, 28:12882121 139-136.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. 2021. Clip4elip: An empirical study of clip for end to end video clip retrieval. Neurocomputing, 508:293-304.
Shuhuai Ren, Aston Zhang, Yi Zhu, Shuai Zhang, Shuai Zheng, Mu Li, Alexander J. Smola, and Xu Sun. 2023b. Prompt pre-training with twenty-thousand classes for open- apen-vocabulary visual recognition. vol- ume 0ass/334404704.
Yiwei Ma. Guohai Xu, Xiaoshuai Sun, Ming Yan, Chao Zhang, and Rongrong 2022. X-elip: End- to-end multi malti-graiied contrastive learning for video- text retrieval. Proceedings of the 30th ACM Interna- tional Conference on Multimedia.
Michael S. Ryoo, A. J. Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. 2021. To- kenlearner: What can 8 learned tokens do for images and videos? ArXiv, abs/2 106. 1297.
Dmitrii Marin, Han-Hao Rick Chang, Anurag Ranjan, Anish K. Prabhu, Mohammad Rastegari, and Oncel Tuzel. 2021. Token pooling in vision transformers. ArXiv, abs/21 10.03860.
Dachuan Shi, Chaofan Tao, Anyi Rao, Zhendong Yang, Chun Yuan, and Jiaqi Wang. 2023. Crossget: Cross- guided ensemble of tokens for accelerating vision- language transformers. ArXiv, 12..2305.1455.
Antoine Miech, Ivan Laptev, and Josef Sivic. 2018. Learning a text-video embedding from incomplete and heterogeneous data. ArXiv, (a0o/8800.0516.
Yuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu, Huan Yang, and Jianlong Fu. 2022. Long-form video- language pre-training with multimodal temporal con- trastive learning. ArXiv, abs/22 10.06331.
Andreca-Maria Oncescu, João F. Henriques, Yang Liu, Andrew Zisserman, and Samuel Albanie. 2020. Queryd: A video dataset with high-quality text and

---

Zhan Tong. Yibing Song, Jue Wang, and Limin Wang. 2022. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre- training. ArXiv, 2222220312222..
Qinghao Ye, Guohai Xu, Ming Yan, Haiyang Xu, Qi Qian. Ji Chao Zhang, and Fei Huang. 2022. Hitea: Hierarchical lemporal-aware video-language pre-training. ArXiv, abs/2212.14548
Alex Wang. Yixiao Ge. Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. 2022a. All in one: Ex- ploring unified video languge pre-training ArXiv, 00//2203.07333.
Zhou Yu. D. Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019. Activitynet-qa: A dataset for understanding complex web videos via question answering. ArXiv, abs/1906.02467.
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jac Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021. Merlot: Multimodal neural script knowledge models. In Neural Information Processing Systems.
Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu- Gang Jiang, and Lu Yuan. 2022b. Omnivl: One foun- dation model for langeel and video-language tasks. ArXiv, abs/2209.07526
Zhihan Zhang. Zhiyi Yin, Shuhuai Ren, Xinhang Li, and Shicheng Li. 2020. Dea: Diversified co-attention towards informative live video commenting. In Natu- ral Language Processing and Chinese Computing.
Qiang Wang. Yanhao Zhang, Yun Zheng, Pan Pan, and Xiansheng Hua. 2022c. Disentangled repre- sentation learning for text-video retrieval. ArXiv, abs/2203.07111.
Dimitri Zhukov, Jean-Baptinn Alayrac, Ramazan Gok- berk Cinbis, David F. Fouhey, Ivan Laptev, and Josef Sivic. 2019, Cross-task weakly supervised learning from instructional videos. 2019 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 3532-3540.
Chaoxia Wu and Philipp Krähenbühl. 2021. Towards long-form video understanding. 2021 IEEE/CVE Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 1884-1894.
Hu Xu, Gargi Ghosh, Po- Yao Huang. Dmytro Okhonko, Armen Aghajanyan, and Florian Metze Luke Zettle-
moyer Christoph Feichtenhofer. 2021. Videoclip:
A Pre-training Details
Contrastive pre-training for zero-shot video-text un- derstanding. In Conference on Empirical Methods in Natural Language Processing.
A.1 Pre-training Datasets.
We perform pre-training on two datasets: Web 2M (Bain et al., 2021) containing 2.5M video-text pairs and Conceptual Captions (CC3M) (Chang- pinyo et al., 2021) consisting of 3M image-text pairs. We include CC3M to improve spatial rep- resentations of videos suggested by Li et al. (2021a). We duplicate images from CC3M for 8 times to make static videos. For WebVid-2M, we randomly sample 8 frames for each video instance. Because a small fraction of video and image URLs from the original datasets are no longer available, the total number of pre-training samples is around 5M. In the pre-training phase, we do not perform token aggregation since the number of frames in the pre-training video data is relatively small.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and X. Wang. 2022. Groupvit: Semantic segmentation emerges from text supervision. 2022 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 181 13-1233..
Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun. Bei Liu, Huan Yang, Jianlong Fu, and Bain- ing Guo. 2021. Advancing high-resolution video- language representation with large-scale video tran- scriptions. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5026- 5035.
Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Rui Song, Houqiang Li, and Jiebo Luo. 2022. Clip-vip: Adapting pre-trained image-text model to video-language representation alignment, ArXiv, abs/2209.06433
A.2 Detailed Pre-training Objectives.
We use the following three classic pre-training ob- jectives.
Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. 2022. Video-text modeling with zero-shot transfer from contrastive captioners. ArXiv, abs/2212.04979.
Tideo-Text Contrastive Loss. Given a batch of B video-text pairs, the contrastive objective aims to pull together the paired videos and texts while pushing apart the others with dissimilar semantics in the feature space. Let v; and t; represent the [CLS] feature of the video and text, respectively.
Antoine Yang, Antoine Miech, Josef Sivie, Ivan Laptev, and Cordelia Schmid. 2020. Just ask: Learning to answer questions from millions of narrated videos. 2021 IEEE/CVF International Conference on Com- puler Vision (ICCV), pages 1666-1677.

---

Dataset
Domain
#Video- Text Pairs
Avg. Len (sec)
Text Len
Duration (h)
WebVid-2.5MM (Bain al., 2021)
open
2.5M
18.0
12.0
13K
QuerYD (Oncescu al., 2020)
open
2K
278.0
243.8
200
Condensed Movie (Bain al., 2020)
movic
34K
132.0
18.0
1.3K
DiDeMo (Hendricks al., 2017)
Flickr
10K
28.0
29.2
87
ActivityNet Captions (Krishna al., 2017)
action
20K
180.0
48.3
849
ActivityNet QA (Yu al., 2019)
action
5K
117.0
8.9
976
Table 7: Statistics of video-language datasets.
The video-to-text contrastive loss Lvrr is:
Captioning Loss. This objective activates the video-grounded text decoder predict the precise tokenized caption in autoregressive way:
exp(v. t./T) log E exp(v, ty/T)'
Lv21
B
M
CCAP Σ
log P (G: Cess V), (4)
where 7 a learnable temperature parameter. Sim- ilarly, the text-to-video contrastive loss L22 is:
where M is the text length. Combining Eq. (2)-(4), the overall objective can be formulated as:
cpp(t/vv/n B 1=1 E, exp(t, vy/T)
T2V
log
L LvTc GVTM + CCAP.
(5)
The video-text contrastive loss is defined as:
A.3 Hyperparameters.
1 Lvrc 2 (Cv2r Lrvv).
(2)
The model is pre-trained for epochs with the Adam (Kingma and Ba, 2015) with a weight decay of 5e-2. The batch size is 384 and the momentum queue size is 57600. The pre-training is conducted four nodes with 32 NVIDIA V100 GPUs (32 GB memory per GPU) in total and each epoch lasts around hours. The learning rate linearly warmed up from 1e-6 5e-6 the first 5000 steps and then gradually cosine decayed to 5e-7 the remaining steps. Temporally consistent random spatial augmentation (Qian et al., 2021) is applied and mixed precision is used for efficient training.
In the implementation Lvrc, the negative sample features are extracted from queue of recent sam- plcs encoded by a momentum encoder (He al., 2020). Moreover, a momentum distillation regular- ization loss (Li et al., 20216) is added to Lvrc for the sake of the potential positives in the negative pairs
Video-Text Text Matching Loss. Video-text matching aims to predict whether pair of video and text is matched or not. For the i-th video-text pair, we first obtain their joint video-text embedding the [ENC] token from the video-grounded text encoder. We then use this embedding to generate a two-class probability p., and calculate the vidco- text matching loss LVTM as:
B Fine-tuning Details
The downstream fine-tuning is conducted on 8 NVIDIA V100 GPUs. The learning rate is 1e-5 with warmup ratio of 0.1. The batch size 16 and the momentum queue size 32. We fine-tune our model for 10 epochs with the Adam optimizer and a weight decay 0.05. For paragraph-to-vinee vivido re- trieval, we use Lvrc and LVTM as training objec- tives. For evaluating paragraph-tor -otraapp--ttvieoooo retrieval models, we select the top 128 candidates based on the vidco-text feature similarity and then rerank the selected candidates by their pairwise VTM scores. For IA. we use the cross-entropy loss for maximizing the generation probability of the cor- rect answer and rank the candidates by their gener- ation probabilities for evaluation.
1 B
LVTM
CE
Here is one-hot vector representing the ground- truth label, and CF(.;) .) is the cross-entrope loss. In the implementation of LvTM, upply online con- trastive hard negative mining (Li al., 2021b). We refer readers the ALBEF paper (Li al., 2021b) for a comprehensive introduction momentum distillation and online contrastive hard negative mining.

---

C Downstream Datasets
Furthermore, Figure 7 presents the model per- formance with different input frames. Surprisingly, increasing the number of input frames from 32 to 96 has minimal impact on the performance of Sin- gularity (Lei et al., 2022) and Frozen (Bain et al., 2021), and even slightly reduced the recall of AL- PRO (Li et al., 2021a) and VINDLU (Cheng et al., 2022). In contrast, our TESTA exhibits linear im- provement in performance with the number of input frames, demonstrating superior scalability.
We finetune and evaluate TESTA on two down- stream tasks of paragraph-to-video retrieval and long-form VideoQA. The details of these datasets are shown in Table 7.
For paragraph-to-vide datasets of DiDeMo (Hendricks et al., 2017), QuerYD (Oncescu et al., 2020), ActivityNet Cap- tions (Krishna et al., 2017), and Condensed Movie (Bain et al., 2020). We evaluate text-to- video retrieval, where the text acts as the query, in terms of R k, which means the recall (%) of the target video through K retrieval efforts.
retrieval, we use
E
Ablation on the Number of Reduced Tokens
In our TESTA ($ 3.2), Rt and Rs specify the num- ber of tokens to be reduced for the temporal and spatial aggregation module, separately. To investi- gate the influence of these hyper-parameters, we vary the number of Rr and Rs, then report the average GFLOPs (blue bars) and recall (red star) on the QuerYD dataset. Figure illustrates the results. On one hand, GFLOPs decrease linearly as R increases, indicating that increasing the reduced token number can improve the efficiency of video encoding. On the other hand, merging too many tokens with large R (e.g., R = 10) will lose mantic information in the final encoded video repre- sentation, thus leading to a declined average recall.
For long-form VideoQA, we use ActivityNet- QA (Yu et al., 2019). The metric is accuracy (%).
85
VINDLU
Singularity
80
ALPRO
Frozen
TESTA (Ours)
32F
75
70
64F
Recall@1
96F
32F
65
60
We evaluate more cases with various Rr and Rs configurations, and plot the GFLLOP-Reeaa Recall trade- off in Figure 7. Based on these results and analysis, we determined the default configuration for our TESTA, i.e., Ry 4 & Rs and for 96-frame inputs, and Rr
55
32F
32F
64F
50
64F
64F
32F
& Rs 12 for 32-framc inputs. This configuration helps our model achieve approximately 50% reduction in computation cost without significant performance decline.
45
0
500
1000
1500
2000
2500
3000
GFLOPs
Figure 5: Comparison of GFLOPs and Recall on the QuerYD dataset. nF denotes using n input frames for fine-tuning and evaluation. The curve of our TESTA is located in the upper left corner, indicating that our model achieves better performance-cost tradeoff compared to other pre-trained models.
F Token Similarity Analysis
We probe the properties of the aggregated tokens by analyzing their similarity. In Figure 8, we count the average similarity between tokens from differ- ent blocks, different dimensions (frame tokens patch tokens), and different aggregation results (ag- gregated or disaggregated).
Recall-GFLOPs Tradeoff of Various Pre-trained Models
For patch tokens (in orange), the overall similar- ity between them is large (higher than 0.5), indicat- ing considerable spatial redundancy. Meanwhile, the aggregated patch tokens (in dark orange) have very high similarity of 0.96, which ensures the semantic purity of the aggregated patch tokens.
In Figure 7, analyze the tradcoff between re- call and GFLOPs for various pre-trained models. The curve of our TESTA is located in the upper left corner, indicating that our model achieves su- perior Recall-GFLOO GFLOPs tradeoff compared to other pre-trained models.
'Here we use R to refer to Rr or Rs for brevity.

---

2400
32
915
2332
91.0
GRloas
2000
2332
593
2000
3832
66.3
2400
1422
25.0
890
2200
1230
685
1850 1751 GFlops
Rr
R5
Figure Ablation on reduced the token number, Rr (temporal aggre- gation), and Rs (spatial aggregation). The average recall is represented by red stars, while GFLOPs are depicted by blue bars. The dotted lines denote the results without any aggregation (Ry 0 and Rs 0). All results are evaluated on Quer with 96 frames.
Figure 7: GFeaPs-Recall tradeoff on QuerYD. We record the performance (dots) of TESTA with various Ry- Rg configurations, and plot the trends (curve) by fiting the dots.
1.0
1
Token Similarity
0.6
0.4
0.0
Aggregated Frame Tokens
Disaggregated
-02
Aggregated Patch Tokens
Disaggregated Patch Tokens
12
Video Encoder Block
Figure 8: Cosine similarity between tokens from Set A and Set B in various video encoder blocks. The blue color indicates frame tokens while the orange color indicates patch tokens. tokens finally being aggregated, we plot their similarity in a dark color.
While for frame tokens (in blue), their simi- larity decreases as the number of blocks increases, which may yield aggregated frames with mixed and diverse semantics. Nevertheless, recall that our frame token is pseudo token ($ 3.2.1) obtained by averaging patch features, which does not elab- orately model frame semantics. Therefore, pared patch tokens, the representation of frame token and their similarity measure needs improve- ment, which we regard as future work.
More Visualization of Aggregation
In this section, provide more qualitative results of our TESTA for video-language understanding. Figure 9 shows another 4 case on the DiDeMo dataset. TESTA effectively aggregates tokens with similar semantics, demonstrating its strong interpretability.

---

Frames
Agg. tokens
Caption: girl pirate walks towards the camera. dog walks up to little girl then walks away.
Frames
Agg. tokens
Caption: Two boys martial arts competition square and bow to each other to start the match.
Frames
Agg. tokens
Caption: Man dog costume throwing bowling ball.
Frames
Agg. tokens
Caption: dog sits at the beach. Camera pans from dog to water. There is a picnic table the scene.
Figure 9: More visualizations of our aggregation on DiDeMo. Frames that are enclosed within the same rectangle, well as patches that share the same inner and border color, merged together.


Reviews:
[{"review_id": "njUL6XUOCu", "reviewer": "Reviewer_E2NS", "summary": "", "questions": "1. Please refer to *Reasons To Reject.\n2. Will you release the code to community?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive assessment and constructive feedback. \n\n>**Q1: Will it be possible to unify as one text encoder but just with different switch options?**\n\nA1: That's an intriguing idea. Currently, our model comprises two text encoder variants: Text Encoder and Video-grounded Text Encoder. These encoders serve different purposes, such as lightweight pre-ranking and fine-grained re-ranking for retrieval inference, respectively (Appendix B). The separation of these encoders simplifies code implementation and aligns with established practices in prior works like ALBEF and BLIP. However, we acknowledge the potential benefits of a unified encoder with switch options and will explore this direction in future research.\n\n>**Q2: TESTA's pre-training relies on BLIP initialization. How much improvement does TESTA achieve over BLIP?**\n\nA2: We provide the **zero-shot** performance of pre-trained BLIP and pre-trained TESTA (TESTA w/o agg.) on MSRVTT (short-form videos) and QuerYD (long-form videos). Additionally, we apply our token aggregation to pretrained TESTA in an off-the-shelf manner (TESTA w/ agg.). The table below summarizes the findings:\n|  model  |  | MSRVTT |  |  | QuerYD |  | Throughput |\n|:----------|:---|:---|:---|:---|:---|:---|:---|\n| | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | video/s |\n| BLIP    | 42.4 | 64.0 | 72.5 | 50.7 | 67.6 | 73.5 | 1.24 |\n| TESTA (w/o agg.) | 42.8 (+0.4) | 64.7 (+0.7) | 75.1 (+2.6) | 64.4 (+13.7) | 82.9 (+15.3) | 86.9 (+13.4) | 1.08 (0.9x) |\n| TESTA (w/ agg.) | 42.4 (+0.0) | 65.1 (+1.1) | 73.5 (+1.0) | 62.1 (+11.4) | 82.2 (+8.7) | 85.4 (+11.9) | 1.68 (1.4x) |\n\nThe results show that **our pre-trained TESTA outperforms BLIP, achieving +1.2 and +14.1 average improvements** on MSRVTT and QuerYD, respectively. Particularly, the gains on long-form video datasets (QuerYD) is quite significant due to our temporal modeling and pre-training on video data. \n\n**After applying our token aggregation, TESTA maintains a considerable performance superiority over BLIP (+0.7 and +7.0 Recall-mean), and also increases throughput by 1.4x**. These results indicate that it is not trivial to directly apply BLIP, an image-text model, to long-form video understanding tasks. By leveraging our pre-training and token aggregation methods, we achieve both improved effectiveness and efficiency compared to BLIP.\n\n>**Q3: Why k-means is not suited for parallelism but the bipartite matching method is ok? How's the computation cost of bipartite operation?**\n\nA3: Our goal is to reduce N tokens to N-R tokens. **K-means**, being an iteration-based algorithm, requires K iterations to find N-R clusters. In each iteration, it computes the similarity between N tokens and N-R clusters. As a result, its computational complexity is $\\mathcal{O}(KN(N-R))$. \n\nOn the other hand, the **bipartite matching** method assigns R tokens to the remaining N-R tokens without iterations. It only needs to calculate the similarity between R tokens and N-R clusters. Consequently, its computational complexity is $\\mathcal{O}(R(N-R))$, making it significantly more efficient than the k-means method.\n\n>**Q4: Will you release the code to the community?**\n\nA4: Absolutely, we are committed to releasing both the code and the pre-trained & fine-tuned checkpoints to the community.\n\n>**Q5: The real-world long videos like Ego4D should be mentioned in the paper.**\n\nA5: Thanks for your suggestions. We will cite these work in our revision.\n\n**We hope our response can address your concerns and we will revise our paper based on your suggestions.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The long-form video-language pretraining is a valuable yet challenging problem, which is less explored compared to common short-term videos.\n2. The paper proposes a novel aggregation strategy to overcome the heavy visual tokens computations. Notably, they use the bipartite matching method to merge the tokens based on their attention key features. They further design two variants to study the aggregation.\n3. The benchmark results over long-video tasks, demonstrate their methods can achieve good efficiency while maintaining strong performance.\n4. Clear ablation studies with convincing numbers and good visualization,  validate their designs.", "reasons_to_reject": "1. In Fig.2, it appears to have three text encoder variants, mainly with shared layers but with slightly different ones. will it be possible to unify as one text encoder but just with different switch options? e.g., FIBER?\n2. Despite the paper achieving strong results on the benchmark, it relies on BLIP initialization, which should be discussed and ablation in experiments, how percentage improvement achieve over the image-text pretraining?\n3. Why k-means is not suited for parallelism but the bipartitle matching method is ok? how's the computation cost of bipartite operation?\n", "questions_to_authors": "1. Please refer to *Reasons To Reject.\n2. Will you release the code to community?", "paper_topic_and_main_contributions": "This paper proposes an efficient end-to-end video-language pretraining framework for long-form videos. Especially, the paper proposes a novel strategy to adaptively aggregate relevant frames and patches, which can significantly reduce the computational cost of visual tokens. Besides, they propose a VLP model with a divided space-time token aggregation.\nThe paper validates their components and reaches satisfactory performance on paragraph-to-video retrieval and long-form VideoQA benchmarks, and provides strong visualization to demonstrate their designs.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The presentation is good."}, "extra_text": {"Missing_References": "The paper discusses several benchmarks on long-video e.g., activitynet, and condensed movies; but they are actually a few minutes long, while the real-world long videos may be several hours e.g., untrimmed egocentric videos (Ego4D), whole movie (MAD). which are more challenging, and should be mentioned in the paper."}}, {"review_id": "sIDfRh2cnt", "reviewer": "Reviewer_jb6J", "summary": "", "questions": "1. Is there any experimental analysis to prove that offline encoding affects the adaptation to downstream tasks? \n2. Are there any experiments to verify the utility of this proposed method on other sota models?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We humbly think that some concerns are caused by misunderstandings, which we will explain in detail below. We hope that our response can clarify the misunderstandings so that you can consider our work more favorably.\n\n>**Q1: The novelty is limited compared to Token Merging**\n\nA1: Compared to Token Merging (ToMe), our TESTA model offers two key innovations in:\n\n**(1) Video Token Definition.** ToMe uses **joint space-time tokens** (2x16x16 cubes), while our TESTA defines **frame tokens** (representing entire frames) and **patch tokens** (16x16 2D patches) for decoupled aggregation. This tailored token design is more efficient for modeling long-form videos.\n\n**(2) Aggregation Method:** ToMe performs **global aggregation** over all tokens, resulting in complexity of $\\mathcal{O}((\\frac{T}{2} HW)^2)$ (T: frame number, H: patch number in height, W: patch number in width). This becomes impractical for long-form video and causes out-of-memory issues beyond 16 frames.  In contrast, TESTA uses **divided aggregation** in time and space, reducing complexity to $\\mathcal{O}(T^2+(HW)^2)$. This allows efficient encoding of much longer videos (more than 128 frames under the same computation quota). The divided scheme also better captures spatial and temporal semantics, resulting in improved performance on long-form video understanding tasks. \n\nTo directly compare performance, we initialized both ToMe and TESTA from the BLIP checkpoint and finetuned on QuerYD (16 frames). We set the hyperparameter R (number of reduced tokens) to ensure matched throughput. The results are as follows: \n| model |  |  QuerYD | | Throughput |\n|:----------|:---|:---|:---|:---:|\n| | R@1 | R@5 | R@10 | video/s |\n| ToMe | 59.9 | 82.2 | 88.6 | 4.56 |\n| TESTA | 64.1 (+4.2) |  85.9 (+3.7) | 90.1 (+1.5) | 4.64 |\n| ToMe (w/o agg.) | 66.1 | 86.4 | 90.4 | 1.56 |\n| TESTA (w/o agg.) | 75.0 (+8.9) | 91.1 (+4.7) | 93.8 (+3.4) | 1.52 |\n\nThe results clearly demonstrate TESTA's effectiveness for long-form video understanding, thanks to our specific design for divided spatial-temporal modeling. We would gladly incorporate this comparison in our revision. \n\n\n>**Q2: The results of the proposed method on VideoQA do not seem to be very impressive compared to HiTeA**\n\nA2: In terms of efficiency, HiTeA utilizes MViTv2 [1] as a video encoder, which employs convolution-based pooling modules for feature downsampling. However, as we stated in Introduction, **this has two drawbacks compared to our approach: (1)** It introduces additional parameters (convolution kernels), requiring extra training. **(2)** Our token aggregation approach offers better interpretability as it allows for easier tracking and recovery of trajectories. \n\nRegarding performance on VideoQA, HiTeA incorporates two additional pre-training objectives (Moment & Temporal Relation Exploration) that benefit QA on specific moments. It achieves comparable performance to our TESTA (45.1 vs 45.0). We believe these pre-training techniques are compatible with our method and can be applied to further enhance our model, which we leave for future work. It is worth noting that our TESTA significantly outperform HiTeA on video retrieval tasks (+4.8 on DiDeMo and +6.6 ANet).\n\n[1] MViTv2: Improved Multiscale Vision Transformers for Classification and Detection. Li et al. CVPR 2022. \n\n>**Q3: I think this work would be more significant if there were more experiments to prove that the proposed token aggregation is a universal module.**\n\nA3: Thanks for your suggestions. We apply our token aggregation technique to ALPRO [1] and perform finetuning on QuerYD (32 frames). The results are as follow:\n| model | R@1 | R@5 | R@10 | GFLOPs  |\n|:----------|:---|:---|:---|:---:|\n| ALPRO | 50.7 | 76.5 | 84.9 | 788 |\n| ALPRO w/ our agg. | 49.5 | 74.8 | 84.4 | 421 |\n\nThe results indicate that by incorporating our token aggregation, ALPRO achieves comparable performance while reducing the computational cost (GFLOPs) to 53% of the original amount. This demonstrates the effectiveness and universality of our approach. In future work, we plan to validate our approach on more models to further support its universality.\n\n[1] Align and Prompt: Video-and-Language Pre-training with Entity Prompts. Li et al. CVPR 2022.\n\n>**Q4: Is there any experimental analysis to prove that offline encoding affects the adaptation to downstream tasks?**\n\nA4: Yes. Our intention is to convey that offline encoding relies on post-aggregation mechanisms, which prohibits full temporal fusion at the beginning. VINDLU [1] conducted experiments in its Section 3 Step 1, revealing that **post-aggregation methods** (mean pooling / late temporal attention) **perform significantly worse (-5.7 acc. on average) compared to full temporal-spatial modeling methods** (Temporal Convolution / Temporal Attention at each layer). Consequently, previous works that directly adapt CLIP's image encoder (e.g., CLIP4clip, XCLIP) often yield inferior results compared to approaches utilizing video encoders with full spatial-temporal modeling (e.g., HiTeA, VINDLU, and our TESTA).\n\n[1] VINDLU: A Recipe for Effective Video-and-Language Pretraining. Cheng et al .CVPR 2023\n\n>**Q5: Are there any experiments to verify the utility of this proposed method on other sota models?**\n\nA5: Yes, we have applied our method to ALPRO, and the results confirm its utility, please refer to A3.\n\n**We hope our response can address your concerns and we will revise our paper based on your suggestions. We would be grateful if you could reassess the paper in light of these clarifications and consider adjusting the scores accordingly.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes a method for solving problems of important practical significance in current video language tasks;\n2. The method presented in this paper works well. The model is able to achieve comparable performance with its counterpart (without adding token aggregation modules), with a computational burden reduction of close to 50%;\n", "reasons_to_reject": "1. Although this paper applies Token Merging to the field of video-language to fuse features from temporal and spatial dimensions, its novelty is still limited;\n2. The results of the proposed method on VideoQA do not seem to be very impressive. For example, HiTeA, with much fewer GFLOPs, reported better performance on ActivityNet-QA than TESTA.\n3. I think this work would be more significant if there were more experiments to prove that the proposed token aggregation is a universal module.\n4. Is there any experimental analysis to prove that offline encoding affects the adaptation to downstream tasks? As far as I know, there are many methods based on using CLIP image encoder to extract frame features, and CLIP itself has strong generalization.", "questions_to_authors": "1. Is there any experimental analysis to prove that offline encoding affects the adaptation to downstream tasks? \n2. Are there any experiments to verify the utility of this proposed method on other sota models?", "paper_topic_and_main_contributions": "This paper investigates the problem of video processing in video-to-language tasks. It argues that past video processing methods for maintaining computational efficiency in video-language learning have the following limitations: 1. Sparse sampling may miss the semantic information in the video. 2. Offline encoding does not transfer the model well to downstream tasks. To solve these problems, this paper proposes an end-to-end video language learning method considering sufficient input video frames.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "l3pGNW9sMv", "reviewer": "Reviewer_YtSY", "summary": "", "questions": "- A. This paper is a direct extension of ToMe[1]. However, the author does not show enough respect for this previous work.\n    - A1. Line 164 \"previous work explored merging similar patches in the image\". Actually, ToMe[1] also conducts video experiments in its Section 5 VIDEO EXPERIMENTS. A clear statement about the difference between the proposed method and ToMe is needed. Plus, a direct comparison in performance is also necessary.\n   - A2. About why matching and not clustering, ToMe has a clear explanation in its Section 3 TOKEN MERGING. In 3.2.2 Aggregation Strategy of the paper, the author has a similar explanation without clearly pointing out that they are following ToMe.\n   - A3. For the Geometry-based Aggregation, I cannot see any difference with the Aggregation strategy of ToMe. As for Importance-based Aggregation, it is much worse than Geometry-based Aggregation. \n   - A4. Actually, it is not something new for aggregating tokens in text-video retrieval, for example, CenterCLIP[2].  \n\n- B. unfair comparison with previous work.\n    - B1. The authors adopt the architecture and pre-trained weights of BLIP. It is necessary to provide the baseline in every table like Table 1.\n    - B2. Please provide the inference time. GFLOPs cannot represent the speed.\n    - B3. I am confused about the #PT Data. For example, Singularity[4] uses CLIP weights and its #PT Data is 5M, while Clip4Clip also uses CLIP weights and its #PT Data is 400M. The authors use BLIP weights, it pre-trained on more than 120M text-image pairs, but its #PT Data is 5M in Table 2. This will mislead readers. Plus, please provide the data type when showing #PT Data numbers, i.e., image or video.\n \n- C. Please provide results on MSRVTT, both retrieval and video QA. Because these results are also reported in the original BLIP paper. The authors can also provide results with CLIP. It would be easier to judge the method if the CLIP results are available.\n\n- D. Does the temporal aggregation will merge two discontinuous frames, for example, frame 1 and frame 31. Or, if frame 1 and frame 31 have a similar appearance, will they be merged? Will this broken the temporal relationship for some fine-grained video dataset, for instance, something-something dataset (see the github link of [4]). \n\n\n\n[1] https://arxiv.org/pdf/2210.09461.pdf\n\n[2] https://arxiv.org/pdf/2205.00823.pdf\n\n[3] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation.\n\n[4] Jie Lei, Tamara L. Berg, and Mohit Bansal. 2022. Revealing single frame bias for video-and-language learning. ArXivbs/2206.03428. https://github.com/jayleicn/singularity", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments. We humbly think that some concerns are caused by misunderstandings, which we will explain in detail below. We hope that our response can clarify the misunderstandings so that you can consider our work more favorably.\n\n>**Q-A1: A clear statement about the difference between the proposed method and ToMe is needed. Plus, a direct comparison in performance is also necessary.**\n\nA-A1: Our paper is NOT a trivial extension of ToMe; rather, we differentiate ourselves from ToMe in two significant ways:\n\n**(1) Video Token Definition:** ToMe uses **joint space-time tokens** (2x16x16 cubes), while our TESTA defines **frame tokens** (representing entire frames) and **patch tokens** (16x16 2D patches) for decoupled aggregation. This tailored token design is more efficient for modeling long-form videos.\n\n**(2) Aggregation Method:** ToMe performs **global aggregation** over all tokens, resulting in complexity of $\\mathcal{O}((\\frac{T}{2} H W)^2)$ (T: frame number, H: patch number in height, W: patch number in width). This becomes impractical for long-form video and causes out-of-memory issues beyond 16 frames.  In contrast, TESTA uses **divided aggregation** in time and space, reducing complexity to $\\mathcal{O}(T^2+(HW)^2)$. This allows efficient encoding of much longer videos (more than 128 frames under the same computation quota). The divided scheme also better captures spatial and temporal semantics, resulting in improved performance on long-form video understanding tasks. \n\nTo directly compare performance, we initialized both ToMe and TESTA from the BLIP checkpoint and finetuned on QuerYD (16 frames). We set the hyperparameter R (number of reduced tokens) to ensure matched throughput. The results are as follows: \n| model |  |  QuerYD | | Throughput |\n|:----------|:---|:---|:---|:---:|\n| | R@1 | R@5 | R@10 | video/s |\n| ToMe | 59.9 | 82.2 | 88.6 | 4.56 |\n| TESTA | 64.1 (+4.2) |  85.9 (+3.7) | 90.1 (+1.5) | 4.64 |\n| ToMe (w/o agg.) | 66.1 | 86.4 | 90.4 | 1.56 |\n| TESTA (w/o agg.) | 75.0 (+8.9) | 91.1 (+4.7) | 93.8 (+3.4) | 1.52 |\n\nThe results demonstrate TESTA's effectiveness for long-form video understanding, thanks to our specific design for divided spatial-temporal modeling. We are glad to incorporate this comparison in our revision. \n\n>**Q-A2: About why matching and not clustering**\n\nA-A2:  We believe it is evident that clustering methods, such as k-means, are not well-suited for parallelism due to their reliance on iterative algorithms. We will cite ToMe's findings to enhance the strength of our argument. \n\n>**Q-A3: For the Geometry-based Aggregation, I cannot see any difference with the aggregation strategy of ToMe**\n\nA-A3: As we stated in Section 3.2.2, our geometry-based aggregation is inspired by ToMe, nevertheless, \n\n**(1)** Our method incorporates decoupling for patch and frame tokens, which is tailored for long-form videos and brings much more efficiency (see A-A1). \n\n**(2)** Our experiments of geometry- and importance-based aggregation provides novel insights into the success factors of geometry-based approaches. We reveal adjacent tokens have higher similarity, indicating they should be separated into different sets during aggregation. This complements ToMe's principles by further elucidating why considering token position benefits aggregation performance.\n\n>**Q-A4: It is not something new for aggregating tokens in text-video retrieval, for example, CenterCLIP**\n\nA-A4: CenterCLIP aggregates video tokens based on clustering methods, which is much less efficient than our matching-based strategy. Furthermore, our TESTA outperforms CenterCLIP by a significant margin (e.g., 53.7 R@1 compared to 46.2) in ActivityNet retrieval. We will include a comparison with CenterCLIP in our revision. \n\n>**Q-B1: It is necessary to provide the baseline in every table like Table 1.**\n\nA-B1: Thanks for your suggestions. We provide the baseline results for Table 2 as follow:\n|  model  | #PT Data | #Frame | GFLOPs | | DiDeMo |  |  | ActivityNet|  |\n|:----------|:---:|:---:|:---:|:---|:---|:---|:---|:---|:---|\n|  |  | | | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 |\n| TESTA | 5M | 32 | **420** | 56.6 | 81.4 | 87.8 | 51.7 | 79.1 | 87.6 |\n| TESTA, w/o agg. | 5M | 32 | 786 | 57.1 (+0.5) | 82.1 (+0.7) | 87.9 (+0.1) | 52.6 (+0.9) | 79.5 (+0.4) | 88.1 (+0.5) | \n| TESTA | 5M | 96 | **1381** | 59.2 |  83.5 | 89.8 | 53.7 | 79.9 | 88.9 |\n| TESTA, w/o agg. | 5M | 96 | 2383 |  59.5 (+0.3) | 83.7 (+0.2) | 90.0 (+0.2) | 53.9 (+0.2) | 80.0 (+0.1) | 88.9 (+0.0) |\n\nThe results and conclusions align with that in Table 1: Although preserving full visual tokens (TESTA, w/o agg.) yields higher recall, it requires 1.8 times more GLFOPs compared to TESTA. As the number of input frames increases from 32 to 96, the GFLOPs of TESTA w/o agg. exceed 2300, but the performance gain diminishes (only +0.2 R@1 on ActivityNet). This indicates the superiority of our method in aggregating redundant tokens in long sequence inputs.\n\n>**Q-B2: Please provide the inference time. GFLOPs cannot represent the speed.**\n\nA-B2: We evaluate the throughput (video/s) of representative models based on 4xA40 GPUs (QuerYD with 64 frames). The results are as follow:\n| Model | R@1 | GFLOPs | Throughput (video/s) |\n|:---|:---:|:---:|:---:|\n| ALPRO | 48.5 | 1581 | 0.22 |\n| Singularity | 50.5 | 1175 | 0.76 |\n| VINDLU | 70.5 | 1490 | 0.73 |\n| TESTA | **79.8** | **942** | **1.21** |\n\nOur model achieves the best efficiency under both GFLOPs and throughput measures. We note that GFLOPs are commonly used as an efficiency measure in previous works like TimeSFormer, VINDLU, MViT, etc., as they are less affected by code implementation. We align with these works. \n\n>**Q-B3: I am confused about the #PT Data.**\n\nA-B3: **The #PT Data statistics in our paper are provided in the original papers and are consistent with previous work, such as VINDLU, HiTeA, etc**. In the case of CLIP4clip, it uses CLIP's pre-trained checkpoint and directly fine-tunes it on the downstream task, so the #PT Data is considered as CLIP's #PT Data (400M). As for Singularity, although it is also initialized from CLIP's checkpoint, it undergoes pre-training on 5M data, hence the #PT Data is considered as 5M. \n\nSimilarly, our model is initialized from BLIP's checkpoint but undergoes pre-training on 5M data, resulting in our #PT Data being 5M. It's important to note that such statistics are normal since BLIP also initializes its model with ViT pre-trained on ImageNet and BERT, but the PT Data of ImageNet and BERT is not taken into account.\n\n\n>**Q-C: Please provide a direct comparison to BLIP and CLIP on MSRVTT dataset.**\n\nA-C: We provide the **zero-shot** performance of pre-trained CLIP4clip, BLIP and TESTA (TESTA w/o agg.) on MSRVTT (short-form videos) and QuerYD (long-form videos) datasets (32 frames). Additionally, we apply our token aggregation to pretrained TESTA in an off-the-shelf manner (TESTA w/ agg.). \n|  model  |  | MSRVTT |  |  | QuerYD |  | Throughput |\n|:----------|:---|:---|:---|:---|:---|:---|:---|\n| | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | video/s |\n| CLIP4clip | 33.8 | 55.7 | 64.8 | 50.0 | 74.5 | 83.3 | 2.49 |\n| | | | | | | | \n| BLIP    | 42.4 | 64.0 | 72.5 | 50.7 | 67.6 | 73.5 | 1.24 |\n| TESTA (w/o agg.) | 42.8 (+0.4) | 64.7 (+0.7) | 75.1 (+2.6) | 64.4 (+13.7) | 82.9 (+15.3) | 86.9 (+13.4) | 1.08 (0.9x) |\n| TESTA (w/ agg.) | 42.4 (+0.0) | 65.1 (+1.1) | 73.5 (+1.0) | 62.1 (+11.4) | 82.2 (+8.7) | 85.4 (+11.9) | 1.68 (1.4x) |\n\nThe results show that **our pre-trained TESTA outperforms BLIP (as well as CLIP4clip), achieving +1.2 and +14.1 average improvements** on MSRVTT and QuerYD, respectively. Particularly, the gains on long-form video datasets (QuerYD) is quite significant due to our temporal modeling and pre-training on video data. \n\n**After applying our token aggregation, TESTA maintains a considerable performance advantage over BLIP (+0.7 and +7.0 Recall-mean),  and also increases throughput by 1.4x.** These results indicate that it is not trivial to directly apply BLIP, an image-text model, to long-form video understanding tasks. By leveraging our pre-training and token aggregation methods, we achieve both improved effectiveness and efficiency compared to BLIP.\n\n>**Q-D: Does temporal aggregation merge two discontinuous frames? Will this break the temporal relationship for some fine-grained video dataset?**\n\nA-D:  Yes, temporal aggregation can merge two discontinuous frames. However, this does not break the temporal relationship due to the following reasons: **(1)** Frame tokens contain time information through time embeddings, so temporal distance is considered along with visual similarity during frame aggregation. Thus, distant frames like frame 1 and frame 31 are not encouraged to be merged. **(2)** Even if two discontinuous frames are aggregated, their temporal information is partially preserved through mean pooling. **(3)** It's important to note that our experiments focused on coarse-grained and long-form datasets, where redundant visual contexts often appear across discontinuous frames (e.g., repeated appearances of hosts). These redundant contexts can be merged without negatively impacting performance.\n\n**We hope our response can address your concerns and we will revise our paper based on your suggestions. We would be grateful if you could reassess the paper in light of these clarifications and consider adjusting the scores accordingly.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper further validates that token aggregation also works for text-video retrieval.  ", "reasons_to_reject": "1. Lack of technique contributions.\n2. Lack of thorough analysis of the proposed method.\n\n", "questions_to_authors": "- A. This paper is a direct extension of ToMe[1]. However, the author does not show enough respect for this previous work.\n    - A1. Line 164 \"previous work explored merging similar patches in the image\". Actually, ToMe[1] also conducts video experiments in its Section 5 VIDEO EXPERIMENTS. A clear statement about the difference between the proposed method and ToMe is needed. Plus, a direct comparison in performance is also necessary.\n   - A2. About why matching and not clustering, ToMe has a clear explanation in its Section 3 TOKEN MERGING. In 3.2.2 Aggregation Strategy of the paper, the author has a similar explanation without clearly pointing out that they are following ToMe.\n   - A3. For the Geometry-based Aggregation, I cannot see any difference with the Aggregation strategy of ToMe. As for Importance-based Aggregation, it is much worse than Geometry-based Aggregation. \n   - A4. Actually, it is not something new for aggregating tokens in text-video retrieval, for example, CenterCLIP[2].  \n\n- B. unfair comparison with previous work.\n    - B1. The authors adopt the architecture and pre-trained weights of BLIP. It is necessary to provide the baseline in every table like Table 1.\n    - B2. Please provide the inference time. GFLOPs cannot represent the speed.\n    - B3. I am confused about the #PT Data. For example, Singularity[4] uses CLIP weights and its #PT Data is 5M, while Clip4Clip also uses CLIP weights and its #PT Data is 400M. The authors use BLIP weights, it pre-trained on more than 120M text-image pairs, but its #PT Data is 5M in Table 2. This will mislead readers. Plus, please provide the data type when showing #PT Data numbers, i.e., image or video.\n \n- C. Please provide results on MSRVTT, both retrieval and video QA. Because these results are also reported in the original BLIP paper. The authors can also provide results with CLIP. It would be easier to judge the method if the CLIP results are available.\n\n- D. Does the temporal aggregation will merge two discontinuous frames, for example, frame 1 and frame 31. Or, if frame 1 and frame 31 have a similar appearance, will they be merged? Will this broken the temporal relationship for some fine-grained video dataset, for instance, something-something dataset (see the github link of [4]). \n\n\n\n[1] https://arxiv.org/pdf/2210.09461.pdf\n\n[2] https://arxiv.org/pdf/2205.00823.pdf\n\n[3] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation.\n\n[4] Jie Lei, Tamara L. Berg, and Mohit Bansal. 2022. Revealing single frame bias for video-and-language learning. ArXivbs/2206.03428. https://github.com/jayleicn/singularity", "paper_topic_and_main_contributions": "This paper proposes to aggregate visual tokens in text-video retrieval. They extend the ToMe[1] method to text-video retrieval.\n\n[1] https://arxiv.org/pdf/2210.09461.pdf", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The core idea of this work is to aggregate similar frames each video and similar patches within each frame, with bipartite matching, to reduce the computation cost of processing long-form videos. Experiments on paragraph-to-video captioning and long-form videoQA, demonstrates the efficiency and performance gain of the approach. While the idea of token merging has been explored previously in ToMe, as mentioned by the reviewers, the adaption of it to long-form video processing requires specific designs such as decoupling frame-level and patch-level aggregation. Thus there is still merit of the approach to the community. One missing point of the work is to test the approach against ToMe on pure video tasks, since the approach works for videos alone as well. AC encourages the authors to provide experiments to make the work in a more complete form.
Decision: Accept-Findings