Video Token Merging for Long Video Understanding
Authors: Seon-Ho Lee
Jue Wang
Zhikang Zhang
David Fan
Xinyu Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Long video understanding
token merging
Primary area: machine_vision

Abstract: As the scale of data and models for video understanding rapidly expand, handling long-form video input in transformer-based models presents a practical challenge. Rather than resorting to input sampling or token dropping, which may result in information loss, token merging shows promising results when used in collaboration with transformers. However, the application of token merging for long-form video processing is not trivial. We begin with the premise that token merging should not rely solely on the similarity of video tokens; the saliency of tokens should also be considered. To address this, we explore various video token merging strategies for long-form video classification, starting with a simple extension of image token merging, moving to region-concentrated merging, and finally proposing a learnable video token merging (VTM) algorithm that dynamically merges tokens based on their saliency. Extensive experimental results show that we achieve better or comparable performances on the LVU, COIN, and Breakfast datasets. Moreover, our approach significantly reduces memory costs by 84% and boosts throughput by approximately 6.89 times compared to baseline algorithms.

Video Token Merging for Long-form Video Understanding
Seon-Ho Lee"
Jue Wang Amazon AGI
Korea University
seonholee@mcl. korea kc
j juewangn@amazon.com com
Zhikang Zhang
David Fan
Xinyu
Amazon AGI
Meta FAIR
Amazon AGI xxnl @amazam. com
zhikang@amazon. com
davidf aodddfan@@mm www.tta.aa.maammmmmmmaaaa.a.ea.a com
Abstract
As the scale of data and models for video understanding rapidly expand, handling long-form video imput in transformer-basee models presents practical challenge. Rather than resorting to imput sampling or token dropping, which may result in information loss, token merging shows promising results when used in collaboration with transformers. However, the application of token merging for long-form video processing not trivial. We begin with the premise that token merging should not rely solely on the similarity video tokens; the saliency of tokens should also be considered. To address this, explore various video token merging strategies for long-form video classification, starting with a simple extension of image token merging, moving region-concentrateed merging, and finally proposing learnable video token merging (VTM) algorithm that dynamically merges tokens based on their saliency. Extensive experimental results show that we achieve better comparable performances on the LVU, COIN, and Breakfast datasets. Moreover, our approach significantly reduces memory costs by 84 % and boosts throughput by approximately 6.89 times compared to baseline algorithms.
Introduction
Over the past few years, the Transformer architecture (Vaswani al. 2017) has risen revolutionary paradigm within natural language processing (NLP) (Devlin et al. [2018) and has seamlessly expanded
its influence into the domain of computer vision (Wang & Torresani] 2022 Bertasius et al. 202 Wang
et al. 2022 Akbari et al.) 2021 Li et al.] 2022: Fan et 2021). This expansion has been exemplified by remarkable achievements recent multi-modaal modality foundation models such as Sora (Brooks et al.
2024), GPT4 (Achiam et al. 2023). and Gemini (gem). showcasing its exceptional performance and versatility across diverse applications.
In contrast to the natural language processing, the visual input has much lower information density and thus tokenizing the RGB image as noveovellppped patches becomes the essential operation in vision transformers (Wang & Torresani 2022, Bertasius et al. 2021 Dosovitskiy et al. 2021). However, the computational cost of transformer exponentially increases with the sequence length, which generates tremendous computation when feeding visual ínput into the large-sealed transformer models with billions of parameters. Due the redundancy in video sequence, this phenomenon becomes more severe with video input, especially for long-form videos. This bottleneck impedes the
Work done during an internship at Amazon Prime Video. Work done while Amazon Prime Video.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

75.9
75.9
Ours
Ours
Accuracy
70.6
Baseline
Accuracy
72.6
ViS4mer
Baseline
S4
65.3
Orthofonner
69.3
Performer
ViS4mer
60.0
66.0
4
10
5
25
35
45
Memory (GB)
Throughput (#sampics/sec)
Figure I: Comparison of GPU memory footprint and throughput against scene prediction accuracy
on the LVU dataset (Wu & Krähenbüh! 2021
further advancement of foundational models in handling long-form video data. Various attempts (Yn ctt al. 2021| Wang et al.] 2021] Meng et al., 2022; Rao et al.) [021] Liang et al., (022) have been proposed improve the efficiency of vision transformer by introducing token selection module. However, these methods are primarily designed for images and may require non-trivial adaptation to the long-form video scenarios due the video-level long-term dependencies and motion dynamics. Moreover, tokens dropped by the token selection module cannot be reused in later layers, which may result in the loss of important information.
In addition to the token selection, token merging techniques (Bolya et al. 2022 Ren et al. 2023; Bolya & Hoffman 2023 Li et al., 2024) have been proposed to increase the efficieccy and effectiveness of transformer- bassd networks. Specifically, they reduce the sequence length by merging similar tokens, thereby decreasing the computational cost. In addition to the efficiency, token merging demonstrates huge advantages by increasing the contextual information so that the model can learn from patterns presented across multiple tokens. Previous token merging algorithms in both the image and video domains use manually designed token partitioning methods and merge tokens based on their similarity. Even though the merged tokens would still keep the original information, they may have different granularity after the merging operation. In this paper, it is argued that different regions in the visual data may have different information density. Since the discriminative information of tokens may be degenerated after merging, some visual tokens should not be merged even if they look similar each other. Rather than relying solely similarity, question whether more unmerged tokens should be used to describe salient areas, while merging more tokens for the background.
In this paper, explore various video token merging (VTM) methods in forg-form vidco classifica- tion task and aim to find the effective token merging method for long-form videos. Previous video token merging method (Li et al. 2024) only decouples the spatial and temporal dimensions, which is unfavorable, especially for lorg-form videos. As the long-ter lon--term dependency plays an important role in the long-form video understanding, spatiotemporal visual tokens should be considered jointly. Sequentially merging token along with one dimension after another may generate biased prior. In our work, we first naively extend the image-based token merging (Bolya al. (022) to the video domain and then propose region-centralized and motion-based token merging algorithms, which estimate the salient region of video sequences, Finally, we develop learnable VTM which predicts the saliency score of each token and adaptively merges spatiotemporal visual tokens in data-driven manner. Experimental results demonstrate that the proposed algorithm improves the effectiveness and the efficiency of the transformer- based network and outperforms the conventional vongo understanding methods with better throughput and less memory usage, as also shown in Figure [
We summarize the contributions of this paper as following:
We explore various video token merging methods including the naive VTM. the region- concentrated VTM, and the motion-based VTM.
We propose the learnable video token merging algorithm, which estimates the saliency score of each token and adaptively merges visual tokens based on their scores.
The proposed algorithm achieves the best or competitive results on various datasets including LVU, Breakfast and COIN. Moreover, we significantly reduce memory costs by 84% and improve the throughput by 6.89 times via the proposed learnalle VTM.

---

2 Related Work
2.1 Long-form Video Modeling
Transformers have demonstrated remarkable prowess in capturing long-term dependencies, as evi- denced in their success in natural language processing (NLP) tasks (Brown et al., 2020 Dai et al.) 2019; Devlin et al. 2018)) However, the intensive computational requirements Stemming from dense self-attention calculations (Vaswani et al.,
2017) pose a significant obstacle not only NLP but also in the domain of computer vision, especially for the long-form videos. Many recent video transformer works (Wang & Torresani] 2022 Liu et al.] | Bertasius et al., (02T) focuses on improving the global attention mechanism. However, they are not designed for dealing with redundant spatial and temporal image tokens that are common in long-form video scenarios. To capture longer temporal information, LF-VILA (Sun et al.) 2022) develops a hierarchical architecture to include more frames in the model. Similarly, McMViT (Wu et al., 2022) utilizes longer temporal information by emerging the previously cached "memory" from the past. A novel alternative to transformers is the Structured State-Space Sequence (S4) model proposed by Gu et al. (202I). which models the long-range de- pendencies by simulating linear time invariant (LTI) system. Subsequently, ViS4mer (Islam & Bertasius 2022) and S5 (Wang et al. al.||2023 (202)) extend S4 model to the long-form video classification task. ViS4mer (Islam & Bertasius] (022) stacks multiple S4 layers with different scales in modeling long-form videos, and S5 (Wang et al. 2023) include an additional selective module to further improve the performance. Unlike these works that focus on the improvement of architecture and attention mechanism, this paper will start from a more basic concept in the transformer, video tokens and how effectively merge them. Even though on S4 model, the scope of this paper is on the well established transformer architecture. We will leave the investigation of video token merging on S4 (Gu et al. 2021) in the future work.
proposed method can theoretically be applied
2.2 Adaptive Token Selection
Adaptive token selection is widely used to improve model efficiency by leveraging a light-weight selection module to pick up the 'useful' tokens while dropping the 'unuseful" ones. In vision transformer, STTS (Wang et al. (02I) utilizes token selection module known as the named scorer network to assign importance scores to each token, subsequently selecting the top-K frames with the highest scores. Building upon this concept, AdaViT (Meng et al. (022) further extends the approach by developing instance-specifi policies. These policies guide the activation of patches, self-attention heads, and transformer blocks, enhancing adaptability and efficiency in processing visual data. STTS, Ada Vit and other similar approaches (Wang et al.) [22T]] Meng et al., [022] Rao et al.) 202T: Liang et al., (222)) drop significant number of tokens in the early decision stage to save more cost, but the dropped tokens cannot be reused in the later layers, which is easier to degenerate the contextual information in the form videos.
2.3 Token Merging
Visual token merging is first proposed in (Bolya et al. 2022) which aims at increasing the throughput of existing ViT models without training. Following works (Ren et al. 2023 Bolya & Hoffman 2023
Liet al. 2024) leverage this idea to save computational cost in different downstream applications, such diffusion model, video and language understanding, and video editing. Specifically, visual tokens are first partitioned into two sets with equal size; for each of the edge tokens in one set, find the most similar token in another set and merge them by average pooling: finally, concatenate two sets back together. Although the token merging is simple and effective, its applications have mostly remained in the image domain. There is no fundamental research work has been explored for the long-form video token merging strategies, where the spatiotemporal tokens are more redundant and embed complicated dependencies locally and globally. In this work, we argue that visual tokens from the long-fore forg-form video should be carefully partitioned and merged based on the salient areas in videos. To this end, we ablate various video token merging algorithms and provide extensive expermental results and analysis.
3 Proposed Algorithm
3.1 Preliminary Token Merging
Token merging (Bolya et al. 2022) aims to reduce the redundancy by merging similar tokens at each transformer block, thereby increasing the effectiveness and efficiency of a transformer-based network. Specifically, token merging has three steps: partitioning, matching, and merging.
3

---

Partitioning: For given set of N tokens X' = {I1, 2..... ZN token merging first partition X into a set of target tokens T and of source tokens S, given by
T = {2; : mod 7 = 0),
(1)
{., j mod # 0}
(2)
where is partition factor and mod denotes the modulo operator. Thus, |T] and T nn Ø
[2] Also, X = TUS
Matching:
Then, for each source token in S, finds the most similar target token in T. Here, the similarity between tokens 24 and v, is defined as the cosine similarity of the corresponding key vectors K; and kj. which are obtained in the most recent self-attention layer. For a source token Iy E S, the index of its matched target token my is obtained by,
m = argmax
(3)
k [kg
Merging: Lastly, token merging merges the tokens based on the matching results. For each target token 24 T, it obtains the merged token yt by using average pooling,
3i + ΣjEM Zj
y:
1 Mal
where M; = (j my =i, Vry E ES) this case, the number of tokens is reduced by (S] after token merging. It can also control the number of reduced tokens by R by reassigning the matching index
V., 3 is the index set of source tokens which matched with v;. In
mj --1
(5)
for all z except for the source tokens with the highest similarity scores.
3.2 Problem Definition
Suppose that a video with L frames is given, where L > 60. To perform the classification or regression on the given video, can use simple transformer-based network, which shown in Figure 2 (a) and (b). It first obtains the tokens X2,...... Xy " xC by using encoder. Here, H. W, and C' denote the height, the width, and the channel dimension of the token tensor, respectively. Then, it utilizes transformer blocks to update the tokens. As its input, i-th transformer block takes the tokens corresponding L; frames without overlapping. where L, S L, S L for i j.j. The prediction head yields the estimation results. However, it requires the prohibitively large memory and computation costs due to the quadratic complexity of the self-attention where D is the dimension of key vectors. Hence, our goal is to increase the efficiency of this baseline network by reducing the number tokens via token merging, while maintaining or even improving the performances of the network by removing the redundant noisy information in the video. To this end, we explore various token merging methods for long video processing.
3.3 Video Token Merging - Exploration
First, combine the standard token merging with the baseline network as intact as possible. To this end, we substitute transformer blocks with VTTM blocks, in which token merging layer is inserted after the dropout layer, as illustrated in Figure 2 (c). In this naive VTM, employ the standard token merging in (I) At each i-th VTM block, naive VTM reduces the R; tokens. For example, at 7 and R S) it gradually removes the 68% of tokens over the network. Hence, the computation cost of the self-attention is reduced from O(22 D2) (( D2) at i-th VTM block. As shown in Table this naive VTM shows better scores than the baseline network, because the token merging reduces the information redundancy in the videos.
Naïve video token merging:
Region-concentraaeed video token merging: Compared to an image, a video contains redundant spatiotemporal tokens. Depending on the tasks, some tokens are more important than others. However, as shown in Figure 3 (a), naïve VTM selects every 7-th token as the target tokens since it exploits uniform token partitioning in ( Also, for token merging, purely relies on the similarity between tokens regardless of the semanties, and thus the self-attention more easily swayed by unnecessary

---

Transformer
Transformer
Transformer
Prediction
Encoder
Block
Block
Block
Head
Video
Tokens
Attention
Dropout
Linear
GeLU
Attention
Dropout
VTM
Linear
GeLU
LN
$
(b)
(c)
Figure 2: The architectures of (a) the baseline network, (b) the transformer block, and (c) the video
token merging block.
Table 1:
Comparison of different VTM methods on the LVU dataset. The best results are boldfaced
and the second-best ones are underlined.
Content (+)
Meta data (t)
User (L)
Algorithm
Relationship
Speaking
Scene
Director
Genre
Writer
Year
Like
View
Baseline
57.14
36.68
69.76
62.61
56.73
49.40
39.86
0.28
4.18
Naive
61.90
36.18
72.09
67.28
55.12
51.19
44.75
0.28
4.01
Boundary
59.52
37.18
69.76
61.68
57.21
50.0
47.55
0.26
4.16
Center
61.90
40.20
74.41
62.61
58.81
51.19
44.05
0.25
4.11
Motion
64.28
37.68
74.41
64.48
58.49
55.95
47.55
0,24
4.13
Learnable
64.28
42.12
75.58
70.09
59.77
53.57
48.55
0.21
4.01
information. Therefore, for better video token merging, it is important to consider the saliency of each token before merging them.
To investigate this issue, we explore center-concentrated video token merging and boundary- concentrated video token merging. The center-conneettaated token merging samples 50% of the W. entire target tokens from the center area with the size of H x 2 which uses more unmerged tokens describe center area and merge more token from the boundary. Specifically, we use the partition factor of I for the center area and 7 for the remaining area. On the other side, we implement the opposite operation for the coundary-concntraaae center-concentraate VTM shows better performances than naïve and general. Since meaningful objects and motions are typically enter-concentrated, this suggests more tokens from salient regions should be unmerged while more of the rest tokens should be merged. Figure (b) shows the partitioning results of center-concentrated VTM.
T
video token merging. As shown in Table [ VTM in
Motion-based video token merging: Even though center-consentratee VTM has shown better performances than naïve VTM, the meaningful tokens are not always located in the center area. Moreover, the hand-crafted partitioning method forces the center-concentrates VTM to select the same number of target tokens from each frame, which not flexible enough when applied scale. Therefore, we explore motion-based video token merging which divides tokens into and S based on the motion information since the moving objects carry important cues in general (Fan et al.) 2023).
Let us assume that we have the magnitude of motion vector v; for each token #4. We compute sampling probability by using softmax
e% Ps
(6)
Σ
N
j==1
5

---

(a)
(b)
(c)
(d)
Figure 3: Visualizations of target tokens of different VTM methods: (a) naive VTM, (b) center- concentrated VTM, (c) motion-based VTM, and (d) learnable VTM. In (d), learnable VTM selects the target tokens around salient objects rather than backgrounds.
Note that the sampling probability is proportional the motion magnitude. Then, we construct T by sampling N tokens from X' with the sampling probability p. for each token TQ.
The goal of VTM is to increase the efficiency of transformer -based network for long video under- standing. Therefore, the motion information should be obtained with negligible time and computation costs. Hence, instead estimating the motion information with an additional module, we use the motion information which is already stored in the video files; most modern video codecs, such as MPEG-4 (Richardson] (004)), H.264 (Richardson) 2004). and HEVC (Wien) 2015). exploit the motion information for efficient compression. The motion decoding takes only 0.3 milliseconds for each frame which is negligible. Figure (c) shows the token partitioning examples of motion-based VTM.
3.4 Learnable Video Token Merging
There are some videos in which unimportant objects have large motions due various factors such as camera movement. Motion-based VTM may not yield good results on those videos. To maximize the generalizability,
develop learnable video token merging method. Instead of depending on the motion information, learnable VTM estimates the saliency score of each token and samples the target tokens based on the estimated scores. Figure shows the architecture of learnable VTM block.
Learnable VTM block contains two forward paths: a main path and an auxiliary path. Let us assume that we have tensor of N tokens X E NxC. In the main path, we first obtain query Q, key K, and value V by
XUg; K= XUK
V= XU,
using learnable projection matrices Ug. Uk. U. E D We perform the standard self-attention on Q, K, and V and yield the updated tokens X' as
X'= softmax(QKT) V.
(8)
vD
Also, from K, we estimate the saliency scores S of tokens by
(9) where U. RD11 is learnable matrix. Also, 8; (-1, 1) for 1 N. Then, compute the sampling probability using (6) with s; instead of o; for each token x; and sample T with the sampling probability in motion-based VTM. After the token partitioning, we match and merge the tokens in X' by using (3) and (4) respectively.
[$1 $2,.....
SN!] sN]] T = tanh(KU,)
However, the learnable matrix U. in 9 can not be trained only with the main path since the partitioning process is not differentiable. To handle this issue, we employ the auxiliary path. This path consists of saliency guided self-attention layer and a merging operation. The auxiliary path takes a tensor of auxiliary tokens Xaux E N as its input. Similar to the main path, we obtain query Qaux key Kaux and value Vaux- Then, perform the saliency guided attention to obtain the updated auxiliary tokens X as
softmax (Qanx KT sux + 1ST
X aux
Vaux
(10)
CC

---

Attention
X"
Merge
Linear
GeLU
Saliency estimation
K
S
Partition
Match
Main path
LN
Xaux
Guided- Attention
X
aux
Merge
Linear
GeLU
Auxiliary path
Figure 4: An overview of the learnable video token merging block. The auxiliary path is used during training only.
Table 2: Comparison of the proposed learnable VTM algorithm with conventional algorithms on the LVU dataset.
Cantent (1)
Mitt dota (1)
User 1U)
Algorithm
Relationsip
Speaking
Gease
Weiter
Year
Vieu
Throughpui
14mer
Kriheabuad
37.76
VidesRFRT
Perfomeer
2011
50.00
38,80
50.45
58.87
49:45
45.21
41.25
3.93
5,93GB
Ontiolbxmit
5000
39.50
66.27
55.14
55.79
47.02
43.35
0.29
3.86
5,50318
1,5T (Klmm & 1e/miiii mure
$2.18
37.31
62.79
56.07
52.70
26
39 19.16
0.31
3.83
41.35GB
ViSteerr Iilan Benuo
57.14
40.79
67.48
62.61
54.71
45.80
44.75
0.26
3.63
5.15GB
25.64
85 [Wang 2 2023
01.58
41.75
09.88
66.40
58.80
50.60)
47.70
0.25
3.51
3,85C18
25,0
50.60)
Lemmmlle
where a N dimensional vector of ones. In the saliency guided attention, the contribution each token controlled by its estimated saliency score; if 8, is positive, i-th token affects more X Xux whereas if 8; is negative has less influence on X In other words, increases the influences of the tokens with high saliency scores in the attention process. Therefore, during training. the network is encouraged to assign high saliency scores to the tokens with meaningful information and low saliency scores the others obtain the better predictions. At the first VTM block, the auxiliary path employs the same input with the main path. From the second VTM block, it takes the output of auxiliary path in the previous VTM block as its input.
Also, it is worth pointing out that the auxiliary path is used for the network training only. Compared to other VTM methods, learnable VTM only introduces additional computation of score estimation module, which is fast and light enough, during test. Therefore, shows almost same inference speed with other VTM methods.
Experiments
4.1 Datasets
LVU (Wu & Krähenbühl, 2021):
contains ~30K videos sampled from ~33 movies on the MovieClips (mov) website. Most videos are 1 to minutes long. It provides the labels for long-video understanding tasks which can be grouped into three major categories:
Content understanding: 'relationship." 'speaking style,' and 'scene/place"
Metadata prediction: 'director,' 'genre,' 'writer,' and 'movie release year'
User engagement: 'YouTube like ratio,' and 'YouTube popularity"
As the evaluation metrics, adopt the top classification accuracy for content understanding and metadata prediction tasks and mean-squared error (MSE) for user engagement tasks.
Breakfast (Kuchne al. 2014): It provides 1,712 videos with the average length of 2.32 minutes and the total length 77 hours. The videos contain 52 individuals and 18 different backgrounds in total. Each video belongs to one of 10 complex cooking activities.
COIN TTng et al. 2019): consists 11,827 videos with the average length of 2.36 minutes, collected from YouTube. Each video belongs to one of 180 distinct procedural tasks.

---

Table 3: Comparison on the Breakfast dataset. PT stands for pretraining.
Algorithm
PT Dataset
#PT Samples
Accuracy
VideoGraph (usssii et al.
2019h
Kineticx-400
306K
65.50
Timeception Hussein et al
2019a
Kinetics-400
136M
71.30
GHRM (hou et al. 2021)
Kinetics-400
495K
75.50
D-sprv Lin el al. (022)
HowTol00h
136M
89.90
ViS4mer (Islam & Bertasius
2022
Kinetics-600
495K
88.17
S5 (Wang et al. 2023)
Kinetics-60n
495K
90.14
S5+LSMCL [Wang et al. 2023
Kinetics-600 -60
495K
90.70
Learnable VTM
Kinetics-600
495K
91.26
Table 4:
Comparison with the state-of ff-thhean art methods on the COIN dataset. PT stands for pretraining.
Here, *
means the reproduction results with the official codes.
Algorithm
PT Dataset
#PT Samples
Accuracy
TSN (Tang et al
2020
Kinetics-400
306K
73.40
D-sprv (Lin ct al.
2022
HowTol 000
136M
90.00
ViS4mer [Islam & Bertasius,
2022
Kinetics- 600
495K
88.41
ViS4mer" Islam & Bertasius
2022
Kinetics-600
495K
87.11
(Wang 20233
Kinetics-600
495K
90.42
S54LSMCL (OICC((War) Wang el al. 2023
Kinetics-600
495K
90.81
Learnable VTM
Kinetics-600
495K
88.55
4.2
Implementation Details
We follow the experimental settings conventional long-form video understanding algorithms (Islam
& Bertasius] 2022: Wang et al. (023). We employ three transformer blocks in the baseline network.
As the encoder, we use ViT-L (Dosovitskiy et al. (02T) pretrained on ImageNet-211
(Ridnik et al.
202I) on the LVU (Wu & Krähenbühl (021) dataset and employ Swin-B (Liu et al. (022)) pretrained
on Kinetics-600 (Kay et al. (017)) on the Breakfast (Kuehne et al. 2014) and COIN (Tang et al.
2019) datasets. Images are resized to 224 x 224 for the feature extraction. Hence, H W 16 and
C² = 1024 for the LVU dataset and H = W = and C = 1024 for the Breakfast and COIN datasets.
The size of the length of input video for each dataset is also same with (Islam & Bertasius 2022
Wang et al.) 2023): we use 60 frames for the LVU dataset and 64 frames for the Breakfast and COIN
datasets. We use the AdamW (Loshchilov & Hutter 2017) optimizer with batch size of 16 and
weight decay of 0.01. We set the learning rate to 0.001. We train the network for 70 epochs by using cosine learning rate scheduler (Gotmare al. 2018 with 10 epochs warm-up. For experiments,
use 8 Tesla V100 GPUs and PoToc..
4.3 Experimental Results
Comparison LVU: In Table 2, we compare the proposed algorithm with the conventional methods on the LVU dataset. With the smallest memory footprint, the proposed algorithm achieves the best scores in 7 of 9 tasks on the LVU dataset. Performer (Choromanski and Orthoformer (Patrick et al.] (021) employ the efficient variants of self-attention to reduce the computation costs. The proposed learnable VTM outperforms these approaches with significant performance margins and less GPU memory usages. It shows the efficiency and effectiveness of our approach. Also, ViS4mer (Islam Bertasius 2022) and S5 (Wang al., 2023) adopt S4 layers instead of self-attention layers to capture the long term dependencies in videos because of its linear computation complexity to the number of input tokens. The promising results of these methods have suggested that S4 layer can be an efficient replacement of self-attention layer for long-form video inputs. However, the higher scores of the proposed algorithm broaden the potential usages of the self-attention layers for various long-form video tasks. Moreover, $5 utilizes LSMCLL, which is pretraining based on the contrastive learning, boost its performances. Nevertheless, without the time-consuming pretraining, the proposed algorithm yields better scores the LVU dataset.
al. 202T

---

Table 5: Comparison of long-video understand- ing results on the LVU dataset according to 7.
Table 6: Comparison of long-video understand- ing results on the LVU dataset according to L1, L2, Lz).
Algorihm
Scene
Director
Like
Throughput
GPU
Baseline
62.61
0.28
6.52
10GB
(Tn, To, Ta)
Throughpet
2
72.09
68.22
0.25
22.13
2.GB
(10. 10,30, 60)
75.58
69.15
0.21
33.62
2.7GB
80)
10
74.41
70.09
0.23
48.89
1,50B
(4,20, 60)
73.25
68.22
0.23
53.75
144BB
Figure 5: Visualizations of video token merging results the LVU dataset. Patches with same inner and border color are merged together. The tokens corresponding to the backgrounds are merged together, thereby increasing the influence of salient tokens in the attention process.
Comparison on Breakfast: Table compares the performances of the proposed algorithm and the conventional techniques the Breakfast dataset. The proposed learnable VTM achieves the best score on this challenging long-range activity classification dataset as well. For pretraining, D-sprv (Lin al. (022) leverages the HowTo100M (Miech al. 2019) dataset which contains much more training samples than our pretraining dataset, Kinetics-600 (Carreira et al. 2018). Nevertheless, we outperform D-sprv (Lin al. 2022) with the accuracy gap 1.36%.
Comparison on COIN: Table compares the scores of the proposed algorithm and the conventional techniques the COIN dataset. We note that the COIN dataset consists of the videos YubTube and more than 1,000 videos are not available anymore. Therefore, ViS4mer, which is one of the state-of-the-ee art method on the COIN dataset, achieves only 87.11% accuracy when is trained the current version of the COIN dataset. may be because of many missing training videos. Even though the comparison is not perfectly fair, we report the results the COIN dataset for reference. The proposed learnable VTM yields better results than ViS4mer with the same training and test data. Also, it shows the comparable score with $5 (Wang al. 2023).
4.4 Analysis
Analysis on 7: Table compares the performances of learnable VTM with different Ty. Compared to the baseline, at all the proposed algorithm improves the performances. Also, at 10. it increase the throughput and the memory efficiency by 7.49 and 6.6 times, respectively. 7 = 6, the proposed algorithm shows the best results.

---

Table 7: Comparison of long-video understanding results of various VTM designs on the LVU dataset.
Algorithm
Scene
Director
Like
Weighted average
72.09
68.22
0.24
Motion
74.41
64.48
0.24
Motion weighted average
74,41
66.35
0.23
Learnable
75.58
70.09
0.21
Table 8: inference.
Comparison of throughput and memory footprint of learnable VTM for training and
Algorithm
Phase
Throughput
Memory
ViS4mer (Islam & Bertasius
2022
Inference
25.64
5.15GB
S5 (Wang ct at. 2023
Inference
25
3.85GB
Learnable VTM
Training
27.84
2.8GB
Learnable VTM
Inference
44.94
1.6GB
Analysis on (L,, L2, L3): Table shows the results of the proposed learnable VTM with different (L1, L2, La). Note that i-th VTM block takes the tokens in L, consecutive frames as its input at time. (L,, , L2, La) = (4, 20, 60). the proposed algorithm yields the lowest scores, since can not capture the long temporal dependency in the early stage of the network, We see that the proposed algorithm yields the best scores at (L1, L2, L3) (6, 30, 60).
To merge tokens, we use average pooling as the default setting in all VTM methods. However, once tokens merged, they may represent more than input patch. Thus, to reflect the token size in merging process, we combine tokens by averaging weighted by their sizes. However, shown in Table this weighted average pooling decrease the performances of learnable VTM. Thus, we exploit the average pooling to merge tokens.
Weighted average pooling:
Motion weighted average pooling: In motion-based VTM, we can combine tokens by averaging weighted by their motion magnitudes. Table 7 shows the performances of motion-band VTM with the motion weighted average pooling. It yields the similar scores with the standard motion- -baedd VTM with the average pooling.
Complexity: Table & compares the throughput and memory footprint of learnable VTM during train- ing and inference. Since the auxiliary path is additionally employed during training, it requires more computation costs. However, even during training, learnable VTM still faster than conventional methods including ViS4mer (Islam & Bertasius 2022) and S5 (Wang et al. (023) and it also requires less amounts of memory than them.
Visualizations: In Figure we visualize the tokens merging results at the end of the network over multiple frames of video. We see that tokens with similar semantics are merged together. Also, tokens corresponding to backgrounds
unnecessary informations are merged more than tokens corresponding salient objects. It is because learnable VTM selects tokens with high saliency scores as the target tokens. More visualization results are available in the supplemental document.
5 Conclusion
In this paper, we investigate the video token merging techniques for long-form video data. Unlike previous algorithms that apply uniform partitioning and merge tokens solely based on the visual similarity, we argue tokens with different saliencies should be treated unequally to avoid undesirable information loss after merging important tokens. To this end, we explore various video token merging methods and receive interesting intuitions from region- coneentrene and motion-based token merging results. Lastly, we propose a learnable video token merging scheme that adaptively samples target tokens and learns discriminative representations from the long-form videos. Compared to the baseline, the proposed algorithm achieves substantial improvements terms of the performance, memory cost and throughput.
10

---

References
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. URL https htt.gg/.pp.....ccc/////caaaaaaaa www.:///dd....eimmmgggmmnmiiiii. www_rr_eerrr.ppppp_
MovicClips. httppps//www..ooieeclppppss
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv, 2023.
Hassan Akbari, Linagzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. VATT: Transformers for multimodal self-supervised learning from raw video, audio and text. arXiv,, 2021.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. space-time attention all you need for video understanding? In ICML, 2021.
Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. In CVPR Workshop, 2023.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token Merging: Your ViT but faster. In ICLR, 2022.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024. URL https://openaai http::///peena..ccorrreeeaar www.researcch////oo video- www.eeeeiioonneeeeeeeee as as- siiildddsmulatorss
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. In NeurlPS, 2020.
Joao Carreira, Eric Noland, Andras note about kinetics-600. arXiv, 2018.
Chloe Hillier, and Andrew Zisserman. A short
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with performers, In ICLR, 2021.
Zihang Dai, Zhilin Yang, Yiming Yang. Jaime Carbonell, Quoc V Le. and Ruslan Salakhutdinov. Transformer-Xe XL: Attentive language models beyond a fixed-length context. arXiv, 2019.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. arXiv, 2018
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
David Fan, Jue Wang, Shuai Liao, Yi Zhu, Vimal Bhat, Hector Santos- Villaa--s Rohith MV, and Xinyu Li. Motion-guided masking for spatiotemporal representation learning. In ICCV, 2023.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Int. Conf. Comput. Vis., 2021.
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. In arXiv, 2018.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00390 2021.
Noureldien Hussein, Efstratios Gavves, and Arnold WM. Smeulders. Timeception for complex action recognition. In CVPR, 2019a.

---

Noureldien Hussein, Efstratios Gavves, and Aroll WM. Smeulders. VideoGraph: Recognizing minutes-long human activities videos. In ICCV Workshop, 2019b.
Md Mohaiminul Islam and Gedas Bertasius. Long movie clip classification with state-space video models. In ECCV, 2022.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. arXiv, 2017.
Hilde Kuehne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In CVPR, 2014.
Xirui Li, Chao Ma, Xiaokang Yang, and Ming-Hsuan Yang. VidToMe: Video token merging for zero-shot video editing. In CVPR, 2024.
Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. MViTv2: Improved multiscale vision transformers for classification and detection. In CVPR, 2022.
Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:2202.0788M 2022.
Xudong Lin, Fabio Petroni, Gedas Bertasius, Marcus Rohrbach, Shih-Fu Chang, and Lorenzo Torresani. Learning recognize procedural activities with distant supervision. In CVPR, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In arXiv, 2017.
Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. AdaViT: Adaptive vision transformers for efficient image recognition. In CVPR, 2022.
Antoine Miech, Dimitri Zhukov, Jean-Baptist Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, pp. 2630-26440, 2019.
Mandela Patrick, Dylan Campbell, Yuki Asano, Ishan Misra, Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and Joao F. Henriques. Keeping your eye on the ball: Trajectory attention in video transformers. In NeurlPS, 2021.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamic ViT: Efficient vision transformers with dynamic token sparsification. In NeurIPS, 2021.
Shuhuai Ren, Sishuo Chen, Shicheng Li, Xu Sun, and Lu Hou. TESTA: Temporal-spatial token aggregation for long-form video-language understanding. In EMNLP, 2023.
lain E. Richardson. H. 264 and MPEG-4 video compression: video coding for next-generation multimedia. 2004.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. ImageNet-21K pretraining for the masses. In NeurIPS, 2021.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. VideoBERT: joint model for video and language representation learning. In ICCV, 2019.
Yuchong Sun, Bei Liu, Hongwei Xue, Ruihua Sone, Huan Yang, and Jianlong Fu. Long-form video-language pre-training with multimodal temporal contrastive learning. In NewrIPS, 2022.
Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. COIN: large-scale dataset for comprehensive instructional video analysis. In CVPR, 2019.
12

---

Yansong Tang, Jiwen Lu, and Jie Zhou. Comprehensive instructional video analysis: The COIN dataset and performance evaluation. IEEE TPAMI, 13(9):3138-3155 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arrivv:776.00000 2017.
Jue Wang and Lorenzo Torresani. Deformable video transformer. In CVPR, 2022.
Jue Wang, Gedas Bertasius, Du Tran, and Lorenzo Torresani. Long-short temporal contrastive learning of video transformers. In CVPR, 2022.
Jue Wang. Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid.
Selective structured state-spaces for long-form video understanding. In CVPR, 2023.
Junke Wang, Xitong Yang, Hengduo Li, Zuxuan Wu, and Yu-Gang Jiang. Efficient video transformers with spatial-temporal token selection. arXiv preprint arXiv:2 2021.
Mathias Wien. High efficiency video coding. Coding Tools and Specification, 24, 2015.
Chao-Yuan Wu and Philipp Krähenbühhi. Towards long-form video understanding. In CVPR, 2021.
Chao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and multiscale vision transformer for efficient
Christoph Feichtenhofer. Mem VIT: long-term video recognition. In CVPR, 2022.
Hongxu Yin, Arash Vahdat, Jose Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. Adavit: Adaptive tokens for efficient vision transformer. arXiv preprint arXiv:2112.07658, 2021.
Jiaming Zhou, Kun- Yu Lin, Haoxin Li, and Wei-Shi Zheng. Graph-based high-order relation modeling for long-term action recognition. In CVPR, 2021.
13

---

A More Implementation Detail
A.1 Network Architecture
Figure 6 illusttraes the detailed architecture of the proposed learnable VTM. For all datasets, the encoder extracts the tokens with 1024 channel dimension. We that the linear layer in each VTM block reduces the channel dimension into half of it. Hence, after third VTM block, each token has 256 channel dimension. Also, the auxiliary path is only used for network training.
Mare
Lansat
VTM Block-2 Main Path
VTM Block Main Puth
Norp
Saliency eslimation
Portition Katzh
+5
Matching
PMaccingg
Xso
Guidel-
Migge
VIM Block A Avelllayy Pat
VIN Block
Sorpa Hul
Atinntoo
Auaillory Path
Nogg
Auailaary pula
VTM Block
VTM Block 2
VIM Block
Figure 6: A network architecture of the proposed learnable VTM.
B
More Experiments
B.1
Analysis on R
Table compares the results of the proposed algorithm at different R. Note that R denotes the number merged tokens at each VTM block. At R S), we merge all source tokens with target tokens. However, there may exist some source tokens which does not have target tokens with similar semantics. Thus, at R [S], undesirable merging of tokens can happen, thereby decreasing the performances. On the other hand, at R 0.5 S), only the half of source tokens are merged with target tokens, and thus some source tokens may not be merged even though they have similar target tokens. Therefore, the proposed algorithm shows the best scores
0.8
Table 9: Comparison of long-video understanding results the LVU dataset according to R.
R
Scene
Director
Like
74.41
66.40
0.22
0.8
75.58
70.09
0.21
0.5 S
72.09
67.28
0.22
14

---

C
More Visualizations
Figure visualizes the token merging results of the proposed learnable VTM on the LVU dataset.
Figure 7: Visualizations of video token merging results on the LVU dataset. Patches with same inner and border color are merged together.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See L65-L71 in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See L.200-287 in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification: NA
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 4.2 and Appendix
A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [No]
Justification: NA
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //ippe.cc/pullic/guiii wwwiiicgguie//eeeeeeeeeis CoodeSubeiiie ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: NA
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 4.2 and Appendix ??.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp:://neuriipee httpp//wwi....uuuu/eee/eee httpp//ssccppp..ttttttttttteeeea
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
19

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should citep the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.rdeewthcee....c.e. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
20

---

Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "bIoJBf543A", "reviewer": "Reviewer_rFDe", "summary": "This paper builds on Token Merging (ToMe) to improve its performance. In particular, the authors explore different ways to partition tokens so that the merging operation can lead to better performance while maintaining speed. They explore region-concentrated merging, motion-vector based merging and a learnable selector, and find that the learnable version works best. To make the network trainable, they employ a creative auxiliary path mechanism  to make everything differentiable. They find that their learnable VTM obtains good results compared to baselines on long form datasets (LVU, Coin, Breakfast), and that it outperforms the other methods they introduce.", "questions": "It’s not really expected that VTM (or any merging method) should score better than full attention, as it has strictly less information. This is backed up in the Token Merging and other follow up papers. Why is it expected (as said on L160) that merging should perform better? It’s supposed to be just faster, with a minimal drop in performance.\n\nThe motion vector method requires extracting pre-computed motion vectors from the encoded video. However, those are computed for 30 FPS, and for the original video size, meaning they don’t actually apply to downsampled (64 frames, 224x224). Was this taken into account? It’s certainly not fast to re-compute these motion vectors if you’re doing random cropping or frame selection.\n\nIs it possible to know the effect on training wall-clock time from this method? This the the metric that practitioners really care about, so including this would potentially strengthen the results of the paper.", "limitations": "The biggest limitation of this compared to baseline Token Merging is that it requires re-training. ToMe could be applied to a pre-trained network out-of-the-box. Learnable VTM cannot do this, making it impossible to apply to a pre-trained video network. The other proposed methods (region, motion vector) can do this though, and this would be a good thing to note somewhere in the paper.\n\nIn my view, the limitation of methods like VTM is that it always reduces a constant number of tokens per video, even though some videos are inherently more compressible than others. For example, Breakfast videos are extremely compressible compared to the average LVU video. However, this is beyond the scope of the paper, and using constant reduction is certainly more convenient, but this would be a good limitation to acknowledge and perhaps address in the future.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The problem this paper addresses is an important one. Videos (especially long ones) have many redundant tokens and reducing their number while maintaining performance is a crucial problem to solve in the field.\n\nThe model itself is well designed and uses a creative auxiliary path to handle a non-differentiable partitioning process. Given the premise of the paper, the model is well-designed and seems to address the issue they propose. \n\nI also appreciate the exploration of different methods, and a comparison on which worked better. This kind of analysis is often missing from papers and I am grateful for the authors for including it.", "weaknesses": "I don’t really agree with the premise of the paper (and am open to a rebuttal to explain if I’m wrong here). Token Merging already explored merging for video in detail. The reason Token Merging is based on similarity is that by combining tokens that are extremely similar, the weighted average of the those tokens should produce an extremely similar result in the attention operation. This was also detailed more in the Token Merging follow-up TomeSD. If you use different criteria such as saliency (which is not really well-defined), this is no longer guaranteed, and from equation (10) it seems like the authors do not use the proportional attention scheme from ToMe (Eq 1 in the original paper). Table 8 doesn’t show the learnable method using this; it seems to just be about the pooling part rather than the attention operation.\n\nI also don’t understand the intuition behind the saliency: shouldn’t we be aiming to combine together tokens that are NOT relevant, so that the transformer can focus more on the relevant tokens, rather than averaging (and thus losing) information from the more salient / important tokens? I’d really appreciate some clarification here. From Figure 3, it doesn’t look like learnable VTM is focusing on visually important tokens: it’s picking ones from the ceiling and wall in addition to the people.\n\nMy main issue is with the evaluation. The evaluation seems not quite fair, especially when measuring memory usage and throughput. Shouldn’t it be compared to baseline merging algorithms, like the naive ToMe? My impression is that the memory usage and throughput from VTM will be exactly the same as ToMe because it uses a similar partitioning scheme and constant factor reduction, which is why it may not be included in the results, but this seems important to include for context. Furthermore, the improvement on metrics is quite small, given that the speed is the same as other merging methods. Is this expected?\n\nAlso, The paper is motivated by “long-term” video, but evaluates on 64 frames, which isn’t really long and in my view, doesn’t merit only evaluating on LVU, Breakfast and COIN. Kinetics-400 has 300 frames per video, and is a more standard benchmark for evaluating video backbones - in fact, the original ToMe paper includes experiments on those datasets, which would make for a more fair comparison. Furthermore, nothing about the method itself is specific to these longer videos. I think evaluating on more standard datasets is crucial to measuring the actual strength of the method, especially compared to baselines like Token Merging. In particular, the long-form datasets are very compressible. \n\nThe paper is not well-written and the grammar needs a lot of revision, making it hard to focus on the content of the paper itself. In addition, a lot of space is spent on methods that are not really used in the final results (center, region, motion vector) and on citing equations from preliminary works (token merging, attention). Given that a claimed contribution is an exploration of these different methods, I would also have expected more detailed ablations and experiments to understand exactly why some of the methods perform better than others.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We do appreciate your constructive comments and will address them faithfully in the final paper. Please find our responses below.\n\n***\n\n> **Token merging for video** \n\nTo the best of our knowledge, there are only a few techniques for video token merging, such as TESTA(EMNLP2023), ToMe(ICLR2023), and VidToMe(CVPR2024). However, all of them are straightforward applications of the standard token merging to video input. Specifically, TESTA and VidToMe employ the same token partitioning, matching, and merging scheme of ToMe. \nIn contrast, we explored multiple token merging methods for video and proposed the learnable token merging method which shows good performances on various long-video classification datasets.\n\n\n> **Saliency**\n\nLearnable VTM uses the estimated saliency scores to divide the tokens into target token set and source token set. The saliency score estimation aims to select a greater number of important tokens (e.g. tokens corresponding to important objects in the scene) as target tokens, thereby increasing the influence of these tokens in the following attention processes after merging. \n\nAs in the standard token merging, we merge tokens based on their similarity. Hence, even though some tokens corresponding to the important objects are selected as target tokens, they are not merged with irrelevant tokens, and thus we do not lose much important information during the merging process. It is also shown in Figure 5. We can see that tokens are merged more in the background area than in the salient area.\n\nAlso, as stated in L206-208, we sample the target tokens based on the probability which is computed from the estimated saliency scores. Hence, it is less likely that the tokens with low saliency scores are selected as target tokens, but not always. As shown in Figure 3, some tokens corresponding to backgrounds or unimportant objects can be selected as target tokens. \n\n> **Comparison** \n\nPlease note that ToMe (ICLR2023) does not provide the results on the long video classification datasets. Kinetics only includes 10s video clip which is spatially heavy and without many temporal dependencies. From previous literature, it is easy to receive good performance with a single frame (What Makes a Video a Video: Analyzing Temporal Information in Video Understanding Models and Datasets, CVPR 2018), which is very different from the scope of this paper.  As stated in L154-155, our naive VTM is a straightforward application of ToMe which exploits the standard token merging method as intact as possible. As the reviewer pointed out, the memory usage and throughput of various VTM methods that we explored in this paper are almost the same. However, with the same memory and speed, a learnable VTM shows better performances than other VTM methods. Moreover, the proposed algorithm outperforms the conventional long-video understanding algorithms with large margins even though it has much better memory-efficiency and speed.\n\n> **Long video understanding**\n\nFor evaluation, we just follow the standard protocol in the field of long video understanding. For fair comparison, we use the same datasets (LVU, COIN, and Breakfast) and the same input setting as in conventional algorithms (ViS4mer, S5) for long video understanding.   \n\n\n> **High performance of VTM**\n\nIn the long video understanding datasets such as LVU, there is a lot of redundant and noisy information in a video. Therefore, as shown in Table 1, the transformer network does not yield reliable prediction results because those redundant tokens may hinder the feature refinement in the attention layers. However, when tokens are merged properly, the influence of important tokens in the attention process increases and thus the network can yield better performances, as shown in our experimental results. This phenomenon has also been revealed in the recently published token selection work (Selective Structured State-Spaces for Long-Form Video Understanding, CVPR 2023). \n\n> **Motion vector computation**\n\nAs stated in L194-195, motion decoding takes only 0.3 milliseconds for each frame which is negligible, since the motion is converted to the token resolution $H \\times W$ by using average pooling.  \n\n> **Training time**\n\nWe list the training time for the LVU and COIN datasets below. The training is finished within a few hours. We use 8 Tesla V100 GPUs and PyTorch for the experiments as stated in L258.\n\n| Methods       | LVU | COIN |\n|---------------|-----|------|\n| Learnable VTM | 4.0h | 3.5h  |\n\n\n\n> **Re-training, fixed compression ratio**\n\nWe agree with the reviewer. We will include this discussion in the revision. \n\n\n***\n\nIf you have any additional concerns, please let us know. We will address them faithfully. Thank you again for your constructive comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z8ndbBvnjK", "reviewer": "Reviewer_sUxH", "summary": "The paper approaches the task of long-video understanding from token reduction perspective. Specifically, Transformer-based approaches suffers from memory bottleneck and quadratic computation complexity with increasing number of tokens, which is even more pressing with long-videos as input. The paper builds on a recently developed token merging mechanism, and proposes a learned saliency measure to modulate what tokens gets merged instead of using a random or hand-crafted saliency measure. The central hypothesis of the work is that typically techniques that use similarity as merging criteria may inadvertently lose out on salient tokens. The paper reports experiments on three conventional long-video benchmarks (LVU, Breakfast and COIN), and shows effectiveness of their approach compared to prior related works both in terms of performance and memory requirement. The paper also ablates the effectiveness of their proposed saliency measure (learned VTM) over hand-crafted measures including motion-based (using optical flow), center-biased and random schemes.", "questions": "- Line 141: why L >= 60?\n- Is saliency projection used in all auxiliary VTM blocks?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written with most of the information presented for ease of understanding \n- The memory requirement is lower than S4, with competitive performance which highlights the importance of token selection in the case of long-videos", "weaknesses": "- Comparison to related token saliency approaches\n    - The paper proposes a scheme to identify salient tokens by using a learned projection matrix $U_s \\in \\mathcal{R}^{D \\times 1}$ with $\\texttt{tanh}$ activation function\n    - However, learnable token saliency methods have also been used in prior works, such as EVEREST [1], which uses a pair-wise learned saliency at feature-level (equation 2) using $\\texttt{Conv3d}$. The resulting approach was shown to be effective in the Masked Autoencoding setup\n    - Having a motion-based merging scheme is a good baseline, but some variants of learnable token saliency could also be tried to gain better understanding how token saliency gets influenced by different approaches   \n\n- Role of $L_1, L_2, L_3$\n    - The paper proposes to take tokens from $L_i$ consecutive frames for the $i^{th}$ VTM block\n    - It seems that choosing the values of $L_i$ is quite crucial given its impact on performance and memory requirement (Table 6) that forms the central claim of the paper\n    - However, the paper highlighted the contribution of token saliency more compared to the choice of $L_i$ hyperparameters\n    - Did the authors experiment with a rather simplistic setup using a single VTM block and/or with all $L_i$ being 60? It would help the readers to gain better understanding of what works in long-videos \n\n\n- How saliency changes with tokens from different number of frames?\n    - It seems that the saliency is being computed at each VTM block. It would be interesting to see how the saliency changes across the three VTM blocks\n    - On that note, what VTM block’s saliency is being visualized in Figure 5?\n\n\n\n### Minor\n- Line 145-146: “$i$-th transformer block takes the tokens corresponding to $L_i$ frames without overlapping”\n    - Confusing when $i$ is referred to as the frame number and the block number of transformer at the same time\n- Line 145-147: $j$ is not defined\n\n\n### Typos\n- Line 24-25: “so the tokenizing the”\n- Line 36: “selectio”\n- Line 114-115: “applications are mostly remained”\n- Line 117: “depedencies”\n- Line 161: “in the videos, .”\n- Line 165: “regarding less of the”\n- Line 177: “sailent”, “the the”\n- Line 306: “sailencies”\n\n\n### References\n[1] “EVEREST: Efficient Masked Video Autoencoder by Removing Redundant Spatiotemporal Tokens”. Sunil Hwang and Jaehong Yoon and Youngwan Lee and Sung Ju Hwang. ICML 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and insightful suggestions, all of which will be addressed faithfully in the final paper. Please find our responses below.\n\n***\n\n> **Difference from EVEREST**\n\nThe proposed learnable VTM is quite different from EVEREST. EVEREST measures the cosine similarity of tokens at the same spatial location in consecutive frames for more efficient mask generation which is used for masked video autoencoder training. Therefore, by its design, it does not consider the content of each token for saliency prediction. In contrast, learnable VTM employs the auxiliary path and the saliency guided attention to learn the saliency of each token. Therefore, the saliency estimation module is optimized to assign the high saliency scores to important tokens to reduce the classification loss in the auxiliary path.\n\n> **Learnable VTM variants**\n\nBelow, we compare the variants of learnable VTM. In 'From $X$’, we estimate the saliency scores from token features $X$ instead of key vectors $K$. In `Multiply’, we use $Q_\\mathrm{aux}K_\\mathrm{aux}^t\\odot\\mathbf{1}S^t$ inside Softmax in Eq.(10) for the saliency guided attention. Here, $\\odot$ denotes the Hadamard product. Both alternatives show decent scores. However, the proposed learnable VTM yields better performances and thus we employ it as our method. \n\n| Methods       | Relationship | Scene | Director | Writer |\n|---------------|--------------|-------|----------|--------|\n| From $X$      | 59.52        | 75.58 | 63.55    | 51.19  |\n| Multiply      | 59.52        | 72.09 | 70.09    | 46.42  |\n| Learnable VTM | 64.28        | 75.58 | 70.09    | 53.57  |\n\n\n\n> **Impact of $L$**\n\nYes, as the reviewer pointed out, $L_i$ is an important hyper-parameter for both efficiency and effectiveness of the proposed algorithm. Therefore, we analyze it in Table 6. However, the performance gain of learnable VTM is not just from the selection of $L$, because the other VTM methods in Table 1 yield low scores at the same $L$ setting. We will discuss this in the revision. \n\n> **$L>60$**\n\nWe evaluate the proposed algorithm with $L=60$ since conventional algorithms use 60 frames for their evaluation. Below, we list the performances of learnable VTM with $L=100$ setting. We see that the proposed algorithm yields good results in this setting as well. \n\n| Methods | Relationship | Speaking | Scene | Director | Genre | Writer | Year  | Like | View |\n|---------|--------------|----------|-------|----------|-------|--------|-------|------|------|\n| $L=60$  | 64.28        | 42.12    | 75.58 | 70.09    | 59.77 | 53.57  | 48.55 | 0.21 | 4.01 |\n| $L=100$ | 67.11        | 42.12    | 75.58 | 68.22    | 61.34 | 52.38  | 48.55 | 0.22 | 3.85 |\n\n \n\n> **Saliency visualization**\n\nWe will include the visualization results in the final copy. \n\n> **Figure 5**\n\nIn Figure 5, we visualize the token merging results at the last VTM block, as stated in L298-299. We will revise this more clearly in the revision.\n\n> **L145-147 and typos**\n\nWe will revise them thoroughly. Thank you for your suggestion.\n\n\n\n***\n\nIf you have any additional concerns, please let us know. We will address them faithfully. Thank you again for your positive comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pyJ4gizUn1", "reviewer": "Reviewer_7bwU", "summary": "This paper explores various video token merging strategies in the context of long-form video classification and finally propose a learnable Video Token Merging algorithm that dynamically merges video tokens based on visual salient areas. The contributions are summarized as follow:\n1.  Explore various video token merging methods including the naïve VTM, the region concentrated VTM, and the motion-based VTM.\n2.  Propose the learnable video token merging algorithm, which estimates the saliency scores of each token and adaptively merge visual tokens based on those scores.\n3. The proposed algorithm achieves the best or competitive results on various datasets.", "questions": "1. Due to the two-paths design, has the training time doubled?\n2. The paper tries to learn saliency scores using matrix $U_s$. How about using $\\sum{QK^T}$ in Equation 8 as saliency scores for each visual token?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper explores various video token merging methods including the naïve VTM, the region concentrated VTM, and the motion-based VTM.\n2. Compare with baseline and rule-based video token merging. The proposed learnable video token merging strategy has large improvement.\n3. The two-paths design to deal with non-differentiable problem in partitioning process is interesting.", "weaknesses": "1. This paper proposes a leanable video token merging strategy. The similiar high-level idea can be found by CTS[1] in image domain. The novelty is insufficient。\n2. This paper focuses on video token merging. However, I do not observe any specific design tailored for the video domain in terms of the methodology. let alone long video.\n\n\n\n[1] Content-aware Token Sharing for Efficient Semantic Segmentation with Vision Transformers", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We do appreciate your constructive comments and will address them faithfully in the final paper. Please find our responses below.\n\n***\n\n> **Difference from CTS**\n\nCTS is not similar to the proposed algorithm, since it is not even a token merging method. CTS is a semantic segmentation algorithm which shares some neighboring tokens expected to belong to the same class in the segmentation network. Hence, CTS does not have any similar key idea such as learning the token saliency or merging tokens. We will do our best to answer your concerns if you provide more detailed points. \n\n> **Design for video domain**\n\nCompared to the images, there is much more redundant or noisy information in the videos, which deteriorates the quality of video understanding. The proposed learnable VTM is a domain-specific algorithm for long-context scenario. To reveal the discriminative information among redundant tokens in the video, we develop the learnable VTM which learns the saliency of each token and keeps the salient tokens, thereby increasing the influence of these tokens in the following attention processes after merging. Also, we explored the motion-based VTM, which selects the tokens with large motions as the target tokens, and thus this token merging method is only applicable to the video. \n\n\n> **Training time**\n\nBelow, we compare the training time. The training time is slightly increased, but not doubled. We use 8 Tesla V100 GPUs and PyTorch for the experiments as stated in L258.\n\n| Naive VTM | Learnable VTM |\n|-----------|---------------|\n| 4.0h      | 3.7h          |\n\n\n> **Attention as saliency score**\n\nThe objective of saliency estimation is to assign high scores to the tokens corresponding to important objects and low scores to them corresponding to backgrounds or noisy information. However, $\\sum QK^t$ would assign high scores to the tokens which have many similar tokens regardless of their semantics. Therefore, we employ the saliency estimation module for the proposed learnable VTM.\n\n***\nIf you have any additional concerns, please let us know. We will address them faithfully. Thank you again for your constructive comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BL9xhf7PFw", "reviewer": "Reviewer_JTuZ", "summary": "- This paper carries out an analysis of token merging[1] in the context of long-form video understanding and proposes learnable video token merging (VTM) to select semantics/saliency guided tokens for merging. \n- In token merging, at each layer, the tokens are divided into two sets source S and target T through uniform sampling. Tokens in S are matched to T based on similarity and merged (usually by average pooling). This paper compares this naive VTM with two other variants where selection of T is guided by informed heuristics: (1) region VTM where tokens at the center of each frame are more likely to be retained, (2) motion VTM where tokens with high motion are more likely to be retained. Through this analysis, authors argue that the strategy to select T plays an important role in the final performance.\n- Motivated by this, authors propose a learnable VTM where it first predicts a saliency score for each input token. The target set T is sampled according to the probability distribution defined by saliency score. Since this partition operation is not differentiable, authors propose a novel training architecture where a parallel auxiliary network is trained alongside. The saliency scores are used to bias the attention score of the aux network, thereby supervising the saliency prediction to focus on important tokens. Aux network can be discarded at test time.\n- Authors carry out a fair evaluation of the learnable VTM on LVU, Breakfast and COIN datasets by comparing against several baselines including ViS4mer, S5, D-sprv. Learnable VTM performs better than baselines in almost all evaluation tasks with low GPU memory usage and high throughput.\n\n[1] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token Merging: Your ViT but faster. In ICLR, 2022.", "questions": "- How is learnable VTM trained? From Fig 4, it looks like there are two outputs from the network. Do you apply the same loss on both outputs?\n- In Fig 4, what does the 'Merge' operation in auxiliary path do? Does it mean that the main and aux - both paths use the same target set sampled by the partition-match of the main path?", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- In learnable VTM, the idea of learnable and auxiliary path is interesting. There is no way to directly supervise the token saliency prediction of the main path because partition operation is non-differentiable. Hence the attention in auxiliary path is influenced by saliency scores of the main path, which encourages the saliency prediction to focus on important tokens.\n- The evaluation is fair and consistent. The authors use the same backbone as the prior works to encode input video frames, thereby ensuring a fair evaluation.\n- The results of learnable VTM on LVU dataset and Breakfast is noticeably better than baselines with less GPU memory usage. However, on COIN dataset, it doesn't perform better than S5 baseline.", "weaknesses": "### Major weaknesses\n- One of the cited contribution is the exploration of region based and motion based VTM (Section 3.3) but it seems trivial. The effectiveness of token selection is already shown in learnable VTM. In light of that, there is an unreasonable focus section 3.3 which is unnecessary.\n- Section 3.4 explains little about the details of learnable VTM, how it is trained, how the gradients flow in the presence of non-differentiable partition function, etc.\n- There are some stretched claims based on qualitative and quantitative results. For example,\n  - In Line 174, authors claim that center VTM performs better than naive VTM. However, according to Table 1, the results are mixed at best.\n  - In Fig 5, authors also claim that the visualization of merged tokens show saliency based merging. However, the figure doesn't support the claim. There are many merged tokens on important salient features and some background tokens are not merged.\n\n### Minor issues\n- Line 19: it should be \"into the domain of video computer vision\" as all cited papers are video learning papers.\n- Is there a difference between notation of C and D? It looks like both are used interchangably to denote token dimension.\n- Table 2: How it throughput measured? fps?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and valuable comments. Please find our responses below.\n\n***\n\n> **Exploration**\n \nCompared to image token merging methods, token merging for video is relatively under-researched. In this work, we investigate various video token merging methods and finally propose a learnable VTM that outperforms all previous techniques. Like our algorithm, there are many conventional algorithms in the video computer vision community that proposed simple but effective algorithms, such as 3D-CNN, inflated 3D kernel, (2+1)D CNN and etc. Hence, we believe that the exploration of various basic token merging methods has its benefits for the research community and the proposed learnable VTM algorithm will serve as a strong baseline. However, as the reviewer suggested, we will reduce Section 3.3 and include more analysis on learnable VTM. \n\n> **Training of learnable VTM**\n\nLearnable VTM is optimized to reduce the classification losses on both main path and auxiliary path. Also, the entire process is differentiable because we merge the matched tokens by using average pooling as in the standard token merging. Partitioning and matching processes do not have to be differentiable for end-to-end backpropagation. These processes do not update the features but just determine which tokens to merge. We will include this explanation in the revision.\n\n> **Stretched claims**\n\nWe agree with the reviewer. We will revise the draft to explain the experimental results precisely. \n\n> **Citations**\n\nWe will revise the draft as suggested.\n\n> **Notation C and D**\n\nWe will revise the notation more clearly.\n\n> **Throughput**\n\nYes, we measure the FPS.\n\n> **Training**\n \nIn learnable VTM, we apply the same video classification loss to both predictions from the main path and the auxiliary path. Therefore, the saliency estimation module is optimized to assign high saliency scores to the important tokens to reduce the classification loss during training, since the tokens with high saliency scores have more influence in the guided attention in the auxiliary path.  \n\n> **Figure 4**\n\nYes, tokens in the auxiliary path are also merged according to the matching results from the main path. \n\n***\nWe will address all these comments faithfully in the final paper. If you have additional comments, please let us know. Thank you again.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their time and efforts for providing constructive reviews. Also, we extend our thanks to the program and area chairs. We have faithfully responded to all comments below.

Meta-review: As the Area Chair overseeing the review of Submission4154, I recommend acceptance for NeurIPS 2024. The paper introduces an innovative learnable Video Token Merging (VTM) method that dynamically adjusts token merging based on saliency, showing significant improvements on long-video classification tasks on datasets such as LVU, Coin, and Breakfast.

Reviewer sUxH praises the method’s approach to computational efficiency and memory optimization, which are critical for processing long video inputs. The exploration of different token merging strategies provides a clear understanding of the proposed method’s advantages.

Reviewer rFDe’s concerns about evaluation and comparison baselines were addressed effectively in the authors' rebuttal. Their commitment to include additional experiments and adapt their approach in response to feedback highlights the paper's potential and thoroughness.

Reviewer 7bwU’s comments on novelty and design considerations for video processing were met with detailed clarifications, underscoring the method’s relevance and adaptability for complex video understanding tasks.

The paper’s theoretical and practical contributions offer valuable insights into optimizing transformer networks for video analysis, making a substantial impact on the field. The positive adjustments and responses to reviewer feedback justify its acceptance at the conference.
Decision: Accept (poster)