Video Token Merging for Long Video Understanding

Authors: seon-ho  lee
jue  wang
zhikang  zhang
david  fan
meta  fair
xinyu  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 21:13:13

Abstract: As the scale of data and models for video understanding rapidly expand, handling long-form video input in transformer-based models presents a practical challenge. Rather than resorting to input sampling or token dropping, which may result in information loss, token merging shows promising results when used in collaboration with transformers. However, the application of token merging for long-form video processing is not trivial. We begin with the premise that token merging should not rely solely on the similarity of video tokens; the saliency of tokens should also be considered. To address this, we explore various video token merging strategies for long-form video classification, starting with a simple extension of image token merging, moving to region-concentrated merging, and finally proposing a learnable video token merging (VTM) algorithm that dynamically merges tokens based on their saliency. Extensive experimental results show that we achieve better or comparable performances on the LVU, COIN, and Breakfast datasets. Moreover, our approach significantly reduces memory costs by 84% and boosts throughput by approximately 6.89 times compared to baseline algorithms.

Main paper: 
 introduction: 
 Over the past few years, the Transformer architecture (Vaswani et al., 2017) has risen as a revolutionary paradigm within natural language processing (NLP) (Devlin et al., 2018) and has seamlessly expanded its influence into the domain of computer vision (Wang & Torresani, 2022; Bertasius et al., 2021; Wang et al., 2022; Akbari et al., 2021; Li et al., 2022; Fan et al., 2021) . This expansion has been exemplified by remarkable achievements in recent multi-modality foundation models such as Sora (Brooks et al., 2024) , GPT4 (Achiam et al., 2023) , and Gemini (gem), showcasing its exceptional performance and versatility across diverse applications.In contrast to the natural language processing, the visual input has much lower information density and thus tokenizing the raw RGB image as non-overlapped patches becomes the essential operation in vision transformers (Wang & Torresani, 2022; Bertasius et al., 2021; Dosovitskiy et al., 2021) . However, the computational cost of transformer exponentially increases with the sequence length, which generates tremendous computation when feeding visual input into the large-scaled transformer models with billions of parameters. Due to the redundancy in video sequence, this phenomenon becomes more severe with video input, especially for long-form videos. This bottleneck impedes the Figure 1 : Comparison of GPU memory footprint and throughput against scene prediction accuracy on the LVU dataset (Wu & Krähenbühl, 2021) .further advancement of foundational models in handling long-form video data. Various attempts (Yin et al., 2021; Wang et al., 2021; Meng et al., 2022; Rao et al., 2021; Liang et al., 2022) have been proposed to improve the efficiency of vision transformer by introducing a token selection module. However, these methods are primarily designed for images and may require non-trivial adaptation to the long-form video scenarios due to the video-level long-term dependencies and motion dynamics. Moreover, tokens dropped by the token selection module cannot be reused in later layers, which may result in the loss of important information.In addition to the token selection, token merging techniques (Bolya et al., 2022; Ren et al., 2023; Bolya & Hoffman, 2023; Li et al., 2024) have been proposed to increase the efficiency and effectiveness of transformer-based networks. Specifically, they reduce the sequence length by merging similar tokens, thereby decreasing the computational cost. In addition to the efficiency, token merging demonstrates huge advantages by increasing the contextual information so that the model can learn from patterns presented across multiple tokens. Previous token merging algorithms in both the image and video domains use manually designed token partitioning methods and merge tokens based on their similarity. Even though the merged tokens would still keep the original information, they may have different granularity after the merging operation. In this paper, it is argued that different regions in the visual data may have different information density. Since the discriminative information of tokens may be degenerated after merging, some visual tokens should not be merged even if they look similar to each other. Rather than relying solely on similarity, we question whether more unmerged tokens should be used to describe salient areas, while merging more tokens for the background.In this paper, we explore various video token merging (VTM) methods in long-form video classification task and aim to find the effective token merging method for long-form videos. Previous video token merging method (Li et al., 2024) only decouples the spatial and temporal dimensions, which is unfavorable, especially for long-form videos. As the long-term dependency plays an important role in the long-form video understanding, spatiotemporal visual tokens should be considered jointly. Sequentially merging token along with one dimension after another may generate biased prior. In our work, we first naively extend the image-based token merging (Bolya et al., 2022) to the video domain and then propose region-centralized and motion-based token merging algorithms, which estimate the salient region of video sequences. Finally, we develop a learnable VTM which predicts the saliency score of each token and adaptively merges spatiotemporal visual tokens in data-driven manner. Experimental results demonstrate that the proposed algorithm improves the effectiveness and the efficiency of the transformer-based network and outperforms the conventional long-video understanding methods with better throughput and less memory usage, as also shown in Figure 1 .We summarize the contributions of this paper as following:• We explore various video token merging methods including the naïve VTM, the regionconcentrated VTM, and the motion-based VTM.• We propose the learnable video token merging algorithm, which estimates the saliency score of each token and adaptively merges visual tokens based on their scores.• The proposed algorithm achieves the best or competitive results on various datasets including LVU, Breakfast and COIN. Moreover, we significantly reduce memory costs by 84% and improve the throughput by 6.89 times via the proposed learnable VTM.2 Related Work 
long-form video modeling: 
 Transformers have demonstrated remarkable prowess in capturing long-term dependencies, as evidenced in their success in natural language processing (NLP) tasks (Brown et al., 2020; Dai et al., 2019; Devlin et al., 2018) . However, the intensive computational requirements stemming from dense self-attention calculations (Vaswani et al., 2017) pose a significant obstacle not only in NLP but also in the domain of computer vision, especially for the long-form videos. Many recent video transformer works (Wang & Torresani, 2022; Liu et al., 2021; Bertasius et al., 2021) focuses on improving the global attention mechanism. However, they are not designed for dealing with redundant spatial and temporal image tokens that are common in long-form video scenarios. To capture longer temporal information, LF-VILA (Sun et al., 2022) develops a hierarchical architecture to include more frames in the model. Similarly, MeMViT (Wu et al., 2022) utilizes longer temporal information by emerging the previously cached "memory" from the past. A novel alternative to transformers is the Structured State-Space Sequence (S4) model proposed by Gu et al. (2021) , which models the long-range dependencies by simulating a linear time invariant (LTI) system. Subsequently, ViS4mer (Islam & Bertasius, 2022) and S5 (Wang et al., 2023) extend S4 model to the long-form video classification task. ViS4mer (Islam & Bertasius, 2022) stacks multiple S4 layers with different scales in modeling long-form videos, and S5 (Wang et al., 2023) include an additional selective module to further improve the performance. Unlike these works that focus on the improvement of architecture and attention mechanism, this paper will start from a more basic concept in the transformer, video tokens and how to effectively merge them. Even though our proposed method can theoretically be applied on S4 model, the scope of this paper is on the well established transformer architecture. We will leave the investigation of video token merging on S4 (Gu et al., 2021) in the future work.Adaptive token selection is widely used to improve model efficiency by leveraging a light-weight selection module to pick up the 'useful' tokens while dropping the 'unuseful' ones. In vision transformer, STTS (Wang et al., 2021) utilizes a token selection module known as the named scorer network to assign importance scores to each token, subsequently selecting the top-K frames with the highest scores. Building upon this concept, AdaViT (Meng et al., 2022) further extends the approach by developing instance-specific policies. These policies guide the activation of patches, self-attention heads, and transformer blocks, enhancing adaptability and efficiency in processing visual data. STTS, AdaVit and other similar approaches (Wang et al., 2021; Meng et al., 2022; Rao et al., 2021; Liang et al., 2022) drop a significant number of tokens in the early decision stage to save more cost, but the dropped tokens cannot be reused in the later layers, which is easier to degenerate the contextual information in the long-form videos.Visual token merging is first proposed in (Bolya et al., 2022) which aims at increasing the throughput of existing ViT models without training. Following works (Ren et al., 2023; Bolya & Hoffman, 2023; Li et al., 2024) leverage this idea to save computational cost in different downstream applications, such as diffusion model, video and language understanding, and video editing. Specifically, visual tokens are first partitioned into two sets with equal size; for each of the edge tokens in one set, find the most similar token in another set and merge them by average pooling; finally, concatenate two sets back together. Although the token merging is simple and effective, its applications have mostly remained in the image domain. There is no fundamental research work has been explored for the long-form video token merging strategies, where the spatiotemporal tokens are more redundant and embed complicated dependencies locally and globally. In this work, we argue that visual tokens from the long-form video should be carefully partitioned and merged based on the salient areas in videos.To this end, we ablate various video token merging algorithms and provide extensive expermental results and analysis.3 Proposed AlgorithmQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See L280-287 in Section 4.3.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminary -token merging: 
 Token merging (Bolya et al., 2022) aims to reduce the redundancy by merging similar tokens at each transformer block, thereby increasing the effectiveness and efficiency of a transformer-based network. Specifically, token merging has three steps: partitioning, matching, and merging.Partitioning: For given a set of N tokens X = {x 1 , x 2 , . . . , x N }, token merging first partition X into a set of target tokens T and a set of source tokens S, given byT = {x i : i mod γ = 0}, (1) S = {x j : j mod γ ̸ = 0} (2)where γ is partition factor and mod denotes the modulo operator. Thus, |T | = |X | γ . Also, X = T ∪S and T ∩ S = ∅.Matching: Then, for each source token in S, it finds the most similar target token in T . Here, the similarity between tokens x i and x j is defined as the cosine similarity of the corresponding key vectors k i and k j , which are obtained in the most recent self-attention layer. For a source token x j ∈ S, the index of its matched target token m j is obtained by,EQUATION)Merging: Lastly, token merging merges the tokens based on the matching results. For each target token x i ∈ T , it obtains the merged token y i by using average pooling,EQUATIONwhere M i = {j : m j = i, ∀x j ∈ S} is the index set of source tokens which are matched with x i . In this case, the number of tokens is reduced by |S| after token merging. It can also control the number of reduced tokens by R by reassigning the matching index asEQUATIONfor all x j except for the source tokens with the R highest similarity scores.Suppose that a video with L frames is given, where L ≥ 60. To perform the classification or regression on the given video, we can use a simple transformer-based network, which is shown in Figure 2 (a) and (b). It first obtains the tokens X 1 , X 2 , . . . , X L ∈ R H×W ×C by using an encoder.Here, H, W, and C denote the height, the width, and the channel dimension of the token tensor, respectively. Then, it utilizes transformer blocks to update the tokens. As its input, i-th transformer block takes the tokens corresponding to L i frames without overlapping, where L i ≤ L j ≤ L for i < j. The prediction head yields the estimation results. However, it requires the prohibitively large memory and computation costs due to the quadratic complexity of the self-attentionO(L 2 H 2 W 2 D 2 ),where D is the dimension of key vectors. Hence, our goal is to increase the efficiency of this baseline network by reducing the number of tokens via token merging, while maintaining or even improving the performances of the network by removing the redundant or noisy information in the video. To this end, we explore various token merging methods for long video processing.Naïve video token merging: First, we combine the standard token merging with the baseline network as intact as possible. To this end, we substitute transformer blocks with VTM blocks, in which token merging layer is inserted after the dropout layer, as illustrated in Figure 2 (c ). In this naïve VTM, we employ the standard token merging in (1)-( 4). At each i-th VTM block, naïve VTM reduces the R i tokens. For example, at γ = 4 and R = |S|, it gradually removes the 68% of tokens over the network. Hence, the computation cost of the self-attention is reduced from 1 , this naïve VTM shows better scores than the baseline network, because the token merging reduces the information redundancy in the videos.O(N 2 D 2 ) to O(( N γ i-1 ) 2 D 2 ) at i-th VTM block. As shown in TableRegion-concentrated video token merging: Compared to an image, a video contains redundant spatiotemporal tokens. Depending on the tasks, some tokens are more important than others. However, as shown in Figure 3 (a), naïve VTM selects every γ-th token as the target tokens since it exploits uniform token partitioning in (1). Also, for token merging, it purely relies on the similarity between tokens regardless of the semantics, and thus the self-attention can more easily swayed by unnecessary information. Therefore, for better video token merging, it is important to consider the saliency of each token before merging them.To investigate this issue, we explore center-concentrated video token merging and boundaryconcentrated video token merging. The center-concentrated token merging samples 50% of the entire target tokens from the center area with the size of H 2 × W 2 , which uses more unmerged tokens to describe center area and merge more token from the boundary. Specifically, we use the partition factor of γ 2 for the center area and 3 2 γ for the remaining area. On the other side, we implement the opposite operation for the boundary-concentrated video token merging. As shown in Table 1 , center-concentrated VTM shows better performances than naïve and boundary-concentrated VTM in general. Since meaningful objects and motions are typically center-concentrated, this suggests more tokens from salient regions should be unmerged while more of the rest tokens should be merged. Figure 3 (b) shows the partitioning results of center-concentrated VTM.Motion-based video token merging: Even though center-concentrated VTM has shown better performances than naïve VTM, the meaningful tokens are not always located in the center area. Moreover, the hand-crafted partitioning method forces the center-concentrated VTM to select the same number of target tokens from each frame, which is not flexible enough when applied at scale. Therefore, we explore motion-based video token merging which divides tokens into T and S based on the motion information since the moving objects carry important cues in general (Fan et al., 2023) .Let us assume that we have the magnitude of motion vector v i for each token x i . We compute the sampling probability by using softmax Note that the sampling probability is proportional to the motion magnitude. Then, we construct T by sampling N γ tokens from X with the sampling probability p i for each token x i . The goal of VTM is to increase the efficiency of transformer-based network for long video understanding. Therefore, the motion information should be obtained with negligible time and computation costs. Hence, instead of estimating the motion information with an additional module, we use the motion information which is already stored in the video files; most modern video codecs, such as MPEG-4 (Richardson, 2004) , H.264 (Richardson, 2004) , and HEVC (Wien, 2015) , exploit the motion information for efficient compression. The motion decoding takes only 0.3 milliseconds for each frame which is negligible. EQUATION)There are some videos in which unimportant objects have large motions due to various factors such as camera movement. Motion-based VTM may not yield good results on those videos. To maximize the generalizability, we develop learnable video token merging method. Instead of depending on the motion information, learnable VTM estimates the saliency score of each token and samples the target tokens based on the estimated scores. Figure 4 shows the architecture of learnable VTM block.Learnable VTM block contains two forward paths: a main path and an auxiliary path. Let us assume that we have a tensor of N tokens X ∈ R N ×C . In the main path, we first obtain query Q, key K, and value V byQ = XU q , K = XU k , V = XU v (7) using learnable projection matrices U q , U k , U v ∈ R C×D .We perform the standard self-attention on Q, K, and V and yield the updated tokens X ′ asEQUATIONAlso, from K, we estimate the saliency scores S of tokens byS = [s 1 , s 2 , . . . , s N ] ⊤ = tanh(KU s ) (9) where U s ∈ R D×1 is a learnable matrix. Also, s i ∈ (-1, 1) for 1 ≤ i ≤ N .Then, we compute the sampling probability using (6) with s i instead of v i for each token x i and sample T with the sampling probability as in motion-based VTM. After the token partitioning, we match and merge the tokens in X ′ by using (3) and (4), respectively. However, the learnable matrix U s in (9) can not be trained only with the main path since the partitioning process is not differentiable. To handle this issue, we employ the auxiliary path. This path consists of a saliency guided self-attention layer and a merging operation. The auxiliary path takes a tensor of auxiliary tokens X aux ∈ R N ×C as its input. Similar to the main path, we obtain query Q aux , key K aux , and value V aux . Then, we perform the saliency guided attention to obtain the updated auxiliary tokens X ′ aux as where 1 is a N dimensional vector of ones. In the saliency guided attention, the contribution of each token is controlled by its estimated saliency score; if s i is positive, i-th token affects more on X ′ aux , whereas if s i is negative it has less influence on X ′ aux . In other words, it increases the influences of the tokens with high saliency scores in the attention process. Therefore, during training, the network is encouraged to assign high saliency scores to the tokens with meaningful information and low saliency scores to the others to obtain the better predictions. At the first VTM block, the auxiliary path employs the same input with the main path. From the second VTM block, it takes the output of auxiliary path in the previous VTM block as its input. Also, it is worth pointing out that the auxiliary path is used for the network training only. Compared to other VTM methods, learnable VTM only introduces additional computation of score estimation module, which is fast and light enough, during test. Therefore, it shows almost same inference speed with other VTM methods.X ′ aux = softmax(Q aux K ⊤ aux + 1S ⊤ ) √ C V aux (10)Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 4.1 Datasets LVU (Wu & Krähenbühl, 2021) : It contains ∼30K videos sampled from ∼3K movies on the MovieClips (mov) website. Most videos are 1 to 3 minutes long. It provides the labels for 9 long-video understanding tasks which can be grouped into three major categories:• Content understanding: 'relationship,' 'speaking style,' and 'scene/place' • Metadata prediction: 'director,' 'genre,' 'writer,' and 'movie release year' • User engagement: 'YouTube like ratio,' and 'YouTube popularity'As the evaluation metrics, we adopt the top 1 classification accuracy for content understanding and metadata prediction tasks and mean-squared error (MSE) for user engagement tasks. Breakfast (Kuehne et al., 2014) : It provides 1,712 videos with the average length of 2.32 minutes and the total length of 77 hours. The videos contain 52 individuals and 18 different backgrounds in total. Each video belongs to one of 10 complex cooking activities. COIN (Tang et al., 2019) : It consists of 11,827 videos with the average length of 2.36 minutes, collected from YouTube. Each video belongs to one of 180 distinct procedural tasks. We follow the experimental settings of conventional long-form video understanding algorithms (Islam & Bertasius, 2022; Wang et al., 2023) . We employ three transformer blocks in the baseline network.As the encoder, we use ViT-L (Dosovitskiy et al., 2021) We use the AdamW (Loshchilov & Hutter, 2017) optimizer with a batch size of 16 and a weight decay of 0.01. We set the learning rate to 0.001. We train the network for 70 epochs by using cosine learning rate scheduler (Gotmare et al., 2018) with 10 epochs warm-up. For experiments, we use 8 Tesla V100 GPUs and PyTorch.Comparison on LVU: In Table 2 , we compare the proposed algorithm with the conventional methods on the LVU dataset. With the smallest memory footprint, the proposed algorithm achieves the best scores in 7 out of 9 tasks on the LVU dataset. Performer (Choromanski et al., 2021) and Orthoformer (Patrick et al., 2021) employ the efficient variants of self-attention to reduce the computation costs. The proposed learnable VTM outperforms these approaches with significant performance margins and less GPU memory usages. It shows the efficiency and effectiveness of our approach. Also, ViS4mer (Islam & Bertasius, 2022) and S5 (Wang et al., 2023) adopt S4 layers instead of self-attention layers to capture the long-term dependencies in videos because of its linear computation complexity to the number of input tokens. The promising results of these methods have suggested that S4 layer can be an efficient replacement of self-attention layer for long-form video inputs. However, the higher scores of the proposed algorithm broaden the potential usages of the self-attention layers for various long-form video tasks. Moreover, S5 utilizes LSMCL, which is a pretraining based on the contrastive learning, to boost its performances. Nevertheless, without the time-consuming pretraining, the proposed algorithm yields better scores on the LVU dataset. Comparison on Breakfast: Table 3 compares the performances of the proposed algorithm and the conventional techniques on the Breakfast dataset. The proposed learnable VTM achieves the best score on this challenging long-range activity classification dataset as well. For pretraining, D-sprv (Lin et al., 2022) leverages the HowTo100M (Miech et al., 2019) dataset which contains much more training samples than our pretraining dataset, Kinetics-600 (Carreira et al., 2018) . Nevertheless, we outperform D-sprv (Lin et al., 2022) with the accuracy gap of 1.36%.Comparison on COIN: Table 4 compares the scores of the proposed algorithm and the conventional techniques on the COIN dataset. We note that the COIN dataset consists of the videos on YouTube and more than 1,000 videos are not available anymore. Therefore, ViS4mer, which is one of the state-of-the-art method on the COIN dataset, achieves only 87.11% accuracy when it is trained on the current version of the COIN dataset. It may be because of many missing training videos. Even though the comparison is not perfectly fair, we report the results on the COIN dataset for reference. The proposed learnable VTM yields better results than ViS4mer with the same training and test data. Also, it shows the comparable score with S5 (Wang et al., 2023) .Analysis on γ: Table 5 compares the performances of learnable VTM with different γ. Compared to the baseline, at all γ, the proposed algorithm improves the performances. Also, at γ = 10, it increase the throughput and the memory efficiency by 7.49 and 6.6 times, respectively. At γ = 6, the proposed algorithm shows the best results. Analysis on (L 1 , L 2 , L 3 ): Table 6 shows the results of the proposed learnable VTM with different (L 1 , L 2 , L 3 ). Note that i-th VTM block takes the tokens in L i consecutive frames as its input at a time. At (L 1 , L 2 , L 3 ) = (4, 20, 60), the proposed algorithm yields the lowest scores, since it can not capture the long temporal dependency in the early stage of the network. We see that the proposed algorithm yields the best scores at (L 1 , L 2 , L 3 ) = (6, 30, 60).Weighted average pooling: To merge tokens, we use average pooling as the default setting in all VTM methods. However, once tokens are merged, they may represent more than one input patch. Thus, to reflect the token size in merging process, we combine tokens by averaging weighted by their sizes. However, as shown in Table 7 , this weighted average pooling decrease the performances of learnable VTM. Thus, we exploit the average pooling to merge tokens.In motion-based VTM, we can combine tokens by averaging weighted by their motion magnitudes. Table 7 shows the performances of motion-based VTM with the motion weighted average pooling. It yields the similar scores with the standard motion-based VTM with the average pooling.Complexity: Table 8 compares the throughput and memory footprint of learnable VTM during training and inference. Since the auxiliary path is additionally employed during training, it requires more computation costs. However, even during training, learnable VTM is still faster than conventional methods including ViS4mer (Islam & Bertasius, 2022) and S5 (Wang et al., 2023) and it also requires less amounts of memory than them.Visualizations: In Figure 7 , we visualize the tokens merging results at the end of the network over multiple frames of video. We see that tokens with similar semantics are merged together. Also, tokens corresponding to backgrounds or unnecessary informations are merged more than tokens corresponding to salient objects. It is because learnable VTM selects tokens with high saliency scores as the target tokens. More visualization results are available in the supplemental document. 
conclusion: 
 In this paper, we investigate the video token merging techniques for long-form video data. Unlike previous algorithms that apply uniform partitioning and merge tokens solely based on the visual similarity, we argue tokens with different saliencies should be treated unequally to avoid undesirable information loss after merging important tokens. To this end, we explore various video token merging methods and receive interesting intuitions from region-concentrated and motion-based token merging results. Lastly, we propose a learnable video token merging scheme that adaptively samples target tokens and learns discriminative representations from the long-form videos. Compared to the baseline, the proposed algorithm achieves substantial improvements in terms of the performance, memory cost and throughput. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: NA Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: NA Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should citep the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: NA Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Network ArchitectureFigure 6 illustrates the detailed architecture of the proposed learnable VTM. For all datasets, the encoder extracts the tokens with 1024 channel dimension. We note that the linear layer in each VTM block reduces the channel dimension into half of it. Hence, after third VTM block, each token has 256 channel dimension. Also, the auxiliary path is only used for network training. Table 9 compares the results of the proposed algorithm at different R. Note that R denotes the number of merged tokens at each VTM block. At R = |S|, we merge all source tokens with target tokens. However, there may exist some source tokens which does not have target tokens with similar semantics. Thus, at R = |S|, undesirable merging of tokens can happen, thereby decreasing the performances. On the other hand, at R = 0.5|S|, only the half of source tokens are merged with target tokens, and thus some source tokens may not be merged even though they have similar target tokens. Therefore, the proposed algorithm shows the best scores at R = 0.8|S|. Justification: See L65-L71 in Section 1.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: As the Area Chair overseeing the review of Submission4154, I recommend acceptance for NeurIPS 2024. The paper introduces an innovative learnable Video Token Merging (VTM) method that dynamically adjusts token merging based on saliency, showing significant improvements on long-video classification tasks on datasets such as LVU, Coin, and Breakfast.

Reviewer sUxH praises the method’s approach to computational efficiency and memory optimization, which are critical for processing long video inputs. The exploration of different token merging strategies provides a clear understanding of the proposed method’s advantages.

Reviewer rFDe’s concerns about evaluation and comparison baselines were addressed effectively in the authors' rebuttal. Their commitment to include additional experiments and adapt their approach in response to feedback highlights the paper's potential and thoroughness.

Reviewer 7bwU’s comments on novelty and design considerations for video processing were met with detailed clarifications, underscoring the method’s relevance and adaptability for complex video understanding tasks.

The paper’s theoretical and practical contributions offer valuable insights into optimizing transformer networks for video analysis, making a substantial impact on the field. The positive adjustments and responses to reviewer feedback justify its acceptance at the conference.

Reviews: review: paper_summary: paper_summary: This paper builds on Token Merging (ToMe) to improve its performance. In particular, the authors explore different ways to partition tokens so that the merging operation can lead to better performance while maintaining speed. They explore region-concentrated merging, motion-vector based merging and a learnable selector, and find that the learnable version works best. To make the network trainable, they employ a creative auxiliary path mechanism  to make everything differentiable. They find that their learnable VTM obtains good results compared to baselines on long form datasets (LVU, Coin, Breakfast), and that it outperforms the other methods they introduce.
strength_weakness: strength_weakness: I don’t really agree with the premise of the paper (and am open to a rebuttal to explain if I’m wrong here). Token Merging already explored merging for video in detail. The reason Token Merging is based on similarity is that by combining tokens that are extremely similar, the weighted average of the those tokens should produce an extremely similar result in the attention operation. This was also detailed more in the Token Merging follow-up TomeSD. If you use different criteria such as saliency (which is not really well-defined), this is no longer guaranteed, and from equation (10) it seems like the authors do not use the proportional attention scheme from ToMe (Eq 1 in the original paper). Table 8 doesn’t show the learnable method using this; it seems to just be about the pooling part rather than the attention operation.

I also don’t understand the intuition behind the saliency: shouldn’t we be aiming to combine together tokens that are NOT relevant, so that the transformer can focus more on the relevant tokens, rather than averaging (and thus losing) information from the more salient / important tokens? I’d really appreciate some clarification here. From Figure 3, it doesn’t look like learnable VTM is focusing on visually important tokens: it’s picking ones from the ceiling and wall in addition to the people.

My main issue is with the evaluation. The evaluation seems not quite fair, especially when measuring memory usage and throughput. Shouldn’t it be compared to baseline merging algorithms, like the naive ToMe? My impression is that the memory usage and throughput from VTM will be exactly the same as ToMe because it uses a similar partitioning scheme and constant factor reduction, which is why it may not be included in the results, but this seems important to include for context. Furthermore, the improvement on metrics is quite small, given that the speed is the same as other merging methods. Is this expected?

Also, The paper is motivated by “long-term” video, but evaluates on 64 frames, which isn’t really long and in my view, doesn’t merit only evaluating on LVU, Breakfast and COIN. Kinetics-400 has 300 frames per video, and is a more standard benchmark for evaluating video backbones - in fact, the original ToMe paper includes experiments on those datasets, which would make for a more fair comparison. Furthermore, nothing about the method itself is specific to these longer videos. I think evaluating on more standard datasets is crucial to measuring the actual strength of the method, especially compared to baselines like Token Merging. In particular, the long-form datasets are very compressible. 

The paper is not well-written and the grammar needs a lot of revision, making it hard to focus on the content of the paper itself. In addition, a lot of space is spent on methods that are not really used in the final results (center, region, motion vector) and on citing equations from preliminary works (token merging, attention). Given that a claimed contribution is an exploration of these different methods, I would also have expected more detailed ablations and experiments to understand exactly why some of the methods perform better than others.
questions: questions: It’s not really expected that VTM (or any merging method) should score better than full attention, as it has strictly less information. This is backed up in the Token Merging and other follow up papers. Why is it expected (as said on L160) that merging should perform better? It’s supposed to be just faster, with a minimal drop in performance.

The motion vector method requires extracting pre-computed motion vectors from the encoded video. However, those are computed for 30 FPS, and for the original video size, meaning they don’t actually apply to downsampled (64 frames, 224x224). Was this taken into account? It’s certainly not fast to re-compute these motion vectors if you’re doing random cropping or frame selection.

Is it possible to know the effect on training wall-clock time from this method? This the the metric that practitioners really care about, so including this would potentially strengthen the results of the paper.
limitations: limitations: The biggest limitation of this compared to baseline Token Merging is that it requires re-training. ToMe could be applied to a pre-trained network out-of-the-box. Learnable VTM cannot do this, making it impossible to apply to a pre-trained video network. The other proposed methods (region, motion vector) can do this though, and this would be a good thing to note somewhere in the paper.

In my view, the limitation of methods like VTM is that it always reduces a constant number of tokens per video, even though some videos are inherently more compressible than others. For example, Breakfast videos are extremely compressible compared to the average LVU video. However, this is beyond the scope of the paper, and using constant reduction is certainly more convenient, but this would be a good limitation to acknowledge and perhaps address in the future.
paper_summary: paper_summary: The paper approaches the task of long-video understanding from token reduction perspective. Specifically, Transformer-based approaches suffers from memory bottleneck and quadratic computation complexity with increasing number of tokens, which is even more pressing with long-videos as input. The paper builds on a recently developed token merging mechanism, and proposes a learned saliency measure to modulate what tokens gets merged instead of using a random or hand-crafted saliency measure. The central hypothesis of the work is that typically techniques that use similarity as merging criteria may inadvertently lose out on salient tokens. The paper reports experiments on three conventional long-video benchmarks (LVU, Breakfast and COIN), and shows effectiveness of their approach compared to prior related works both in terms of performance and memory requirement. The paper also ablates the effectiveness of their proposed saliency measure (learned VTM) over hand-crafted measures including motion-based (using optical flow), center-biased and random schemes.
strength_weakness: strength_weakness: - Comparison to related token saliency approaches
    - The paper proposes a scheme to identify salient tokens by using a learned projection matrix $U_s \in \mathcal{R}^{D \times 1}$ with $\texttt{tanh}$ activation function
    - However, learnable token saliency methods have also been used in prior works, such as EVEREST [1], which uses a pair-wise learned saliency at feature-level (equation 2) using $\texttt{Conv3d}$. The resulting approach was shown to be effective in the Masked Autoencoding setup
    - Having a motion-based merging scheme is a good baseline, but some variants of learnable token saliency could also be tried to gain better understanding how token saliency gets influenced by different approaches   

- Role of $L_1, L_2, L_3$
    - The paper proposes to take tokens from $L_i$ consecutive frames for the $i^{th}$ VTM block
    - It seems that choosing the values of $L_i$ is quite crucial given its impact on performance and memory requirement (Table 6) that forms the central claim of the paper
    - However, the paper highlighted the contribution of token saliency more compared to the choice of $L_i$ hyperparameters
    - Did the authors experiment with a rather simplistic setup using a single VTM block and/or with all $L_i$ being 60? It would help the readers to gain better understanding of what works in long-videos 


- How saliency changes with tokens from different number of frames?
    - It seems that the saliency is being computed at each VTM block. It would be interesting to see how the saliency changes across the three VTM blocks
    - On that note, what VTM block’s saliency is being visualized in Figure 5?



### Minor
- Line 145-146: “$i$-th transformer block takes the tokens corresponding to $L_i$ frames without overlapping”
    - Confusing when $i$ is referred to as the frame number and the block number of transformer at the same time
- Line 145-147: $j$ is not defined


### Typos
- Line 24-25: “so the tokenizing the”
- Line 36: “selectio”
- Line 114-115: “applications are mostly remained”
- Line 117: “depedencies”
- Line 161: “in the videos, .”
- Line 165: “regarding less of the”
- Line 177: “sailent”, “the the”
- Line 306: “sailencies”


### References
[1] “EVEREST: Efficient Masked Video Autoencoder by Removing Redundant Spatiotemporal Tokens”. Sunil Hwang and Jaehong Yoon and Youngwan Lee and Sung Ju Hwang. ICML 2024.
questions: questions: - Line 141: why L >= 60?
- Is saliency projection used in all auxiliary VTM blocks?
limitations: limitations: N/A
paper_summary: paper_summary: This paper explores various video token merging strategies in the context of long-form video classification and finally propose a learnable Video Token Merging algorithm that dynamically merges video tokens based on visual salient areas. The contributions are summarized as follow:
1.  Explore various video token merging methods including the naïve VTM, the region concentrated VTM, and the motion-based VTM.
2.  Propose the learnable video token merging algorithm, which estimates the saliency scores of each token and adaptively merge visual tokens based on those scores.
3. The proposed algorithm achieves the best or competitive results on various datasets.
strength_weakness: strength_weakness: 1. This paper proposes a leanable video token merging strategy. The similiar high-level idea can be found by CTS[1] in image domain. The novelty is insufficient。
2. This paper focuses on video token merging. However, I do not observe any specific design tailored for the video domain in terms of the methodology. let alone long video.



[1] Content-aware Token Sharing for Efficient Semantic Segmentation with Vision Transformers
questions: questions: 1. Due to the two-paths design, has the training time doubled?
2. The paper tries to learn saliency scores using matrix $U_s$. How about using $\sum{QK^T}$ in Equation 8 as saliency scores for each visual token?
limitations: limitations: Yes
paper_summary: paper_summary: - This paper carries out an analysis of token merging[1] in the context of long-form video understanding and proposes learnable video token merging (VTM) to select semantics/saliency guided tokens for merging. 
- In token merging, at each layer, the tokens are divided into two sets source S and target T through uniform sampling. Tokens in S are matched to T based on similarity and merged (usually by average pooling). This paper compares this naive VTM with two other variants where selection of T is guided by informed heuristics: (1) region VTM where tokens at the center of each frame are more likely to be retained, (2) motion VTM where tokens with high motion are more likely to be retained. Through this analysis, authors argue that the strategy to select T plays an important role in the final performance.
- Motivated by this, authors propose a learnable VTM where it first predicts a saliency score for each input token. The target set T is sampled according to the probability distribution defined by saliency score. Since this partition operation is not differentiable, authors propose a novel training architecture where a parallel auxiliary network is trained alongside. The saliency scores are used to bias the attention score of the aux network, thereby supervising the saliency prediction to focus on important tokens. Aux network can be discarded at test time.
- Authors carry out a fair evaluation of the learnable VTM on LVU, Breakfast and COIN datasets by comparing against several baselines including ViS4mer, S5, D-sprv. Learnable VTM performs better than baselines in almost all evaluation tasks with low GPU memory usage and high throughput.

[1] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token Merging: Your ViT but faster. In ICLR, 2022.
strength_weakness: strength_weakness: ### Major weaknesses
- One of the cited contribution is the exploration of region based and motion based VTM (Section 3.3) but it seems trivial. The effectiveness of token selection is already shown in learnable VTM. In light of that, there is an unreasonable focus section 3.3 which is unnecessary.
- Section 3.4 explains little about the details of learnable VTM, how it is trained, how the gradients flow in the presence of non-differentiable partition function, etc.
- There are some stretched claims based on qualitative and quantitative results. For example,
  - In Line 174, authors claim that center VTM performs better than naive VTM. However, according to Table 1, the results are mixed at best.
  - In Fig 5, authors also claim that the visualization of merged tokens show saliency based merging. However, the figure doesn't support the claim. There are many merged tokens on important salient features and some background tokens are not merged.

### Minor issues
- Line 19: it should be "into the domain of video computer vision" as all cited papers are video learning papers.
- Is there a difference between notation of C and D? It looks like both are used interchangably to denote token dimension.
- Table 2: How it throughput measured? fps?
questions: questions: - How is learnable VTM trained? From Fig 4, it looks like there are two outputs from the network. Do you apply the same loss on both outputs?
- In Fig 4, what does the 'Merge' operation in auxiliary path do? Does it mean that the main and aux - both paths use the same target set sampled by the partition-match of the main path?
limitations: limitations: NA

Comments: title: Post-rebuttal comment
Response by Authors
Raising Rating + Followup response
comment: Thank you for your additional feedback. Please find our follow-up response below.
***

> **Comparison with token reduction methods**

Token merging and token sampling/dropping are two alternatives to reduce the valid number of visual tokens in the transformers, which reduce the computational cost of the self-attention layer. However, the scope of adaptive token sampling and its similar works are different from our work. They focus on improving the efficiency of the transformer while maintaining similar performance. In this paper, we argue that these methods may not be favorable in the long-form video understanding. Images as well as short-form videos are easy to build short-term dependencies within one image or highly overlapped video frames from a few seconds long, e.g. the basketball and basketball court, the sky and cloud. Thus, a parameter-free or a light weight module can produce reasonably good results for merging and sampling. On the other hand, in long-form videos, it is more challenging to capture dependencies between sparsely sampled frames, e.g. leveraging dependencies to predict the genre and production year of a movie. 
As this work specifically focuses on the token merging method, we only include previous merging ideas in the comparison. We believe it is a great idea to expand the scope of this work in the future including adaptive token sampling and Perceiver as two representatives in the concept of general token reduction analysis. 

> **Figure 3**

Also, as stated in L206-208, we sample the target tokens based on the probability which is computed from the estimated saliency scores. Hence, it is more likely that the tokens with high saliency scores are selected as target tokens, but not always. Therefore, in Figure 3, some tokens corresponding to backgrounds are selected as target tokens. However, we can see that target tokens are selected more around salient objects (two main characters). Moreover, it is desirable that some tokens from backgrounds are selected as target tokens so that source tokens corresponding to background can be merged with those similar target tokens. Otherwise, source tokens from background areas would be merged with unsimilar target tokens corresponding to salient objects, which causes significant loss of important information. 

> **Paper organization**

Thanks for your advice. As suggested, we will move the explanation on different token merging methods to Appendix and use the saved space to address the comments from all reviewers.

***
Thank you again for your time and effort for reviewing our paper. We do appreciate it. If you have additional comments, please let us know.
Thank you for your constructive review and insightful suggestions, all of which will be addressed faithfully in the final paper. Please find our responses below.
***

> **Design for video domain**

We kindly request Reviewer 7bwU to clarify the definition of ‘special design for video’. In recent transformer-based networks and many foundation models, various inputs, including images, short-form videos, long-form videos, and text, are all processed in the form of tokens, but they just have different modalities and properties. If the absence of special design for video means that our method can be ‘generally’ applied to different modality tokens without raising errors, it is true. However, as stated in Section 1 and our previous response, the proposed algorithm is designed to handle long-contextual information in the video more efficiently and effectively. Compared to relatively well-structured text inputs and images/short-videos with relatively small amounts of information, it is more challenging to capture short-term and long-term dependencies across tokens in the long video since the long video contains many tokens which convey complex semantics individually or collectively. Therefore, we design our algorithm by merging tokens adaptively based on saliency scores for capturing the token dependency more easily while minimizing the information loss. Also, please note that the proposed algorithm achieves better results than all conventional algorithms for long-video understanding on various datasets. We believe that it indicates that the proposed algorithm has efficient design for long-video understanding. 


> **Training Time**

We compare the throughput (FPS) in Table below. The throughput of learnable VTM with the auxiliary path (during training) is almost half, compared to that of naive VTM or that of learnable VTM without the auxiliary path (during inference). This is also shown in Table 9 in Appendix B.2. However, when measuring training time, there are various different factors such as data preprocessing, data loading, logging, and back-propagation. These factors take much longer than the forward path, and thus the total training time of learnable VTM is not doubled. 

|            | Naive VTM | Learnable VTM (training) | Learnable VTM (inference) |
|------------|-----------|--------------------------|---------------------------|
| Throughput | 45.39     | 27.84                    | 44.94                     |



***
If you have any additional concerns, please let us know. Thank you again for your comments.
Thanks for a detailed rebuttal.

### Major comments
- It looks like learnable VTM is very similar to other token sampling method ('Adaptive Token Sampling For Efficient Vision Transformers'). However, such prior token sampling method is parameter free and can be used on any pretrained transformers. Learnable version of such sampling is perceiver module ('Perceiver IO: A General Architecture for Structured Inputs & Outputs'). Authors haven't compared with either of them. Can the authors provide more comments on this?
- The visualization of saliency in Fig. 3 doesn't support the claim that the salient patches are being selected. Rather, it looks like that the network is repurposing the unimportant tokens to store information (this has been explored in 'Vision transformers need registers'). This also reaffirms by belief that Learnable VTM may be just a resampler and hence doesn't actually learn vision saliency.

### Minor comments
- I agree that the exploration is on different token merging techniques is valueable in the supplementary material. However, it would be better if its length in main paper is reduced.
1. Design for video domain

The author, although exploring motion-based VTM, ultimately adopts a learnable VTM. In the learnable VTM, I do not see any special design for the video.

2. Training time

Could you explain why maintaining two paths during training does not result in a relationship of twice as much?
There was a minor error in above Table for training time. The correct table is as below. We do apologize for the confusion.

| Naive VTM | Learnable VTM |
|-----------|---------------|
| 3.7h      | 4.0h          |

***
Thank you again for your time and effort for reviewing our paper. We do appreciate it.
Thank you for your positive response. Please find our follow-up response below.
***

> **Saliency**

We agree with the reviewer. We will revise the draft as suggested.

> **Training time**

In Table below, we list the training time on the LVU dataset. The training time of the learnable VTM is much faster than that of the baseline network without token merging. Also, compared to the naive VTM, which is the straightforward application of ToMe to the baseline network, the learnable VTM takes a slightly longer time. However, it is not a huge gap, which significantly diminishes the benefits of the proposed algorithm. 

| No Merging | Naive VTM | Learnable VTM |
|------------|-----------|---------------|
| 7.2h       | 3.7h      | 4.0h          |

> **Long video vs short video**

Long-form video understanding tasks not only require to model more frames but also need to capture the complex spatiotemporal short-term and long-term dependencies of semantics. To facilitate this, the learnable VTM finds out important tokens among numerous in the video through the learned token partition and increases the influence of these tokens in the attention processes after merging. By doing so, it obtains better understanding than the ‘ToMe’ on the long-form video understanding tasks, as shown in the experimental results. 

As stated in L51-52, in this work, we aim to explore various video token merging methods on the long-video understanding datasets (LVU, Breakfast, and COIN) and to find out the effective video token merging technique for long-video understanding. Thus, we don’t use all types of video benchmarks for the evaluation. However, we explore various token merging methods (including ToMe) on the long-form video understanding benchmarks and also compete with recent SOTA methods for long video understanding. 

As the reviewer pointed out, the proposed algorithm can be evaluated on the short-video understanding benchmarks (such as Kinetics-400), even though it is out of the original scope of this paper. Please note that Kinetics-400, as pointed out in the “What Makes a Video a Video: Analyzing Temporal Information in Video Understanding Models and Datasets, CVPR 2018”, is a relatively simple dataset with low information density; it contains one simple action for each video and thus complex information across many frames are not required to obtain good classification results on this dataset. Therefore, the proposed algorithm may not yield meaningful performance gap over other token merging baselines on the Kinetics-400 dataset, because this dataset may be less sensitive to information loss caused by suboptimal token merging than the long video understanding datasets which contain enormous amount of complex information. Due to the time limitation, we may not be able to finish the experiment within the rebuttal period. However, we will include performance of short-video understanding benchmarks as the supplementary of this paper. 

***
Thank you again for your time and effort for reviewing our paper. We do appreciate it. If you have additional comments, please let us know.
Thanks for the detailed rebuttal, I really appreciate your efforts. I will raise my rating for now to a 4. I have some further doubts that I'd appreciate clarification on. 

__Saliency__

I think I understand this better now but I really don't think saliency is the right word. From what I understand, the point of this is to learn some scoring function to do a better job of partitioning, since the hypothesis of the paper is that the partitioning scheme of ToMe is suboptimal. Maybe "learned partition" coudl be a better word? Saliency is too broad / vague of a term.

__Training Time__

When I asked for the training time, I meant compared to standard ToMe (no learned module) and a baseline with no merging. From the results of the original ToMe paper they demonstrated very large speedups on training time. It seems that learnable VTM would remove those gains, which would be a big reason to not use this method. Can you comment on this?

__Long Video vs Short Video__

I still don't really see why this method is tailored for long video at all, and Reviewer 7bwU agrees. Nothing about learnable VTM specifically is more effective than other methods with more frames . Furthermore, as I mentioned, you can train a model on Kinetics-400 with 64 frames as well and we should be able to notice some sort of improvement. What exactly about this method is specialized for long videos and thus merits not evaluating on Kinetics (where there are more baseline results)?