Don't Look Twice: Faster Video Transformers with Run-Length Tokenization
Authors: Rohan Choudhury
Guanglei Zhu
Sihan Liu
Koichiro Niinuma
Kris M. Kitani
Laszlo Attila Jeni
Venue: NeurIPS 2024 spotlight (2024)
Keywords: video understanding
vision transformers
efficient transformers
Primary area: machine_vision
TL;DR: We make transformers 40% faster on video, with no performance drop, by identifying consecutive runs of tokens repeated in time, and treating them as a single token with variable length.

Abstract: Video transformers are slow to train due to extremely large numbers of input tokens, even though many video tokens are repeated over time. Existing methods to remove uninformative tokens either have significant overhead, negating any speedup, or require tuning for different datasets and examples. We present Run-Length Tokenization (RLT), a simple approach to speed up video transformers inspired by run-length encoding for data compression. RLT efficiently finds and removes `runs' of patches that are repeated over time before model inference, then replaces them with a single patch and a positional encoding to represent the resulting token's new length. 
Our method is content-aware, requiring no tuning for different datasets, and fast, incurring negligible overhead. 
RLT yields a large speedup in training, reducing the wall-clock time to fine-tune a video transformer by 30% while matching baseline model performance. RLT also works without training, increasing model throughput by 35% with only 0.1% drop in accuracy.
RLT speeds up training at 30 FPS by more than 100%, and on longer video datasets, can reduce the token count by up to 80\%. Our project page is at  rccchoudhury.github.io/projects/rlt.

Don't Look Twice: Faster Video Transformers with Run-Length Tokenization
Rohan Choudhury'
Guanglei Zhu
Sihan Liu
Koichiro Niinuma"
Kris M. Kitani
László A. Jeni
Carnegie Mellon University
Fujitsu Research
rchoudhu@andrew.. cmu edu
Abstract
Transformers slow to train videos due extremely large numbers of input tokens, even though many video tokens are repeated over time. Existing methods remove such uninformative tokens either have significant overhead, negating any speedup, or require tuning for different datasets and examples. We present Run-Length Tokenization (RLT), simple approach speed up video transformers inspired by run-length encoding for data compression. RLT efficiently finds and removes 'runs' of patches that repeated over time prior model inference, then replaces them with a single patch and a positional encoding to represent the resulting token's new length. Our method is content-aware requiring no tuning for different datasets, and fast, incurring negligible overhead. RLT yields a large speedup training, reducing the wall-clock time fine-tune video transformer by 30% while matching baseline model performance. RLT also works without any training, increasing model throughput by 35% with only 0.1 11 drop in accuracy. RLT speeds up training at 30 FPS by more than 100%, and on longer video datasets, can reduce the token count by up to 80%. Our project page at https: http:///ccccccur...cccc/o gi github. io/ st/projects/rtt//
Introduction
Vision transformers [11] have enjoyed enormous success in modeling images and videos due to their scaling properties and minimal inductive bias. Unfortunately, training these models videos, which generally have orders magnitude more tokens than images, is significantly more expensive. One contributing factor is that video transformers tokenize videos by splitting them into uniformly sized spatiotemporal patches [2, 3]. then embed them into a latent token space. As a result, the number of tokens depends only on the video's length and resolution. Researchers are thus forced to work with very short videos (<10s), as well as significantly downsample them to low semnes-per-scood (FPS) and low spatial resolution.
One promising solution to this problem is to reduce the number of input tokens. Compared to language input, videos significantly less dense in information: many works observe that videos consist mostly of redundant or uninformative tokens [15, 39, 43]. However, existing methods that aim to reduce input tokens vision transformers have had limited adoption. Learned pruning methods [34, 52] reduce model complexity measured by GFLOPS, but either incur significant overhead during training, require padding handle changing numbers of tokens, negating any speed-up during training. Random masking [1, 27], though fast, decreases accuracy and thus requires more training time to match performance. Moreover, although methods like random masking and Token Merging [5] do lead to wall-clock speedups, they are not content-aware: they only remove fixed number of
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Input
Frames
Standard Tokenization
Random Masking
Run- Run-Length Tokenization (Ours)
Figure I: Toy Example. Given set of input frames, with each square representing a patch, standard tokenization always produces the same number of tokens. RLT compares temporally consecutive patches and removes redundant ones, storing a single token and the run-length instead.
tokens per video, and will reduce the same number of tokens from a high-speed, high-action clip as from still image repeated over time.
We argue that content-awarenet can help more effectively reduce the number input tokens. As an example, imagine an hour-long video of lecture. Most of the frames exactly the same over time, displaying single slide. Existing methods would produce the same number of tokens from this as from an hour of motion-hear GoPro footage, even though the two videos have significantly different amounts of content. On the other hand, video compressors, such as H.264 and H.265 [46, 41], are explicitly content-aware. rather than encoding frames independently, they encode pixel differences between consecutive frames, drastically reducing video size when there no change.
We propose Run-Length Tokenization (RLT), which combines simpler version of this idea with classical run-length encoding to tokenize videos for transformers. Our insight is that we can efficiently identify 'runs" of input patches that are repeated over time, enabling to reduce the number of tokens based on video content. When tokenizing the video, we compare consecutive patches in time and group together patches with sufficiently small differences. We then remove the "repeated" patches, and treat the remaining tokens as having variable length. Similar to how the string aaaabb can be run-length encoded as a4b2, additional overhead while retaining some the information lost from removing the redundant tokens. Despite its simplicity, RLT works remarkably well with it, 40% faster wall-clock time than baseline ViT's while matching performance.
add length information each of the tokens, which incurs
fine-tune video transformer in
Our contributions are as follows: we (1) propose RLT, an alternative method to tokenize videos vision transformers, (2) thoroughly compare its performance and compare RLT's speed to prior methods, finding significant improvements, (3) evaluate RLT's performance high-FPS and longer videos, and (4) ablate design choices and qualitatively visualize RLT's output. We believe RLT can be a key step to significantly accelerate and further scale video understanding.
Related Work
Video Transformers. Vision Transformers [11] have been successfully adapted video [2, 13, 26, 38] but are generally trained and evaluated on short (<10s) video clips with relatively few frames. efficiently handle videos, many works incorporate video-specific inductive biases their architectures [29, 22, 56], such as memory [49, 35], compression cues [48], or modified attention mechanisms [30, 53). Other methods, especially in video generation, project the video to smaller latent space [20] and then split it into patches. We instead use the standard VIT formulation but apply a different tokenization scheme, reducing the number of input tokens improve speed while maintaining performance.

---

Split input patches
Compute
3 2
1
4. Add length embedding takens and pass into
Prune polches with low difference compuic temporal run-kngth
Figure 2: RLT Overview. RLT works by comparing temporally consecutive patches, and retaining those with L1 difference above threshold т. The remaining tokens are augmented with length encoding to signify their 'run-length' and passed to the transformer.
Video Tokenization. Prior vision transformers, video architectures were designed to take in fixed size input [14, 40, 55J. However, transformers can handle arbitrary numbers of input tokens [44], and training on variable-sized inputs standard in language modeling [23]; this has been used train vision transformers with variable resolutions [4, 10]. However, video transformers still generally use the spatiotemporal patch tokenization scheme introduced in [2, 3] which content-agnostic the number of tokens depends only on the video's 's length and resolution. Some works attempt to reduce input size by compressing the video to a latent space, then tokenizing [12, 33, 6], but the number of tokens still depends strictly on the input video dimensions. On the other hand, standard video compressors like HEVC [41] and AVC [46] are content-aware: they actively consider the differences between consecutive frames for more efficient compression. Our work applies this idea to video transformers by condensing static tokens and tracking their length.
Faster ViTs with Fewer Tokens. Several works have attempted to remove uninformative tokens from vision transformers. One line of work identifies such tokens either through learned modules or attention scores [34, 52, 28, 21], and prunes them at each layer. Although transformers can handle variable sized inputs, these methods require padding as token counts change unpredictably with each layer. Other works combine tokens instead of pruning them ([5, 31, 51]). Most of these works require training model for pruning or merging. with the exception of Token Merging [5], which demonstrates strong results at inference time. Inspired by the success of masked pre-training ([17, 45, 43, 15]), another line of work uses random masking to speed up training. Although masking leads to worse performance after the same number of batches, the dramatic speedup enables training more epochs in less time [1, 10, 27, 50]. In contrast, our method matches the performance of base models with the same amount of data with large speedups. and be stacked with random masking for even more speed benefits. Closely related to our method are EVEREST [18] and STA [36] which both exploit temporal similarity to identify redundant tokens. However, like [5] both these works require setting constant number of tokens to remove from each video, while RLT can remove varying numbers of tokens based the video content.
3 Method
Consider a vision transformer that takes as input a video V E RCXTxxxW The standard tok- enization scheme splits V into set P of uniformly sized, non-overlapping patches, each with size Cx x Dy x D., with P. called the tubelet size. These patches are projected to a lower dimension dembed with an MLP &, resulting in Np tokens, with each corresponding to distinct spatiotemporal location. This results the same number of tokens for any input video that has the same size.
In contrast, our goal is to identify input patches that are extremely similar, then compress these redundant patches, increasing throughput and training time. Our approach is illustrated in Figure In particular, we focus on temporally consecutive patches, those which have the same z, y location

---

and differ by one timestep. These "static patches" correspond to visual content that does not change or move over time, and such tokens can be easily compressed.
3.1 Removing Static Patches
Token Similarity. Unlike prior works, we aim to reduce the number of total input tokens by comparing patches rather than tokens. By operating on patches, we do need to run the patch embedding & or any layer of the model. As a result, we do not need to freeze parts of the model or propagate gradients through the pruning operation, which would require padding and negate potential speedups. This contrasts with prior works which progressively prune or combine tokens after each layer in the transformer. Furthermore, by identifying redundant patches, we can pre-compute the token distributions of various datasets and sizes of examples, allowing us to employ techniques like seample-packing [23]. Finally, operating on visual patches is more interpretable and is similar the heuristics used by video encoders [41, 46].
We next define a criterion for determining whether two consecutive patches are static. Consider Two temporally consecutive patches Pi, P2 that correspond to spatial location (æ, y) and temporal locations (,, l2 with l2 11 Dr., For tubelet sizes with value P: > 1, each patch consists of
multiple frame crops, so that P1 = [P zy- pl,+1 and P2 static if
.... P..pP De D. Given a threshold т, we consider P1
|
ply 11 <T
(1)
with Pay p being the temporally last spatial crop of in Pi2 and Pay the first spatial crop of P This operation compares the "start" of the P1 to the "end" of Ps. with the idea being that if the first crop of token P1 matches the last crop of token P2. the patches in between likely match as well. Notably, is a hyperparameter that needs to be tuned, but is dataset-agnostic it simply encodes how much change between patches is allowed before they are considered different. 5 controls the trade-off between speed and accuracy; while higher values reduce significantly more tokens, they treat tokens that are perceptibly different as being the same, redueing accuracy. We use r > 0 since imperceptible artifacts can occur, and follow standard procedure by running ImageNet normalization before comparing patches. We typically use r = 0.1, and provide experiments and visualizations on its effect in Section 4.3 and Appendix B.
Pruning Procedure. To identify all static tokens, we run the prior comparison on all pairs of tem- porally consecutive patches in P obtaining their differences and only retaining those with difference less than т. We always include the entirety of the first frame since there is no previous patch to compare it to. This results in a binary mask Matatie, which can then apply with
= P Metatic
(2)
with Pl containing Np tokens and P consisting of Np tokens. Note that Np. S Np is always true; with RLT. we can never have more tokens than in the standard tokenization procedure, so the worst-case performance matches the standard vision transformer. RLT also incurs essentially no overhead as the entire process can be implemented entirely with parallelizable PyTorch [32] operations on the GPU, so training and inference are strictly faster.
The simplicity of RLT is a major advantage: in contrast to other methods, we can take advantage of transformers' ability to handle variable input sizes, and do not need to provide any additional padding. Because we make no changes to the model itself, a video transformer using RLT can make use of hardware optimizations like Flash Attention [8. 9] and memory efficient kernels [25].
Notably, the pruning procedure is content-aware some videos with large amounts of static content will result in significantly fewer input tokens than videos with significant amounts of camera or subject motion. This is a desired outcome, and we discuss how to handle training with dynamic input sizes in Section 3.3.
3.2 Run-length Positional Encoding
Although we have reduced the number of input patches, we know that each patch represents a 'run" of static patches, with length 1 corresponding to static content, and length T corresponding to input time dimension length. Without information about the length of the 'run' of static patches, the transformer may not be able to compensate for information removed during the pruning procedure.

---

To address this, Bolya et al. [5] introduced Proportional Attention, which weights each token by the number of tokens in each group. On the other hand, we opt let the model learn this information: we treat each token as having variable length that we can communicate through new positional encoding. Specifically, we use factorized encoding, described Dehghani al. [10], with one encoding Payt containing positional information and the other Φr corresponding to the length. We use a learnable length bias Φ consisting of a single parameter size (T', dembed). ). For a given 'run' of repeated patches, we always retain the initial patch Payt, and thus can compute the new length L as the distance from zyt to the nearest entry Matatic along the t-axis. Concretely, for Pa Pays
4 min(t' t), where Mstatic y, t') 1, >
This operation can also be efficiently implemented on the GPU, adding no overhead. Then, the full positional encoding becomes
(4) with the [6,] representing the indexing operator. We add the positional encoding (TT)) to each token after running the patch embedding network &. Unlike the pruning procedure, since we use learnable length encoding $L. we propagate gradients to the positional embedding, enabling the model to learn how to optimally encode variable length tokens during fine-tuning.
(T.)) &ayt (T:) + (z[[.]
3.3 Handling Dynamic Input Sizes
Since RLT is content-aware, the number of tokens varies significantly per example. Although transformers can natively handle any input size [44], prior methods like DynamicViT [34] or A- ViT[52] produce different numbers of tokens at each layer; this requires padding or attention masking to handle batched inference during training. In our case, only the input token count variable, but the number of tokens stays constant throughout the network, closer to the setting of NaViT [10]. Furthermore, since we know the input size before running the network, we can employ example packing [23], an idea from language modeling where multiple inputs with variable sizes are packed together, and tokens from individual examples attend only to each other.
At training time, the input to the transformer consists of a batch of tokenized videos, V1, V2,,.. 12.....B. VB. each with size Ti, B.. Rather than pass input (B, max; T;, dembed) to the network, we concatenate the video tensors to produce V' = Vi @ 1/2 @ VB...BB. resulting in input size (1, > Tú, dembeed.. We then construct a block-diagonal attention mask so that tokens only attend to other tokens from the same video, which we add during the attention operation. Since every token in V' is attending only to tokens from the same example, this does not reduce throughput and is also compatible with existing hardware-effiiient attention implementations. To compute the class prediction in action recognition, we split each example out and compute its prediction the mean of each example token, as in [43]. We then project it to dimension Nc, resulting in output of size (B, .c) to which we can apply standard cross-entropy losses during training.
We note that typically example packing results in a constant number of input tokens, with a variable number of input examples. A key difference between RIT and Dehghani et al. [10] is that data augmentations such as RandAugment [7] can alter the visual content and thus number of tokens of input videos, rendering greedy example packing strategies inapplicable during data loading. We opt to use a constant number of examples per GPU, with high enough batch size sufficiently reducing variance in input size.
4 Experimental Results
To analyze RLT's impact on performance and speed, conduct several experiments on standard action recognition tasks. We measure the speedup on model training at several scales in Section 4.1 as well as RLT's effect as a drop-in addition at inference time in Section 4.2. We perform ablations in Section 4.3, then evaluate RLT's effect on higher FPS videos and long video datasets in Section 4.4. Finally, we provide qualitative visualizations in Section 4.5.
4.1 Training
In Table we evaluate RLT's impact on the performance of video transformers during training and its resulting speedup. We fine-tune ViT-B and ViT-L from pre-trained VideoMAE [43, 45] checkpoints,
5

---

Kinetics-400
Model
Acc
FT time(8 GPU)
Speedup
Acc
FT time(8 GPU)
Speedup
ViT-B
80.1
14.4h
1.0x
70.3
10.1h
1.0x
ToMeras
80.0
13.4h
1.1x
69.7
9.4h
1.1x
Random (0.7)
79.2
10.2h
1,4x
69.3
7.2h
1,4x
RLT (Ours)
80.1
10.2h
1.4x
70.2
7.2h
1,4x
ViT-L
84.8
21.6h
1.0x
74.3
15.2h
1.0x
ToMe
84.4
18.3h
1.2x
74.3
12.9h
1.2x
Random
83.1
15.4h
1.4x
74.3
10.8h
1.4x
RLT (Ours)
84.7
15.4h
1.4x
74.4
10.8h
1.4x
Table I: Training results action recognition. RLT significantly reduces fine-tuning time with Something-v2..
comparable performance to the baseline on both Kinetics-400 and
comparing the speed and performance with standard tokenization, random masking, and RLT. We evaluate random masking by removing & tokens, with being the mean number tokens pruned by RLT on a given dataset. For the most fair speed comparison, all evaluated models trained with mixed-prectte -peeiisp,,,,,nn, memory-efficient attention and Flash Attention where possible using an 8xH100 node, as well the optimized data loader from AVION [54] to avoid data loading bottlenecks. We use the standard Vision Transformer rather than more complex architectures such as TimesFormer [3] or MViT [26]; we found that was significantly simpler and more efficient, matching observations from Ryali et al. [38]. We limit our analysis to tinettunigg due to computational constraints. We compare against the baseline vision transformer, as well as Token Merging and STA [36]. We also include a random masking baseline where the masking fraction is set to the average number of tokens removed by RLT. which a stronger baseline than using a fixed standard fraction such as 0.5.
Compared to standard tokenization, RLT achieves a speed-up of up to 40%, even with heavily optimized implementations. RLT achieves the best trade-off between performance and speed, with better performance than random masking while achieving the same speedup. This demonstrates that the choice of which tokens to remove makes nontrivial difference, and that properly identifying redundant tokens is important.
Compared to standard tokenization, RLT achieves speed-up of up 40%, even with heavily opti- mized implementations. RLT achieves the best trade-off between performance and speed, with better performance than random masking while achieving the same speedup. In particular, RLT is much faster to train than Token Merging since it compatible with hardware-optimizz such Flash-Attentio [8, 9]. Unlike random masking, RIT matches the performance of the baseline ViT after the same number of training batches, while random masking requires significantly more epochs catch up. RLT matches baseline performance across multiple scales, indicating that RLT does not degrade performance while considerably accelerating training.
implementations
4.2 Inference-Tinn Results
Although RLT was designed to speed up training, it can be used as drop-in replacement for standard tokenization, similar to Token Merging[5]. Table we compare the top-1 accuracy, GFLOPs and throughput with RLT to standard tokenization and Token Merging [5]. We also compare against random masking for completeness, although intended only for training time
[27]. For the most
fair comparison, we randomly mask out P tokens for each example, where P is the mean number of tokens used by RLT; for Kinetics-400 and SSv2 this was P 0.72, We do not compare to Icarned pruning methods like V-V [52] since those only present results on images. We measure throughput in clips-per-secone
with each model running on single clip at a time. In practice, video models are evaluated on multiple temporal and spatial crops; following VideoMAE[43] we measure GFLOPs on single clip and measure accuracy with 4 temporal and 3 spatial crops.
Across model sizes, RLT consistently delivers the best tradeoff between speed and accuracy. The benefit becomes more pronounced as model size increases, as at larger parameter counts, the attention operation begins to dominate the computation. Compared to baselines, RLT is significantly faster than Token Merging and outperforms all other baselines on accuracy. Token Merging cannot make use of Flash Attention and other optimizations due to its reliance weighted attention operation, slowing

---

Kinetics-400
Somnehing-Somethingg
Model
Ace
GFLOPS
Clips/s
Speedup
Acc
GFLOPSS
Clips/s
Speedup
ViT-B
80.5
180
31.4
1.0x
70.8
180
31,4
1.0x
ToMera
80.4
131
34.4
1.09
69.1
131
34.4
1.09
STAres
80.4
131
34.4
1.09
69.1
131
34.4
1.09
Random
80.1
120
53.0
1.68x
69.3
120
53.0
1.68x
RLT (Ours)
80.6
120
52.6
1.67 x
69.8
120
52.6
1.67x
VIT-L
84.8
598
11.5
1.0x
74.3
598
11.5
1.0x
STAr
80.4
308
34.4
1.09
69.1
308
34.4
1.09
ToMers
84.3
285
19.3
1.68
73.6
285
19.3
1.68x
Random
84.1
405
18.8
1.63x
73.3
405
18.8
1.63
RLT (Ours)
84.6
405
18.71
1.62x
74.1
405
18.71
1.62x
VIT-H
86.8
1192
6.65
1.0x
ToMeraa
86.1
766
8.51
1.27x
STA 80.4
611
34.4
1.09
Random 85.1
816
9.66
1.45x
RLT (Ours)
86.3
816
9.66
1.45x
Table 2: Inference-only results action recognition. With batch size 1, RLT with 0.1 consistently achieves the closest performance to the baseline, comparable or faster than Token Merging random masking. We omit ViT-H results existing pre-trained checkpoints.
v2 due to lack of
37
Model
Acc
FT Time
WFBB T-C
ViT-B
80.1
14.4h
35
T-0.05
RLT (no length)
80.1
10.2h
Top1 101-Accuracy
84-
T-0.3
T=c.35
RLT
80.
10.2h
T=622
RLT (no length, w/random)
79.3
8.1h
RLT (w/random)
79.8
8.1h
92-
ViT-L
84.8
21.6h
RLT (no length)
84.6
15,4h
81
RLT
84.6
15,4h
1.0
3.2
1.4
166
1.8
RLT (no length, w/random)
84.2
11.3h
Relative Speed-up baseline)
RLT (w/random)
83.3
11.3h
Figure 3: Varying Difference Thresh- old. When comparing the tradeoff between speedup factor and accuracy, RLT is close to baseline performance for low values of T, with sharp drop-off after = 0.1.
Table 3: Effect of length encoding. When fine- tuning with RLT only, length encoding has mini- mal effect, but helps significantly when combined with random masking.
down in comparison to RLT. Although worse than RLT, random masking performs surprisingly well, likely due the fact that most tokens in videos redundant. Random masking can also be combined with RLT for further speed benefits, with smaller resulting performance gaps than in [27]. However, achieving the optimal performance-throughhee tniing for each dataset, while RLT is natively content-aware, achieving higher accuracy at similar speeds without tuning. Similarly, Token Merging [5] requires changing the parameter based on the model size and is not content aware, limiting its speed-up in highly static videos.
tradeoff with random masking requires
4.3 Ablations
We ablate design choices for RLT in Figure 3 and Table 3, measuring the impact of the difference threshold and length encoding design choices at multiple model scales during training.
7

---

Model
FPS
Acc
FT Time
Dataset
FPS
#Tokens
RLT
VIT-L
7.5
84.8
21.6h
K400
7.5
3.8 x10
2.7 x 10 (-29%)
RLT
7.5
84.6
15.4h
1.41x
K400
15
7.5 x10
4.8 x 10 (-36%)
VIT-L
15
85.8
45.2h
K400
30
1.5 x10
8.2 x 10 (-45%)
RLT
15
85.8
27.4h
1.72x
VIT-L
30
86.3
110h
SSv2
7.5
2.6 x10
1.8 x 10 (-31%)
RLT
30
86.2
52.3h
2.1x
SSv2
15
5.2 x x00*
3.2 x 10t (-38%)
SSv2
30
1.0 x10
5.7 x 10t (-48%)
VIT-L
7.5
74.3
15.1h
RLT
7.5
74.4
10.8h
1.39x x
EK-100
3.5
1.1 x x10*
7.2 x x10 (-36%)
VIT-I.
15
75.4
41.4h
COIN
30
9.8 x 10
2.8 x 109 (-71%)
RLT
15
75.3
24.1h
1.7x
Breakfast
15
1.3
109
2.7 x 10 (-79%)
VIT-L
30
76.1
99.8h
RLT
30
76.1
47.5h
2.0x
Table 4: Per-Dataset Token Reduction. RLT reduces tokens significantly across datasets, with higher reductions on higher FPS. On long-video datasets like COIN and Breakfast with mostly static content, RLT achieves almost 80% reduction, demonstrating its promise for scaling training.
Table 5: Training higher FPS. RLT en- ables training efficiently for higher FPS, al- lowing us to go beyond the standard low FPS paradigm. As FPS increases, RLT delivers larger and larger speed-ups over the baseline for training, with no decrease in accuracy.
Difference Threshold. The only tunable hyperparameter in RLT is the threshold which controls the sensitivity to change between temporally consecutive tokens. Lower values indicate higher sensitivty to change. We vary tau and compare the final action recognition accuracy throughput and wall-clock time for several configurations, both for training and inference. These results are shown in Figure 3. We find that using T 0.1 offered the best tradeoff in speed and performance: it matches the baseline performance while delivering a 37% speedup in training. Lower values of lead similar performance, but with less of speedup. while high values deliver larger speedups at a cost to performance. We attribute this to the existence of 'difference cut-off':: at some point, the tokens are too different to be grouped together, and the resulting tokens do not obey the assumptions made by RLT. We also note that T is dataset-agnostic: it simply describes how much pixel difference is needed consider two 16x16 patches different, and the same value of leads different reductions across datasets based on the video content.
Length Encoding. We ablate the effect of our length encoding mechanism in Table 3. When using RLT by itself, length encoding has minimal effect. However, when combining RLT with random masking, we note a clear improvement. Due RLT's structured and predictable pruning. length encoding may be unnecessary: the transformer is able to mostly understand the length of various tokens by their associated spatial positional encoding. However, once random masking is introduced, the structure is removed, and the length encoding adds crucial information. Since including the length encoding is strictly more information and has no negative effect, we default to including it.
4.4 Longer Videos and Higher FPS
Standard action recognition datasets consist of short clips with downsampled FPS: an input example typically spans 2 seconds. One potential advantage of RLT is that by reducing the total number of tokens, training becomes more tractable for both longer videos and higher FPS. We evaluate the effect of training with RLT in Table 5 on action recognition datasets with higher FPS along with their training time. As before, we fine-tune these models from pre-trained VideoMAE checkpoints. Although these checkpoints were pre-trained at 7.5 FPS, we can still compare with the baseline performance to observe differences in training time or quality. Similar to the result from Table 1, find that ViTs trained with RLT can match performance but train significantly faster, with the speed-up increasing with the FPS.
We next analyze the number of total tokens in RLT compared to the baseline for several video datasets in Table 4, including datasets with longer videos as well as higher FPS. Matching the result from Table 5, at higher FPS, RLT consistently reduces the tokens by higher proportion. This matches
8

---

Figure 4: Sample Visualizations. Tokens that are compressed are visualized in gray. RLT retains tokens that change between frames while removing redundant tokens. In the top example, RLT captures the static background, and in the bottom example, due to camera motion and the motion of the girl, almost tokens are modified. Video visualizations are available at the project page.
our intuition, since tokens between two redundant tokens at lower FPS are likely to be similar and also be removed. Furthermore, longer video datasets, RLT can reduce the number of tokens by significantly larger margins, with reductions of up to 80% on COIN and Breakfast. These datasets in particular consist of videos filmed with fixed cameras and largely static backgrounds, demonstrating RLT's potential to drastically speed up transformers on these types of videos. Although in practice, researchers do not typically train of video decoding on academic clusters, RLT presents a promising way to efficiently train on these videos at scale.
videos with large number of frames due to the heavy cost
4.5 Visualizations
We provide some qualitative visualizations of the tokens RLT removes in Figure 4. As desired, input patches that are repeated over time are pruned by RLT. This intuitively matches with how humans often pay less attention to static tokens over time. In the top example, most of the background is black, with some motion taking place in the foreground. RLT is able remove the constant black portions, drastically reducing the number of tokens. Similarly in the second example, RLT ensures that the tokens containing motion, with the boy's 's hands and instrument, are not modified, but prunes the static background. In the lower two examples, the person using the drill and the girl in the foreground move around significantly, reducing the amount of tokens that can be compressed. such cases where there is significant subject camera motion, RLT removes fewer tokens, resulting in similar token

---

7= 0.05
7
0.15
0.2
Figure Effect T. r. With low values of r, the clearest repeated patches are ablated, but imperceptible variations can prevent some visibly similar tokens from being pruned. Above 0.1, some tokens with slight movement are pruned.
counts to standard tokenization. However, the sensitivity of RLT to small perturbations and motion depends entirely on the hyperparameter. We provide further example visualizations and visualize the effect different values of F in Appendix and on our project page. In Figure 5 we demonstrate the effect that the hyperparameter has on the input tokens. We that increases, more and more patches are included, and after 7 = 0.1, some patches that have change them are pruned incorrectly. On the other hand, 0 includes many patches with essentially imperceptible change, which is also undesired.
5 Conclusion
Summary We present Run-Lengt Tokenization (RLT), simple alternative standard video tokenization for video transformers that replaces temporally redundant tokens with a single token of variable length. RLT decreases transformer training and inference wall-clock time by up to 40%m achieves a better speed-accuracy tradeoff than prior works, and is simple to implement and combine with other methods. RLT demonstrates strong results during finetuning, especially at higher FPS, and even works well when applied to models without any training.
Limitations Though RLT works well, it relies a heuristic to compare temporally consecutive tokens, which could include extra tokens that are unused by the transformer. While RLT speeds up video transformers significantly, cannot be used for dense vision tasks, such point tracking or video generation, that require the same number of output tokens as input tokens; RLT reduces tokens before running the model and does not replace them. Furthermore, RLT does not handle camera motion well: in a video with constant camera motion, few tokens will be removed, leading to no speedup. Future work will be necessary to overcome these limitations, and hope that RLT can inspire more research on efficient vidco transformers.
Acknowledgments and Disclosure of Funding
This work is supported by Fujitsu Research of America, and RC is supported by the NSF GRFP.
10

---

References
[1] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang. Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 34:24206-2422 2021.
[2] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: video vision transformer. In Proceedings of the IEEE/CVF international conference computer vision, pages 6836-6846, 2021.
[3] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In Proceedings of the International Conference on Machine Learning (ICML), July 2021.
[4] Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornllihh, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. Flexivit: One model for all patch sizes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14496-14506, 2023.
[5] Daniel Bolya, Cheng-Yane Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Representations, 2023.
[6] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL https://openai: pttp:://ppenai..cmmrreeeaa com/research/ video- lereeo-gnneraiion-ooooee
sessworld-simuleed
[7] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702-703, 2020.
[8] Tri Dao. Flashattenntinn-2: Faster attention with better parallelism and work partitioning, arXiv preprint (rXiiv2307.0869/ 2023.
[9] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memoery-fffccient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 1111646444659,, 2022.
[10] Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdulmohsin, al. Patch n' pack: Navit, vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems, 36, 2024.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint
arXiv:2010.1199 2020.
[12] Patrick Esser, Robin Rombach, and Bjor Ommer. Taming transformers for high- high-resolution image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2873-12883, 2021.
[13] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 6824-6835. 2021.
[14] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVE international conference on computer vision, pages 6202-6211, 2019.
[15] Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders as spatiotemporal learners.
2022.

---

[16] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pages 5842- 5850, 2017.
[17] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[18] Sunil Hwang, Jachong Yoon, Youngwan Lee, and Sung Ju Hwang. Everest: Efficient masked video autoencoder by removing redundant spatiotemporal tokens.
[19] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 17X0001705.06550, 2017.
[20] Diederik Kingma. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
[21] Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, al. Spvit: Enabling faster vision transformers via latency- aware soft token pruning. In European conference on computer vision, pages 620-640. Springer, 2022.
[22] Bruno Korbar, Du Tran, and Lorenzo Torresani. Scsampler: Sampling salient clips from video for efficient action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6232-6242, 2019.
[23] Mario Michael Krell, Matej Kosec, Sergio P Perez, and Andrew Fitzgibhon. Efficient sequence packing without cross-contamination Accelerating large language models without impacting performance. arXiv preprint arXiv:2107.022 000022, 2021.
[24] Hilde Kuchne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal direceee human activities. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 780-787, 2014.
[25] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hack- able transformer modelling library. https: / com/faccboookesoarrdoooooooo www.ccchffoooerreeeoom 2022.
[26] Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the JEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 4804-481 4814, 2022.
[27] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 23390 20490,,,000,-20 2023.
[28] Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:2202.07880 2022.
[29] Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video under- standing. In Proceedings of the IEEF/CVF international conference computer vision, pages 7083-7093, 2019.
[30] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
12

---

[31]
Dmitrii Marin, Jen-Hao Rick Chang, Anurag Ranjan, Anish Prabbu, Mohammad Rastegari. and Oncel Tuzel. Token pooling vision transformers. arXiv preprint arXiv:2110.03860 2021.
[32] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[33] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023,
[34] Yongming Rao, Wenliang Zhao, Benlin I. Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamievit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34: 14:137971,3949 2021.
[35] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.007/1 207144 2024.
[36] Yong Ren, Chenxing Li, Manjie Xu, Wei Liang. Gu, Rilin Chen, and Dong Yu. Sta- v2a: Video-to-aue audio generation with semantic and temporal alignment. arXiv preprint arXiv:2409.08600 2024.
[37] Cedric Renggli, André Susano Pinto, Neil Houlsby, Basil Mustafa, Joan Puigcerver, and Carlos Riquelme. Learning merge tokens in vision transformers. arXiv preprint 1201, 2022.
[38] Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Yao Huang, Vaibhay Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, al. Hiera: A hierarchical vision transformer without the Learning, pages
In International Conference on Machine
PMLR, 2023.
[39] Michael S Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Tokenlearner: What can 8 learned tokens do for images and videos? arXiv preprint 1rXiv:2106.11297 2021.
[40] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems, 27, 2014.
[41] Gary J Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (heve) standard. IEEE Transactions on circuits and systems for video technology, (612))16499166
2012.
[42] Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou, Coin: A large-scale dataset for comprehensive instructional video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2216-1216, 2019.
[43] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self- supervisdd video pre-training. Advances in neural information processing systems, 35:10078-10000 2022.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[45] Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He. Yi Wang, Yali Wang. and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pages 14549-1455 14560, 2023.
[46] Thomas Wiegand, Gary J Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the h. 264/ave video coding standard. IEEE Transactions on circuits and systems for video technology, 13(7))560-576, 2003.
13

---

[47] Ross Wightman. Pytorch image models. pytorch- image- models, 2019.
https://github..cm com http://ttuub.ccoo//iigttttaaa
[48] Chao-Yuan Wu, Manzil Zaheer, Hexiang Hu, R Manmatha, Alexander J Smola, and Philipp Krähenbühl. Compressed video action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6026-600 60356035,40 2018.
[49] Chao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Memvit: cemory-augmenter multiscale vision transformer for efficient long-term video recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13587-13597, 2022.
[50] Zhirong Wu, Zihang Lai, Xiao Sun, and Stephen Lin. Extreme masking for learning instance and distributed visual representations. arXiv preprint arXiv:2206. 2022.
[51] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134-18144, 2022.
[52] Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 10809-10818, 2022.
[53] Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. In Proceedings of the IEEE/CVF international conference computer vision, pages 2998-3009 2021.
[54] Yue Zhao and Philipp Krähenbühl. Training a large video model on single machine in day. arXiv preprini arXiv:2309. 16669, 2023.
[55] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Proceedings of the European conference on computer vision (ECCV), pages 803-818, 2018.
[56] Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox. Fco: Efficient convolutional network for online video understanding. In Proceedings of the European conference computer vision (ECCV), pages 695-712, 2018.
A Implementation Details
Our code, demos and associated blog post are all located on our project page. In this section, we provide further details on implementation details our experiments.
Architecture. All models used were based on the timm [47] Vision Transformer implementation, and all fine-tuning experiments were done with pre-trained checkpoints from VideoMAE [43] and VideoMAEv2 [45]. As mentioned in 3.3, we compute output predictions for action recognition by taking the mean across the output tokens, rather than producing separate class token.
Baselines. The baselines we compared to are Token Merging [5] and random masking [27]. For all random masking experiments, we set the masking ratio p to match the mean RLT token reduction for given dataset.
example, on Kinetics-400 at 7.5 FPS, RLT with 7 0. reduces the number of tokens by 28%, so we randomly drop 28% of the tokens during training. We use the recommended
values of from the Token Merging paper, except on ViT-H, where we use 32 due to the larger depth of the model.
14

---

Datasets. We train and evaluate RLT on Kinetics-400 (K400) [19] and Something-Something (SSv2) [16]. Both datasets are video classification datasets, with K400 having 400 classes and SSv2 having 174. K400 has 240k training examples and 40k test examples, while SSv2 has 170k training examples and 30k test examples. We also included experiments measuring the token reduction on the Breakfast [24] and COIN [42] datasets, both of which are smaller-scale datasets involving longer videos that range from 2-5 minutes. In particular, these datasets contain lots of fixed-camers videos with static backgrounds, leading to particularly high token reductions from RLT.
Training Recipe. We not change hyperparameters when finetuning models with different tokenization strategies, we found the provided set to be optimal in our experiments. We follow the recommended training recipes from VidcoMAE for each model size, namely training for up to 100 epochs, with batch size 256, learning rate with warm-up to 1 10 for 5 epochs, then cosine annealing down to x 10-6. 6, We also use RandAugment, random erasing, CutMix, and standard cropping/scaling and flipping. We do not use MixUp since it severely affect the efficacy of RLT, and we found that removing and only using CutMix did not affect our experiments. We also used random erasing with single value rather than noise, enabling some of the erased tokens to be removed by RLT.
All experiments were conducted with 8xH100 Nvidia GPUs with 128 CPU cores, with 16 workers per GPU. The inference-time results were computed on single GPU, along with the throughput and FLOPS analysis. One important detail is that data loading often bottleneck. We mainly relied on the fast video data loader from AVION [54], but NVIDIA DALI also works very well. However, only recomend to use DALI on A100 newer chips, as earlier generations have insufficient number of dedicated decoder hardware. Each training run for the paper is specified in hours, but this does not include few months of work testing and debugging. We used single node for all work on this paper.
B More Visualizations
We include some additional visualizations here qualitatively demonstrate which tokens RLT prunes, as well as to analyze the qualitative effect of varying the difference threshold In each figure, the whitened patches represent those RLT identified static, and that are not passed the transformer. In Figure 6, we visualize a diverse range of samples and note that RLT consistently prunes out patches that repeat across consecutive frames. One case where RLT fails remove many tokens the 4th example from the top, which is from ski jumper using a GoPro; the constant camera motion means that RLT is unable identify almost any repeated patches.
We highly encourage readers to visit our project page for video visualizations that better convey the effect of RLT.
15

---

Figure 6: More examples. We visualize RLT's 's effect on videos ranging from TV shows, movies, action sequences on a GoPro, and sports. RLT consistently prunes out tokens that are repeated and static, and includes all patches that change between frames, retaining as much information as possible while cutting the number of tokens significantly.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims in the abstract and introduction are concretely linked results in our Results section, and aspirational goals are clearly denoted.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include a limitations section at the end of the paper and discuss some of the weaknesses of our method that we hope future work can address.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [NA]
Justification:
Our paper includes no theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our paper includes detailed technical instructions, lists of hyperparameters and model architectures and experimental results. We also include our code in this submission and will open-source the code upon releasing the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
18

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We include the code in the submission which has all the necessary files to run the experiments from the paper. We have dataset contribution and thus do not include any data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /twpps.cc/pppppeee// h //t////ooeeeeesseeeeeeeeee iitmmiisionn onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include data splits, hyperparameters and rational for choosing them in the Implementation details section of the appendix. This is also provided with the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Our experiments deterministic, and none of the experiments require statis- tical significance analysis; we ensure all experiments are random-sceded and reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We include details about our compute usage in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https https://neett /ttp:r////////p/// cc cctppcccccc//cccccccccccccccccct httolllc//Ehiiseeee schicsshudellnes""
Answer: [Yes]
Justification:
We reviewed and adhere to the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [No]
The impact of our work simply speed up existing methods, in this case vision transformers. We do not see any clear way for this to be maliciously used or for this to directly negatively impact society and thus did not discuss this in the main text.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: Our experiments are relatively small-scale and based on existing pre-trained models, and thus requires no such safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All pre-trained models were cited for their authors, along with the codebases used. We reference these in the main text, supplement and code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
21

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode com conddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide details about our method in the main text and supplement, and the code contains documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We include human subject research in our work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We have no user studies and thus did not need an IRB.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "P6L4g9YuVc", "reviewer": "Reviewer_B8qa", "summary": "The authors present a compression or optimization technique applicable to video transformers for both training and inference paradigms. Through empirical evaluation, the work showcases efficiency gains achieved for fine-tuning video transformer models and also showcases inference time efficiency without any training at all, with minimal quality degradation. The code has been made available for reproducibility.", "questions": "Out of curiosity, how did the formulation of RLT come up?", "limitations": "NA", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper introduces run length tokenization (RLT) as a mechanism to reduce static tokens in training time dynamically. RLT takes into account spatial changes with respect to the temporal aspect and effectively condense static tokens adding run length information in the input embedding. RLT draws inspiration from widely used video compression techniques such as HEVC and AVC which are content aware, making RLT also content aware as part of the input embedding of the video transformer. \n\nThe originality of the work is commendable and the presentation quality of the work is exceptional. The efficiency gains achieved using this tokenization mechanism are state of the art, in both training and inference time with no performance degradation makes this work exceptionally useful to the extended community to train and deploy video transformer models more efficiently than previous literature in the field.", "weaknesses": "No additional weaknesses apart from the limitations pointed out by the authors in Section 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your kind comments towards our work, and thank you for your detailed review. We are glad you found our work to be exceptionally presented, and our work to be high quality. To answer your question, the inspiration for this work came from one of the authors’ habit of watching live podcasts on YouTube. These are often hours long but have a still camera and a static background. Videos with large amounts of stationary content are prime candidates for methods like RLT - their visual information can be drastically compressed with minimal loss.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cQmbqsXA3A", "reviewer": "Reviewer_N711", "summary": "Current video models usually need to process every patch or tubelet of every frame, no matter if the video is very dynamic or contains patches that almost never change (e.g. static backgrounds). This submission proposes to simply omit tubelets that do not change significantly between frames, and optionally add a run-length embedding to them that signifies for how many frames that token does not change. The result is a model that only needs to process the tokens that are changing in a video, which can lead to significant speedups depending on the video content, at no loss in performance on human action recognition tasks.", "questions": "- The term \"tokenization\" is loaded with a lot of meaning, and in the introduction it's easy to assume the method performs VQ-VAE or KL-VAE like tokenization, while the method operates on spatio-temporal patches, i.e. tubeletes, that are linearly projected. I suggest making it clearer what is considered a token.\n- How well would RLT perform at inference time if the run-length for all tokens is forced to be just 1, no matter the threshold?", "limitations": "The authors adequately address potential limitations.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The method is very well motivated and aims to tackle an important problem, e.g. as the paper states, encoding a lecture with static background shouldn't require the same sequence length as a busy GoPro video. Dropping tubelets based on the temporal differences is better motivated than e.g. randomly dropping tokens irrespective of the video complexity.\n- The RLT formulation ensures that in the worst case where every tubelet changes in every frame, the inference time and performance is as large as a non-RLT baseline (assuming the changes are greater than the threshold). With this formulation, the sequence length is quite short for static videos, and the same for very busy videos.\n- The architectural modifications required are minimal, and only consist of implementing sequence packing and adding a learnable run-length embedding. This enables fine-tuning existing video models rather than having to train from scratch.\n- As far as I can tell, the run-length \"positional\" embedding is novel\n- On Kinetics-400 and Something-Something-v2, RLT can maintain performance while being significantly faster to train. Compared to a random approach that is content-unaware, RLT works better.", "weaknesses": "- In Table 3 it is shown that for the larger ViT-L, adding run length embeddings significantly hurts performance. On the smaller ViT-B model it achieves the same performance, but the significant performance drop on the large model calls into question how the run length encoding influences train and test dynamics, and how well tuned these ablations are.\n- How well does the approach work when training from scratch compared to fine-tuning existing checkpoints? How easy is it to fine-tune in the additional run-length embedding? Would we see a similar performance drop of ViT-L as in Table 3 when training from scratch?\n- Table 2 is quite messy. \"ViT-B\" should be greyed out, the Xs in the greyed out rows are black, and the bolding is inconsistent (ViT-L acc on Kinetics is best, and only 4 values from the entire table are bolded).\n- Table 3 is not very clear. Shouldn't the second rows for each model size be called \"RLT minus length\" since the default RLT setting uses run length embeddings, and the third row is just \"RLT\"? The description of \"Rand\" and \"length\" are not very clear. The gain from combining length and rand does not seem very significant compared to the base performance.\n- Table 4: How much can RLT compress the sequence length of other datasets? It should be quite cheap to evaluate this. \n- It's also not clear how much influence the threshold value has on different datasets. It would be important to know if the optimal threshold value differs much between datasets, or if there is a safe range of values it can be set to.\n- When training a model with RLT, should one train for the same number of tokens or same number of samples? It's not clear how much of a difference this makes between datasets.\n- Is there a case where the proposed thresholding approach fails? How sensitive is it to the norm used? What if there are small but crucial differences that are below the threshold but important for the task at hand? The paper only evaluates on human action recognition tasks which could be blind to such differences.\n- The method of calculating patch differences is to compute the 1-norm between the beginning and end of the two adjacent patches respectively. However there could potentially be some cases where rapid motion through a patch could conceivably be missed by this method. Did the authors test whether this was ever the case or explore alternative difference calculation methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your exceptionally thorough, helpful and clear review of our paper. We are glad you found our work to be “very well motivated”, “novel” and found it to “work better than other methods like random masking”.  We address each of your concerns individually below.\n\n__Concern 1: Learned embeddings seem to hurt performance on ViT-L.__\n\nWe address this in the global rebuttal. In fact, RLT achieves about the same performance when trained with the learned embedding, and the learned embedding is especially helpful when combined with random masking.\n\n\n__Concern 2a: Did you ever test whether the thresholding would fail? There are cases where this could be a concern.__\n\nWe found that a single threshold worked well for all experiments. We measured this by visually inspecting pruned patches (as in Figure 6) and measuring the performance as a function of threshold (Figure 3) However, as you note, this is an approximation for checking whether two patches are similar. We agree it could potentially affect performance for specific tasks where pixel-level details matter: for example, video generation or pose estimation. In this paper we focused on action recognition because it is the de facto task for measuring the quality of learned video representations, and our results demonstrate that RLT does not decrease the quality of these representations. As such, we believe our use of a single threshold justifies the potential tradeoff on certain downstream tasks.\n\n__Concern 2b: How sensitive is RLT to the norm used? How did this compare across datasets?__\n\nIt doesn’t matter which norm is used - we found there was no appreciable difference in performance with L1 or L2. In our case, we took inspiration for using the L1 norm and a single dataset-agnostic threshold from the way video compressors [1] are implemented. Using and tuning for other types of norms could result in small differences, but L1 is simple, fast to compute, and widely used.\n\nThe threshold for whether the average intensities in a block of pixels has sufficiently changed should be agnostic of the data distribution. Though the threshold value itself matters, we find that the content of the dataset videos determine the reduction from RLT itself. We include results for different values of the threshold in Table 6 of the global rebuttal PDF, supporting that the change in performance for different values of the threshold is similar for Kinetics and Something-Something v2. On the other hand, datasets with almost no motion (e.g Breakfast) will result in many pruned tokens even for extremely small thresholds.\n\n__Concern 3: How much can RLT compress the sequence length of different datasets?__\n\nWe provide this analysis in Table 5 of the PDF. To be clear, each example in a dataset is compressed to a different number of tokens based on its content. We find that the average compressed sequence length does vary across datasets depending on their content; as mentioned in the main text, the Breakfast and COIN datasets have minimal motion, and RLT is able to successfully prune many more tokens.\n\n__Concern 4: Should RLT be trained for the same number of tokens or the same number of samples?__\n\nFor the fairest comparison, we trained on the same number of samples. Since RLT significantly reduces the number of tokens with the same number of samples, we find that training for the same number of total tokens leads to improved performance. We demonstrate this in Table 4 of the PDF, and will include these in the final version of the paper.\n\n__Concern 5: How well does the RLT work when training from scratch? How easy is it to fine-tune the length embedding?__\n\nThis is an excellent question, and one we would have liked to investigate as well. However, pre-training video transformers requires large amounts of compute that was outside of our resources. Secondly, masked pre-training (which is how all the checkpoints we fine-tuned were pretrained) already involves removing a large proportion of tokens, and thus RLT would have minimal speedup during pre-training. However, our hypothesis is that pre-training with a learned embedding would be particularly helpful during pre-training, since we noticed that training with random masking benefitted from a learned embedding, supported by Table 2 of the PDF. \n\n__Concern 6: Tables 2 and 3 are not very clear.__\n\nThank you for pointing this out. We apologize for the lack of clarity and will gladly clean these issues up in the updated version of the manuscript. We have included fixed versions of these tables in the results PDF.\n\n__Question 1: Use of the term “tokenization\" is unclear.__\n\nThank you for bringing this to our attention. We agree that the term “tokenization” is unclear, and perhaps “patchification” would be a better alternative. We will gladly update this throughout the text to better distinguish our method from VQ-VAE and other tokenizers, as you mention.\n\n__Question 2: How would RLT perform if the length of all tokens is 1, no matter the threshold?__\n\nWe are not entirely sure we understand the question, but it appears that what you refer to here is RLT without the length encoding. This result is in Table 4 of the main text, and performs about the same at inference time, and slightly worse when combined with random masking. If we have misunderstood the question, please feel free to clarify further.\n\nGiven that we have addressed the primary concerns raised in your review, would you consider adjusting your score?\n\n[1] Wiegand, T., Sullivan, G.J., Bjontegaard, G. and Luthra, A., 2003. Overview of the H. 264/AVC video coding standard. IEEE Transactions on circuits and systems for video technology, 13(7), pp.560-576.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rkZW0gT3rO", "reviewer": "Reviewer_tWGP", "summary": "In this paper, the authors propose run-length tokenization, an efficient video patch merging strategy to speed up the video transformer", "questions": "### Strength\n1. The paper is well-written and easy to follow\n2. The proposed RLT is intuitive and works well\n\n### Weakness.\n1. My main concern is the experiment part is far from solid. In Tables 1 and 2, there are only two simple baselines Token Merging and random masking. Pruning is a well-studied research topic and there are many methods, the Token Merging also has many following works. Two simple baselines are not convincing. For the rest experiments, no baseline is provided. The lack of correct baselines hinders the correct evaluation of the proposed method.\n2. Following 1, the token merging baseline lacks details, is it conducted within each frame? or along the temporal dimension like RLT.\n3. The proposed method is somewhat too simple and intuitive, ie, merging the patches with similar pixels. \n4. Some experiment results are conflicting and confusing. In Table 3, the length encoding seems nonhelpful or even harmful for the RLT, while it is claimed as a contribution of the proposed methods.", "limitations": "Please refer to Questions", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Please refer to Questions", "weaknesses": "Please refer to Questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply thank you for your helpful review and greatly appreciate your feedback. We are glad you found that our method  is “intuitive and works well”, and that the paper was “well written and easy to follow”. We address each of your concerns below.\n\n__Concern 1a: Evaluation: Two simple baselines are not enough.__\n\n It is difficult to respond to this without direct citations of comparable pruning works or merging follow-ups. To our knowledge, the vast majority of pruning methods (e.g A-Vit [1], EVit [2], DynamicVit [3]) require training a learned token selection module. This prevents their application to out-of-the-box models while also preventing speedups during training as they require padding. Furthermore, these methods almost all only present results on images, preventing us from comparing them. On the other hand, Token Merging and random masking are two widely used methods that have been demonstrated to work well on videos, and present the fairest comparison to our work.\n\nHowever, the most applicable baseline we did find was Semantic-Aware Temporal Accumulation (STA) [5] which is applicable to pre-trained models out-of-the-box, but performs poorly when used during training. We re-implemented STA, optimized it further to use Flash-Attention, and then compared it to RLT in Tables 1 and 3 of the global rebuttal PDF. We find that STA is slower and performs worse in all cases than RLT. We will gladly include this result in the main text.\n\n__Concern 1b: For other experiments, no baselines are provided.__\n\nWe presume that this comment refers to Tables 3, 4 and 5 of the main text. Table 3 is an ablation, and thus does not make sense to compare to other baselines. Table 4 and 5 measures the reduction in tokens in different datasets and FPS configurations. However, Token Merging, random masking and STA all reduce a constant number of tokens controlled by a single hyperparameter. Comparing the reduction from RLT to these baselines in this context does not make sense. We will reword our analysis in the paper to make this easier to follow for readers.\n\n__Concern 1c: The token merging baseline lacks details.__ \n\nAs stated in the Token Merging paper [4], in videos merging is conducted across both spatial and temporal dimensions. We will clarify this further in the paper.\n\n__Concern 2: The proposed method is too simple and intuitive, i.e. merging patches with similar pixels.__\n\nIf a method yields significant performance improvements and is novel, it is a valuable contribution, regardless of its simplicity. In particular, Reviewers B8qa and N711 note our method is “very well motivated” and has “commendable originality”, and you note RLT’s intuitiveness in the Strengths section of the review. All three reviewers agree RLT contributes significant speedups. __Crucially, RLT does not merge patches progressively__. It finds which patches to remove before running the model. This key component enables it to remove different numbers of tokens for each input while avoiding the overhead of padding, which hobbles methods like A-Vit or DynamicVit. This point is central to the paper, and we will be sure to rewrite parts of the paper to emphasize this point better. Given our method’s novelty, simplicity and significant performance improvements, we believe that it provides an elegant solution to a problem that will be easy for many researchers to adopt in practice.\n\n__Concern 3: Some experimental results are conflicting.__ \n\nWe understand your concerns about the results on the learned length-encoding. As we state in the global rebuttal, supported by results in the PDF attachment, we observe no real difference with the RLT when training, but do observe a noticeable difference when combining other techniques like random masking with RLT. As we expect practitioners to combine RLT with other methods for maximizing speed, we decided to include these results to better inform the community. We will be sure to clarify the role of the learned positional embedding further in the text.\n\nGiven that we have addressed the primary concerns raised in your review, we kindly request that you adjust your score.\n\n[1] Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J. and Hsieh, C.J., 2021. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34, pp.13937-13949.\n\n[2] Liang, Y., Ge, C., Tong, Z., Song, Y., Wang, J. and Xie, P., 2022. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:2202.07800.\n\n[3] Yin, H., Vahdat, A., Alvarez, J.M., Mallya, A., Kautz, J. and Molchanov, P., 2022. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10809-10818).\n\n[4] Bolya, D., Fu, C.Y., Dai, X., Zhang, P., Feichtenhofer, C. and Hoffman, J., 2022. Token merging: Your vit \nbut faster. International conference on learning representations.\n\n[5] Ding, S., Zhao, P., Zhang, X., Qian, R., Xiong, H. and Tian, Q., 2023. Prune spatio-temporal tokens by semantic-aware temporal accumulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 16945-16956).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their insightful comments and are grateful for their feedback. We are glad they found our work to be __“commendably original, very well-motivated and intuitive”__ (RB8qa
, RN711, RtWGP), __“state-of-the-art, useful, significantly faster and better“__ (RB8qa, RN711,RtWGP) and __“exceptionally presented and well-written”__ (RB8qa, RtWGP). In this global rebuttal, we review the main contribution, address a common concern and outline the additional results provided in the PDF. We address each reviewer's individual concerns in the Author Rebuttals.

To reiterate, RLT provides a way to dramatically speed up video transformers by identifying redundant spatiotemporal patches before running the model. By removing those patches, we find that we can achieve significant speedups without loss of performance on inference for pre-trained models and during training. Our formulation of RLT avoids the common pitfall of having to use padding during training while devoting more computation to busier videos and more static ones. 

__Concern (RN711,RtWGP): the results in Table 4 are confusing - why does the learned encoding hurt on ViT-L?__

We understand your concerns about the results on the learned length-encoding. The result in Table 4 of the main manuscript is a typo. RLT in fact achieves a top-1 accuracy of 84.0 when trained with the run-length encoding, an extremely small drop-off in performance. As we state in the text of the paper, and demonstrate in Table 2 of the results PDF, the results on ViT-L match the pattern of those of ViT-B, and show the learned encoding is especially helpful when combined with masking. We deeply apologize for this oversight on our part, and will be sure to update the manuscript accordingly.

__New Results__: 

We include the results from several experiments requested by Reviewer N711 and Reviewer tWGP. Table 2 contains the updated results for the run-length encoding ablation with the correct numbers. Table 1 and 3 compare RLT to STA, another pruning baseline that can also apply to pre-trained models. Table 4 compares training RLT for the same number of samples vs the same number of tokens, and Table 5 shows the average sequence length produced by RLT across multiple datasets.

Given our responses to each reviewers’ concerns, we humbly request that you consider adjusting your scores, and look forward to a fruitful discussion period.

Meta-review: This well-written paper introduces a novel method for reducing the computational burden of video analysis models. 

The proposed method omits patches/tubelets that do not change significantly between frames, and uses a run-length embedding to capture the information about how much data is omitted. The result is a model that only needs to process the tokens that are changing in a video, which can lead to significant speedups depending on the video content, with little or no loss in performance on human action recognition tasks. The computational savings is greatest when the input video is static, but significant savings can be had in typical videos where the important activity take up only a small part of the total time, or where the motion activity is localized to a small area of the frame.

The method reduces static tokens in training time dynamically. RLT takes into account spatial changes with respect to the temporal aspect. The RLT method inspired by popular video compression techniques such as HEVC and AVC which are content aware, making RLT also content aware as part of the input embedding of the video transformer.

The rebuttal, and subsequent discussion, resolved most of the concerns of the initial reviews. One reviewer expressed concern that the method was "too simple". But, as the authors responded, this is not a weakness. Many important advances in ML have been at heart apparently simple ideas (e.g. "dropout"). In this case the contribution may perhaps be simple, but it is not trivial, and is worth being made known to the ML community.

The other main critique made by the reviewers had to do with the apparent loss in performance when RLT is used. However, the authors point out that there was a typo and the actual performance drop is minimal. It should be kept in mind that the main benefit of the approach is the significant speedup attained when running video transformer models.
Decision: Accept (spotlight)