ElasticTok: Adaptive Tokenization for Image and Video
Authors: Wilson Yan
Volodymyr Mnih
Aleksandra Faust
Matei Zaharia
Pieter Abbeel
Hao Liu
Venue: ICLR 2025 Poster (2025)
Keywords: adaptive representation
adaptive tokenization
autoencoder
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Efficient video tokenization remains a key bottleneck in learning general purpose vision models that are capable of processing long video sequences. Prevailing approaches are restricted to encoding videos to a fixed number of tokens, where too few tokens will result in overly lossy encodings, and too many tokens will result in prohibitively long sequence lengths. In this work, we introduce ElasticTok, a method that conditions on prior frames to adaptively encode a frame into a variable number of tokens. To enable this in a computationally scalable way, we propose a masking technique that drops a random number of tokens at the end of each frames's token encoding. During inference, ElasticTok can dynamically allocate tokens when needed -- more complex data can leverage more tokens, while simpler data only needs a few tokens. Our empirical evaluations on images and video demonstrate the effectiveness of our approach in efficient token usage, paving the way for future development of more powerful multimodal models, world models, and agents. Video examples of using ElasticTok can be found on our website: http://largeworldmodel.github.io/elastictok

Published as a conference paper at ICLR 2025
ELASTICTOK: ADAPTIVE TOKENIZATION FOR IMAGE AND VIDEO
Wilson Yan* Vlad Mnih
Aleksandra Faust! Matei Zaharia² Pieter Abbeel Hao L
UC Berkeley
!Google DeepMind *Carnegie Mellon
ABSTRACT
Efficient video tokenization remains key bottleneck in learning general purpose vision models that are capable of processing long video sequences. Prevailing approaches are restricted to encoding videos to fixed number of tokens, where too few tokens will result in overly lossy encodings, and too many tokens will result in prohibitively long sequence lengths. In this work, introduce ElasticTok, a method that conditions on prior frames to adaptively encode a frame into a variable number of tokens. To enable this in computationally scalable way, propose a masking technique that drops random number of tokens at the end each frames's 's token encoding. During inference, Elastic'Tok dynamically allocate tokens when needed more complex data can leverage more tokens, while simpler data only needs few tokens. Our empirical evaluations images and video demonstrate the effectiveness of our approach in efficient token usage, paving the way for future development of more powerful multimodal models, world models, and agents. Video examples of using Elastic Tok can be found on our website: largeworldmodel.gine wwt.oleee..gguuuioooooooo htt.thuubb./llasiitto.....
t+ds
Time
1=716
GT
Recon
% tokers Used
h
Time
Figure 1 ElasticTok adaptively represent video based on information available. (Top) Ground- truth video frames. (Middle) Reconstructed frames with varying token usage. (Bottom) The bottom section depicts how ElasticTokk dynamically adjusts token allocation over time, with the percentage of tokens used correlating to different content complexities in the video.
INTRODUCTION
Efficient video tokenization remains key bottleneck in learning general purpose vision models that are capable of processing long video sequences a crucial aspect towards developing intelligent agents for the visual world. Prevailing approaches (Van Den Oord et al., 2017; Esser et al., 2021; Yu et al., 2023a; Yan et al., 2021) are restricted to encoding videos to a fixed number tokens irrespective of the visual content of the original video. As result, being able reliably encode with little information loss requires increasing the number of tokens to account for the worst case, highly
To whom correspondence should be addressed.

---

Published as a conference paper at ICLR 2025
complex visual imputs. However, this in turn causes many wasted tokens simpler data, and leads to significant computational challenges for downstream training, where encoding a video or trajectory require tens or even hundreds of millions of tokens, resulting in substantial computational costs and inefficiency (Liu et al., 2024a; Brooks et al., 2024; Liu et al., 2024b). On the other hand, using a too few number of tokens will result in lossy encodings, and fundamentally limit the capabilities of vision models when processing more complex visual data.
Intuitively, we would want do learn a model that adaptively encodes visual input in data-dependent manner to variable length encodings, similar to how existing works in image and video compres- sion (Richardson, 2004; Li al., 2023a; Christopoulos et al., 2000; Chen al., 2017) compress data to varying number of bytes. Taking inspiration from this, we introduce ElasticTok, method that conditions on prior frames adaptively encode a frame into variable number of tokens. To enable this in computationally scalable way, we propose masking technique that drops random number of tokens the end of each frame's token encoding. During inference, Elastic'Tok can dynamically allocate tokens when needed more complex data can leverage more tokens, while simpler data only needs a few tokens.
Our empirical evaluations demonstrate the effectiveness of ElasticTokk, highlighted below.
We show that Elastic'ook can leverage adaptive tokenization to efficiently represent long videos with up 2-5x fewer tokens.
We show that ElasticTok enables flexibility in downstream vision-language tasks, allowing users to allocate tokens based their compute budget.
We show that ElasticTokk allows leveraging different objectives during inference adaptively trade off various semantic aspects of images.
2 BACKGROUND
2. BLOCKWISE RINGATTENTION
Blockwise RingAttention (Liu et al., 2024b) a training technique for efficiently scaling transformers on long- long-context data. Each rank along the sequence parallel sharding axis retains non-overlapping, equal sizes slices of the full sequence. During an attention operation, each sequence parallel rank calculates its own corresponding Qi, K;, Vi, and computes its own portion of the full attention as: softmax (Q? [K....., Ks))[V Vsl, where S is size of the sequence parallel dimension. Instead
Single Block (Images Short Video)
Multi-Block (Longer Video)
Decoder
Decoder (Block Causal Mask)
Sample Letent Mask
Encoder
Encoder (Block Causal Mask)
Figure 2 ElasticTok adaptively encodes image and video to variable length outputs based on the complexity of the input data. Single block uses an Incoder-Decoder pipeline with a sampled latent mask. Multi-bloc' extends this with Block Causal Mask to handle longer video sequences.
2

---

Published as a conference paper at ICLR 2025
of naively all- galthrthniing K, V. RingAttention proposes to iteratively pass K, V2 blocks in ring- structure. where rank i will pass its current key-value block to rank (i + 1) mod S. Importantly, RingAttention enables overlap between compute and communication by computing the partial attention over the current block while communicating the next block. this paper, we leverage Ring to scale method to long sequences of videos.
METHOD
In this section, we present ElasticTokk, a scalable adaptive visual tokenizer than can efficiently encode image and video. At high level, we leverage a specialized random masking scheme when training any standard autoencoder to learn our elastic encodings. In the following sections, we provide more detailed description of our model first covering the simpler single block case (image short video), and then moving on to the multi block case (longer video). An overview of our method is shown in Figure 2.
3.1 TRAINING
First, we consider a single-block case in which we want to learn clastic codes over an image or short video chunk (e.g.. 4 frames). Let r represent a sequence of input frames, where images are considered as frame videos. Our proposed method extends upon any existing variants of autoencoders (VQ-VAE (Van Den Oord et al., 2017), FSQ (Mentzer et al., 2024), VAE (Kingma, 2013)) by incorporating additional masking of the encoder tokens.
For each training input T, first uniformly sample the number of tokens to keep é U[( U[(MMin...... Mmarr) }), where Mmin and Miar define the supported range of encoding lengths. Mmaz is generally set N, the maximum number of tokens the encoder can output (e.g., 2048 or 4096), and Mmin a lower bound such as 128 or 256. We found this lower bound necessary as we found that sampling too few tokens could destabilize training. After sampling E, we then compute its mask m, defined as a binary vector of length N with the first E values set to 1. Imput x and mask m are then fed into the encoder E(x, m) to compute z. is masked as 2m @ m, and fed through the decoder D(=m) to produce á. Finally, our model optimizes reconstruction loss, with potentially auxiliary losses depending on the exact autoencoder being used ge . KI. for VAE or VQ loss for VQ- (AE). Although method is generally architecture invariant, opt use ViTs (Dosovitskiy, 2020) as our encoder and decoders for simplicity.
We can extend our method to process longer video by breaking up video into blocks, with sizes defined by the number of frames used for the single block model (e.g 4 frames per block). The training process remains the same the single block case, with two key differences: (1) masks for each multi block video sequence are sampled independently for each block, and (2) a block causal mask (block size N) is used in the encoder and decoder. The number of blocks is constrained only by the transformer's 's context size. We utilize a prior long-context method (Liu et al., 2024b) to train ElasticTok on longer videos. Architecturally, there are no added parameters which found useful when progressively training our model from single block to multi block.
Exact details of our model's forward pass are described in Algorithm
3.2 INFERENCE
There are two primary ways to use ElasticTok for inference - by specify a target encoding length, or a target reconstruction threshold.
Target encoding lengths. This method simple to implement, as it only requires generating the correct mask and running the input through the encoder. However, although such inference is simple and efficient, is difficult from a user's perspective due to lack of knowledge of how many tokens to specify.
Target reconstruction threshold. This method presents a more intuitive inference approach that will automatically adaptively allocate tokens between different inputs based specified reconstruction threshold (e.g., target pixel-wise MSE loss). Visual content that is easy to reconstruct may require fewer tokens meet the threshold whereas more complex inputs may require more tokens. Doing so
3

---

Published as a conference paper at ICLR 2025
requires search process over different encoding lengths determine lowest encoding length that still satisfies the given threshold.
We consider few diffeerent search algorithms, detailed below:
Full Search: Exhaustive search over every possible number of tokens lengths, and treat the result discrete latent with max 4096 tokens.
ground truth. We
Binned Search (100): Similar to Full Search, but only evaluate on 100 uniformly spaced token lengths.
Binary Search: We perform search using binary search, where the token length is the "array index", and the evaluated reconstruction loss is the "array value." Note that this assumes that reconstruction loss monotonic with respect token length, which generally close to true, but not fully accurate, hence error in the result produced.
Neural Regression: We collect 100k examples of paired (image video, token length) data, and finetune our autoencoder to directly regress the number of tokens given an image or video.
Each search algorithm has its own trade-off between accuracy and efficiency we further examine this relationship in Section 4.4.
In the multi block case, we iteratively perform the search process each block in a block autore- gressive manner, using caching similar to in language models. Algorithm provides more details on the exact inference process.
Further ablations (Table 5 in Appendix F) suggest that can additionally achieve 2x inference speedup at slight cost to encoding quality by not conditioning the mask on the encoder this would require only running the encoder once, and then performing iterative search through masking the encoding and only running the decoder. This may be a better choice depending on the size of the downstream pretrained model.
Algorithm 1
Training
Algorithm Inference
Required: Video Patch size Tp x Hp Wp.
Required: Tokens per block
Required: Video x. Patch size Tр H, x W,.
Required:
Encoder E. Decoder D)
Required: Tokens per block Z.
Required: Min Max token lengths Mmin Mmaz
Required: Encoder E. Decoder D
// batch size, timesteps, height, width, channels
Required: Target reconstruction threshold
// is B X T H x W X XC PatchifyRearrange(aa B
batch size, timesteps, height, width, channels x T x W
// T/T,, H/H, W/W,
+ PatchifyReearannel (t) // X D
I1 D Tp Нd * W CC
11 T. T/T, H H W/W,
N₂ + L/Z
11 D T, H, W
for i in {0, ......,A No - do
<- L/Z
Sample & (1{ ([MMinn Mmaz})
Initialize KV cache of length L for i in , No 1} do
Initialize masks m. &- Oz
Fill masks m. & end for
// Soc Scction 2.2 for search algorithms E; mi, c + (earchAAgo((:zz(4412 c,t, E) Encode 2; <- E(5,2.(41)2, mi, c) m;
Concatenate masks
(mo. ..., MNN-1)
Encode +- E(x,, m))
Dz
end for
Mask out along sequence length 2m. +- ④ m Decode & <- ()2m)
Return = f- (zo...., 1.0....,2N,-1 2N,
Compute loss L( x) egg., MSE LPIPS
4 EXPERIMENTS
In this section, we introduce evaluation setup and present the results of pretraining ElasticTok to adaptively represent images and videos, as well as performance on downstream tasks.
4.1
EXPERIMENTAL SETUP
Model details. We train 200M parameter discrete and continuous autoencoders on images and video sequences. For learning discrete representations, FSQ (Mentzer et al., 2023), while
4

---

Published as a conference paper at ICLR 2025
for continuous representations, and decoders, consisting of a single patchify strided convolutions (the transpose for the decoder), followed by transformer blocks identical to those in LLaMA 2 (Touvron et al., 2023). Attention is applied in a block-causal manner, with block size equal to the number of tokens per video block. The encoder is additionally conditioned on the token mask (represented and I's with different learned embedding vectors, and then added to the video patchify output hidden states. The bottleneck representations are computed through linear projection on the encoder output. Similar to prior works (Esser et al., 2021), we optimize both pixel-wise reconstruction loss (MSE) and a perceptual loss (LPIPS) (Johnson et al., 2016). Notably, we do not use GAN (Goodfellow et al., 2014 4), as we find it more difficult to stably train when progressively extending to much longer sequences ge.g. 1000+ frames), which we leave to future work, as not the main focus of this paper.
VAE (Kingma, 2013). Both models employ ViT encoders
binary mask) by replacing 0's
Training details. All models are jointly trained on 256 256 images and 24 FPS videos. Each video block consists of 4 frames, and joint training is conducted by treating images as single-frame videos, replicating each image times to match the block size. Each block is encoded into 2048 or 4096 tokens by the continuous and discrete autoencoders, respectively. Following prior research in LWM (Liu et al., 2024a) on scaling sequence length, we begin by training our models on single-blos block cases (images and short videos) and progressively extend the context length cover up 40-second videos (51 (512K to IM tokens). We train our long video models using v4-512 TPUs from Google Cloud on the COYO-700M image dataset and a custom dataset consisting of 6M videos scraped from the web. We additionally train an image-only model on ImageNet using v4-256 TPUs. Details for further training details can be find in Appendix B, and data details in Appendix C
Baselines. We compare our proposed method against fixed token baselines trained at varying token capacities. The architecture, training processes, and total FLOPs are exactly the same our method with the only difference being restricted to one fixed mask, instead of sampling variable masks.
Evaluation Search Algorithm Unless otherwise noted, all results use the Binary Search algorithm as described in Section 3.2.
GT
Recon
GT
Recon
GT
Recon
GT
Recon
47% Tokens (962)
15% Takens (318)
78% Tokens (1595)
100% Tokens (2048)
=0s
Time
(=21s
Recon GT
100
Time
Figure ElasticTok adaptively encodes image and video to variable length outputs based on the complexity of the input data (using ElasticTok- VAE). The top rows shows examples of ElasticTokk on images. Below shows a video example with: (Top) Ground-truth video frames. (Middle) Reconstructed frames with varying token usage. (Bottom) The bottom section depicts how ElasticTok dynamically adjusts token allocation over time, with the percentage of tokens used correlating different content complexities in the video.
5

---

Published as a conference paper at ICLR 2025
Comparison on ImageNet
Comparison Video
100
Satisfy Recon Threshold (%)
Satisfy Recon Threshold (%)
80
80
3.5
2.4k
60
60
40
1.3x
20
20
40
60
80
100
20
40
60
80
100
% Tokens Used
% Tokens Used
0.001 (Baseline)
0.003 (Baseline)
0.015 (Baseline)
0.001 (Ours)
0.003 (Ours)
0.015 (Ours)
Figure 4 Performance comparison between baseline and Clostic'Tok-FSQ on ImageNet and Video. The y-axis shows the percentage of samples that satisfy the reconstruction threshold, while the x-axis represents the percentage of tokens used. (Left) On image, ElasticTok achieves a 3.5x and 1.3x efficiency boost different reconstruction thresholds. (Right) On video, ElasticTok shows 5x and 2.4x improvement over the baseline, maintaining superior performance while using fewer tokens. Figure 10 in Appendix D shows reference examples of reconstruction quality for an image at different thresholds.
4.2
MAIN RESULTS
Intuitively, the advantages of our elastic tokenization allow variable length codes that depend on the visual content of a given image or video. Figure shows some qualitative examples that demonstrate this, All images and videos provided the same MSE reconstruction quality (0.003) threshold to satisfy, and require different number of tokens depending on the complexity the images. Simpler images such the blue cushion require fewer tokens, while more complex images such as the painting recipe have details that require more tokens to properly encode. For the provided video example, we generally see larger token usage spikes in the event of a scene change or faster motion, such as when the phone animates a transition screen, when the finger scrolls up in the newsfeed. More qualitative examples can be found at: wargeworddmodel.giheee witthub.o//llstiice........
Figure 4 shows quantitative comparisons between method and different fixed token baselines trained at each token length. In order to show the benefits of elastic token representations, we compute quantitative metric that measures the percentage of images videos blocks in which model satisfies a specified reconstruction (MSE) threshold. As shown, our method can leverage its clastic representations achieve similar reconstruction satisfactory percentages compared to baselines while using 1.3x 5x fewer tokens, depending on the given threshold. Generally, more lax (0.015) reconstruction thresholds present larger performance improvements due more room to use fewer than max tokens. In contrast, more strict (0.001) thresholds usually almost always require all tokens. Different thresholds may be useful for different cases, where more lossy encodings can be used for simple VQA tasks, while more accurate reconstructions are useful for tasks such as image / video generation. Table 3 shows similar results for the continuous variant our model, compared against single baseline model fixed to 50% token usage.
Figure shows qualitative examples of progressive reconstruction quality as we increase the number of tokens. Different visual inputs will saturate reconstruction quality at different number of tokens. This flexibility allows us to use large number of tokens for very hard images (e.g inpuus with detailed fine-grained text), while saving tokens on inputs that are easier to reconstruct. Figure 6 shows the performance of our method as we increase the sequence length of the model performing better as sequence length increases due to being able leverage long context for more accurate reconstruction. However, performance degrades at the max context length (IM tokens), which we hypothesize is due to not enough training due relatively limited compute budget for that context size.

---

Published as a conference paper at ICLR 2025
GT
More Takens Used
100%
Loss progressively declines as more tokens are used Tok-FF)) The top row illustrates the impact text clarity, while the bottom row shows the effect on image sharpness. The graphs on the right quantify the reconstruction loss relative to token usage percentage, showing a rapid decline more tokens are consumed.
Figure 5
Performance on Increasing Sequence Length
Satisfy Recon Threshold (%)
75-
10
10a
12
Number of Frames In Context
Figure 6 Progressive performance increase with more framees (ElasticTokk TokiicTooFFOO) Performance improves with increasing sequence length, peaking around 100 frames before a slight decline. (Note the log scale for the x-axis)
Takeaway: ElasticTok offers significant advantages in image and video reconstruction by using variable-length codes that depend on content complexity. Qualitatively, simpler images require fewer tokens, while more complex ones need more tokens. Quantitatively, ElasticTok achieves similar reconstruction quality compared fixed token baselines while using up to fewer tokens
4.3 DOWNSTREAM TASKS
In order to evaluate the quality of our learned representations, we finetune a pretrained language model with visual tokens, and evaluate text-image and vidoo VQA tasks. We use OpenLLaMA-3B (Geng and Liu, 2023), and finetune with visual tokens from the continuous variant of model (max 2048 tokens). We finetune the entire model for 80M text-image pairs from COY0-700m (Byeon et al., 2022), and chat finetune with data from Chen et al. (2023). For video, we continue to train on WebVid (Bain et al., 2021) and finetune on Iideo-ChatGPT (Maaz et al., 2023) instruct data. Figure 7 shows evaluation results on GQA (Ainslic et al., 2023). POPE (Li et al., 2023b), MSVD-QA (Xu et al., 2017), and MSRVTT-QA (Xu et al., 2017) input inference tokens. Benchmark performance generally increases at the number inference tokens increases, suggesting the usefulness of our model as means for users to potentially be able to flexibly choose how many tokens use, compute accuracy trade-off, especially useful for users with more limited compute API call budgets. Lastly, Table shows that our VLM finetuned on adaptive tokens match the performance of VLM finetuned fixed token (50% tokens) baseline tokenizer. suggesting that there little to no loss in accuracy when switching to ElasticTok the tokenizer, with additional added adaptivity benefits.
varying number of
7

---

Published as a conference paper at ICLR 2025
Downstream Image/ Benchmarks
GQA
POPE
MSVD
MSRVTT
80
Ours
54%
82%
52%
37%
Accuracy (%)
70
* POPE
MSVD
Baseline
54%
82%
53%
37%
60
MSRVIT
50
Table 1 Comparison of our method with baseline image and video benchmarks (ElasticTokk- VAE). Our method can match the performance of the baseline trained a fixed number (100%) of tokens. However, baseline models are restricted require full pretraining a new model for every possible token length. whereas ElasticTok only requires single model to generalize to all token lengths.
40-
30
100
% Tokens Used
fixed token output, and
Figure 7 The accuracy and compute trade- off with varying percentages of tokens used (ElasticTokk-VAl VAE). This allows users to adjust the accuracy based on computational budget.
4.4 INFERENCE
Speed. Table 2 shows comparisons of the accuracy and inference speed (NFFs) for each of these methods. Error rate is computed as the relative error between the ground truth number of tokens used, and the number of tokens returned by each method. NFE (Number of Function Evaluations) is the number of forward passes that needed. In generally, methods closer to exhaustive search (Full / Binned Search) are more accurate, while methods that orders of magnitudes faster generally have much higher error around 10%. Users with less compute available for inference may want to use the faster inference methods at a slight cost to token encoding accuracy. Users with lot of compute can leverage more exhaustive approaches while retaining fast inference speed by computing search for token lengths as parallel batches.
Target Objective. One benefit of computing elastic tokens that we can switch to any target objective during inference, and can adaptively tokenize visual content based certain visual preferences, or aspects that users want to prioritize by allocating more tokens to. For example, Figure shows qualitative examples comparing running inference using MSE objective versus using CLIP loss (image-image image cosine distance) On average. thresholds are set so that both models as the same average number of tokens over the dataset, but OCR capabilities of CLIP allow it to show preference towards allocating more tokens in reconstructing the text (bottom images), while deprioritizing other images such as the ain-ggrined details in the bottom image. In general, any scalar function usable, and does not need to be differentiable (e.g., running OCR detection, and computing a more direct text reconstruction metric, or running segmentation and priortizing object clarity).
Takeaway: FlasticTok allows multiple adaptive inference methods: full search is the most accurate but slow, while faster methods like neural regression slightly sacrifice accuracy (5- (5-10% error) for speed. Users with less compute can opt for faster methods with slight accuracy loss. Additionally, inference objectives be adapted prioritize specific content, such as focusing text over other image features, allowing for flexible token allocation based on user preferences.
4.5 FREQUENCY ANALYSIS
We hypothesize that visual content that tends to have more fine-grained details, and high-frequncy features tends to use higher number of tokens. To investigate this, we run inference on 2k videos (5s, 32 blocks), and also compute an approximate frequency metric for each video block. For each frame in a video block, we convert it to grayscale, run a Sobel edge detection filter (horizontal vertical), and compute average gradient magnitudes. Figure 9 shows scatter plots and correlation for the single and multi block cases. For the single block case, the correlation between encoding length and amount of figgffrey detail is quite high (0.77). The multi block case is lower (0.67), which is most likely
8

---

Published as a conference paper at ICLR 2025
due to slight decorrelation from being able to leverage past frames (conditional encoding) in videos.
MSE
CLIP
Inference Method
Error Rate
NFEs
Full Search
0%
4096
Binned Search (100)
0.5%
100
Binary Search
7%
12
76% Takens
28% Tokens
Neural Regression
9%
Table 2 Comparison of inference methods showing their respective error rafes, number of function evaluations (NFEs) (ElasticTok-FFS Note that while Full and Binned Search are more computationally expensive, they could also bene- fit more from parallel function evaluations com- pute is available.
MIRACLE MONTH
MIRACLE MONTH
MIRACLE MONTH
30% Tokens
50% Tokens
Comparison of different loss func- tions for inference (ElasticTok-FSO), Inference with a CLIP model (cosine distance) prioritizes textual reconstruction (bottom) while deprioritiz- ing other detailed visual features (top).
Figure 8
Takeaway: Pearson correlation shows that ElasticTok adaptively allocate number of tokens for videos with more detailed visual content and high-frequeng, features require more tokens for encoding.
5 RELATED WORK
Adaptive Representation. There have been a lot of research studies on learning adaptive or ordered representations. Early work nested dropout (Rippel et al., 2014) proposes using dropout (Srivastava et al., 2014) in the context of autoencoders to learn ordered representations. index is sampled from a prior distribution, and all units with index larger than the sampled one dropped. Similar to our ElasticTook, this method imposes an inherent ordering on the representation dimensions. Units that are dropped less frequently encode more important information, while those that are dropped more often encode less critical details. Other works study adaptive architectures based on context size (Kim and Cho, 2020), slimmable neural networks that train network by sampling multiple sub-networks of different channel numbers simultaneously, where the weights are shared among different widths (Lee and Shin, 2018; Yu al., 2018), 8), adaptive width and depth in BERT (Hou et al., 2020), or dropping random layers during training to increase robustness to pruning (Fan et al., 2019; Huang et al., 2016). Dieleman et al. (2021) learns a variable-rate representation audio applied to speech. Transframer (Nash et al., 2022) uses vuriable-length DCT representations with transformers for general image and frame-level video prediction tasks. Matryoshka representation learning (Kusupati et al., 2022) explores adding nested substructures inside the standard Transformer block. Graves (2016) explores using recurrent neural networks to learn adaptive computation for learning tasks. However, none of the previous works consider learning elastic representations in autoencoders for long sequences like videos. Our work provides a scalable solution for representing videos with elastic representations.
this approach, an
Tokenization. Representing visual images with a set of discrete tokens has been extensively studied, such in VQVAE (Van Den Oord et al., 2017; Razavi et al., 2019) and VQGAN (Yu et al., 2021). Recent research has highlighted several shortcomings of traditional tokenization methods, including low vocabulary utilization. In response, alternative approaches like FSQ have been explored (Mentzer et al., 2024; Yu et al., 2023b). These discrete visual tokens facilitate integration with next-token prediction in language models and multimodal models (Yu et al., 2023b; Liu et al., 2024a; Xie et al., 2024; Wang et al., 2024). Parallel to this, other studies have investigated the use of continuous embeddings for representing images and videos (Lin et al., 2023; Liu et al., 2023; Zhu et al., 2023). Other research proposes next-scale prediction (Tian et al.. 2024), which employs hierarchical multi- scale modeling to first capture coarse details, followed by finer ones. However, this approach requires manually defined hierarchy and uses fixed number of tokens. Our work, while complementary

---

Published as a conference paper at ICLR 2025
to these efforts, focuses adaptive representation and can be directly applied to both discrete and continuous approaches.
Frequency Analysis [Image// Shart Video (Single Block))
Frequency Analysis (Long Video (Multi Block))
100
% Tokens Used
80
%6 Tokens Used
80-
20
Pearson = 0.771
Pearson r- 0.676
20
40
60
80
100
40
60
80
100
Frequency Magnitude
Frequency Magnitude
Figure 9 Comparison of token usage versus frequency magnitude in single-block and multi- block frequency analysis (ElasticTok-FSQ). Both scatter plots show a strong positive correlation between frequency magnitude and token usage in single-block setting a multi-block -lock setting. The red lines represent the linear regression fits for each case.
Compression. Adaptive learning for images and video have also been well-studied in the context of variable-rate compression. JPEG (Christopoulos et al., 2000) remains one of the most popular lossy image compression algorithms in the world, using a combination of DCT with quantization followed by entropy coding. For video, codecs such as H264 (Wiegand et al., 2003), H265 (Sze et al., 2014), VP9 (Mukherjee et al., 2015), and AV1 (Han et al., 2021) are popularly used compression algorithms. Prior works have additionally explored extending neural networks to learn more effective compressors. Theis et al. (2022) and Minnen et al. (2018) introduce methods that leverage CNNs to learn effective variable bit-rate compression algorithms over images competitive with JPEG. Li et al. (2023a), Mentzer et al. (2022), and Li et al. (2021) extend neural networks to learn how to effectively compress videos. While similar to our work in that these methods also learn adaptive encodings, these compression methods are more difficult to utilize for downstream training and generation tasks due to lack of direct compatibility. In contrast, our work builds upon existing tokenization strategies (VAE, FSQ) that have shown to work well for such downstream tasks, well take advantage of the adaptive representations to better utilize compute.
6 DISCUSSION AND CONCLUSION
In this work, we propose an elastic representation approach address the inefficiencies of traditional video encoding approaches through an adaptive encoding method that selectively encodes new information based on the context of previous frames. By dynamically allocating resources, it reduces computational costs while maintaining high-quality video representation. The proposed technique of dropping tokens at the end of each sequence allows the model to prioritize essential information, ensuring scalability and efficiency during inference. TlasticTok demonstrates strong performance in both video representation and downstream tasks. Lastly, we identify some limitations of our method, as well possible directions for future work.
Masking schemes: Our model currently slightly underperforms baselines at the tail ends of encoding lengths, which we hypothesize may be partially due to conflicting representations that the model needs to learn in each case (low frequency, global encodings vs high frequency, local encodings). Preliminary investigations showed that changing the way tokens are distributed (as opposed to our method of left-aligned tokens) improved reconstruction accuracy. Future work may also explore learnable encoding masks dynamically select which tokens to keep for each input.
Other temporal modalities: Although our primary focus was image and video, our method is generally mognsity-agnostic and can be extended to any temporal data, such audio and decision-making trajectories (e.g., state, action, reward).
Our focus was on leveraging temporal encoding to achieve more efficient tokenization while retaining high reconstructed visual quality, However, our method can be explored more broadly as a means of encouraging meaningful lossy encodings. For example, robotics models can learn text-conditioned visual encoders that capture only telk-rele information from an image. In other scenarios, different reconstruction objectives could prioritize facial textual reconstruction while ignoring background reconstruction accuracy.
Conditional encoding and different training objectives:
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We gratefully acknowledge the Google TPU Research Cloud (TRC) program for providing access to TPUs. Pieter Abbeel holds concurrent appointments as a Professor at UC Berkeley and as an Amazon Scholar. This paper describes work performed at UC Berkeley and is not associated with Amazon. We extend our thanks to Xinyang Geng, Jimmy Shi, Tim Rocktäschel, Joao Carreira, and Satinder Singh for their valuable discussions and insightful feedback on an earlier version of this paper, and to Doina Precup for valuable support of this work.
REFERENCES
Joshua Ainslie, James Lee- Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.1333 13245, 2023.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024. URL
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Sae- hoon Kim. Coyo-700m: Image-text pair dataset. petps://githot com coonkkaaobbaii
coyo-dataset, 2022.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4y: Improving large multi-mode models with better captions. arXiv preprint arXiv:2311.12799 2023.
Tong Chen, Haojie Liu, Qiu Shen, Tao Yue, Xun Cao, and Zhan Ma. Deepcoder: A deep neural network based video compression. In 2017 IEEE Visual Communications and Image Processing (VCIP), pages 1-4. IEEE, 2017.
Charilaos Christopoulos, Athanassios Skodras, and Touradj Ebrahimi. The jpeg2000 still image coding system: overview. IEEE transactions on consumer electronics, 46(4): 103-1 1127, 2000.
Sander Dieleman, Charlie Nash, Jesse Engel, and Karen Simonyan. Variable-rate discrete representa- tion learning. arXiv preprint arXhv:2103.06008 2021.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv22010.11929 2020.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for nigh-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth demand with structured dropout. arXiv preprint arXiv: 1909. 1556, 2019.
Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama. URL: https://github compdnpente--esvrehoqpnnee
2023.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair,
Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Alex Graves. Adaptive computation time for recurrent neural networks. arXiv: arXiv:1603.0800 1603.6833.8.... 2016.
arXiv preprint
11

---

Published as a conference paper at ICLR 2025
Jingning Han, Bohan Li, Debargha Mukherjee, Ching Ching-Han Chiang. Adrian Grange, Cheng Chen, Hui Su, Sarah Parker, Sai Deng. Urvang Joshi, et al. A technical overview of av Proceedings of the IEEE, (09(9):1435-146) 2021.
Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33: 9782-9793, 2020.
Gao Huang, Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages
Springer, 2016.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 694-711. Springer, 2016.
Gyuwan Kim and Kyunghyun Cho. Length-adaptive transformer: Train once with length drop, use anytime with search. arXiv preprint arXiv:2010.07003, 2020.
Diederik Kingma. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard- -Syddr,r Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning, Advances in Neural Information Processing Systems, 35:30233-30249, 2022.
Hankook Lee and Jinwoo Shin. Anytime neural prediction via slicing networks vertically. arXiv preprint 1rXXv:1807.02609 2018.
Jiahao Li, Bin Li, and Yan Lu. Deep contextual video compression. Advances in Neural Information Processing Systems, 34:181 1-155,,, 2021.
Jiahao Li, Bin Li, and Yan Lu. Neural video compression with diverse contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22616-22628 2023a.
Yifan Li, Yifan Du, Kun Zhou, Jimpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023b.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint M.Xiv:24022.82665 2024a.
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near- infinite context. International Conference on Learning Representaations//LLR)) 2024b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, arXiv preprint arXiv:2304.0844 2023.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint 23IV:22306.05224. 2023.
Fabian Mentzer, George Toderici, David Minnen, Sung-Jin Hwang, Sergi Caelles, Mario Lucic, and Eirikur Agustsson. Vct: A video compression transformer. arXiv preprint arX1v:2206.07307, 2022.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2300. 1.505,, 2023.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: VQ-VAE VAE made simple. In The Twelfth International Conference on Learning Representations, 2024. URL https:/ wpwpreveeeveeeeeeeeeeee net forum? www.:s8i--h---m 3.. hhttts::/oppnrrviii
12

---

Published as a conference paper at ICLR 2025
David Minnen, Johannes Ballé, and George D Toderici. Joint autoregressive and hierarchical priors for learned image compression. Advances in neural information processing systems, 31, 2018.
Debargha Mukherjee, Jingning Han, Jim Bankoski, Ronald Bultje, Adrian Grange, John Koleszar, Paul Wilkins, and Yaowu Xu. A technical overview of vp9-the the latest open-source video codec. SMPTE Motion Imaging Journal, 124(1):44-55 54, 2015.
Charlie Nash, Joao Carreira, Jacob Walker, lain Barr, Andrew Jaegle, Matcusz Malinowski, and Peter Battaglia. Transframer: Arbitrary frame prediction with generative models. arXiv preprint arXiv:2203.09453 09449, 2022.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
lain E Richardson. H. 264 and MPEG-4 video compression: video coding for next-generation multimedia. John Wiley & Sons, 2004.
Oren Rippel, Michael Gelbart, and Ryan Adams. Learning ordered representations with nested dropout. In International Conference on Machine Learning, pages 1746-1754. PMLR, 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-195 1958, 2014.
Vivienne Sze, Madhukar Budagavi, and Gary J Sullivan. High efficiency video coding (hevc). In Integrated circuit and systems, algorithmss and architectures, volume 39, page 40. Springer, 2014.
Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with compressive autoencoders. In International conference on learning representations, 2022.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-seale prediction. arXiv preprint arXiv:2404. 02905, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXivv2307.09288 2023.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang. Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409, /889, 2024.
Thomas Wiegand, Gary J Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the h. 264/ave video coding standard. IEEE Transactions on circuits and systems for video technology, 13(7): 560-576, 2003.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijic Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. /2528, 2024.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference Multimedia, pages 1645-1653, 2017.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104. 10157, 2021.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. arXiv preprint arXiv: 01882208928 2018.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arXiv:21 2021.
13

---

Published as a conference paper at ICLR 2025
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10459-10469, 2023a.
Lijun Yu, José Lezama, Nitesh Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
14

---

Published as a conference paper at ICLR 2025
A
MODEL CONFIGURATION
ElasticTok- VAE (Continuous)
ElasticTok-FSQ (Discrete)
Parameters
210M
210M
Frame Resolution
256 x 256
256 256
Block Size (Maxx)
2048 tokens (4 frames)
4096 tokens (4 frames)
Mmin
128 tokens
256 tokens
Max Number of Frames
1024
1024
Patch Size (T, H. W)
(2, 8, 8)
(1, 8)
Hidden Size
1024
1024
FFN Size
2048
2048
Encoder Layers
10
10
Decoder Layers
10
10
RoPE Theta
5000000
50000000
Max Sequence Length
512K
IM
FSQ Dims
N/A
8, 8, 5, (64k codes)
VAE Dim
8
N/A
KL Weight
le-8
N/A
B TRAINING DETAILS
The tables below showing trainin details for each of models. Our Long Video model is trained on mix of images and video (Batch Split), and each run (e.g. Long Video (2)) is initialized from the previous run (e.g. Long Video (1)). For the discrete (FSQ) model, each block has 4k tokens (256 blocks = 1M tokens), and the continuous (VAE) model has 2k tokens in each block (256 blocks 512K tokens).
ImageNet
Long Video (1)
Long Video (2)
Long Video (3)
Batch Size
256
256
256
256
# Blocks
1
1
2
4
Batch Split (Image/ / Video)
100%/0%
50%/50%
50%/50%
25%/755%
Total Iterations
200k
200k
80k
50k
Learning Rate
10
2 x 10TH
X 10
x 10TH 4
Optimizer
AdamW
AdamW
AdamW
AdamW
Weight Decay
1 x 10-4
1 x 10-4
x 10-4
x 10-4
Warmup Iterations
10k
10k
5k
2k
Long Video (4)
Long Video (5)
Long Video (6)
Long Video (7)
Batch Size
16
8
# Blocks
16
64
128
256
Batch Split (Image/ / Video)
25%/75%
25%/75%
25%/775%
25%
Total Iterations
10k
1k
500
200
Learning Rate
2 x 10
2 x 10-4
2 x 10-4
2 x 10 -4
Optimizer
AdamW
AdamW
AdamW
Adam
Weight Decay
10-4
1 x 10-4
1 1 10-4
1 10-4 25
Warmup Iterations
lk
200
100
C DATA CURATION DETAILS
Image Data We use COYO- 700M (Byeon et al., 2022) for our text-image data. We filter out images with less than 256 x 256 images. After accounting for stale links, left with roughly 350M
15

---

Published as a conference paper at ICLR 2025
text-image pairs. For better text correspondance, we further generate synthetic captions for each image using the mooondream2 model.
Video Data To collect our video data, we first scrape Common Crawl to collect video links. After deduplication, download each video and split then into individual scenes using PySceneDetect. For each scene we run various filtering metrics, such as OCR detection, NSFW scoring, motion score, and aesthetic scoring. Filtering metrics are aggregated over each scene in video, and used to select a final subset of 6M videos ranging from 4 seconds to 2 minutes in length.
D RECONSTRUCTION EXAMPLES BY MSE
We include the reconstructed images at different Mean Squared Error (MSE) loss thresholds in Figure 10 as a reference for how the images appear at various MSE thresholds.
GT
Recon With MSE
0.04
0.02
0.01
0.007
0.004
Comparison of reconstructed image quality at varying MSE thresholds. The ground truth (GT) image is displayed the left, followed by reconstructions different MSE thresholds (0.04, 0.02, 0.01, 0.007, and 0.004), showing progressive improvement fidelity as the threshold decreases.
Figure 10
E DETAILS ON NEURAL REGRESSION MODEL
To collect training data, we first run inference with the tokenizer for a pre-defined noise target of 0.003 on 100K sequences to get pairs of image/video and token lengths. To train the neural regression model, we augment the pretrained encoder model with alternating downsampling and transformer blocks. We downsample by factor of over sequence each time each time using simple size stride 4 ID conv, and each set of transformer blocks consists of 4 layers. The final output is pooled, and fed through a hidden layer MIP to output a single scalar per block. The regressor is optimized using MSE loss, and regresses normalized token length in the range [-1.
F FURTHER EXPERIMENTS
Table 3 Performance of ElasticTok- VAE videos. Values in the table show the percentage of reconstructed video blocks that satisfy a given reconstruction threshold. The baseline is a 50% fixed token baseline, and our method uses variable token lengths with an average of 50% token usage over the dataset.
Threshold
0.001
0.003
0.015
Baseline
40%
78%
99%
Ours
50%
88%
99%
Table 3 above shows the same metrics as Fig 4 but on our continuous variant ElasticTok- VAE model. Similar FlasticTok-FSQ. the VAE variant can reliably encode more videos satisfy given
16

---

Published as a conference paper at ICLR 2025
reconstruction threshold while using the same number of tokens (on average) FLOPs identical continuous variant baseline.
Table Quality baseline comparison with ElasticTok-FSQ on videos. Values shown in the table represent the average percentage of tokens used by each model to achieve the same worse-case MSE (compute MSE over each video block, and take the max MSE over the all blocks the video). Our method uses fewer tokens to acheive the same worse-case reconstruction as the baseline.
Threshold
0.001
0.003
0.015
Baseline
78%
57%
15%
Ours
75%
37%
9.4%
Table 4 above shows running a quality baseline comparison between our method and a baseline. For method, encode each evaluation video using given reconstruction threshold, and compute the average percentage of tokens per frame. For the baseline, we find the needed fixed tokens per frame such that the worse-case reconstruction MSE over the whole video matches the worst-case MSE of ElasticTokkss 's reconstruction. Intuitively, fixed token baselines perform worse due to allocated fixed number of tokens per video block, and must optimize for the worst-case scenario. In contrast, ElasticTok can save tokens when the videos transition to simpler to encode scenes while allocating more tokens more complex scenes.
Table 5
Ablation comparing with and without encoder conditioning on the masked tokens. Values in the table show the percentage of reconstructed video blocks that satisfy given reconstruc- tion threshold under the constraint that both models use the same average percentage of tokens.
Threshold
0.001
0.003
0.015
With Enc Cond
36 %
79.6%
98.5%
Without Enc Cond
33.1%
75.3%
98.4%
Table shows comparison of ElasticTok with and without encoder conditioning on the mask. ElasticTokk with conditioning slightly outperforms the model without conditioning. As such, it may be more efficient to train the model without encoder conditioning depending on expected size of the downstream models this present trade-off in terms of slightly better encoding quality at the cost of slower inference speed.
17

---

Published as a conference paper at ICLR 2025
G
MORE PROGRESSIVE RECONSTRUCTION EXAMPLES
2566
296
342
396
458
531
614
710
B22
951
1101
1274
1474
1 1706
1974
2284
2643
3059
3539
4096
Figure 11 Progressive reconstruction of a given images as you increasing the number of tokens (left-to-right in the encoding). Images are in raster scan order, and the number of tokens used the bottom right of each image (from 256 4096 interpolated with 20 values on log scale)
18

---

Published as a conference paper at ICLR 2025
256
296
342
396
458
531
614
710
822
951
1101
1274
1474
1706
1974
2284
2643
3059
3539
4096
Figure 12 Progressive reconstruction of given images you increasing the number of tokens (left-to-right in the encoding). Images are in raster scan order, and the number of tokens used the bottom right of each image (from 256 4096 interpolated with 20 values log scale)
19

---

Published as a conference paper at ICLR 2025
256
296
342
396
458
531
614
710
822
9511
1101
1274
1474
1706
1974
2284
2643
3059
3539
4096
Figure 13 Progressive reconstruction of given images you increasing the number of tokens (left-to-right in the encoding). Images are in raster scan order, and the number of tokens used the bottom right of each image (from 256 4096 interpolated with 20 values log scale)
20

---

Published as a conference paper at ICLR 2025
BY TMA I.M. OF REAVS OY MREAE HATHI 1AN
B EN OTIAL URE IL A OF DEEAY
UNI CECIINE
UMIT DEINE
DE TOR 1 EVALTT BUSS 248-
LST B M & UN UFBI 53 OF LOL
256
296
R42
396
458
TO EN DE DIG
TO ME COME. AE SALE IF
TIES SN DN CITWE
TB SL SA NE COTME
SHINE DRNNE ABI BEES A OTI AIDO BYYLE JOUN BNR.
ANI BELANE OF NER ONCCC MLWS SAN OA
UN WEMEE OF
AN EEUIE OF
NE ATTITT
NET AAD RPN IS JOWN EAR.
NEW HIRST SE LANI 448-
EELLIC AN-
531
614
710
822
95.1
EE SN. LEIING
BIBE SPAN CONEE
ES SINW DOIIE
FOLR SWE CRIME IT PEANEE TER VISIT EERUVE LANT EAT
DB SMATO DAIME AS 1S MUSE & NEN AFFL EAALIN LUW EAR-
1E DELANS NET PACCT JNN BIR
IS IS DER ARRIIT DULLI IDUR 14-
IN TIE NEJLEE OF
NEY MATT 42 DUTSS JOM EIR
1101
1274
1474
1 1706
1974
VIC MN CRINNG
TIEE SPANNE CRINE BF BS IF
WIR SNEVE DFAIE IN FI MOMEE OF AUFOT SAPLIUS JOCK EAR
THERE SEE NE CRIMG AT SEADEE OF NEF HOLST DULLI LANA E4-
F. DRMEE 13 IS SE F UN UFFIT SPDO6 1O BAR-
ANT BLANSE OF AW JPLLES LON BIG
GER AFRIIT BUBLINS 1033 EN-
2284
2643
3059
3539
4096
Figure 14 Progressive reconstruction of a given images as you increasing the number of tokens (left-to-right in the encoding). Images are in raster scan order, and the number of tokens used is in the bottom right of each image (from 256 4096 interpolated with 20 values on log scale)
21


Reviews:
[{"review_id": "Wiat8Y8cK5", "reviewer": "Reviewer_bdcm", "summary": "- The paper presents an approach to dynamically assign different number of tokens per different image/video depending on the input complexity.\n- During training, a random mask is applied to the encoded image tokens, and the unmasked tokens are used to reconstruct the image. - For training the video pipeline, similar random masking is applied at per block (stack of 4 images) levels, with the masking function being independent per block.\n- During inference, an iterative search is performed over a different number of tokens to be masked, and the max mask that satisfies the required constraint (reconstruction loss < target reconstruction loss) is selected.\n- Interesting results – analyzing the performance of image/video reconstruction quality and fine-tuned VLM performance at different token counts.", "questions": "My main concern are weakness points highlighting lack of clarity or details over the left-most token masking strategy. This raises concerns regarding efficacy of the trained model. But I feel the authors should be able to address this easily both in text and by visualizing the tokens.\n\nThe second concern is that independent per frame or per block masking strategy doesn't seem to be a scalable or efficient training approach. Intuitive understanding of the authors and maybe some statistics of the masking done during training could help alleviate these issues.\n\nRest, writing should be improved and this should be also fixable.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- The paper studies an interesting topic of adaptive tokenization for images and videos. As mentioned, such adaptive tokenization is crucial for long-context tasks.\n- While similar adaptive and compression architectures have been studied in the past, the application to auto-encoding of images and videos is a novel contribution. Related works in adaptive representations and compression are well covered.\n- Results include interesting analysis of token requirement for reconstruction and VLM tasks.", "weaknesses": "- The paper writing needs improvement –\n    - **A background section on Ring attention** (Liu et al., 2024b) paper should be the first paragraph of the method section since Ring attention is a big component of the proposed approach. Most important are the **details of the autoregressive video model**.\n    - Algorithm 1 and 2 says variable \"x\" (output of PatchifyRearrange) shape is B $\\times$ L $\\times$ D, what exactly is L ? From the next line ($N_b$ = the number of blocks), it seems that \"x\" should be of shape: B $\\times$ ($\\frac{L}{B}$) $\\times$ D i.e. number of tokens per block $\\times$ number of blocks $\\times$ feature dim. Please clarify this and **define all variables**.\n    - The captions of Tables 1 and 2 and the corresponding paragraph writing can be improved. It seems that table 1 is for image dataset and table 2 is for video dataset, but this is not mentioned anywhere. Please clarify and **fix these writing issues**.\n\n- Questions regarding masking strategy: Encoders / Tokenizers like VAE and VQGAN map tokens to patches and also assign 2D positional encodings to each patch. Thus, in some sense, each token (although captures enough receptive field via convolution or self-attention) is responsible for the reconstruction of the corresponding patch. Architectures like MAE mask random patches/tokens, performing a local task of inpainting (because at least some token is unmasked in the vicinity of a masked token). But the proposed approach always masks the leftmost tokens meaning that a fixed section of the image is always masked, irrespective of the content complexity of that particular section of the image. This raises questions on the choice of masking and doubts about the efficacy of the trained model. Visualizing what the leftmost tokens capture could help understand the issue. Details about position encodings would also be great.\n\n- Does the encoder take the sampled mask as input? While the text says so, the architecture figure shows that the mask is applied after full-length encoding.\n\n- For image reconstruction, searching for the mask at inference time makes sense, but for video level, searching for masks seems too computationally expensive. The only possible approach seems neural regressive in that case. More details on the neural regression approach would be great, including details on whether the neural masking approach is independent per frame, or it takes into account the context like the autoregressive video encoder.\n\n- For video training, independent per-block mask sampling will lead to a combinatorially large number of masking choices per video, with a majority of them being not very useful for training. For example, strong masking of the first frame, and no masking of the immediate next frame might be a waste if there is no motion from one frame to another. Thus, a next-frame masking strategy should also have better inductive biases in order to train larger models efficiently.\n- More details about the video dataset will be great. Will the exact video dataset be released to support the reproduction of the proposed approach?\n- It seems that all paper results are from the continuous model. Results or ablations from the quantized FSQ model would be appreciated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s9T9LXQ1PQ", "reviewer": "Reviewer_xU6C", "summary": "This paper proposes ElasticTok, a method for adaptive image/video tokenization that encodes visual data into variable-length tokens.  During training, the method drops a random number of tokens at the end of the token encoding. During inference, ElasticTok can use a variable length of tokens according to the need. Experiments show the advantages of the proposed method compared with the baseline.", "questions": "See Weakness. \n\nBesides, the applications of elastic token compression are still not so convincing to me. Are there more applications that clearly benefit from using elastic token compression than from using other compression methods?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposes an effective way to compress the number of tokens adaptively.\n\n2. The length of used tokens aligns well with human intuition: when the image is more complex, more tokens are used.\n\n3. The proposed method can effectively represent long videos with up to 2-5x fewer tokens.", "weaknesses": "1. Lack of comparison. The proposed method should be compared with other state-of-the-art token compression methods considering the compress rate, reconstruction quality, and downstream task performance.\n\n2. Loss of details in reconstruction. From the demos, the reconstruction quality is not very satisfactory. For example, the loading icon on the cellphone has totally disappeared/distorted. \n\n3. More analysis of the drop in image quality when using the elastic token compression is desired. Consider measuring the quality drop between the original image and the reconstructed image using PSNR/SSIM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OBQ8VacTQz", "reviewer": "Reviewer_xRJN", "summary": "This paper proposes a video representation learning approach that learns a compressed representation via an autoencoder. Randomly dropping the last few tokens of each frame during training encourages important information to stay in the first few tokens. This allows users to dynamically adjust the number of tokens used during inference, balancing the trade-off between faster inference and higher output quality based on the task's requirements.", "questions": "I think it might be helpful to mention which search method was used for each evaluation (e.g. in Figure 4). I was assuming exhaustive search was used for most if not all of them.\n\nI could not find details on how the regressor is fine-tuned. Does it see the context of the entire video? Does the regression loss change the encoder weights?\n\nExperiment suggestion: a heuristic-based regressor baseline (e.g. use the amount of motion as a proxy to determine the number of tokens needed)\n\nCitation suggestion: \"Adaptive Computation Time for Recurrent Neural Networks\" by Alex Graves", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Exploiting temporal coherence to learn a more compact video representation is a well-motivated problem.\nFigure 4 does indeed show that the dropout strategy can reduce the number of tokens as intended, although it requires a computationally exhaustive search to find the right number of tokens for each frame.", "weaknesses": "The \"neural regression\" evaluation (Table 4) does not seem very helpful. If I understand correctly, the error rate is the rate the regressor failed to correctly predict the optimal number of tokens found by the exhaustive search. It would have been good to also show the reconstruction accuracy. Also it would have been good to show the total inference time or FLOPS for each search method. It can be architecture-dependent but how much computation can be saved using fewer tokens in the decoder should be more carefully presented considering claims about efficient computation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "64Dj4a9uIq", "reviewer": "Reviewer_Qyem", "summary": "This paper presents a method to encode images or videos with variable length tokens which can then be decoded in a manner that minimizes a reconstruction loss (or any other target loss). This is achieved by dropping (or masking) some portion of tokens during training and inference. The method assumes a fixed maximum token length l and chooses to use the first k (k<=l) tokens for encoding purposes. In case of longer videos, a block processing mechanism is adapted where a different number of tokens are used for each block.", "questions": "For Fig.4 comparisons, do the authors first search for the ideal token length for each input for their method?\n\nIn Table3, if I understand correctly the baseline is trained with 50% of tokens and the method is trained with varying number of tokens. How is the inference done? Do both methods use 50% of tokens during inference?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea of using adaptive token lengths for image/video encoding (or other sequential data) is valid especially since there is a desire to generate/analyze longer sequences of data.\n- The paper is well written, it's easy to follow. The authors have also shared code.\n- Provided experiments are reasonable with each experiment evaluating a clear aspect of the proposed approach.", "weaknesses": "- For me the biggest limitation of the method seems like although the paper claims to \"adaptively\" select the number of tokens given the input, it seems the current proposed strategies (e.g., binary search, trying to match a threshold etc.) still require various computation and hence might limit the applicability in certain scenarios. For example, adaptive token lengths might potentially be most useful in long generation tasks but it's not clear yet how one can adaptively decide how many tokens to generate.\n- As the authors also mention in their limitations, the current masking strategy is pretty naive in the sense that it always uses the first k tokens. This may or may not be ideal depending on the input content.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)