ElasticTok: Adaptive Tokenization for Image and Video

## Abstract

Abstract Efficient video tokenization remains a key bottleneck in learning general purpose vision models that are capable of processing long video sequences. Prevailing approaches are restricted to encoding videos to a fixed number of tokens, where too few tokens will result in overly lossy encodings, and too many tokens will result in prohibitively long sequence lengths. In this work, we introduce ElasticTok, a method that conditions on prior frames to adaptively encode a frame into a variable number of tokens. To enable this in a computationally scalable way, we propose a masking technique that drops a random number of tokens at the end of each frames’s token encoding. During inference, ElasticTok can dynamically allocate tokens when needed – more complex data can leverage more tokens, while simpler data only needs a few tokens. Our empirical evaluations on images and video demonstrate the effectiveness of our approach in efficient token usage, paving the way for future development of more powerful multimodal models, world models, and agents. Video examples of using ElasticTok can be found on our website: largeworldmodel.github.io/elastictok

## 1 Introduction

Efficient video tokenization remains a key bottleneck in learning general purpose vision models that are capable of processing long video sequences – a crucial aspect towards developing intelligent agents for the visual world. Prevailing approaches (Van Den Oord et al., , Esser et al., , Yu et al., , Yan et al., ) are restricted to encoding videos to a fixed number of tokens irrespective of the visual content of the original video. As a result, being able to reliably encode with little information loss requires increasing the number of tokens to account for the worst case, highly complex visual inputs. However, this in turn causes many wasted tokens on simpler data, and leads to significant computational challenges for downstream training, where encoding a video or trajectory can require tens or even hundreds of millions of tokens, resulting in substantial computational costs and inefficiency (Liu et al., , Brooks et al., , Liu et al., ). On the other hand, using a too few number of tokens will result in lossy encodings, and fundamentally limit the capabilities of vision models when processing more complex visual data.

Intuitively, we would want do learn a model that adaptively encodes visual input in a data-dependent manner to variable length encodings, similar to how existing works in image and video compression (Richardson, , Li et al., , Christopoulos et al., , Chen et al., ) compress data to a varying number of bytes. Taking inspiration from this, we introduce ElasticTok, a method that conditions on prior frames to adaptively encode a frame into a variable number of tokens. To enable this in a computationally scalable way, we propose a masking technique that drops a random number of tokens at the end of each frame’s token encoding. During inference, ElasticTok can dynamically allocate tokens when needed – more complex data can leverage more tokens, while simpler data only needs a few tokens.

Our empirical evaluations demonstrate the effectiveness of ElasticTok, highlighted below.

- •
We show that ElasticTok can leverage adaptive tokenization to efficiently represent long videos with up to 2-5x fewer tokens.
- •
We show that ElasticTok enables flexibility in downstream vision-language tasks, allowing users to allocate tokens based on their compute budget.
- •
We show that ElasticTok allows leveraging different objectives during inference to adaptively trade off various semantic aspects of images.

Figure: Figure 2: ElasticTok adaptively encodes image and video to variable length outputs based on the complexity of the input data. Single block uses an Encoder-Decoder pipeline with a sampled latent mask. Multi-block extends this with a Block Causal Mask to handle longer video sequences.
Refer to caption: extracted/5917882/figure/elastic.png

## 2 Method

In this section, we present ElasticTok, a scalable adaptive visual tokenizer than can efficiently encode image and video. At a high level, we leverage a specialized random masking scheme when training any standard autoencoder to learn our elastic encodings. In the following sections, we provide a more detailed description of our model – first covering the simpler single block case (image / short video), and then moving on to the multi block case (longer video). An overview of our method is shown in Figure 2.

### 2.1 Training

First, we consider a single-block case in which we want to learn elastic codes over an image or a short video chunk (e.g., 4 frames). Let $x$ represent a sequence of input frames, where images are considered as $1$ frame videos. Our proposed method extends upon any existing variants of autoencoders (VQ-VAE (Van Den Oord et al., ), FSQ (Mentzer et al., ), VAE (Kingma, )) by incorporating additional masking of the encoder tokens.

For each training input $x$, we first uniformly sample the number of tokens to keep $\ell\sim U(\{M_{min},\dots,M_{max}\})$, where $M_{min}$ and $M_{max}$ define the supported range of encoding lengths. $M_{max}$ is generally set as $N$, the maximum number of tokens the encoder can output (e.g., 2048 or 4096), and $M_{min}$ a lower bound such as 128 or 256. We found this lower bound necessary as we found that sampling too few tokens could destabilize training. After sampling $\ell$, we then compute its mask $m$, defined as a binary vector of length $N$ with the first $\ell$ values set to $1$. Input $x$ and mask $m$ are then fed into the encoder $E(x,m)$ to compute $z$. $z$ is masked as $z_{m}=z\odot m$, and fed through the decoder $D(z_{m})$ to produce $\hat{x}$. Finally, our model optimizes a reconstruction loss, with potentially auxiliary losses depending on the exact autoencoder being used (e.g., KL for VAE or VQ loss for VQ-VAE). Although our method is generally architecture invariant, we opt to use ViTs (Dosovitskiy, ) as our encoder and decoders for simplicity.

We can extend our method to process longer video by breaking up video into blocks, with sizes defined by the number of frames used for the single block model (e.g 4 frames per block). The training process remains the same as the single block case, with two key differences: (1) masks for each multi block video sequence are sampled independently for each block, and (2) a block causal mask (block size $N$) is used in the encoder and decoder. The number of blocks is constrained only by the transformer’s context size. We utilize a prior long-context method (Liu et al., ) to train ElasticTok on longer videos. Architecturally, there are no added parameters which we found useful when progressively training our model from single block to multi block.

Exact details of our model’s forward pass are described in Algorithm 1.

### 2.2 Inference

There are two primary ways to use ElasticTok for inference – by specify a target encoding length, or a target reconstruction threshold.

Target encoding lengths. This method is simple to implement, as it only requires generating the correct mask and running the input through the encoder. However, although such inference is simple and efficient, it is difficult from a user’s perspective due to lack of knowledge of how many tokens to specify.

Target reconstruction threshold. This method presents a more intuitive inference approach that will automatically adaptively allocate tokens between different inputs based on a specified reconstruction threshold (e.g., target pixel-wise MSE loss). Visual content that is easy to reconstruct may require fewer tokens to meet the threshold whereas more complex inputs may require more tokens. Doing so requires a search process over different encoding lengths to determine lowest encoding length that still satisfies the given threshold.

We consider a few different search algorithms, detailed below:

- •
Full Search: Exhaustive search over every possible number of tokens lengths, and treat the result as ground truth. We use our discrete latent with max 4096 tokens.
- •
Binned Search (100): Similar to Full Search, but only evaluate on 100 uniformly spaced token lengths.
- •
Binary Search: We perform search using binary search, where the token length is the "array index", and the evaluated reconstruction loss is the "array value." Note that this assumes that reconstruction loss is monotonic with respect to token length, which is generally close to true, but not fully accurate, hence error in the result produced.
- •
Neural Regression: We collect 100k examples of paired (image / video, token length) data, and finetune our autoencoder to directly regress the number of tokens given an image or video.

Each search algorithm has its own trade-off between accuracy and efficiency – we further examine this relationship in Section 3.4.

In the multi block case, we iteratively perform the search process for each block in a block autoregressive manner, using caching similar to in language models. Algorithm 2 provides more details on the exact inference process.

Figure: Algorithm 1 Training

Figure: Algorithm 2 Inference

## 3 Experiments

In this section, we introduce our evaluation setup and present the results of pretraining ElasticTok to adaptively represent images and videos, as well as its performance on downstream tasks.

### 3.1 Experimental Setup

Model details.
We train 200M parameter discrete and continuous autoencoders on images and video sequences. For learning discrete representations, we use FSQ (Mentzer et al., ), while for continuous representations, we use a VAE (Kingma, ). Both models employ ViT encoders and decoders. Similar to prior works (Esser et al., ), we optimize both a pixel-wise reconstruction loss (MSE) and a perceptual loss (LPIPS) (Johnson et al., ). Notably, we do not use a GAN (Goodfellow et al., ), as we find it more difficult to stably train when progressively extending to much longer sequences (e.g. 1000+ frames), which we leave to future work, as it is not the main focus of this paper.

Training details.
All models are jointly trained on $256\times 256$ images and 24 FPS videos. Each video block consists of 4 frames, and joint training is conducted by treating images as single-frame videos, replicating each image 4 times to match the block size. Each block is encoded into 2048 or 4096 tokens by the continuous and discrete autoencoders, respectively. Following prior research in LWM (Liu et al., ) on scaling sequence length, we begin by training our models on single-block cases (images and short videos) and progressively extend the context length to cover up to 40-second videos (512K to 1M tokens). We train our long video models using v4-512 TPUs from Google Cloud on the COYO-700M image dataset and a custom dataset consisting of 6M videos scraped from the web. We additionally train an image-only model on ImageNet using v4-256 TPUs. Details for further training details can be find in Appendix B, and data details in Appendix C

Baselines.
We compare our proposed method against fixed token baselines trained at a varying token capacities. The architecture, training processes, and total FLOPs are exactly the same as our method with the only difference being restricted to one fixed mask, instead of sampling variable masks.

Figure: Figure 3: ElasticTok adaptively encodes image and video to variable length outputs based on the complexity of the input data. (Top) Ground-truth video frames. (Middle) Reconstructed frames with varying token usage. (Bottom) The bottom section depicts how ElasticTok dynamically adjusts token allocation over time, with the percentage of tokens used correlating to different content complexities in the video.
Refer to caption: x2.png

### 3.2 Main Results

Intuitively, the advantages of our elastic tokenization allow variable length codes that can depend on the visual content of a given image or video. Figure 3 shows some qualitative examples that demonstrate this. All images and videos are provided the same MSE reconstruction quality (0.003) threshold to satisfy, and require different number of tokens depending on the complexity the images. Simpler images such as the blue cushion require fewer tokens, while more complex images such as the painting or recipe have details that require more tokens to properly encode. For the provided video example, we generally see larger token usage spikes in the event of a scene change or faster motion, such as when the phone animates a transition screen, or when the finger scrolls up in the newsfeed. More qualitative examples can be found at: [largeworldmodel.github.io/elastictok](http://largeworldmodel.github.io/elastictok).

Figure 4 shows quantitative comparisons between our method and different fixed token baselines trained at each token length. In order to show the benefits of elastic token representations, we compute a quantitative metric that measures the percentage of images or videos blocks in which a model satisfies a specified reconstruction (MSE) threshold. As shown, our method can leverage its elastic representations to achieve similar reconstruction satisfactory percentages compared to baselines while using 1.3x - 5x fewer tokens, depending on the given threshold. Generally, more lax (0.015) reconstruction thresholds present larger performance improvements due to more room to use fewer than max tokens. In contrast, more strict (0.001) thresholds usually almost always require all tokens. Different thresholds may be useful for different cases, where more lossy encodings can be used for simple VQA tasks, while more accurate reconstructions are useful for tasks such as image / video generation. Table 3.2 shows similar results for the continuous variant of our model, compared against a single baseline model fixed to $50\%$ token usage.

Table 3.2 shows results comparing ElasticTok against a quality baseline. In which we first run inference using our model with a given threshold, and then compute which token length baseline is needed to match the quality (worst-case MSE over video blocks) of our model. Note that our model can use a different number of tokens each video block, where as the baseline uses the same number of tokens for all blocks. The adaptivity of our model helps match baseline model quality with lower average token usage.

Figure 5 shows qualitative examples of progressive reconstruction quality as we increase the number of tokens. Different visual inputs will saturate reconstruction quality at a different number of tokens. This flexibility allows us to use a large number of tokens for very hard images (e.g.inputs with detailed fine-grained text), while saving tokens on inputs that are easier to reconstruct.
Figure 6 shows the performance of our method as we increase the sequence length of the model – performing better as sequence length increases due to being able to leverage long context for more accurate reconstruction. However, performance degrades at the max context length (1M tokens), which we hypothesize is due to not enough training due to our relatively limited compute budget for that context size.

Figure: Figure 4: Performance comparison between baseline and ElasticTok on ImageNet and Video. The y-axis shows the percentage of samples that satisfy the reconstruction threshold, while the x-axis represents the percentage of tokens used. (Left) On image, ElasticTok achieves a 3.5x and 1.3x efficiency boost at different reconstruction thresholds. (Right) On video, ElasticTok shows a 5x and 2.4x improvement over the baseline, maintaining superior performance while using fewer tokens. Figure 10 in Appendix D shows reference examples of reconstruction quality for an image at different thresholds.
Refer to caption: x3.png

**Table 1: Performance comparison between baseline and ElasticTok across different reconstruction thresholds with the continuous token model. Percentage shows the proportion of video blocks that satisify the given reconstruction threshold. Our model can satisfy more video blocks.**
|  | ElasticTok-VAE (Continuous) | ElasticTok-FSQ (Discrete) |
| --- | --- | --- |
| Parameters | 210M | 210M |
| Frame Resolution | $256\times 256$ | $256\times 256$ |
| Block Size ($M_{max}$) | 2048 tokens (4 frames) | 4096 tokens (4 frames) |
| $M_{min}$ | 128 tokens | 256 tokens |
| Max Number of Frames | 1024 | 1024 |
| Patch Size (T, H, W) | (2, 8, 8) | (1, 8, 8) |
| Hidden Size | 1024 | 1024 |
| FFN Size | 2048 | 2048 |
| Encoder Layers | 10 | 10 |
| Decoder Layers | 10 | 10 |
| RoPE Theta | 5000000 | 50000000 |
| Max Sequence Length | 512K | 1M |
| FSQ Dims | N/A | 8, 8, 8, 5, 5, 5 (64k codes) |
| VAE Dim | 8 | N/A |
| KL Weight | 1e-8 | N/A |

### 3.3 Downstream Tasks

Figure: Figure 7: The accuracy and compute trade-off with varying percentages of tokens used. This allows users to adjust the accuracy based on computational budget.
Refer to caption: extracted/5917882/figure/downstream_plot.png

**Table 3: Comparison of our method with baseline on image and video benchmarks. Our method can match the performance of the baseline trained on a fixed number ($100\%$) of tokens. However, baseline models are restricted to a fixed token output, and require full pretraining a new model for every possible token length, whereas ElasticTok only requires a single model to generalize to all token lengths.**
|  | ElasticTok-VAE (Continuous) | ElasticTok-FSQ (Discrete) |
| --- | --- | --- |
| Parameters | 210M | 210M |
| Frame Resolution | $256\times 256$ | $256\times 256$ |
| Block Size ($M_{max}$) | 2048 tokens (4 frames) | 4096 tokens (4 frames) |
| $M_{min}$ | 128 tokens | 256 tokens |
| Max Number of Frames | 1024 | 1024 |
| Patch Size (T, H, W) | (2, 8, 8) | (1, 8, 8) |
| Hidden Size | 1024 | 1024 |
| FFN Size | 2048 | 2048 |
| Encoder Layers | 10 | 10 |
| Decoder Layers | 10 | 10 |
| RoPE Theta | 5000000 | 50000000 |
| Max Sequence Length | 512K | 1M |
| FSQ Dims | N/A | 8, 8, 8, 5, 5, 5 (64k codes) |
| VAE Dim | 8 | N/A |
| KL Weight | 1e-8 | N/A |

### 3.4 Inference

Speed.
Table 3.4 shows comparisons of the accuracy and inference speed (NFEs) for each of these methods. Error rate is computed as the relative error between the ground truth number of tokens used, and the number of tokens returned by each method. NFE (Number of Function Evaluations) is the number of forward passes that are needed. In generally, methods closer to exhaustive search (Full / Binned Search) are more accurate, while methods that orders of magnitudes faster generally have much higher error around $5-10\%$. Users with less compute available for inference may want to use the faster inference methods at a slight cost to token encoding accuracy. Users with a lot of compute can leverage more exhaustive approaches while retaining fast inference speed by computing search for token lengths as parallel batches.

Target Objective.
One benefit of computing elastic tokens is that we can switch to any target objective during inference, and can adaptively tokenize visual content based on certain visual preferences, or aspects that users want to prioritize by allocating more tokens to. For example, Figure 8 shows qualitative examples comparing running inference using an MSE objective versus using a CLIP loss (image-image cosine distance) On average, thresholds are set so that both models as the same average number of tokens over the dataset, but OCR capabilities of CLIP allow it to show preference towards allocating more tokens in reconstructing the text (bottom images), while deprioritizing other images such as the fine-grained details in the bottom image. In general, any scalar function is usable, and does not need to be differentiable (e.g., running OCR detection, and computing a more direct text reconstruction metric, or running segmentation and priortizing object clarity).

**Table 4: Comparison of inference methods showing their respective error rates, number of function evaluations (NFEs). Note that while Full and Binned Search are more computationally expensive, they could also benefit more from parallel function evaluations if compute is available.**
|  | ElasticTok-VAE (Continuous) | ElasticTok-FSQ (Discrete) |
| --- | --- | --- |
| Parameters | 210M | 210M |
| Frame Resolution | $256\times 256$ | $256\times 256$ |
| Block Size ($M_{max}$) | 2048 tokens (4 frames) | 4096 tokens (4 frames) |
| $M_{min}$ | 128 tokens | 256 tokens |
| Max Number of Frames | 1024 | 1024 |
| Patch Size (T, H, W) | (2, 8, 8) | (1, 8, 8) |
| Hidden Size | 1024 | 1024 |
| FFN Size | 2048 | 2048 |
| Encoder Layers | 10 | 10 |
| Decoder Layers | 10 | 10 |
| RoPE Theta | 5000000 | 50000000 |
| Max Sequence Length | 512K | 1M |
| FSQ Dims | N/A | 8, 8, 8, 5, 5, 5 (64k codes) |
| VAE Dim | 8 | N/A |
| KL Weight | 1e-8 | N/A |

### 3.5 Frequency Analysis

We hypothesize that visual content that tends to have more fine-grained details, and high-frequncy features tends to use a higher number of tokens. To investigate this, we run inference on 2k videos (5s, 32 blocks), and also compute an approximate frequency metric for each video block. For each frame in a video block, we convert it to grayscale, run a Sobel edge detection filter (horizontal / vertical), and compute average gradient magnitudes. Figure 9 shows scatter plots and correlation for the single and multi block cases. For the single block case, the correlation between encoding length and amount of high-frequency detail is quite high ($0.77$). The multi block case is lower ($0.67$), which is most likely due to slight decorrelation from being able to leverage past frames (conditional encoding) in videos.

Figure: Figure 9: Comparison of token usage versus frequency magnitude in single-block and multi-block frequency analysis. Both scatter plots show a strong positive correlation between frequency magnitude and token usage in a single-block setting a multi-block setting. The red lines represent the linear regression fits for each case.
Refer to caption: extracted/5917882/figure/freq_plot.png

## 4 Related Work

Adaptive Representation.
There have been a lot of research studies on learning adaptive or ordered representations. Early work on nested dropout (Rippel et al., ) proposes using dropout (Srivastava et al., ) in the context of autoencoders to learn ordered representations. In this approach, an index is sampled from a prior distribution, and all units with an index larger than the sampled one are dropped. Similar to our ElasticTok, this method imposes an inherent ordering on the representation dimensions. Units that are dropped less frequently encode more important information, while those that are dropped more often encode less critical details. Other works study adaptive architectures based on context size (Kim and Cho, ), slimmable neural networks that train a network by sampling multiple sub-networks of different channel numbers simultaneously, where the weights are shared among different widths (Lee and Shin, , Yu et al., ), adaptive width and depth in BERT (Hou et al., ), or dropping random layers during training to increase robustness to pruning (Fan et al., , Huang et al., ). Dieleman et al. () learns a variable-rate representation on audio applied to speech. Transframer (Nash et al., ) uses variable-length DCT representations with transformers for general image and frame-level video prediction tasks. Matryoshka representation learning (Kusupati et al., ) explores adding nested substructures inside the standard Transformer block. However, none of the previous works consider learning elastic representations in autoencoders for long sequences like videos. Our work provides a scalable solution for representing videos with elastic representations.

Tokenization.
Representing visual images with a set of discrete tokens has been extensively studied, such as in VQVAE (Van Den Oord et al., , Razavi et al., ) and VQGAN (Yu et al., ). Recent research has highlighted several shortcomings of traditional tokenization methods, including low vocabulary utilization. In response, alternative approaches like FSQ have been explored (Mentzer et al., , Yu et al., ). These discrete visual tokens facilitate integration with next-token prediction in language models and multimodal models (Yu et al., , Liu et al., , Xie et al., , Wang et al., ). Parallel to this, other studies have investigated the use of continuous embeddings for representing images and videos (Lin et al., , Liu et al., , Zhu et al., ). Other research proposes next-scale prediction (Tian et al., ), which employs hierarchical multi-scale modeling to first capture coarse details, followed by finer ones. However, this approach requires a manually defined hierarchy and uses a fixed number of tokens. Our work, while complementary to these efforts, focuses on adaptive representation and can be directly applied to both discrete and continuous approaches.

Compression. Adaptive learning for images and video have also been well-studied in the context of variable-rate compression. JPEG (Christopoulos et al., ) remains one of the most popular lossy image compression algorithms in the world, using a combination of DCT with quantization followed by entropy coding. For video, codecs such as H264 (Wiegand et al., ), H265 (Sze et al., ), VP9 (Mukherjee et al., ), and AV1 (Han et al., ) are popularly used compression algorithms. Prior works have additionally explored extending neural networks to learn more effective compressors. Theis et al. () and Minnen et al. () introduce methods that leverage CNNs to learn effective variable bit-rate compression algorithms over images competitive with JPEG. Li et al. (), Mentzer et al. (), and Li et al. () extend neural networks to learn how to effectively compress videos. While similar to our work in that these methods also learn adaptive encodings, these compression methods are more difficult to utilize for downstream training and generation tasks due to lack of direct compatibility.
In contrast, our work builds upon existing tokenization strategies (VAE, FSQ) that have shown to work well for such downstream tasks, as well as take advantage of the adaptive representations to better utilize compute.

## 5 Discussion and Conclusion

In this work, we propose an elastic representation approach address the inefficiencies of traditional video encoding approaches by introducing an adaptive encoding method that selectively encodes new information based on the context of previous frames. By dynamically allocating resources, it reduces computational costs while maintaining high-quality video representation. The proposed technique of dropping tokens at the end of each sequence allows the model to prioritize essential information, ensuring scalability and efficiency during inference.
Our ElasticTok method demonstrates strong performance in both video representation and downstream tasks.

Lastly, we identify some limitations of our method, as well as possible directions for future work.

- •
Masking schemes: Our model currently slightly underperforms baselines at the tail ends of encoding lengths, which we hypothesize may be partially due to conflicting representations that the model needs to learn in each case (low frequency, global encodings vs high frequency, local encodings). Preliminary investigations showed that changing the way tokens are distributed (as opposed to our method of left-aligned tokens) improved reconstruction accuracy. Future work may also explore learnable encoding masks to dynamically select which tokens to keep for each input.
- •
Other temporal modalities: Although our primary focus was on image and video, our method is generally modality-agnostic and can be extended to any temporal data, such as audio and decision-making trajectories (e.g., state, action, reward).
- •
Conditional encoding and different training objectives: Our focus was on leveraging temporal encoding to achieve more efficient tokenization while retaining high reconstructed visual quality. However, our method can be explored more broadly as a means of encouraging meaningful lossy encodings. For example, robotics models can learn text-conditioned visual encoders that capture only task-relevant information from an image. In other scenarios, different reconstruction objectives could prioritize facial or textual reconstruction while ignoring background reconstruction accuracy.

## Acknowledgments

We gratefully acknowledge the Google TPU Research Cloud (TRC) program for providing access to TPUs. We extend our thanks to Xinyang Geng, Jimmy Shi, Tim Rocktäschel, Joao Carreira, and Satinder Singh for their valuable discussions and insightful feedback on an earlier version of this paper, and to Doina Precup for valuable support of this work. Pieter Abbeel holds concurrent appointments as a Professor at UC Berkeley and as an Amazon Scholar. This paper describes work performed at UC Berkeley and is not associated with Amazon.

## Appendix A Model configuration

|  | ElasticTok-VAE (Continuous) | ElasticTok-FSQ (Discrete) |
| --- | --- | --- |
| Parameters | 210M | 210M |
| Frame Resolution | $256\times 256$ | $256\times 256$ |
| Block Size ($M_{max}$) | 2048 tokens (4 frames) | 4096 tokens (4 frames) |
| $M_{min}$ | 128 tokens | 256 tokens |
| Max Number of Frames | 1024 | 1024 |
| Patch Size (T, H, W) | (2, 8, 8) | (1, 8, 8) |
| Hidden Size | 1024 | 1024 |
| FFN Size | 2048 | 2048 |
| Encoder Layers | 10 | 10 |
| Decoder Layers | 10 | 10 |
| RoPE Theta | 5000000 | 50000000 |
| Max Sequence Length | 512K | 1M |
| FSQ Dims | N/A | 8, 8, 8, 5, 5, 5 (64k codes) |
| VAE Dim | 8 | N/A |
| KL Weight | 1e-8 | N/A |

## Appendix B Training Details

The tables below showing trainin details for each of our models. Our Long Video model is trained on a mix of images and video (Batch Split), and each run (e.g. Long Video (2)) is initialized from the previous run (e.g. Long Video (1)). For the discrete (FSQ) model, each block has 4k tokens (256 blocks = 1M tokens), and the continuous (VAE) model has 2k tokens in each block (256 blocks = 512K tokens).

|  | ImageNet | Long Video (1) | Long Video (2) | Long Video (3) |
| --- | --- | --- | --- | --- |
| Batch Size | $256$ | $256$ | $256$ | $256$ |
| # Blocks | $1$ | $1$ | $2$ | $4$ |
| Batch Split (Image / Video) | $100\%/0\%$ | $50\%/50\%$ | $50\%/50\%$ | $25\%/75\%$ |
| Total Iterations | 200k | 200k | 80k | 50k |
| Learning Rate | $2\times 10^{-4}$ | $2\times 10^{-4}$ | $2\times 10^{-4}$ | $2\times 10^{-4}$ |
| Optimizer | AdamW | AdamW | AdamW | AdamW |
| Weight Decay | $1\times 10^{-4}$ | $1\times 10^{-4}$ | $1\times 10^{-4}$ | $1\times 10^{-4}$ |
| Warmup Iterations | 10k | 10k | 5k | 2k |

|  | Long Video (4) | Long Video (5) | Long Video (6) | Long Video (7) |
| --- | --- | --- | --- | --- |
| Batch Size | $64$ | $16$ | $8$ | $4$ |
| # Blocks | $16$ | $64$ | $128$ | $256$ |
| Batch Split (Image / Video) | $25\%/75\%$ | $25\%/75\%$ | $25\%/75\%$ | $25\%/75\%$ |
| Total Iterations | 10k | 1k | 500 | 200 |
| Learning Rate | $2\times 10^{-4}$ | $2\times 10^{-4}$ | $2\times 10^{-4}$ | $2\times 10^{-4}$ |
| Optimizer | AdamW | AdamW | AdamW | AdamW |
| Weight Decay | $1\times 10^{-4}$ | $1\times 10^{-4}$ | $1\times 10^{-4}$ | $1\times 10^{-4}$ |
| Warmup Iterations | 1k | 200 | 100 | 25 |

## Appendix C Data curation details

#### Image Data

We use COYO-700M (Byeon et al., ) for our text-image data. We filter out images with less than $256\times 256$ images. After accounting for stale links, we are left with roughly 350M text-image pairs. For better text correspondance, we further generate synthetic captions for each image using the [mooondream2](https://huggingface.co/vikhyatk/moondream2) model.

#### Video Data

To collect our video data, we first scrape Common Crawl to collect video links. After deduplication, we download each video and split then into individual scenes using [PySceneDetect](https://www.scenedetect.com/). For each scene we run various filtering metrics, such as OCR detection, NSFW scoring, motion score, and aesthetic scoring. Filtering metrics are aggregated over each scene in a video, and used to select a final subset of 6M videos ranging from 4 seconds to 2 minutes in length.

## Appendix D Reconstruction Examples by MSE

We include the reconstructed images at different Mean Squared Error (MSE) loss thresholds in Figure 10 as a reference for how the images appear at various MSE thresholds.

Figure: Figure 10: Comparison of reconstructed image quality at varying MSE thresholds. The ground truth (GT) image is displayed on the left, followed by reconstructions at different MSE thresholds (0.04, 0.02, 0.01, 0.007, and 0.004), showing progressive improvement in fidelity as the threshold decreases.
Refer to caption: x6.png

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 1
questions: My main concern are weakness points highlighting lack of clarity or details over the left-most token masking strategy. This raises concerns regarding efficacy of the trained model. But I feel the authors should be able to address this easily both in text and by visualizing the tokens.

The second concern is that independent per frame or per block masking strategy doesn't seem to be a scalable or efficient training approach. Intuitive understanding of the authors and maybe some statistics of the masking done during training could help alleviate these issues.

Rest, writing should be improved and this should be also fixable.
soundness: 3
strengths: - The paper studies an interesting topic of adaptive tokenization for images and videos. As mentioned, such adaptive tokenization is crucial for long-context tasks.
- While similar adaptive and compression architectures have been studied in the past, the application to auto-encoding of images and videos is a novel contribution. Related works in adaptive representations and compression are well covered.
- Results include interesting analysis of token requirement for reconstruction and VLM tasks.
summary: - The paper presents an approach to dynamically assign different number of tokens per different image/video depending on the input complexity.
- During training, a random mask is applied to the encoded image tokens, and the unmasked tokens are used to reconstruct the image. - For training the video pipeline, similar random masking is applied at per block (stack of 4 images) levels, with the masking function being independent per block.
- During inference, an iterative search is performed over a different number of tokens to be masked, and the max mask that satisfies the required constraint (reconstruction loss < target reconstruction loss) is selected.
- Interesting results – analyzing the performance of image/video reconstruction quality and fine-tuned VLM performance at different token counts.
weaknesses: - The paper writing needs improvement –
    - **A background section on Ring attention** (Liu et al., 2024b) paper should be the first paragraph of the method section since Ring attention is a big component of the proposed approach. Most important are the **details of the autoregressive video model**.
    - Algorithm 1 and 2 says variable "x" (output of PatchifyRearrange) shape is B $\times$ L $\times$ D, what exactly is L ? From the next line ($N_b$ = the number of blocks), it seems that "x" should be of shape: B $\times$ ($\frac{L}{B}$) $\times$ D i.e. number of tokens per block $\times$ number of blocks $\times$ feature dim. Please clarify this and **define all variables**.
    - The captions of Tables 1 and 2 and the corresponding paragraph writing can be improved. It seems that table 1 is for image dataset and table 2 is for video dataset, but this is not mentioned anywhere. Please clarify and **fix these writing issues**.

- Questions regarding masking strategy: Encoders / Tokenizers like VAE and VQGAN map tokens to patches and also assign 2D positional encodings to each patch. Thus, in some sense, each token (although captures enough receptive field via convolution or self-attention) is responsible for the reconstruction of the corresponding patch. Architectures like MAE mask random patches/tokens, performing a local task of inpainting (because at least some token is unmasked in the vicinity of a masked token). But the proposed approach always masks the leftmost tokens meaning that a fixed section of the image is always masked, irrespective of the content complexity of that particular section of the image. This raises questions on the choice of masking and doubts about the efficacy of the trained model. Visualizing what the leftmost tokens capture could help understand the issue. Details about position encodings would also be great.

- Does the encoder take the sampled mask as input? While the text says so, the architecture figure shows that the mask is applied after full-length encoding.

- For image reconstruction, searching for the mask at inference time makes sense, but for video level, searching for masks seems too computationally expensive. The only possible approach seems neural regressive in that case. More details on the neural regression approach would be great, including details on whether the neural masking approach is independent per frame, or it takes into account the context like the autoregressive video encoder.

- For video training, independent per-block mask sampling will lead to a combinatorially large number of masking choices per video, with a majority of them being not very useful for training. For example, strong masking of the first frame, and no masking of the immediate next frame might be a waste if there is no motion from one frame to another. Thus, a next-frame masking strategy should also have better inductive biases in order to train larger models efficiently.
- More details about the video dataset will be great. Will the exact video dataset be released to support the reproduction of the proposed approach?
- It seems that all paper results are from the continuous model. Results or ablations from the quantized FSQ model would be appreciated.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: See Weakness. 

Besides, the applications of elastic token compression are still not so convincing to me. Are there more applications that clearly benefit from using elastic token compression than from using other compression methods?
soundness: 3
strengths: 1. This paper proposes an effective way to compress the number of tokens adaptively.

2. The length of used tokens aligns well with human intuition: when the image is more complex, more tokens are used.

3. The proposed method can effectively represent long videos with up to 2-5x fewer tokens.
summary: This paper proposes ElasticTok, a method for adaptive image/video tokenization that encodes visual data into variable-length tokens.  During training, the method drops a random number of tokens at the end of the token encoding. During inference, ElasticTok can use a variable length of tokens according to the need. Experiments show the advantages of the proposed method compared with the baseline.
weaknesses: 1. Lack of comparison. The proposed method should be compared with other state-of-the-art token compression methods considering the compress rate, reconstruction quality, and downstream task performance.

2. Loss of details in reconstruction. From the demos, the reconstruction quality is not very satisfactory. For example, the loading icon on the cellphone has totally disappeared/distorted. 

3. More analysis of the drop in image quality when using the elastic token compression is desired. Consider measuring the quality drop between the original image and the reconstructed image using PSNR/SSIM.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: I think it might be helpful to mention which search method was used for each evaluation (e.g. in Figure 4). I was assuming exhaustive search was used for most if not all of them.

I could not find details on how the regressor is fine-tuned. Does it see the context of the entire video? Does the regression loss change the encoder weights?

Experiment suggestion: a heuristic-based regressor baseline (e.g. use the amount of motion as a proxy to determine the number of tokens needed)

Citation suggestion: "Adaptive Computation Time for Recurrent Neural Networks" by Alex Graves
soundness: 2
strengths: Exploiting temporal coherence to learn a more compact video representation is a well-motivated problem.
Figure 4 does indeed show that the dropout strategy can reduce the number of tokens as intended, although it requires a computationally exhaustive search to find the right number of tokens for each frame.
summary: This paper proposes a video representation learning approach that learns a compressed representation via an autoencoder. Randomly dropping the last few tokens of each frame during training encourages important information to stay in the first few tokens. This allows users to dynamically adjust the number of tokens used during inference, balancing the trade-off between faster inference and higher output quality based on the task's requirements.
weaknesses: The "neural regression" evaluation (Table 4) does not seem very helpful. If I understand correctly, the error rate is the rate the regressor failed to correctly predict the optimal number of tokens found by the exhaustive search. It would have been good to also show the reconstruction accuracy. Also it would have been good to show the total inference time or FLOPS for each search method. It can be architecture-dependent but how much computation can be saved using fewer tokens in the decoder should be more carefully presented considering claims about efficient computation.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: For Fig.4 comparisons, do the authors first search for the ideal token length for each input for their method?

In Table3, if I understand correctly the baseline is trained with 50% of tokens and the method is trained with varying number of tokens. How is the inference done? Do both methods use 50% of tokens during inference?
soundness: 3
strengths: - The idea of using adaptive token lengths for image/video encoding (or other sequential data) is valid especially since there is a desire to generate/analyze longer sequences of data.
- The paper is well written, it's easy to follow. The authors have also shared code.
- Provided experiments are reasonable with each experiment evaluating a clear aspect of the proposed approach.
summary: This paper presents a method to encode images or videos with variable length tokens which can then be decoded in a manner that minimizes a reconstruction loss (or any other target loss). This is achieved by dropping (or masking) some portion of tokens during training and inference. The method assumes a fixed maximum token length l and chooses to use the first k (k<=l) tokens for encoding purposes. In case of longer videos, a block processing mechanism is adapted where a different number of tokens are used for each block.
weaknesses: - For me the biggest limitation of the method seems like although the paper claims to "adaptively" select the number of tokens given the input, it seems the current proposed strategies (e.g., binary search, trying to match a threshold etc.) still require various computation and hence might limit the applicability in certain scenarios. For example, adaptive token lengths might potentially be most useful in long generation tasks but it's not clear yet how one can adaptively decide how many tokens to generate.
- As the authors also mention in their limitations, the current masking strategy is pretty naive in the sense that it always uses the first k tokens. This may or may not be ideal depending on the input content.

Meta-review: The paper proposes an adaptive approach to encoding frames into tokens to substantially reduce the number of tokens to be processed in downstream tasks. 

Reviewers appreciated the novel application of adaptive tokenization to auto-encoding images and videos. They considered the token reduction effective and contributions sufficiently validated through experiments.

Initially, reviewers were concerned about the effectiveness of the encoding due to the need of a search for the right number of tokens at inference. This, and further concerns about the masking strategy, image quality, and implementation details were clarified during the discussion phase. Although all concerns were clarified and reviewers raised their scores correspondingly, no reviewer championed the paper and the overall score remains marginally above acceptance.

Decision: Accept (Poster)