FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
Authors: Roman Bachmann
Jesse Allardice
David Mizrahi
Enrico Fini
Oğuzhan Fatih Kar
Elmira Amirloo
Alaaeldin El-Nouby
Amir Zamir
Afshin Dehghan
Venue: ICML 2025 poster (2025)
Keywords: tokenization
variable rate compression
image generation
computer vision
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We present FlexTok, an image tokenizer capable of resampling images into 1D token sequences of flexible length.

Abstract: We introduce FlexTok, a tokenizer that projects 2D images into variable-length, ordered 1D token sequences. For example, a 256x256 image can be resampled into anywhere from 1 to 256 discrete tokens, hierarchically and semantically compressing its information. By training a rectified flow model as the decoder and using nested dropout, FlexTok produces plausible reconstructions regardless of the chosen token sequence length. We evaluate our approach in an autoregressive generation setting using a simple GPT-style Transformer. On ImageNet, this approach achieves an FID<2 across 8 to 128 tokens, outperforming TiTok and matching state-of-the-art methods with far fewer tokens. We further extend the model to support to text-conditioned image generation and examine how FlexTok relates to traditional 2D tokenization. A key finding is that FlexTok enables next-token prediction to describe images in a coarse-to-fine "visual vocabulary", and that the number of tokens to generate depends on the complexity of the generation task.

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
Roman Bachmann 12 Jesse Allardice David Mizrahi" 1 Enrico Fini' Oguzhan Fatih Kar ?
Elmira Amirloo
Alaaeldin El-Nouby' Amir Zamir Afshin Dehghan
https: epfl ch
loken
2 takens
tokens
tokens
16 tokens
32 takens
64 tokens
128 tokens
256 tokens
golden
Raster 2D-grid tokens with autoregressive decoding
golden retriever
FlexTok
Flexbble longth 10 taken
sequences with autoregressive decoding
Figure 1. Comparison of partial sequence generation: Raster-scan 2D-grid tokenizer vs. FlexTok. FlexTok resamples images into ID sequence of discrete tokens of flexible length, describing images in coarse-to-line manner. When training autoregressive (AR) models on F FlexTok token sequences, the class conditioning (here "golden retriever") can be satisfied by generating as few as 8 tokens, whereas AR models trained 2D tokenizer grids (here, LlamaGen (Sun al., 2024)) need always generate all tokens, complexity of the condition or image.
the
Abstract
TiTok and matching state-of-the-and methods with far fewer tokens. We further extend the model support to text-conditioned image generation and examine how F FlexTok relates to traditional 2D tokenization. A key finding is that FlexTok enables next-token prediction to describe images coarse-to-fine "visual vocabulary", and that the number of tokens generate depends the complexity of the generation task.
We introduce FlexTok, a tokenizer that projects 2D images into variable-length, ordered ID token sequences. For example, 256 256 image can be resampled into anywhere from to 256 discrete tokens, hierarchically and semantically compress- ing its information. By training a rectified flow model as the decoder and using nested dropout, FlexTok produces plausible reconstructions re- gardless of the chosen token sequence length. We evaluate approach in autoregressive gen- eration setting using simple GPT-style Trans- former. On ImageNet, this approach achieves an FID across 8 to 128 tokens, outperforming
1. Introduction
Image generation has advanced significantly in both quality and inference speed. Tokenization plays a crucial role in reducing the computational cost of training generative mod- els (Van Den Oord et al., 2017; Esser al., 2021). Recently, autoregressive (AR) image generation has shown competi- tive performance when scaled to billions of parameters (Sun et al., 2024).
"Equal contribution 'Apple "Swiss Federal Institute of Technology Lausanne (EPFL). Correspondence to: Roman Bachmann Froman.bachmann @ ch?>, Jesse Allardice <jal- lardico@apple.com.. David Mizrahi <d.mizrahi @ com>.
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Generative models such as diffusion (Rombach et al., 2022: Peebles & Xie, 2023), masked (Chang et al., 2022; 2023),

---

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
token
2 tokens
tokens
tokens
16 tokens
32 tokens
64 tokens
128 tokens
256 tokens
Original RGB
Figure 2. Reconstruction examples using FlexTok d18-d28 trained on DFN. Notice how most of the images' semantic and geometric content is captured by fewer than 16 tokens. The first tokens already capture the high-level semantic concepts (c.g., gray bird, people in colorful garments, mountain scene, yellow flower), while more tokens are required to reconstruct more intricate scene details (e.g.. position and clothing of every person, brushstroke placement, etc.). To showcase out-of-distribttion reconstruction, we generated the original images using Midjourney v6.1 (Midjourney, 2024).
and autoregressive (Chen et al., 2020; Yu et al., 2022) mod- els traditionally operate on 2D grids of continuous dis- crete tokens. These representations maintain strong spatial alignment with the original pixel patches (Esser et al., 2021; Mentzer et al., 2023), but this means that the representation size is proportional to the image size, rather than depen- dent on the complexity of the image. Recent work (Yu al., 2024a) has demonstrated the benefits of 1D tokeniza- tion schemes for improving computational efficiency while maintaining competitive quality. Similar to existing 2D grid tokenization schemes, these ID token sequences are fixed- length, regardless of the underlying image complexity. In other words, no matter whether an image simply depicts a single object or a busy scene with intricate details, it is encoded into the same number of tokens. In turn, condi- tional image generator trained to predict these tokens must always produce the full set of tokens, no matter the semantic complexity of the condition.
Our contributions are follows:
Flexible-length tokenization: We present a 1D tokeniza- tion approach that can compress images into any from to 256 tokens. Through a combination of nested dropout and causal attention masking, our tokens are naturally or- dered from coarse to fine details, enabling high-quality re- construction even with very few tokens while providing progressively more detailed representations as more tokens are used.
A new "visual vocabulary" to describe images: Unlike classical 2D tokenizers which describe image content at each x.y location, F FlexTok uses hicrarchical approach in which high-level aspects such as semantic and geometric concepts naturally emerge to be ordered first, while subse- quent tokens add finer details. This allows a coarse condi- tioning to be fulfilled with relatively few predicted tokens, while more detailed condition requires generating a larger number of tokens. For example, shown in Figure 7, ImageNet- 1k classes can be generated with as few as 8 to- kens. In contrast, text conditions, which may involve more complex and varied prompts, can benefit from predicting up to 256 tokens.
In this paper, we present F FlexTok, a novel variable-length 1D tokenizer that is able to encode images into an ordered and content-dependent sequence of tokens. As visualized in Figure earlier tokens capture the most high-level semantic and geometric information, while additional tokens progres- sively add finer details. The variable-length sequences, i.e. truncated subsequences of length 1, 2, 4, .... 256, can be decoded into plausible images using an end-to-end trained rectified flow decoder. Figure 1 contrasts the rigid raster- sean ordering of traditional 2D tokenizers (top) with our hierarchical ID approach that enables coarse-to-fine genera- tion (bottom).
End-to-end trained rectified flow decoder: To enable high-quality image reconstruction across varying token lengths, we incorporate a rectified flow objective into our tokenizer decoder. This proves essential for maintaining reconstruction quality even at extreme compression rates.
2

---

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
2. Related Work and Background
Although effective producing 1D token sequences, TīTok requires two-stage training process and represents im- ages using fixed number of tokens. Compared to TiTok, F FlexTok ok shows a superior reconstruction and generation quality (see Figure 4 and Table 1). To address the issue of the token sequence length depending entirely on the image height and width instead of its complexity, a range of con- current works have proposed adaptive tokenization methods. While F PlexTok can compress images into little single token, TlasticTok's (Yan et al., 2024) FSQ tokenizer is limited to a minimum of 256 tokens and ALIT (Dug- gal et al., 2024) to 32 tokens. Generation is a key use case of tokenizers, and unlike ElasticTokk, ALIT, or One-D- Piece (Miwa et al., 2025), we show in Section 5 that training on F FlexTok tokens can produce strong generative models. Compared our focus on AR models, ViLex (Wang et al., 2024b) and CAT (Shen et al., 2025) focus more on learning continous tokens to use in diffusion models. We discuss further differences with respect to FlexTok in detail in Appendix D.
The primary role of an image tokenizer in generation tasks is to create compressed latent representations of images. Gen- erative models need learn a distribution over their outputs, but doing so directly in continuous, high-dimensional spaces (e.g,, raw image pixels) is challenging. Compression helps by mapping images to a more compact latent space that removes imperceptible details (Rombach et al., 2022), while discretization enables the generative model to output per- token categorical distributions that can be sampled from.
Vector-quantized autoencoders (VQ-VAES) (Van Den Oord et al., 2017; Razavi et al., 2019; Esser et al., 2021) have become standard framework for learning these discrete representations. VQ-VAE models operate through three core components: (1) an encoder Enc that maps input images X E HHxW' x3 to D-channel latent embeddings Z = Ene(X) E RixwxD (usually < H and w < W), (2) quantizer Quant that maps these continuous latents to discrete codes from learned codebook, and (3) decoder Dec that reconstructs the image X Dec(Quant( (Z)) This approach has proven versatile, finding applications in image (Chang et al., 2022; 2023; Li et al., 2022), au- dio (Baevski et al., 2019), and video generation (Villegas et al., 2022; Hu et al., 2023; Kondratyuk et al., 2023), novel view synthesis (Yan et al., 2021), and large-scale multimodal pretraining (Lu et al., 2022: 2023; Mizrahi et al., 2023; Bach- mann et al., 2024; Chameleon, 2024; Wang et al., 2024a).
3. Method
FlexTok is an autoencoder with discrete 1D bottleneck, see Figure 3 for overview. A ViT encoder maps 2D image patches into ID sequence using register tokens (Darcet et al., 2023; Yu et al., 2024a). The registers are discretized using FSQ (Mentzer et al., 2023), and then used as condi- tioning for a rectified flow model tasked with reconstructing the image. We use causal attention among register tokens followed by nested dropout (Rippel et al., 2014) to induce ordering in the bottleneck representation. Paired with rectified flow decoder, this design enables the model to decode any nested subset of tokens into plausible images.
Various improvements to this framework have been pro- posed, exploring different architectures (Yu et al., 2021),
objective functions (Esser et al., 2021; Hu et al., 2023), codebook structures (Zhang et al., 2023; Yu et al., 2023),
and the use of diffusion models as decoders (Shi et al., 2022; Mizrahi et al., 2023; Xu et al., 2024; Zhao et al., 2024). A key advancement is finite scalar quantization (FSQ) (Mentzer et al., 2023), which replaces the learned codebook with a projection to a mmal--dimensiona latent space and quantizes values using fixed bins along each di- mension. This approach maintains reconstruction quality while being simpler to implement and train.
3.1. ID tokenization with a rectified flow decoder
Register encoder and discrete bottleneck. Similar to TiTok (Yu et al., 2024a), the encoder uses register tokens to resample the patched 2D VAE latents into a 1D sequence of discrete tokens. Specifically, we concatenate VAE la- tent patches with a set of learnable register tokens that act as read-write storage for the encoder. After encoding, the register tokens act as the bottleneck representation for the autoencoder, while the encoded patches are discarded. The register tokens are quantized into discrete tokens using FSQ. Instead of pixels, we operate on latent representations of a VAE-GAN, similar to SDXL's VAE (Podell et al., 2023), to abstract away the perceptual compression from this in- vestigation. Using the VAE-GAN latent space enables us to greatly simplify our design space and reduce computa- tional requirements as training generative models on pixels is expensive (Rombach et al., 2022).
Instead of maintaining a 2D spatial structure in the latent space, TiTok (Yu et al., 2024a) creates 1D) representations using learned register tokens R E k (Darcet et al., 2023). During encoding, these K register tokens are con- catenated with image patch embeddings P e R for processing in a VIT encoder. The encoder output re- tains only the register token embeddings as Zin. which capture the image content in compact sequence. For re- construction, the image patches are replaced with a grid of h x learnable mask tokens M E Rx These mask tokens, guided by the quantized register embeddings, are transformed by the decoder into the reconstructed image: x = Dec( Quant( ZiD) @ M).
3

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
Stage 1: FlexTok training
Stage 2: AR training
Predicted flow
Quantization (FSQ)
ViT Encoder with Registers
1
2
3
4
Rectified Flow Decoder
1
2
4
Nested dropout
"a blue porsche
AR Transformer
1
X
1
M
M
VAE latents
Registers
Noised latents
S
3
Figure 3. FlexTok overview. Stage 1: P'lexTok resamples 2D VAE latents to ID sequence of discrete tokens using ViT with registers (Darcet et al., 2023). The -SS-ntuatized bottleneck (Mentzer et al., 2023) representation used to condition a rectified flow model that decodes and reconstructs the original images. F FlexTok leams ordered token sequences of flexible length by applying nested dropout (Rippel et al., 2014) on the register tokens. Stage 2: We train class- and text-conditional autoregressive Transformers to predict ID token sequences in coarse-to-fine manner. As tokens predicted, the generated image becomes specifie, encoding high-level concepts first (e.g.. presence of a car) followed by finer details (e.g., car shape, brand, color).
Rectified flow decoder. The decoder's purpose to gen- perceptually plausible images conditioned on the pressed latent representations. When the bottleneck size is small, training decoder with only reconstruction loss can result in blurry reconstructions. To mitigate this, F FlexTok uses a rectified flow model that conditioned on the quan- tized register tokens Quant I)) by concatenating them with noised VAE latent patches X; (1 X) + le, where Xo are the clean VAE latents, E [0, ran- dom time step, and (O, 1) is random sample from the noise distribution. The decoder's objective is to pre- dict the flow U = E Xo given the partially noised VAE latents and the encoded register tokens. We minimize the rectified flow loss LRF = U - (E Xo)ll? given the predicted flow U- Dec (araeQt ZID) @ X.). We find that adding a REPA (Yu al., 2024b) inductive bias loss CREPA between intermediate decoder layer and DINOv2- L (Oquab et al., 2023) features significantly improves vergence time and downstream generation performance (see Table 3 and Fig. 12). The total FlexTok loss we optimize CriexTok = CRF + AREPA CREPA, with AREPA = 1. See Section 4 and Appendix C.2 for more implementation de- tails.
mance deteriorates when the compression degree is high, see Figure 4 and Appendix K.3 for visual comparisons FlexIos Since we target AR generation with F FlexTok, we additionally introduce nested left-to-right ordering structure that naturally aligns with next-token prediction. We propose two techniques to introduce a 1D ordering and variable length into the token sequences.
Nested dropout. We train FlexTok to produce or- dered representation by randomly dropping the encoded reg- ister tokens in a nested manner during training (Rippel et al., 2014; Kusupati et al., 2022; Cai et al., 2024). Specifically, given register token sequence of length K, we randomly sample the number of units to keep Kteep E {I, K} and remove the K Kteep last tokens by masking them By training the tokenizer in this manner, the encoder learns to compress the image content into the register tokens in an ordered manner, while the rectified flow decoder learns to reconstruct images given the variably-sized token sequences. As we show in Figure 5, this design enables FlexTok to capture the most important aspects of images in very few tokens. Simplistic images require few tokens to be faithfully compressed, while complex ones require longer token se- quences. We note here that this ordering is not handcrafted and emerges purely from performing nested dropout reg- ister tokens, and computing rectified flow and REPA losses. We mainly ablate two strategies of dropping tokens a nested manner. In the first, we sample the number of tokens to keep uniformly, as described above, In the second variant, uniformly sample them from exponentially increasing set, e.g. Kknep [1, 2, 4, 8, 16, ... K}. The latter variant addresses an issue with uniform nested dropout, where the last register tokens are passed to the decoder very rarely, meaning they are effectively trained for only a small fraction of gradient updates.
3.2. Learning ordered 1D token sequences of flexible length
ID tokenizers like TYTok (Yu al., 2024a) require training different tokenizers for each desired number of register to- kens. As shown in the concurrent work ALIT (Duggal et al., 2024), fixed token sequence lengths do not take into account the inherent complexity of an image. Simple images can be compressed into as few as 32 tokens, while more com- plex ones require more tokens to faithfully reconstruct them. Neither TYTok nor ALIT demonstrate flexible tokenization below 32 tokens, and their image reconstruction perfor-

---

FlexTok:
Resampling Images into 1D Token Sequences of Flexible Length
Causal attention masks. Orthogonal to the use of nested dropout, adding causal attention mask the encoded registers enforces causal dependency structure between them (Ge et al., 2023). In this setting, the encoded image patches can all attend to each other but not to the registers, the register tokens can attend to all the patches, but the i-th register token may only attend to the j-th register token if i 2 j. In addition, the use of causal masks enables users to more efficiently encode images if they know ahead of time that they only want to keep Kkeep <
flow model that receives noised VAE latent patches and the (randomly masked) registers as input, and is tasked to predict the flow. We use adaLN-zero (Peebles & Xie, 2023) to condition the patches and registers separately on the current timestep, and REPA (Yu et al., 2024b) with DINOv2-L (Oquab et al., 2023) features to speed up con- vergence. We use 2x2 patchification in both the F FlexTok encoder and decoder. which combined with the VAE's 8x downsampling yields total 16x downsampling from pix- els to patch tokens. All models are trained at resolution of 256x256 pixels. The encoder and decoder dimensions w are parameterized using their respective depths d, us- ing a fixed aspect ratio of 64, i.e. w = 64 d. We train three F FlexTok versions with different encoder and de- coder sizes (separated by a hyphen), d12-d12, dl 8-d18 and d18 -d28 after sweeping optimal hyperparameters at small scale using μP (Yang al., 2022). Depending on the downstream use case, we train FlexTox models on ImageNet-IK lk (Russakovsky et al., 2014) for subsequent class conditional generation, and on DEN (Fang et al., 2023) for subsequent text-to-image modeling. See Appendix C.2 for further F FlexTok implementation and training details.
K tokens.
3.3. Autoregressive image generation
To evaluate different design choices of F FxexTok and com- pare to relevant baselines, we measure both reconstruction and generation performance. To that end, we train autore- gressive Transformers to perform class-conditional genera- tion on ImageNet- 1k (Russakovsky et al., 2014), following LlamaGen (Sun et al., 2024), and text-to-image generation on DFN-2B (Fang et al., 2023).
4. Implementation
We break down the implementation into three distinct stages. In Stage 0, we train VAE models (Rombach et al., 2022) with continuous latents to perceptually compress images into 2D token grids. In Stage 1, we then train F exTok tokenizers resample these continuous 2D token grids into discrete ID token sequences of flexible length. Finally, in Stage 2, we train autoregressive class-to-image and text-to- image models to evaluate the effectiveness of FlexTok in generative tasks.
Stage 2: AR Transformer training. Our autoregressive Transformer follows LlamaGen's 's .lama-inspired architec- ture (Sun et al., 2024; Touvron et al., 2023), using pre- normalization with RMSNorm (Zhang & Sennrich, 2019) and a SwiGLU feedforward (Shazeer, 2020). Since our tokens lack a 2D grid structure, we use learned absolute positional embeddings instead of 2D RoPE (Su et al., 2024).
For class conditioning, we add learned class embedding to an [SOI] token (Tian et al., 2024) and concatenate it with the image token sequence. The AR model, ranging from 49M to 1.3B parameters, predicts the token sequence from the F FlexTok tokenizer. To enable comparisons with LlamaGen and TiTok, we train without j.P.
Stage O: VAE training. The rectified flow decoder (see Section 3.1) is a key design element of FlexTok that en- ables decoding arbitrary token subsequences. However, training such models directly in pixel space is computation- ally expensive (Rombach et al., 2022). The goal of this stage is therefore to facilitate Stage 1 modeling by perceptu- ally compressing images into more compact representations. We follow the architecture of the SDXL VAE (Podell et al., 2023), and train versions with 4, 8, and 16 channels on the DFN dataset (Fang et al., 2023). All subsequent experiments use the 16 channel VAE with a downsampling factor of 8. Please see Appendix C.I for more VAE training details, and Table 2 for ablations on the number of latent channels.
For text-conditioned generation, our AR decoder cross- attends to text embeddings from FLAN-T5-XL (Chung et al., 2024), projected to the model dimension via an MLP (Chen et al., 2023). We scale these text-conditioned AR models up to 3B parameters, using jPP to maintain consistent behavior across scales.
Following standard practice. we employ conditioning dropout during training to enable classifier-free guidance at inference. For text-conditioned models, this involves randomly replacing text inputs with an empty string. For additional implementation and training details, see Ap- pendix C.3.
Stage 1: FlexTok training. The FlexTok architec- ture consists of a Transformer encoder and decoder using maximum of 256 registers tokens. After applying a 6- dimensional FSQ (Mentzer et al., 2023) bottleneck with levels [8, B, 8, 5, 5, 5] (for an effective vocab- ulary size of 64 000), the encoded registers are randomly truncated using nested dropout. The decoder is a rectified
5. Experiments
In this section, we experimentally evaluate the reconstruc- tion performance of FlexTok across different numbers of
5

---

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
token
2 tokens
4 tokens
tokens
16 tokens
32 takens
64 tokens
128 takens
256 tokens
TiTok L-32,
TiTok B-64,
X
X
x
x
X
Original RGB
TiTok S-128
ALIT (semi-large)
X
X
X
FlexTok d18-d88
Figure 4. Image reconstruction comparison between three different TYTok (Yu et al., 2024a) models, ALIT (Duggal et al., 2024), and FlexTok. Compared to other ID tokenizers, FlexTok is able to tokenize images a highly semantic and ordered manner, all the way down to single token, and all in single model. For more visual comparisons, Appendices K.I K.3.
0.25
0.5
6
0.20
DreamSim
0.4
MAE
0.15
0.3
0.10
0.2
2-
0.05
32 Number of tokens
8
16
64
128
256
1
2
4
16
64
128
256
32 Number of tokens
8
16
64
128
256
Number of tokens
FlexTok d12-d12
FlexTok d18 d18-d18
FlexTok d18-d28
Figure 5. FlexTok rate-distortion tradcoff. We show ImageNet-Ik reconstruction metrics for three different FlexTok sizes. The tokens used, the closer the reconstructions get to the original RGB images. Scaling the tokenizer size significantly improves reconstruction FID, but is not as crucial in terms of MAE and DreamSim score. For cach of the different F FlexTok model sizes we use the optimal inference hyperparameters detailed in Appendix F, We show additional reconstruction metries in Table 8.
tokens, assess its applicability for class-condiiional and text- conditional image generation, and compare it relevant baselines. We show that FlexTok effectively com- press images into ID sequences of flexible length, enabling a novel "visual vocabulary" where images can be specified and generated a coarse-to-fine manner.
using the first few tokens capture high-level semantic fea- tures. As more tokens used, both the alignment with the original image becomes more fine- ine-grained and the image- wise reconstruction metrics (MAE and DreamSim) improve rapidly.
Please see Appendices K. to K.3 for additional reconstruc- tion examples and comparisons. For linear probing experi- ments on the token sequences, see Appendix B.
5.1. Flexible-length tokenization
We demonstrate F FlexTok's 's variable-rate tokenization ca- pability by evaluating its reconstruction performance on nested token sequences of different lengths. We perform comparisons using FlexTok models trained on ImageNet- 1k. testing them on 256x256 pixel crops from the valida- tion set (Russakovsky et al., 2014). In Figure 5 we mea- sure reconstruction metrics (rFID, MAE, DreamSim) for KESep E 2, 4, 16....,256] tokens. FlexTok is able to generate plausible images, as measured through the rFID against the ImageNet- Ik validation set, with as little as a single token.
We find the following properties particularly noteworthy: (1) By performing nested dropout on the registers, hierarchy emerges in which high-level concepts are ordered first. (2) Through training F FlexTok with a Rectified Flow decoder, any token subsequence can be decoded into a plausible image. (3) The token sequences specify distribution over images that gets more and more specific with more tokens, see Figures 34 and 35 for visual examples.
5.2. Coarse-to-fine generation with increasing specificity
As shown in Section 5.1, FlexTok compresses images into ordered token sequences. This naturally leads to explore the implications of predicting these sequences for autoregressive image generation. By training class- and text-
As shown in Figure with FlexTok d18 d18-d28 model trained on DFN, and in Figure 4 with a F FlexTok d1 8-d28 model trained on ImageNet-Ik, reconstructions
6

---

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
taken
tokens
tokens
a tokens
16 tokens
32 tokens
64 tokens
128 tokens
258 tokens
golden retriever (207)
volcano (980)
tlamingo (130)
'A board topped bread, meat and vegetables."
'A blue Porsche 356 parked in front of a yellow brick wall"
'a corgi's head depicted as an explosion nebula"
Figure 6. Image generation examples with varying numbers of tokens. Images generated with both class (top rows) and text conditioning (bottom rows) demonstrate that F -eeeTok--aaee based models achieve high quality all the way down to a single token, and all within single model. The conditioning alignment strengthens more tokens are generated. For example with the prompt "a corgi's head depicted as an explosion of a nebula", the first two tokens capture the high-level concept a artistic depiction of a dog, while adding tokens adds further details such the dog breed and the nebula background. For visualizations, Appendices K.4 and K.S.
Generation Specificity
Generation Quality
Top-1 on INIK
10
20
80
CLIPScore (COCO)
9FID (INIK train 50k)
9FID (COC val 30k)
30
70
10
60
26
50
2
8
16
32
64 128 256
4
32 64 128256
1
2
8
16
32
64 128 256
Number of tokens
Number of tokens
Number of tokens
FlexTok + 1.3B AR (INIK)
FlexTok AR (DFN)
2D Grid Tok 3B (DFN)
Figure 7. Conditioning alignment and generation quality vs. number tokens. Left: For class- conditional generation with a 1.3B AR model, compute DINOv2-L (Oquab al., 2023) top-1 accuracy generated images conditioned ImageNet-IK class labels. Center: For text-conditional generation with a 3B AR model, we show CLIPScore relative to input prompts from the COCO 30k validation set (Lin et al., 2014), using CLIP base model. Right: We measure class-conditional gFID ImageNet-IK. 1k', and text-conditional gFID on COCO. The AR models use guidance scales of 1.0 (no guidance) and 2.5, respectively. We follow the optimal inference parameters described in Appendices and G.
Note. For evaluation the conditiodd image generation results, follow common practice measuring the generation FID (gFID) of 50K generated samples relative to the reference statistics calculated over the entire training split of the ImageNet- -kk dataset (Dhariwal & Nichol, 2021).

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
1 token
4 tokens
26 tokens
64 tokens
250 tokens
CLIPScore
Different random seeds
Rectified Flow Decoder
Simple prompe
64
128
256
'A single red apple white background"
Number of tokens
Transformer
Mean Pairwise
DreamSim
Different random seeds
Recti fied Flow Decoder
Detailed prompt
0.0
8
16
32
64
128 256
Number of tokens
"graffit racket
AR Transformer
A single red apple on a white background
brick wall"
graffiti of a rocket ship on
Figure Image generation with simple and detailed prompts. Images generated with FLexTok-basee models show that the number of tokens needed to fulfill the conditioning depends on prompt complexity. For simple prompt, the desired image is achieved with few as 4-16 tokens (as measured by CLIPScore), and semantic variation between different decoded images (as measured by pairwise DreamSim scores) vanishes quickly. In contrast, detailed prompt requires the full 256-token sequence to fully meet the conditioning and shows greater variation at lower token counts as the FlcxTok rectified flow decoder compensates for missing details. For cach prompt, the FlexTok tokens generated just once using the AR Transformer and then decoded with 10 random seeds in the rectified flow decoder.
(NIK train 50K)
29.0
Training loss
8.5
CLIPScore [NN1 val.
8.0
28.0
7.5
27.5
7.0
gFID
27.0
6.5
26.5
100K
200K
300K
322
04
1.20
256
16 32
64
128
256
Training iterations
Number of tokens
Number of tokens
AR Size:
49M
85M
201M
393M
679M
1.33B
Figure 9. Class-conditioned AR model scaling. We show training loss, gFID and image generation CLIPScore values for the class- conditional models with the FlexTok d 8-188 tokenizer. calculate the CLIPScore using the text label of the classes from the ImageNet-IB IK validation set, and do not use classifier- free guidance for the AR Transformer (CFG scale 1.0). We show additional generation metries for the 1.33B AR model Table
conditional models, we find that F exeookk token sequences act as a "visual vocabulary". allowing autoregressive mod- els to describe images with increasing levels specificity. Unlike conventional autoregressive models that generate images in a fixed raster-scan order on token grids, our approach enables progressive refinement of image details. We also observe a clear relationship between conditioning complexity and token requirements. Simple conditions, like ImageNet class labels, can be fulfilled with as few as 16 tokens, while more complex ones, like open-ended text prompts, benefit from generating up to 256 tokens.
results in Figure 7 confirm this trend, showing that align- ment between the conditioning signal and the generated images improves with higher token counts. We measure alignment using DINOv2-L (Oquab et al., 2023) classifi- cation accuracy for class conditioning and CLIPScore for text conditioning. Notably, text-image alignment contin- ues to improve as additional tokens are generated, whereas classification accuracy tapers off after the first few tokens and plateaus around 32 (Figures 7 and 9). Furthermore, we observe that generation quality remains consistent across token sequence lengths, as measured by gFID, which we attribute the strength of our rectified flow decoder.
In Figure 6, we show that images generated by class- or text- conditional models become increasingly specific their conditioning more tokens are produced. Our quantitative
Similarly, Figure 8 shows that the number of tokens needed to generate prompt-aligned images varies significantly based
8

---

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
on prompt complexity. Simple prompts like "a red apple" can produce satisfactory results with just 4 to 16 tokens, while detailed prompts like "graffiti of rocket ship" ben- efit from using the full 256-token sequence. When fewer tokens are used, the model can still produce realistic im- ages, but at the cost of greater variation between different random seeds. This variation decreases much more quickly for simple prompts than for detailed ones as token count increases, suggesting a fundamental relationship between prompt complexity and token requirements.
Table I. System-level comparison on ImageNet-1k class- conditional generation. FlexTok generates variable length token sequences from to 256 tokens long. The FlexTok tok- enizers are combined with 1.33B parameter AR Transformers for class-conditioned image generation. For each F FlexTok model we follow the optimal inference parameters and use guidance in the AR model, as described in Appendices and G. 1 indicates F FlexTck results for sequence of 32 tokens. The "-re"
llssiffer-free
suffix indicates the use of rejection sampling.
Tokenizer
# tokens
Codebook size
rFID
gFID 4
Taming VQ-GAN-re
16x
16384
4.98
5.20
5.3. Scaling autoregressive model size
MaskGIT VQ-GAN
16x16
1024
2.28
6.06
Open MAGVIT-v2
16x16
262
1.17
2.33
As shown in Figure 7, using our largest autoregressive mod- els, the alignment between condition and generations gen- erally improves with more predicted tokens. In this sec- tion, we investigate the scaling behavior of autoregressive class-conditional models trained on FlexTok d18-d28 tokens, focusing on how model size impacts image-caption alignment and image fidelity. In Figure 9, we show that increasing the AR model size consistently improves the measured training loss. However, for the prediction of the first few (1-8) tokens, the generation FID (gFID) values for the decoded images are effectively independent of the autoregressive model size, indicating that these short initial parts of the sequences are easily learned even by small mod- els. In contrast, for long sequences (> 128 tokens) the task becomes more challenging, and performance scales strongly with model size for both gFID and CLIPScore.
LlamaGen
16x16
16384
2.19
3.06
TiTok-L
32
4096
2.21
2.77
TYTok-B
64
4096
1.70
2.48
TYTokSS
128
4096
1.71
1.97
Fl TxeTTkk d12-d12
1-256
64000
4.20t
3.831
FlexIok d18-d18
1-256
64000
1.611
2.021
FlexIok d18-d28
1-256
64000
1.451
1.861
128 tokens (Figure 7). A full 256-token sequence with FlexTok yields better text-image alignment (CLIPScore) than the 2D grid tokenizer, despite slight gFID regression compared the 2D grid-based model.
6. Discussion & Conclusion
In this work, we show the potential of a flexible sequence length tokenizer for image reconstruction and generation. Beyond enabling high-fidelity reconstructions with very few tokens, we demonstrate through training class- and text- conditional AR models that the FlexTok token sequences specify a "visual vocabulary" that enables generation in coarse-to- tonfin ordering.
This highlights an important trade-off between the FlexTok and AR model: as more tokens are generated, the AR model takes on a larger role in shaping the image. With few tokens, FlexTok's 's flow decoder drives most of the generation, resulting in low gFID even with smaller AR models. However, as the token count increases, an equally powerful AR model is needed to maintain strong perfor- mance.
Our experiments suggest that depending on the complexity of the generation task, a model may be trained to stop the generation early as soon as the condition is fulfilled. While FlexTok ok can semantically compress images into as little as single token, representing highly dense or structured content like text requires more tokens and training objectives that prioritize semantically meaningful concepts. We believe that these present exciting research directions to speed up and improve autoregressive image generation using adaptive compute budgets.
5.4. System-level comparison
In Table 1, we compare FlexTok models against several relevant baselines. We evaluate the tokenizers by training autoregressive models to perform ImageNet-Ik -Ik class-conditional generation. Compared to previous 1D superio
approaches FlexTok achieves superior reconstruction and generation quality at each token budget, all in a single model.
Looking ahead, we anticipate that FlexTck-lle tokeniz- ers, which adapt to the intrinsic complexity of the input data, could be applicable to other domains with high redundancy, such as audio and video. Training generative models on representations that can be both very compact and semantic, or very long and detailed, may enable further explorations into long- horizon video generation, understanding, as well as visual reasoning.
For comparison, we train 2D grid-based tokenizer with flow matching decoder, matching the F FlexTok di 8-d28 tokenizer in parameter count, training steps, and source dataset. Unlike FlexTok d1 B-8-228, ,, it does not use register tokens, causal masking, or nested dropout. Text- conditioned image generation using an FlexTok tokenizer outperforms the 2D grid baseline when generating 2 to

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
Acknowledgments
Chen, J., Yu, J., Ge, C., Yao, L.,, Xie, E., Wu, Y., Wang, Z., Kwok, J., Luo, P., Lu, H., et al. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023.
We thank Justin Lazarow and Miguel Angel Bautista Mar- tin for their valuable feedback on earlier versions of this manuscript. This work was supported as part of the Swiss Al initiative by grant from the Swiss National Supercom- puting Centre (CSCS) under project ID a08 on Alps.
Chen, M., Radford, A., Wu, J., Jun, H.,, Dhariwal, P., Luan, D., and Sutskever, I. Generative pretraining from pix- els. In International Conference on Machine Learning, 2020. URL https: serpiissematiiissss org/ hatp/CorpusD:::: : 219781060.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Chung, H. W. Hou, L., Longpre, S., Zoph, B., Tay, Y., Fe- dus. W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling eatruction-finetumee language models. Journal of Machine Learning Research, 25(70):1-53, 2024.
References
Darcet, T., Oquab, M., Mairal, J., and Bojanowski, P. Vi- sion transformers need registers. ArXiv, Abs/2309.16588 2023. URL https: a semanticschola.: org/ htt//CorpuTD::2331111
Bachmann, R., Kar, O. F., Mizrahi, D., Garjani, A., Gao, M., Griffiths, D., Hu, J., Dehghan, A., and Zamir, A. 4M-21: An any-to-any vision model for tens of tasks and modalities. Advances in Neural Information Processing Systems, 2024.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Baevski, A., Schneider, S., and Auli, M. vq-wav2vec: Self- supervised learning of discrete speech representations. arXiv preprint (00::::9100055555 2019.
Dosovitskiy, A. and Brox, T. Generating images with percep- tual similarity metrics based on deep networks. Advances in neural information processing systems, 29, 2016.
Burgess, N., Milanovic, J., Stephens, N., Monachopou- los, K., and Mansell, D. H. Bfloat16 process- ing for neural networks. 2019 IEEE 26th Sympo- siun on Computer Arithmetic (ARITH), pp. 88-91, 2019. URL https:/ aep..ssanticsccoll..... ort//oorpuSDD:204813
Duggal, S., Isola, P., Torralba, A., and Freeman, W. T. Adap- tive length image tokenization via recurrent allocation. 2024. URL https: www.ssicccaatclll.a...........ae org/
El-Nouby, A., Muckley, M., Ulirich, K., Laptev, I., Verbeek, J., and Jégou, H. Image compression with product quantized masked image modeling. Trans. Mach. Learn. Res., 2023, 2022. URL https: / //api //api.semante semanticschole............coceee caailarroor/CorpusD... 263797879.
Cai, M., Yang, J., Gao, J. and Lee, J. Ma- tryoshka multimodal models. ArXiv, abs/2405. 17430, 2024. URL http:::/api..smmatiiiesssee.eeee semanticschole.. ott//CorpusDD:220063333
Chameleon, Т. Chameleon: Mixed-modal carly-fusion foun- dation models. arXiv preprint arXiv:2405.099 08 2024.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Chang, H., Zhang, H., Jiang, L., Liu, C., and Free- man, W. T. Maskgit: Masked generative image trans- former. 2022 JEEF/CVF Conference on Computer Vi- sion and Puttern Recognition (CVPR), pp. 11305-11315, 2022. URL https: / api semanticscholar. 2468002244663311 6.
Esser, P. Kulal, S., Blattmann, A., Entezari, R. Muller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., Podell, D., Dockhorn, T., English, Z., Lacey, K., Goodwin, A., Marek, Y., and Rom- bach, R. Scaling rectified flow transformers for high- resolution image synthesis. ArXiv, abs/2403.03206, 2024. URL https: / /appi. semanticscholar. org/ http:/CooIID::2222470020
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M. Murphy, K. P., Freeman, W. T., Rubinstein, M., Li, Y., and Kr- ishnan, D. Muse: Text-to-image generation via masked generative transformers. ArXiv, 0is/2301100704 2023. URL https: /atp:::////pipapee/// org/CorrustD::25533 http://ppp.smmntiiccccce.a
Fang, A., Jose, A. M., Jain, A., Schmidt, L., Toshev, A., and Shankar, V. Data filtering networks. arXiv preprint WwXiv:230..74422 2023.
10

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
Fini, E., Shukor, M., Li, X., Dufter, P. Klein, M., Haldimann, D., Aitharaju, S., da Costa, V. G. T., Béthune, L., Gan, Z. et al. Multimodal autoregres- sive pre-trate vre-training of large vision encoders. arXiv preprint arXiv:2411. 144022, 2024.
Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A. Image-to- image translation with conditional adversarial networks. In Proceedings of the IEEE conference computer vi- sion and pattern recognition, pp. 1125- 1134, 2017.
Kolesnikov, A., Pinto, A. S., Beyer, L.,, Zhai, X.,
Fu, S., Tamir, N. Y., Sundaram, S., Chai, L.., Zhang, R., Dekel, T., and Isola, Dreamsim: Learning new dimensions of human visual similar- ity using synthetic data. ArXiv, abs/2306.09344 2023. URL https:/ http:////......aaa.aaeeeeee www.setiincoolll.............
Harmsen, J., and Houlsby, N. Uvim: A unified modeling approach for vision with learned guiding codes. ArXiv, abs/2205.10337, 2022. URL https:
/ http.//maantcccoooo...oo/ooolpll 248964962.
Kondratyuk, D., Yu, L., Gu, X., Lezama, J., Huang, J., Hornung, R., Adam, H., Akbari, H., Alon, Y., Birod- kar, V., Cheng. Y., Chiu, M.-C., Dillon, J., Essa, I., Gupta, A., Hahn, M., Hauth, A., Hendon, D., Mar- tinez, A., Minnen, D. C., Ross, D. A., Schindler, G., Sirotenko, M., Sohn, K., Somandepalli, K., Wang. H., Yan, J., Yang, M., Yang, X., Seybold, B., and Jiang, L.. Videopoet:
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-Or, D. An image is worth one word: Personalizing text-to-image genera- tion using textual inversion. ArXiv, abs/2208.01618, 2022. URL sttps://api... http:s/:/aap.smmatiiccheeee semanticscholar. http://oouI:::11::::/:
large language model for zero-shot video generation. ArXiv, 1112312.11425. 2023. URL httpss:/aph httttp::/aapi.emmatiissssss www.sttcssastooooeeeeeee. org/ htss/CoopuIID:264335555
Ge, Y., Ge, Y., Zeng, Z., Wang, X., and Shan, Y. Planting seed of vision in large language model. ArXiv, abs/2307.08041, 2023. URL https: slaapissemattisccool.e www.aattiiccccola...oor/oooopppp org/CorpusID: 259937351.
Kusupati, A., Bhatt, G., Rege, A., Wallingford, M., Sinha, A., Ramanujan, V., Howard-Snyder, W., Chen, K., Kakade, S. M., Jain, P. and
Han, J., Liu, J., Jiang, Y., Yan, B., Zhang. Y., Yuan, Z., Peng, B., and Liu, X. Infinity: Scaling bitwise au- toregressive modeling for high-resolution image synthe- sis. 2024. URL https: api. eew.ticcchoolll.......... otr//oorpusD::24511..
Farhadi, A. Matryoshka representation learn- ing. In Neural Information Processing Systems, 2022. URL atttps://ap http:::///....aa.....a.aa cacaonn...coa.r.. org/ CorpusID:25268334
Li T., Chang, H., Mishra, S. K., Zhang, H., Katabi, D., and Krishnan, D. Mage: Masked generative en- coder to unify representation learning and image syn- thesis. 2023 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 2142-2152, 2022. URL https: apii semanticscholar. org/ CorpustD:223553224
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Neural Infor- mation Processing Systems, 2017, URL https: /ttes://ppp.. /api. semanticscholes org. wwt.:laaa..gg/oooussssisas 326772.
Ho.
Classifier-free diffusion guidance. URL https:
ArXiv, abs/2207.12598, 2022.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Mierosoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
//api. seapi.ssemantcschool... wwtpeoooo//Coooorr// 249145348.
Hoogeboom, E., Agustsson, E., Mentzer, F,, Ver- sari, I.. Toderici, G., and Theis, L. High-fidelity image compression with score-based generative mod- els. ArXiv, abs/2305.18231 2023. URL https: //api. semanticscholaa org/CCrpssII: 258960316.
Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. ArXiv, abs/2209.03003. 2022. URL https: / aapp.semant.ii. www.eettiicccccc..ccccocoo www.o//rrrrr/oooostttttao 252111177.
Hu, A., Russell, L., Yeo, H.. Murez, Z., Fe- doseev, G., Kendall, A., Shotton, J., and Cor- rado, G. Gaia-l: A generative world model for autonomous driving. ArXiv, abs/2309.17080,, 2023. URL https::/aapp..eaattaimmm org//CorpusD::23311
Loshchilov, I. and Hutter, F. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2017. URL http://appp semanticscholarr oew.

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
Lu, J., Clark, C., Zellers, R., Mottagii, R., and Kemb- havi, A. Unified-io: A unified model for vision, lan- guage, and multi-modal tasks. ArXiv, 0bs/2206.089116 2022. URL https: http:/pp...ammaacccccoooooo arg/CorpusTD> 2498848229
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dock- horn, T., Muller, J., Penna, J., and Rombach, R. Sdxl: Improving latent diffusion models for high- resolution image synthesis. ArXiv, abs/2307.01952 2023. URL https: http:///aappps.aaeeeeeesmee org/ /CcCCCoopuI::::55111111
Lu, J. Clark, C., Lee, S., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., and Kembhavi, A. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. 2024 IEEE/CVF Conference Computer Vision and Pultern Recognition (CVPR), pp. 26429-26445, 2023. URL https:/ wtt.appp...aaiiiiooomimoo icscholas.. hog//oorpuIID:266550
Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., and Carmon, Y. Resolving discrepancies in compute- optimal scaling of language models. arXiv preprint arXiv:2406. 19146, 2024.
Ma, N., Goldstein, M., Albergo, M. S., Boffi, M., Vanden- Eijnden, E., and Xie, S. Sit: Exploring flow and diffusion- based generative models with scalable interpolant trans- formers. In European Conference on Computer Vision, 2024. URL https:/ http:///aapp..aaaii..ee.mea eemanicschoolaa org/CorpusTDD petooorppsD:::::: 267007777
PyTorch Team: Horace He, Driss Guessous, Yanbo Liang, Joy Dong. FlexAttention: The Flexibility of PyTorch with the Performance of FlashAttention, Au- gust 2024. URL http:::////ttoc.rrg/ogg Flexattention/.
Mentzer, F,, Toderici, G., Tschannen, M., and Agusts- son, E. High-fidelity generative image compres- sion. ArXiv, 0ls/2006.09965, 2020. URL https: //api. i. semanti wwc.onicccccha...... CrgpCorps 219721015.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. Zero- shot text-to-image generation. ArXiv, abs/2102.12099 2021. URL https: /api. wwwant org Sxg/CoorpusDD:23205555
Mentzer, F., Minnen, D., Agustsson, E., and Tschannen, M. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309.1550: 2023.
Razavi, A., Van den Oord, A., and Vinyals, O. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Midjourney. Midjourney version 6.1 update, 2024.
URL https:/ http:://udatssmmm/e version-6-11
com/
Rippel, O., Gelbart, M. A., and Adams, R. P.
Learning ordered representations with nested dropout. ArXiv, abs/ 10ss/400..9 2014. URL https::/aap semant cschol ar. wrgg/orrpssor// 11: 0333238.
Miwa, K., Sasaki, K., Arai, H., Takahashi, T., and Ya- maguchi, Y. One-d-piece: Image tokenizer meets quality compression. arXiv preprint arXiv:2501.10064 2025.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Mizrahi, D., Bachmann, R., Kar, O. Yeo, T., Gao, M., Dehghan, A., and Zamir, A. 4M: Massively multimodal masked modeling. In Advances in Neural Information Processing Systems, 2023.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M. S., Berg, A. C., and Fei-Fei, L. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115:211-253 2014.
Oquab, M., Darcet, T. Moutakanni, T., Vo, H. Q. Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W. Howes, R. Huang, P.-Y. B., Li, S.W... Misra, I,, Rabbat, M. G., Sharma, V., Synnaeve, G., Xu, H., Jégou, H., Mairal, J., Labatut, P. Joulin, A., and Bojanowski, P. Dinov2: Learning robust visual features without supervision. ArXiv, abs/2304.07193,, 2023. URL https: wewp...smattischooo.....o org/CoorpusID:25811000
Sadat, S., Hilliges, O., and Weber, R. M. Blimi- nating oversaturation and artifacts of high guidance scales in diffusion models. ArXiv, abs/2410.02416, 2024. URL attpss:/aai..smmmttiscct... org/CorpusTD:273098.0
12

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
Shazeer, N. M. Glu variants improve transformer.
h htt.s//att..cccoolrrrrrorrrrosss 2 269982015.
ArXiv, abs/2002.05202,, 2020. URL https:
wet.lsssmaaticccchoe org/CorpusID: 211096588.
Wang. X., Zhang, X., Luo, Z., Sun, Q. Cui, Y., Wang, J., Zhang, F,, Wang, Y., Li, Z., Yu, Q. et al. Emu3: Next-token prediction all you need. arXiv preprint WrXiv:249. 2024a.
Shen, J., Tirumala, K., Yasunaga, M., Misra, I,, Zettle- moyer, I.., Yu, I.,, and Zhou, C. Cat: Content-
adaptive image tokenization. 2025. URL https: //api. deemnnticschon 275336854.
org/CorpusID;
Wang. X., Zhou, X., Fathi, A.. Darrell, T., and Schmid, C. Visual lexicon: Rich image features in language space. 2024b. URL https: //t::///ppppp semanticschon org/ /o:CCCorruT::246111100
Shi, J. Wu, C., Liang J., Liu, X., and Duan, N. Divae: Photorcalistic images synthesis with de- noising diffusion decoder. ArXiv, abs/2206.0035 00386, 2022. URL https: sttpppsseattticcoooo...oooo org/Corpus TI: (2:4:opp 249240430.
Xu, Y. Corso, G., Jaakkola, T., Vahdat, A., and Kreis, Disco-diff: Enhancing continuous diffusion mod- els with discrete latents. ArXiv, abs/2407.03300, 2024. URL https: /api. semanticscholar: org /ct./pcsSD::200924444533
Su. J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputinng. 568:127063, 2024.
Yan, W., Zhang, Y., Abbeel, P., and Srinivas, A. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157 2021.
Sun, P., Jiang, Y., Chen, S., Zhang, S., Peng, B., Luo, P., and Yuan, Z. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprini arXiv:2406.0m 06525, 2024.
Yan, W., Zaharia, M., Mnih, V., Abbeel, P., Faust, A., and Liu, H. Elastictok: Adaptive tokeniza- tion for image and video. ArXiv, abs/2410.08368, 2024. URL https://app semanticscholar: org/ /tt//CoouSII::7332277777
Taubman, D. S. and Marcellin, M. W. JPEG 2000: Im- age compression fundamentals, standards and practice. Kluwer Academic Publishers, 2001.
Tian, K., Jiang, Y., Yuan, Z., Peng, B., and Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.0200 2024.
Yang. G., Hu, E., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J. W., Chen, W., and Gao, J. Ten- sor programs Tuning large neural networks via zero- shot hyperparameter transfer. ArXiv, abs/2203.033466 2022. URL /ttttsss//appt. www.nticschoootoee org /ttpCCopuII::2422222002
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. /3971, 2023.
Yu, J., Li, X., Koh, Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y.. Baldridge, J., and Wu, Y. Vector-quantized image modeling with improved vggan. arXiv preprint 2021.
Van Den Oord, A., Vinyals, O., et al. Neural discrete rep- resentation learning. Advances neural information processing systems, 30, 2017.
Yu, J. Xu, Y., Koh, Y., Luong, T., Baid, G., Wang, Z. Vasudevan, V., Ku, A., Yang, Y., Ayan, B. K., Hutchinson, B., Han, W. Parekh, Z., Li, X., Zhang, H.. Baldridge, J., and Wu, Y. Scal- ing autoregressive models for content-rich text-to- image generation. Trans. Mach. Learn. Res., 2022, 2022. URL pestps:/aatt org CocCCooooiD:29992200 http//ppp...a.a....mme..mm semant cashhola......
Villegas, R., Babaeizadeh, M., Kindermans, P.-.,, Moraldo, H., Zhang, H., Saffar, M. T., Castro, S., Kunze, J., and Er- han, Phenaki: Variable length video generation from open domain textual description. ArXiv, abs/2210.02399, 2022. URL https:htp//oooc httpsss semanttccccoll.r. org/CorpusID 252715594
Wallace, G. K. The jpeg still picture compression standard. IEEE transactions on consumer electronics, 38(1 x-vviii xxxiv, 1992.
Yu, I.,, Lezama, J., Gundavarapu, N. B., Versari, I.., Sohn,
K., Minnen, D., Cheng, Y., Birodkar, V., Gupta, A., Gu, X., et al. Language model beats diffusion -otttssinnttkkeizerrree is key to visual generation. arXiv preprint arXiv:2310.05737, 2023.
Wang, X. and Aitchison, L. How to set adamw's weight decay as you scale model and dataset size. ArXiv, abs/2405.1 36698, 2024. URL https:
13

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
Yu, Q. Weber, M., Deng, X., Shen, X., Cremers, D., and Chen, L.-C. An image is worth 32 tokens for reconstruc- tion and generation. arXiv preprint wrxiv:2406.07550, 2024a.
Yu, S., Kwak, S., Jang, H., Jeong, Huang, J., Shin, J., and Xie, S. Representation alignment for generation: Training diffusion transformers easier than you think. 2024b. URL https: semantiscchooooooe.coooooocccc org/Corpus ID: 27:22222222.
Zern, J., Massimino, P., and Alakuijala, Webp im- age format. RFC, 9649:1-46, 2024. URL https:
dedevelopers.goolle........ com/ sppeed/wbbp///
Zha, K., Yu, L.. Fathi, A., Ross, D. A., Schmid, C., Katabi, D., and Gu, X. Language-guided image to- kenization for generation. ArXiv, abs/2412.05796, 2024. URL
http:/ppp..smmaniccccoool.l... Stt/CooppuSD::24597722
Zhang, B. and Sennrich, Root mean square layer nor- malization. Advances in Neural Information Processing Systems, 32, 2019.
Zhang, J., Zhan, F., Theobalt, C., and Lu. Regularized vector quantization for tokenized image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pp. 18467-18476, 2023.
Zhao, L., Woo, S., Wan, Z., Li, Y., Zhang. H., Gong, B. Adam, H., Jia, X., and Liu, T. e-vae: De- noising visual decoding. ArXiv, abs/2410.04081, 2024. URL https: http://aap..eemmniccchll..l..... orr/CooppsID::27118333
14

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
Appendix
A. Ablation of FlexTok Design Choices
We explore the design space offered by the F FlexTok framework. The effects of the VAE choice, structure applied to the tokens to induce an ordering, the decoder loss formulation, and the use of inductive bias losses are all investigated in the goal of converging on a high-quality and compact ID tokenizer that can resample images into variable-length token sequences.
Default ablation setup.
Unless otherwise stated:
All models trained on images of resolution 256x256.
We use 16-channel VAE and produce the VAE latent space by sampling from the Gaussian distributions, rather than just taking the mode.
We use FSQ quantization with 6 dimensions, each bucketed into levels [8, 8, 8, 5, 5, 5], for an effective vocabulary size of 64 000.
We use 2x2 patchification inside the FlexTck encoder and decoder that acts the VAE latent space which itself has downsample factor of 8. This yields an effective spatial downsample factor of 16 from pixels to patch tokens.
All FlexTok ablation models have encoder and decoder sizes dl 2-1221 and are trained for 50B patch tokens on ImageNet-Ile (Russakovsky al., 2014). We measure one patch token 16x16 pixel patch.
To evaluate the ablation models for class conditional image generation, we train 393M parameter AR Transformers on the resulting token sequences. The AR models are trained for 94B tokens on ImageNet- 1 (300 epochs).
Evaluation Setup We evaluate our tokenizers using an arry of metrics which probe their reconstruction and generation properties. To evaluate image reconstruction we calculate reconstruction FID (rFID) (Heusel et al., 2017), DreamSim (Fu et al., 2023), and Mean Absolute Error (MAE) on the full validation split of the ImageNet-Ik Ik dataset (Russakovsky et al., 2014). During generation, we use a classificr-free guidance scale of 1.5 for the AR model. For evaluation of the class-conditioned image generation results, we follow the common practice of measuring the generation FID (gFID) of 50K generated samples relative to the reference statistics calculated over the entire training split of the ImageNet- 1k dataset (Dhariwal& & Nichol, 2021).
A.1. VAE choice ablation
We observe a strong correlation between the VAE reconstruction quality and the number of latent channels (Tab. 2), and find that increasing the number of latent channels above 4 can significantly improve the F exTok reconstruction quality (Fig. 10). This finding is in line with the observations that the VAE latent channel size should be scaled with size of the subsequent latent diffusion model (Esser et al., 2024). To optimize for the largest-sized tokenizer's 's performance, we select the 16-channel VAE for our final setup.
Table VAE reconstruction performance. We measure MSE and rFID on the COCO-30X validation set (Lin et al., 2014).
Model
Arch.
Latent Channels
MSE
rFID $
SDXL VAE
SDXL VAE
0.0038
1.13
SDXL VAE
4
0.0043
1.53
SDXL VAE
8
0.0028
0.66
ours
SDXL VAE
16
0.0013
0.35
15

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
0.40
0.105
MAE(INIK va 50k)
0.100
0.35
FFD (NN1 val
DreamSimiln Smmmin
VAE Model:
SDXL-VAE 4ch
0.30
0.090
8ch
0.085
0.25
0.080
0.20
10 Guidance scale
15
Guidance scale
Guidance scale
Figure 10. Guidance scale ablation for different VAE choices. We train PleeXTok d12-d12 models on different VAE choices. Wc
adaptive projected guidance (Sadat al., 2024), and evaluate on the ImageNet-IK (Russakovsky al., 2014) validation
A.2. Resampling strategy ablation
Table Ablation of resampling strategies. compare grid tokenization with ID tokenization, the use rectified flow decoder. the training noise schedule, as well as the use of an auxiliary REPA (Yu et al., 2024b) loss. We compare on reconstruction MAE,
DreamSim, rFID, as well ImageNet-Ik class-conditional generation gFID.
Structure
Formulation
REPA
MAE 4
DreamSim
rFID
gFID 4
16 (2D)
MSE
X 0.059
0.288
51.27
35.93
16 (2D)
Rectified flow
mode(0.25)
X
0.083
0.272
32.85
23.29
(ID)
Rectified flow
mode(0.25)
X
0.078
0.220
23.28
22.16
(ID)
Rectified flow
uniform
X
0.078
23.63
21.35
(ID)
Rectified flow
logit-normal
X
0.082
0.208
19.03
18.69
(ID)
Rectified flow
mode(0.25)
0.075
0.128
7.40
We first compare the resampling strategy (2D) grid tokenization versus 1D register tokenization), as well as the use of rectified flow decoder compared to simple decoder optimized with MSE. Note that in none of these experiments do we apply any ordering strategies. We reserve those investigations for Appendix A.4.
The results in rows and 2 in Table show that the use of a rectified flow decoder significantly improves both rFID and gFID, while getting worse MAE. When switching from 2D ID tokenizer (row 3), we improvements across all metrics, with the largest improvements being rFID. We also ablate three different choices of noise schedules following Esser et al. (2024), namely uniform noise schedule (row 4), logit-normal schedule with location
0 and scale s = (row 5), and mode sampling schedule with scale 0.25 (row 3). While the logit-normal schedule shows the strongest performance, we note that we observe inference instabilities due to the earliest and latest timesteps being undertrained. We chose the mode sampling schedule for our models due to the comparatively better reconstruction performance.
A.3. Use of inductive bias loss.
We observe that rectified flow decoders converge slowly and investigate the use of inductive bias losses (Hu et al., 2023; Yu et al., 2024b) to 1) improve convergence time, and 2) distill the semantic inductive biases strong pre-trained vision model into the tokenizer to make the tokens more predictable. Following REPA (Yu et al., 2024b), we train a three-layer MLP read out the activations of the decoded 2D patches in the first decoder layer, and project them to the DINOv2-L feature dimension. In addition the rectified flow loss, add a cosine similarity loss with weight 1.0 between the predicted features and the reference DINOv2-L features.
The last row of Table 3 compared to row shows that the use of REPA significantly improves perceptual reconstruction metrics like DreamSim and rFID, as well as generation performance as measured by gFID. As shown in Figure 12, the use of an inductive bias loss significantly improves convergence time to high quality reconstructions. This is mirrored in our visual comparison in Figure 13, where reconstructions models with REPA are significantly higher fidelity and more semantic. Interestingly, we find that though the use of REPA improves the reconstruction and generation metrics significantly, it does not improve the convergence of the reconstruction loss during training of the tokenizer (Figure 11). Given the striking improvements in downstream reconstruction and generation metrics, use REPA for all experiments from here
16

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
1.4
No REPA
Reconstruction loss
With REPA
1.2
1.0
0.8
0.6
20K
40K Training iterations
60K
80K
Figure 11. REPA (Yu et al., 2024b) ablation loss curves. We train FlexTok d12-d12 models on ImageNet 1k, with and without
REPA. The reconstruction loss shown here does not include the REPA loss contribution.
Inception score (INIK val 50k)
REPA
0.0925
50k)
60
With REPA
MAE ([NIK val 50c)
0.0900
120
LPIPS (IN1K va 50k)
0.0875
100
0.0850
40
80
0.0825
0.24
0,0800
60
20
0.0775
0.22
0.0750
20K 40K
60K 80K
20K
40K
60K
80K
20K
40K 60K 80K
20K
40K
60K a0k
Training lteration
Training iteration
Training Iteration
Training iteration
Figure 12. REPA (Yu et al., 2024b) ablation evaluation curves.
We train FlexTok d12-d12 models on ImageNet-1k, with and
without REPA. The evaluation metrics measured the ImageNet-Ik (Russakovsky al., 2014) validation
token
tokens
tokens
tokens
16 tokens
tokens
64 tokens
tokens 256 tokens
Original RGB
without REPA
with REPA
without REPA
with REPA
without REPA
REPA
Figure 13, Visual comparison of REPA (Yu et al., 2024b) ablation models. We train FlexTok d12-d12 models on ImageNct- lk, with and without REPA show ImageNet-Ik Ik validation set reconstructions different number tokens.
17

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
A.4. Structuring the ID tokens and inducing ordering
We ablate various ways induce an ordering into the register tokens, starting with the ID rectified flow model trained with REPA, shown in the last row of Table 3 and first row of Table 4.
Table Comparison of strategics to induce an ordering. We ablate the use of causal masks on the register tokens, as well as multiple nested dropout variants.
MAE 4
DreamSim
+
rFID +
gFID +
Number of tokens
32
256
4
32
256
4
32
256
4
32
256
Baseline
X
X
0.075
x
x
0.128
X
X
5.98
x
X
7.40
Causal register mask
X
0.080 x
x
0.118 x
X
X
3.28
x
X
5.04
Uniform nested dropout
0.268
0.148
0.080
0.522
0.275
0.132
19.97
6.48
4.01
17.99
5.26
4.83
+ "Pow2" dropout
0.209
0.159
0.088
0.440
0.323
0.175
10.39
8.09
7.22
9.51
6.63
5.78
"Unifpow2" dropout 0.225
0.149
0.079
0.578
0.278
0.125
30.17
6.78
3.52
28.93
5.48
4.97
First, we ablate the use of a causal attention mask (see Section 3.2) applied to the encoder register tokens, see row 2 in Table Compared to unstructured registers, this results in a significant improvement both rFID and gFID. While the causal mask implicitly induces an ordering over the register tokens, it does not enable use of this model as flexible-length tokenizer. To that end, we ablate three different nested dropout (Rippel et al., 2014) schedules (see Section 3.2). In "uniform nested dropout" we randomly draw Kkeep E 2, 4, ..., KJ. For the "pow2" setting, we uniformly sample from powers of two, i.e. Keep {1, 2, 4, 8, .....K }.. Finally, for "unifpow2", next-highest power-of-two. Each of these choices puts more or less weight on different subsets of the token sequences. For example, when performing uniform nested dropout each valid number of tokens is only trained on roughly 256 of all samples, while "pow2" and "unifpow2" reduce the number of possible sequence lengths from 256 down to 9. "unifpow2" favors larger token sequences, which is preferable when dealing with images that require more tokens. That said, its rFID and gFID for lower number of tokens lower due to being relatively undertrained, and "pow2" can present more balanced approach across the range of sequence lengths (see Figure 14). This improvement of the reconstruction quality for short sequences does come at the cost of slight regression the reconstruction and generation performance with the fully 256 token sequences. In Figure 15 we show guidance scale sweeps for all ablation models.
sample as in "uniform nested dropout" but round to the
Dream DreamSim([N1 val 50k)
IFID (IN1 50k)
MAE(INIK val 55KK)
0.25
0.20
0.4
Num tokens:
10
Unifpow2
0.15
0.3
Pow2
0.2
0.10
32
128
8
32
128
32 Number of Tokens
128
Number of Tokens
Number of Tokens
Figure 14. Pow2 ys Unifpow2 nested dropout schedule as a function of the number of tokens. We train FlexTok d12 612-612 models on ImageNet-Ik Ik with different nested dropout schedules and evaluate the Imagenet- Ik validation Pow2 presents balanced approach across sequence lengths, while Unifpow2 is preferable for higher sequence lengths.
18

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
0.090
0.30
Ablations:
DreamSin(IN1K val 50k)
2D No Flow
FFID (IN1K val 50k)
MAE(INIK vol 50k)
0.085
20
0.25
1D No Flow
10
0.075
1D Unifarm Naise
10
0.20
1D log-normal Noise
0.070
0.065
0.15
Registers
Nested Dropout
Pow2
10
20
10
20
10
20
Uniform Pow2
Guidance
Guidance
Guidance scale
Figure 15. Classifier-free guidance ablations. For all ablation models, we perform guidance scale sweeps using adaptive projected
guidance (Sadat al., 2024). Evaluation metrics measured the ImageNet- 1k validation set.
19

---

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
B. Evaluating the Representation Quality of FlexTok Tokenizers
In this subsection, we evaluate the quality of the representations learned by the FlexTok tokenizer. Specifically, we perform linear evaluation on the quantized register tokens produced by the FlexTok encoder (i.e., the tokens passed as input to the flow model). Importantly, we do not use patch representations, focusing only on the quantized register tokens. For linear evaluation, we train linear classifier on ImageNet-1k* 1k's training set and evaluate it the test set, keeping the tokenizer frozen throughout. Since a single representation is needed for the linear classifier, we follow the approach of Yu et al. (2024a), where the quantized register tokens are concatenated to form a single feature vector per image. For the rest, we follow the probing setup in Fini et al. (2024). Given the 6-dimensional FSQ latents, this results in feature vectors of size x num num.register.token
To tune the linear evaluation recipe, we conduct a hyperparameter search over the following grid: Learn- ing rate [1 10-3,5 10-1, x 10-4), weight decay E [0.1, 0.02], and minimum crop scale [0.1,0.2 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]. Using the optimal hyperparameters found (learning rate 5 x 10m4, weight de- cay 0.05, crop scale (0.4, 1.0), we perform sweep over the number of register tokens to collect the results shown in Figure 16. These experiments are conducted for all models (FlexTok d12-d12 FlexTok d18 d18-d18 d188, FlexTok d1 d18-d28) with a batch size of 1024, using 8 H100 or A A10 GPUs for each experiment.
Linear Probe Accuracy (INIK val 50k)
65
FlexTok d18-d28
60
FlexTok d18-d18
FlexTok d12-d12
TiTok L-32
50
45
40
35
30
25
Number of tokens
Figure Linear probing experiments. ImageNet-Ik top-1 accuracy on frozen tokenizer trunk when varying number of tokens.
Observations. These experiments are interesting because there is guarantee that the representations learned by the tokenizer will be linearly separable. However, as our results demonstrate, the representations are indeed linearly separable. The results, presented Figure 7, reveal several key insights:
Improved performance with additional tokens: Increasing the number of register tokens consistently improves per- formance. This behavior contrasts with TYTok, where the opposite occurs. The main difference lies in the training approach: TiTok trains a separate model for each configuration (i.e., number of tokens), while we learn a unified model capable of handling all configurations. In our model, when adding more tokens we retain the previous ones unchanged, e.g for Kiep = 16 the first tokens will be the same as in the case Kkeep = Consequently, the linear layer has strictly more information improve linear separability. In the worst case, the linear layer can simply ignore the additional tokens if they do not contribute useful information. For this reason we expect that performance will monotonically increase with more tokens, which is verified by the experimental evidence. In contrast, TTTok trains different model for each token configuration, resulting in distinct feature spaces for each model.
Superior peak performance: The best performance achieved by our model significantly surpasses TYTok, with our top-performing configuration reaching 64.6% top-1 accuracy ImageNet.
Trends across encoder and decoder sizes: A larger encoder improves linear separability when all register tokens are activated; conversely, a larger decoder size leads to a slight degradation linear separability.
20

---

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
C. Implementation and Training Details
In this Section, we list implementation and training details for the VAEs in Appendix C.I, the resamplers in Appendix C.2, and the autoregressive models in Appendix C.3.
C.1. VAE details
We train a series of VAE models following the Stable Diffusion methodology (Rombach et al., 2022). The variational autoencoder architecture the same as the SDXL VAE (Podell et al., 2023) except we investigate varying the size of the latent dimension from 4 up and 16 channels. For the discriminator loss we use a 3-layer Patch-GAN discriminator (Isola et al., 2017) that is applied after 5k steps of training and combine it with perceptual (Dosovitskiy & Brox, 2016), KL and reconstruction losses. We train all the VAΣs using a learning rate of le-3, weight decay of 0.05, model EMA decay of 0.992, spatial Nlon-ssampinng factor of 8, KL loss weight of 1c-6, discriminator loss weight of 0.5, reconstruction and perceptual loss weights of 1.0, a batch size of 128 and for 305k steps (40B patches) on images from the DFN dataset (Fang et al., 2023). During inference, we produce the VAE latent space by sampling from the Gaussian distribution, rather than just taking the mode.
The trained VAEs are evaluated on the COCO validation split using both mean squared error and reconstruction Fréchet inception distance (rFID) (Heusel et al., 2017) metrics (see Table 2). We find that increasing the latent channel dimension significantly improves the performance of the reconstructions produced by the VAEs. Critical for the upper bound of the performance for the subsequent tokenizer and autoregressive models, the VAE trained with 16 channels achieves a competitive rFID of 0.354. Furthermore, when training FlexTok d12-d12 tokenizers on these four VAE variants, we show in Figure 10 that choosing Stage 0 VAEs with a low number of channels can have detrimental consequences on the Stage FlexTok reconstruction performance. This finding is consistent with Esser et al. (2024), and we choose to train FlexTok models using our 16-channel VAE to avoid prematurely putting an upper-bound on its performance.
C.2. FlexTok details
See Table 5 for a detailed breakdown of the resampler tokenizer architecture and training settings. All encoders and decoders are Transformers, whose hidden dimension w is parameterized by the number of layers using a fixed aspect ratio of 64, Le. = 64 d. The number of attention heads is set to d. Both the encoder and decoder operate on 2 x 2 patches of VAE latents to reduce the sequence lengths they need to process. The registers are randomly initialized and learnable parameter of shape K x d, concatenated with the VAE patches. We use Flex (Py (PyCocch Team: Horace He, Driss Guessous, Yanbo Liang, Joy Dong, 2024) to create an encoder attention mask in which all patch tokens can attend to each other but not the registers, the registers can attend to all patch tokens, but the i-th register token can only attend to the j-th register token if i j.
We use FSQ as the quantization bottleneck with levels [8, 8, 8, 5, 51, for an effective codebook size of 64 000. When performing nested dropout, we replace the dropped tokens with learnable mask token.
The FlexTok decoders are rectified flow models that are conditioned on the encoder's 's register tokens by concatenating them with the noised VAE latent patches. Unlike the encoder, the decoder computes full self-attention between the registers and patch tokens, and only the patch tokens are output. The VAE latents are sampled according to a noise schedule that follows Stable Diffusion 3's mode sampling with heavy tails scheme, using scale parameter 5 = 0.25, We use adaLN-Zero (Peebles & Xie, 2023) to condition the registers and image patches using separate sets of adaLN weights applied to the same continuous timestep embeddings. As shown in Table 5, the adaL.N parameters make up half the total decoder parameters, but we note that their contribution to the overall decoder FLOPS is negligible they are computed on the time embedding tokens (once for the registers and once for the noised patches). We leave the reduction of this parameter cost to future research. To enable the use of classifier-free guidance (Ho. 2022), we randomly replace the entirety of the encoded registers by a learned null-condition with probability 0.2. The decoder's is target is the flow, see Section 3.1.
In addition to the rectified flow loss LRF, we use REPA (Yu et al., 2024b) to speed up convergence of F FlexTok. Specifically, we project the first decoder layer activations using a 3-layer MLP with the same dimension as the decoder and ratio 4.0, upsample them to size 37 x 37, and compute the cosine similarity with 1024-dimensional DINOv2-L (Oquab al., 2023) features. The REPA loss CREPA is weighted equally to the rectified flow loss, i.e. L = LRF + 1.0 CREPA
The resampler models are either trained on ImageNet-Ie 1k (Russakovsky et al., 2014) for downstream use in class-conditional
21

---

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
Table FlexTok training settings. Model and training configuration for three different model sizes of our resampler tokenizers. Appendix C.2 for further training details.
Configuration
PlexTok d12-d12
FlexTok d18-d18
FlexTok d1 d18-d28
Encoder depth dene
12
18
18
Decoder depth ddec
12
18
28
Encoder dim. Vene
768
1152
1152
Decoder dim. Wdee
768
1152
1792
Encoder Transformer parameters
84.9M
286.7M
286.7M
Decoder Transformer parameters
84.9M
286.7M
1.1B
Decoder adaLN (Peebles & Xie, 2023) parameters
84.9M
286.7M
1.IB
Max. num. registers
256
Register attention mask
Causal (see Section 3.2)
Register nested dropout mode
Powers two:
16, 64, 128, 256 (see Section 3.2)
FSQ (Mentzer et al., 2023) levels
[8, 8, 8, 5,
VAE channels
VAE downsampling factor
8
Patch size
x
Feodforward activation
SwiGLU (Shazeer, 2020)
Rectified flow decoder
Decoder adero
time emb. (Peebles & Xie, 2023)
Noise mode sampling
0.25
param. ((Esser al., 2024), Sec. 3.1)
Condition dropout prob.
0.2
REPA (Yu et al., 2024b) layer
1
REPA (Yu al., 2024b) model
DINOv2-L. (Oquab et al., 2023)
REPA (Yu al., 2024b) projection
3-layer MLP with decoder dim.
REPA (Yu et al., 2024b) loss weight
1.0
Training length (n tokens)
200B
Warmup length (n tokens)
4B
Warmup learning rate
1e-6
Learning rate schedule
Cosine decay
Model EMA decay
0.998
Optimizer
AdamW (Loshchilov Hutter, 2017)
Opt. momentum
31 B2 0.9, 0.99
Learning rate ?)
5.62e-4
Batch size
2048
μuP (Yang et al., 2022) base dim.
256
Weight decay timescale Tuter (Wang& & Aitchison, 2024)
Tater 381 470
Gradient clipping norm
1.0
Dataset
ImageNet- 1k (Russakovsky et al., 2014) or DFN-2B (Fang et al., 2023)
Image resolution
256
RandomRes zzedrop.
Augmentations
KandomHorizontalF11ig
Normalize
Data type
bfloat16 (Burgess al., 2019)
image generation, DFN-2B (Fang al., 2023) for training autoregressive text-to-image models. The image-caption dataset contains mixture of original and synthetic captions. During training, we randomly select crops of size 256 using random scales in [0.8, 1.0] and aspect ratio in [0.75, 1.3333]. The resulting images are randomly flipped horizontally with probability of 0.5, and normalized to the range [-1,1].
All models are trained for total of 200B tokens seen, where one token counted as 2 2 VAE patch, i.e. a 16 16 grid of pixels. For images size 2562, this amounts to 256 tokens per sample. We linearly warm up the learning rate for
22

---

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
tokens, and decay it using cosine decay. The model is trained using the A (Loshchilov & Hutter, 2017) optimizer, and swept the learning rate and batch size at small scale using μPP (Yang al., 2022), which transfer directly to all other settings. We note here that we did not sweep these hyperparameters for every resampler setting, but only the base setting of a rectified flow resampler without REPA, causal register masks, nor nested dropout. We automatically set the weight decay following the interpretation that Adam W with weight decay can be understood as an exponential moving average (EMA) of recent updates (Wang& & Aitchison, 2024). Concretely, we compute the weight decay such that it corresponds to averaging over all training iterations, setting Titer to the total number of iterations Nlster The weight decay À is computed as ( using learning rate 7). For the final model evaluations, we additionally use an EMA of the weights with decay rate 0.998. See the exact settings in Table and the ImageNet- I training curves in Figure 17.
1.8
FlexTok size:
d12-d12
1.6
d18-d18
Training loss
d18-d28
1.4
1.2
1.0
0
50K
100K
150K
200K
250K
300K
350K
Training iterations
Figure 17. FlexTok training loss curves. The FlexTok of different sizes shown here are trained for 200B tokens on ImageNet- Ik (Russakovsky et al., 2014). We plot the total loss, i.c. Lat + LREEA.
Inception score [INI val 50k)
(IN1K val 50k)
30
MAE (INIK val 50k)
180
LPIPS (INIK val 50kk)
FlexTok
d12-412
25
0.095
160
d18-418
140
0.28
d18-428
20
0.090
120
15
100
0.085
80
0.24
5-
60
0.080
100K 200K 300K Training iteration
100K 200K 300K Training iteration
100K 200K 300K
100K
200K 300K
Training iteration
Training iteration
Figure 18. FlexTok evaluation curves. We evaluate the resamplers during training at periodic intervals on the entire ImageNet-Ik validation set, showing rFID, MAE, Inception score, and LPIPS.
C.3. Autoregressive model details
See Table 6 and Table for detailed breakdown of the class conditional and text-conditional autoregressive transformer architectures and training settings. The AR component of the models casual decoder Transformers, similar to the FlexTok tokenizer modules, the hidden dimension w is parameterized by the number of layers d using fixed aspect ratio of 64, i.e. w = 64-d d. The number of attention heads is set to d. We use a MLP ratio of 4 for the FFN hidden dim relative to the attention hidden.
In the class-conditional models we do not apply yP (Yang et al., 2022) and instead scale the learning rate inversely with the model width. To mitigate overfitting to the ImageNet- 1k dataset we apply dropout with 0.1 probability to the FFN, attention, and projection modules in the Transformer decoder blocks and apply random homizontal flipping of the images. Additionally text-conditional models we do apply jP, use a learning rate of le-2 for all model sizes, don't t use dropout in the Transfomrer decoder blocks, take square center crops and apply no data augmentations to the training images. When scaling up the AR transformer for the text conditioned models we warmup the learning rate for the same number of tokens as the model's parameter count (Porian al., 2024).
produce 10 random crops per image prior to tokenization(Sun al., 2024). Whereas, for the
23

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
Table Class conditioned AR training settings. Model and training configuration for different model sizes of our AR transformers.
Configuration
AR 49M
AR 85M
AR 201M
AR 393M
AR 679M
1.33B
Num. non-embedding Parameters
49M
85M
201M
393M
679M
1.33B
Decoder depth ddee
10
12
16
20
24
30
Decoder dim. Wdee
640
768
1024
1280
1536
1920
Cross Atm. dim.
n/a
n/a
n/a
n/a
n/a
n/a
MLP Ratio
Max. Sequence Length
256
Attention mask
Causal
Vocab Size
64,000
Foodforward activation
SwiGLU (Shazeer, 2020)
Positional Encoding
Learned Embedding
Conditioning dropout prob.
0.1
FFN, Attm, and Projection dropout proh.
0.1
Training length (n tokens)
94B (300 Epochs) 9.4B
Warmup length (n tokens) Initial warmup learning rate
le-6 Cosine decay
Learning rate schedule
Optimizer
Adam W (Loshchilov & Hutter, 2017)
Opt. momentum
31. 82 0.9, 0.95
Learning rate !) Final learning rate
1.2E-3
IE-3
7.5E-4
6E-4
SE-4
4E-4
1 7 1F-2
Batch size
1024
μPP (Yang et al., 2022) base dim.
n/a
Weight decay
0.05
Weight decay timescale Titer (Wang & Aitchison, 2024)
n/a
Gradient clipping norm
1.0
Dataset
ImageNet- (Russakovsky et al., 2014)
Image resolution
256
10 options RandomResi zedCrop,
Augmentations
RandomHor 2
Normalize bfloat16 (Burgess al., 2019)
Data type
24

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
Table Text conditioned AR training settings. Model and training configuration for different model sizes of our AR transformers.
Configuration
AR 113M
AR 382M
AR 1.15B
AR 3.06B
Num. non-embedding Parameters
113M
382M
1,15B
3.06B
Decoder depth daec
12
18
26
36
Decoder dim. deee
768
1152
1664
2304
Cross Attn. dim.
12
18
MLP Ratio
4
Max. Sequence Length
256
Attention mask
Causal
Vocab Size
64,000
Feedforward activation
SwiGLU (Shazeer, 2020)
Positional Encoding
Learned Embedding
Conditioning dropout prob.
0.1
FFN, Attn and Projection dropout prob.
0.0
Training length (n tokens)
284B
Training FLOPss
1.93E+20
6.39E+20
1.90E+21
5.00E+21
Warmup length (n tokens)
Initial warmup learning rate
1e-6
Learning rate schedule
Cosine decay
Optimizer
AdamW (Loshchilov & Hutter, 2017)
Opt. momentum
81,30 0.9, 0.95
Learning rate 7)
le-2
Final learning rate
x) IE-2
Batch size μP (Yang et al., 2022) base dim.
8192
256
Weight decay
0.05
Weight decay timescale Titer (Wang & Aitchison, 2024)
Niter 135 594
Gradient clipping norm
1.0
Dataset
DFN-2B (Fang et al., 2023)
Image resolution
256
Augmentations
n/a
Data type
bfloat16 6 (Burgess et al., 2019)
25

---

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
D. Related Work
Image tokenization. The goal of tokenization is to convert high-dimensional images into a more compact sequence of token representations, making diffusion and flow (Rombach et al., 2022; Podell et al., 2023; Ma et al.. 2024; Esser et al., 2024; Yu et al., 2024b), masked (Chang et al., 2022; 2023; Li et al., 2022; Lu al., 2022: 2023; Mizrahi et al., 2023; Bachmann et al., 2024), autoregressive (Chen et al., 2020; Ramesh et al., 2021; Yu et al., 2022; Sun al., 2024) image modeling more tractable. Usually, these tokens are learned using autoencoder objective with díscrete (Van Den Oord et al., 2017; Razavi et al., 2019) or continuously regularized (Rombach et al.. 2022) bottleneck. In addition, auxilliary perceptual and discriminator losses (Esser et al., 2021; Yu et al., 2021) are commonly used to abstract away imperceptible details, while latent bias losses (Hu et al., 2023) have been shown to facilitate downstream token prediction. In the discrete case, vector quantization (VQ) (Van Den Oord et al., 2017) has been the de facto standard, but more recent techniques such as LFQ (Yu al., 2023) and FSQ (Mentzer al., 2023) show promising scaling trends.
Tokenizers with rectified flow decoding. A core property of image tokenizers is the ability to reconstruct perceptually plausible images from the heavily compressed tokens. While discriminator losses (Esser et al., 2021) are commonly used for this, their training can be notoriously difficult (Yu et al., 2024a). Instead, we opt to train the decoder as a rectified flow (Liu et al., 2022) model, similar to other works (Shi et al., 2022; Mizrahi et al., 2023; Bachmann et al., 2024; Xu et al., 2024; Zhao et al., 2024), as this approach has proven successful in scalable image generation with both coarse and fine conditioning a property that is particularly useful for our model, FlexTok, which can handle a wide range of conditioning specificity.
Structured tokenization. So far, most of the aforementioned image tokenization methods project images into fixed-size 2D grid. Methods like VAR (Tian et al., 2024) and Infinity (Han et al., 2024) showed impressive scaling trends when projecting images into structured multi-scale latent representation, while TiTok (Yu et al., 2024a), TexTok (Zha et al., 2024), and DisCo-Diff (Xu et al., 2024) explored doing away with the 2D grid entirely and projecting images into an unstructured but highly compact 1D sequence. Still, the number of tokens an image is represented with is either fixed in the case of Titok, or depends on the image resolution in the case of Infinity.
Structured and adaptive tokenization. To address the issue of the token sequence length depending entirely on the height and width instead of the image complexity, range of concurrent works have proposed adaptive tokenization methods:
ElasticTokk (Yan et al., 2024) is a joint image and video tokenizer that performs nested dropout during training between a pre-specified minimum and maximum number of tokens. The authors note that the minimum bound is set to 128 or 256 due to instabilities at lower values. Compared to TlasticTok, we observe stable training of FlexTok down to a single token. In addition, by training the decoder with a rectified flow objective, FlexTok is able to reconstruct high-fidelity images at any number of tokens. Comparatively, ElasticTok requires much higher number of tokens to reconstruct images with high perceptual quality due to being trained only with an MSE and LPIPS loss.
ALIT (Duggal et al., 2024) presents adaptive length tokenizer that recurrently encodes images into a growing sequence of tokens. This process can be dynamically halted, leading to adaptively sized image representations, Compared to ALIT, Tok encodes images in single forward pass, and we efficiently enforce an ordering through causal masks and nested dropout. Since the tokenizers are trained on a 100-class subset of ImageNet- Ik and not evaluated on generative tasks, its scaling and generation properties are not entirely experimentally demonstrated. In comparison, we scale both the FlexTok and downstream AR models, and demonstrate strong generative modeling capabilities.
One-D-Piece (Miwa et al., 2025) is architecturally similar to TYTok (Yu et al., 2024a) and FlexTok, in that it uses a register encoder and decoder and is trained using nested dropout on token sequences between I and 256. While the One-D-Piece models achieve good reconstruction metrics at high number of tokens, the model fails to produce plausible images at lower number of tokens, unlike F FlexTok. In addition, it requires a two-stage training approach like TiTok, while F FlexTok's 's Stage is trained end-to-end. Similar to ElasticTokk and ALIT, the authors do not evaluate the use of One-D-Piece tokens to train generative models.
ViLex (Wang et al., 2024b) uses a pre-trained text encoder and diffusion model to learn variable-sized sets of "text tokens" that encode a given image. These soft tokens can be combined with image prompts to generate novel images, similar to textual inversion (Gal et al., 2022). The goal of F FlexTok lies more in learning a generic set of coarse-to- fine- fine-grained token sequences, which can both be highly semantic for short sequences and highly detailed for longer
26

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
sequences. We believe that the of pre-trained diffusion models the decoder presents exciting future research directions.
CAT (Shen et al., 2025) presents a nested VAE architecture that can adaptively compress images into 8x, 16x, and 32x spatially down-sampled representations. Compared FlexTok's 's discrete 1D representations, these token grids are 2D and continuous.
Ordered representation learning. Besides image generation, learning ordered representations has been a long-studies topic. Nested dropout (Rippel et al., 2014) learns a variably sized bottleneck representation through uniformly dropping latents from one side, while Matryoshka Represenation Learning (Kusupati et al., 2022) proposes to sample the representation dimensions from powers of two, and decodes each with weight shared decoder. Matroshka Multimodal Models (Cai et al.. 2024) proposes instead to adaptively pool 2D vision encoder representations into smaller 2D grid sizes. UViM (Kolesnikov et al., 2022) proposes to use nested dropout to learn codes that are more robust and easier to model with a downstream language model.
Variable-rate compression. In lossy image compression, how to trade compression strength (rate) for reconstruction performance (distortion) has been a long-studied topic. Classical lossy image compression codecs (Wallace, 1992; Zern et al., 2024; Taubman & Marcellin, 2001) and more recent neural compression algorithms (Mentzer et al., 2020; El-Nouby al., 2022; Hoogehoom al., 2023) effectively enable users choose this trade-off and are able to compress simple images smaller file sizes compared to more complex ones. While classical JPEG images transmitted and decoded in raster-scan order, progressive schemes like progressive JPEG (Wallace, 1992) structure images in coarse-to- to--tt--ft fine manner, allowing users to very quickly reconstruct a low-quality version of an image.
27

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
E. Additional ImageNet Reconstruction and Generation Metrics
To complement the ImageNet-Ik Ik reconstruction metrics (rFID, MAE, and DreamSim) shown in Figure 5, we provide additional image-wise reconstruction metries in Table including PSNR, SSIM, and LPIPS, measured on the ImageNet- Ik validation set. We also supplement the class-conditional generation results from Figure (right) and Figure 9 with additional metrics in Table 9, which includes CLIP score, gFID, sFID, Inception score, precision, and recall measured against the complete ImageNet-Il k training set.
Table 8. FlexTok d1 818-d28 ImageNet-Ik reconstruction metrics. For the largest F FlexTok 618 618-d28 model trained on ImageNet- Ik, we show reconstruction metrics full validation set measure distribution-leve differences (reconstruction FID), well as image-wise distortions in pixel-space (MAE, PSNR, and SSIM) and feature-space (LPIPS and DreamSim). Given vocabulary size of 64 000, each token takes up 2 bytes storage.
# tokens
# bytes
rFID
MAE
PSNR t
SSIM
LPIPS
DreamSim
2
4.01
0.273
9.35
0.187
0.701
0.546
4
3.09
0.237
10.25
0.222
0.651
0.494
8
2.43
0.197
11.51
0.254
0.582
0.417
16
1.90
0.185
11.90
0.269
0.532
0.321
16
32
1.61
0.154
13.05
0.304
0.462
0.271
32
64
1.45
0.134
13.96
0.330
0.407
0.227
64
128
1.37
0.126
14.34
0.343
0,380
0.207
128
256
1.20
0.102
15.90
0.407
0.293
0.158
256
512
1.08
0.081
17.70
0.489
0.219
0.114
Table Class-conditional 1.33B AR model with FlexTok d18-d28 generation metrics on ImageNet-1k, We show generation metrics for 1.33B class-conditional AR model trained tokens from FlexTok dl 2d28. The generation FID, sFID, Inception score, precision, and recall are measured using 50k generated samples by comparing against the full ImageNet- train set statistics, while
the CLIP score computed the full validation set instead.
# tokens
CLIP score
gFID
sFID +
IS f
Precision t
Recall
26.84
3.14
6.54
236.47
0.83
0.53
2
27.10
2.51
5.33
238.07
0.82
0.57
4
27.51
2.00
4.86
226.77
0.80
0.60
8
28.84
1.82
4.53
266.48
0.82
0.61
16
29.12
1.75
4.32
277.45
0.82
0.61
32
29.10
1.71
4.46
284.99
0.82
0.61
64
29.14
1.76
4.53
286.40
0.82
0.61
128
29.03
1.89
5.24
275.63
0.82
0.61
256
28.94
2.45
6.76
258.33
0.80
0.61
28

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
F. FlexTok Inference Hyperparameter Sweeps
For trained FlexTok models, we sweep different inference-time hyperparameters relating to rectified flow decoder, such as the number of denoising steps in Figure 19 and the massifier-frre guidance scale when using adaptive projected guidance (APG) (Sadat al., 2024) Figure 21. We further show guidance sweeps when using a vanilla guidance formulation in Figure 20, and compare guidance scales for largest ID and 2D tokenizers in Figure 22.
F.I. FlexTok flow decoder timestep sweeps
We sweep the number of denoising steps between I and 100 on FlexTok d18-d28 (trained on ImageNet-Ik, for different number of tokens. By default, set the guidance scale to 7.5 and use APG (Sadat al., 2024). We find that denoising for more than 25 steps leads to significant diminishing returns in terms of the reconstruction quality metrics, shown in Figure 19. rFID and DreamSim metrics plateau, but the mean absolute error to the original images continues to increase, especially for low number of tokens. We choose 25 denoising steps for all subsequent ablations and use it as our default value for all model sizes, as it provides a good balance between inference speed and quality.
0.30
DreamSim (INIK val 50K)
0.8
rFID (IN1K val 50K)
100
MAE (INIK val 50KK)
0.20
16
10
0.15
32
64
0.10
0.2
20
60
80
100
40
60
100
20
60
80
100
Number of denoising steps
Number denoising steps
Number of denoising steps
Figure 19. Number of inference denoising steps ablation. We sweep number of inference steps FlexTok d18-d22 (trained on ImageNet- 1k) and find that 25 denoising step provides a good balance between inference speed and quality.
F.2. Ablating standard classifier-free guidance
We find that using standard classifier-free guidance (Ho, 2022) in the flow matching decoder results in narrow basins of optimal reconstruction performance that are highly dependent the number of tokens provided to the decoder (see Figure 20). We instead opt to use APG (Sadat et al., 2024), as discussed in Appendix F.3.
Dreamsin (IN1K val 50K)
0.6
Num. tokens:
MAE (INIK val 50K)
10
0.25
0.5
0.20
IFID (NN1
16
0.15
0.3
32
64
0.10
0.2
128
256
0.1
5
Guidance scale
scale scale
Guidance scale
Figure Standard classifier-free guidance ablation. We sweep guidance scales for FlexTok dl d18-228 (trained on ImageNet- and find optimal guidance scale basins to be narrow and vary strongly across different number of tokens used.
F.3. FlexTok normalized guidance sweeps
We would like to avoid cases where the optimal guidance scales varies greatly across different number F FeexTok tokens. As observed in Appendix F.2, standard classifier- free guidance can exhibit narrow optimality basins. For that reason, we explore the use of normalized guidance schemes like adaptive projected guidance (APG) (Sadat et al., 2024). We choose
29

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
hyperparameters for rescaling threshold the guidance scale. Compared to standard guidance, using APG (Sadat al., 2024) yields optimal performance basins which are shallower, more aligned across differing numbers of tokens, and have overall improved rFID values. The basins of smaller Flle FlexTok models are less aligned than the ones of larger models, and their optimal guidance values are higher. No matter the model size, the DreamSim metric improves and plateaus with higher guidance scales, while the MAE degrades slightly.
2.5, parailel component as 1/ 0, and momentum
-0.5, and sweep
FlexTok d12-d12 (ImageNet-1k
FFID (INIK val 50K)
MAE(INIK 50l 50K)
0.25
0.5
10
0.20
DreamSim((NII
0.4
0.3
30
Guidance scale
Guidance scale
Guidance scale
FlexTok d18-d18 (ImageNet-1k)
0.30
DreamSinnNNIKK val 50k)
0.6
FRD (IN1K 50k)
MAE(NIK val 50%) 50k)
0.25
1.0
0.20
0.4
0.15
0.3
0.10
20
10
Guidance scale
Guidance scale
Guidance scale
FlexTok d18-d28 (ImageNet-1k)
DreamSim(NNKK val 50k)
FFID (INIK 50K)
MAE(INIK val 50k)
10
0.20
0.4
0.15
0.3
0.2
0.10
20
Guidance scale
scale
FlexTok d18-d28 (DFN-2B)
0.30
IraamSim(CCCO.oo 30.30.5 [k)
IID (COCO 30.5k)
MAE/COCO val 5k)
0.6
0.25
0.5
1.0
0.20
0.15
0.10
0.2
20
10
15
15
25
scale
Num. tokens:
128
256
Figure 21. Adaptive projected guidance (Sadat et al., 2024) ablation. We sweep guidance scales for different FlexTok model sizes and number of tokens and find that APG results smoother guidance scale basins than standard guidance.
30

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
F.4. 1D 2D tokenizer classifier-free guidance ablation
For our largest tokenizer size FlexTok 8-8 (trained on DFN-2B), compare classifier-free guidance basins with controlled 2D grid tokenizer baseline the COCO 30.5k validation set. Both models use APG (Sadat et al., 2024). As shown in Figure 22, find that in terms of rFID and DreamSim score, the ID tokenizer outperforms the 2D baseline by large margin across guidance scale values, but observe that the 2D grid tokenizer performs better in terms of MAE.
0.086
DreamSim(COC0 val 30.5k)
IFl (COOO val 30,5k)
MAEICOCO val 10.300.00 5k)
0.145
0.084
0.082
0.140
Model:
1D
0.080
0.135
20
0.078
0.130
0.076
10
15
20
25
10
15
20
Guidance scale
Guidance scale
Guidance scale
Figure 22. 1D vs 2D FlexTok d18-d28 guidance scale ablation. We sweep the APG guidance scale for FlexTok d18-d28 (trained on DFN-2B) and the controlled 2D tokenizer baseline. The 1D FlexTok d1 d18-d28 model is evaluated using the full 256 tokens.
F.5. Final FlexTok inference byperparameters
From these observations we make the final choices of the inference hyperparameters for each of the model sizes and training datasets and use these for all subsequent image generation experiments. We detail them in Table 10 for ImageNet-IK tokenizers and 11 for DFN-2B tokenizers. We note that common timestep and guidance-scale distillation techniques for diffusion and flow models are applicable to FlexTok decoders, and we expect that such steps could reduce the computational requirements of running the decoder significantly.
Table 10. ImageNet-1k FlexTok inference settings. Chosen tokenizer inference configurations used for the three different model sizes trained on ImageNet-Ik.
Configuration
FlexTok d12-d12
FlexTok d18-d18
FlexTok
8-d28
# denoising steps
25
25
25
Adaptive Projected Guidance (APG) (Sadat et al., 2024)
True 15
True 7.5
True 7.5
Decoder guidance scale
Table DFN FlexTok inference settings. Chosen tokenizer inference configurations used for the three different model sizes trained on DFN-2B. Configuration
FlexTok 218--28
2D Grid d18-d28
denoising steps
25
25
Adaptive Projected Guidance (APG) (Sadat et al., 2024)
True
True
Decoder guidance scale
7.5
5.0
31

---

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
G. Autoregressive Class-Conditional Image Generation Hyperparameter Sweeps
For class-conditional AR Transformers trained on top FlexTok tokenizers we sweep a variety of inference-time hyperparameters. We implement classifier-free guidance the AR model as a update of the prediction logits given by Tuncond + 8 (Feond Tuncond). Here Zoond and Tuncond are the logits with and without the conditioning supplied the model. Taking fully trained AR 1.33B and FlexTok d18-d28 s= 1.0, results in the best gFID values irrelevant of the top-k sampling or number of tokens generated before decoding to image space (Figures 23 and 24). A optimal gFID value with no guidance sets AR models with F FlexTok tokenizers apart from previous examples in the literature. For example the optimal CFG value for the LlamaGen models was 2.0 for their class-conditional models (Sun et al., 2024). The ordering introduced by the causal mask and nested-dropout in the tokenizer model. There are significant inference compute advantages
find that using no guidance, i.e. CFG scale
could be the source of the property the subsequent CFG applied, in particular the batch doesn't I need be increased by factor of accommodate the unconditional samples. Besides the interesting dependence the Massifier-free guidance scale find that increasing the top-k sampling threshold improves the gFID values. For optimal generation FID we find that no top-k sampling should be used.
1P (IN1K train 50k)
Top-k: k:
4000
8000
16000
32000
64000
Guidance scale
Figure 23. Class-conditional 1.33B model with FlexTok d18 d18-d28 guidance and top-K ablation. gFID measured with respect
to the full ImageNet-Ik train set statistics.
gFID (IN1K train 50k)
10
Num. tokens:
1
2
4
8
16
32
64
128
256
Guidance scale
Figure 24. Class-conditional 1.33B AR model with FlexTok d18-d28 guidance and number of tokens ablation. gFI measured
with respect to the full ImageNet-lk train set statistics.
32

---

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
H. Autoregressive Class-Conditionss Image Generation Model Size
By scaling up the AR model size in class conditioned models we observe consistent improvements in the training loss (Figure 25). Following the optimal inference parameters observed in Appendix G, no CFG and no top-k sampling. we evaluate the effect of scaling the AR model size on top of the FlexTok d18-d28 tokenizer. Figure 26 shows that generation performance when producing only few tokens is independant of model size. However the longer the sequence the more the larger AR model size limits token decoding errors. When generating the full 256 tokens scaling up the AR model size significantly improves the gFID values. We note that even with the largest AR model investigated here, with 1.33B parameters, we still observe a slight regression in the gFID values as the number of decoded tokens increases about 128.
AR size:
49M
Training loss
10:
85M
201M
393M
9.
679M
50K
100K
150K
200K
250K
300K
350K
Training iterations
Figure 25, Autoregressive class-conditional image generation loss curves. The AR models of diffcrent sizes shown here are trained for 94B tokens ImageNet-Il Ik using the FlexTok d 8-88 tokenizer trained on ImageNct- -
AR size:
gFID (IN1K train 50k)
8
49M
gFID (IN1K train 50k)
85M
201M
6
393M
6
679M
5
1.33B
4
2
2
32
128
10a Number of Parameters
10
Number of tokens
Figure 26. Autoregressive class-conditional image generation model scaling, Left shows the gFID values for each model size varying numbers of generated tokens. The right figure shows the gFID with 256 tokens against the parameter count of the AR model. The gFID values measured with respect the full ImageNet-Ik kk train set statisties. All AR models trained using the F1 FlexTok d1 818--28 tokenizer trained on ImageNet-1k, During generation we use the optimal inference parameters detailed Appendix G.
Scaling up the tokenizer size significantly improves the generation quality (Figure 27). With a fixed AR model size of 1.33B we scale up the tokenizers size from F FlexTok d12-d12 to F FlexTok d18-d28 and find improvements in the measured gFID values. For some tokenizer sizes the reconstruction quality of the tokenizer upper bounding the performance of the generative model.
33

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
FlexTok Size:
4.5
gFID (IN1K train 50k)
d12-d12
gFID (INIK train 50k)
8
d18-d18
d18-d28
4.0
3.5
4
3.0
2
2.5
128
109
Number of tokens
Number of Parameters
Figure 27. Autoregressive class-conditional image generation FlexTok tokenizer scaling. Left shows the gFID values for 1.33B AR models trained with each the different sized lexTok tokeninizers at varying numbers generated tokens. The right figure shows the gFID with 256 tokens against the parameter in the TexTok tokenizer. The gFID values measured with respect the full ImageNet- Ik train set statistics. During generation we use the optimal inference parameters detailed in Appendix G.
34

---

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
I. Autoregressive Text-Conditional Image Generation Inference Hyperparameter Sweeps
For text conditional AR Transformers trained with FlexTok tokenizers we sweep the hyperparameter classifier-free guidance scale and number of decoded tokens. Taking fully trained AR 3.06B and FlexTok d18-d28 we find that using classifier-free guidance improves the gFID for all lengths of generated token sequences (Figure 28). This observation in contrast to the class conditioned FlexTok based models where using CFG hurt gFID performance. The difference in the complexity of the conditioning and any distribution shift between the DFN training and COCO validation datasets could be contributing factor here. In addition to the CFG improving gFID values we also find that the CLIPScore increases with increasing guidance.
We find that a CFG seale of 2.5 optimizes the gFID values across the variety of number of tokens generated. Using this CFG scale we see gFID values which sharply drops and then gradually increase as the number of tokens generated increases (Figure 29). In comparison find that the text-image alignment measured by the CLIPScore only improves as the number of tokens is increased.
score (COCO val 30K)
32
gFiD (COCO val 30K)
Num. tokens:
30
1
2
28
4
8
26
16
32
24
64
LiP
22
128
10
256
2
4
Guidance scale
Guidance scale
Figure 28. Text-conditional 3.06B AR with FlexTok d18-d28 guidance and number of tokens ablation. gFID (left) and CLIPScore
(right) metrics measured with respect the COCO validation set varying numbers of tokens.
CLIP SCore 30k)
32
B CO00 val 30k)
15
30
IND (COCO val
28
10
8
32
128
8
32
128
8
32
128
Number of tokens
Number of tokens
Number of tokens
Figure 29. Text-conditional 3.06B AR with FlexTok d18-d28 image gemeration vs number of tokens. All metrics are computed using CFG scale of 2.5 during generation. Left shows the rFID of the DFN trained P11 exexook d1 8-d28 the COCO validation set. The AR image generation gFID (middle) and CLIPScore (right) on the COCO validation set as functions of the number of generated tokens.
Figure 30 shows the 2D grid based tokenizer has similar dependence of the evaluation metrics the CFG scale used. We select a CFG scale of 2.5 as a balance between optimizing the gFID and CLIPScore metrics. Additionally this selected CFG scale is the same as the value used for the FlexTok based models enabling a balanced comparison the two approaches.
35

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
32
CLIP CLIP Score (COCO val 30K)
gFID (COCO val 30K)
31
30
Num. tokens:
256
29
28
10
2
3
5
Guidance scale
Guidance scale
Figure 30. Text-conditional 3.06B AR with 2D grid tokenizer guidance ablation. gFID (left) and CLIPScore (right) metrics measured with respect to the COCO validation set varying numbers of tokens.
36

---

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
J. Autoregressive Text Conditional Image Generation Model Size
Following the optimal inference parameters observed in Appendix I, we evaluate the effect of scaling the AR model size while fixing the Fl FlexTok dl 8-d28 tokenizer. Scaling up the model size from 113M 3.06B parameters results in lower final loss values being achieved for both the FlexTok and 2D grid tokenizer (Figure 31 and 32). When generating the full 256 tokens, scaling up the AR model size significantly improves the gFID values (Figure 33).
AR size:
113M
362M
Training loss
1.158
3.068
0
20K
40K
60K Training iterations
80K
100K
120K
Figure 31. FlexTok autoregressive text-conditional image generation loss curves. The AR models of different sizes shown here are trained for 284B tokens on DFN using the eeeookk 8 8-d28 tokenizer trained DFN.
11.0
AR size:
10.5
113M
362M
Training loss
10.0
1.158
3.06B
9.0
8.5
20K
40K
60K
80K
100K
120K
Training iterations
Figure 32. 2D grid tokenizer autoregressive text-conditional image generation loss curves. The AR models of different sizes shown here trained for 284B tokens DFN using the FlexTok di d-d28 tokenizer trained DFN.
FlexTok (256 Tokens) Grid (256 Tokens)
FRD (COCO val 30K)
16
10a
10a
Number of Parameters
Figure 33. Autoregressive text conditional image generation model scaling. The gFID values for each model size as functions of the parameter count of the AR model. During AR generation we generate 256 tokens so that the FlexTok and 2D grid tokenizer can be compared and use the optimal inference parameters detailed in Appendix I. The gFID values measured with respect to the COCO validation set. AR models are trained using tokenizers trained on DFN.
37

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
K. Additional Visualizations
K.I. FlexTok image reconstruction for different numbers of tokens multiple samples per token sequence
token
tokens
tokers
tokens
16 tokens
32 tokens
64 tolkans
128 tokens
256 tokens
Sample #1
Original RGB
Sample #2
Sample
Sample
Sample
token
takens
tokens
takens
18 tokers
32 takens
84 tokens
128 takens
258 tokens
Sample
Original RGB
Sample
Sample
Sample
Sample #5
18
Sample
Original RGB
Sample
Sample
Sample
Sample #5
Figure 34, FlexTok samples for different number of tokens. We show FlexTok d1 8 B-d28 (trained on ImageNet- reconstructions for different numbers tokens for ImageNet-Il Ik validation set samples. We draw random samples from the rectified flow decoder given the same token sequences. FlexIok token sequences define distribution images that gets narrower. and more specific to the original image. the more tokens are used.
38

---

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
16
Sample
Original RGB
Sample
Sample
Sample #4
Sample #5
token
takens
tolkens
takens
18 takers
32 tokens
takens
128 tokens
258 tokens
Sample #1
Original RGB
Sample
Sample
Hares
Sample
Sample
token
tokens
tokens
tokens
16 tokens
32 tokens
64 tokens
128 tokens
256 tokens
Sample #1
Original
Sample #2
Sample
Sample
Sample
Figure 35. FlexTok samples for different number tokens. show FlexTok d1 8 8-628 (trained on ImageNet- 1k) reconstructions for different numbers of tokens for ImageNet-16 Ik validation set samples. We draw 5 random samples from the rectified flow decoder given same token sequences. F1 Flexin ox token sequences define a distribution over images that gets narrower, and more specific to the original image, the more tokens used.
39

---

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
K.2. FlexTok image reconstruction for different numbers of tokens and tokenizer sizes
token
tokens
4 tokens
8 tokens
16 tokens
32 tokens
64 tokens
128 tokens
256 tokens
FlezTok
d12-d12
Original RGB
FlexTok
d18-d18
FlexTok d18-d28
taken
takens
4 tokens
8 tokens
16 tokens
32 tokens
64 takens
128 tokens
256 tokens
FlexTok d12-d12
Original RGB
FlexTok d18-d18
FlexTok d18-d28
token
tokens
4 tokens
8 tokens
tokens
32 tokens
64 tokens
128 tokens
256 tokens
FlexTok d12-d12
Original RGB
FlexTok d18-d18
6
LenoeTes
FlexTok d18-d88
token
tokens
tokens
8 tokens
tokens
32 tokens
tokens
128 tokens
256 tokens
FlexTok d12-d18
Original RGB
FlexTok d18-d18
FlexTok d18-d28
Figure 36. FlexTok reconstructions for different numbers of tokens and model sizes. We show FlexTok (trained on ImageNet- 1k) reconstructions for different numbers of tokens and model sizes (d12-d12, d18-10 3--11. 8,, dl 8-d28) for ImageNet-II Ik validation set samples.
40

---

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
token
tokens
4 tokens
8 tokens
16 tokens
32 tokens
64 tokens
128 tokens
256 tokens
FlexTok
d18
Original RGB
FlexTok d18-d18
cai
FlexTok d18-d26
token
tokens
4 takens
8 tokens
16 tokens
32 tokens
64 tokens
128 tokens
256 tokens
FlexTok d12-d12
Original RGB
FlexTok d18-d18
FlexTok d18-d28
token
tokens
tokens
8 tokens
tokens
32 tokens
64 tokens
tokens
256 tokens
FlexTok d12 d12-d18
Original RGB
FlexTok d18-d18
FlexTok d16-d88
token
tokens
tokens
8 tokens
tokens
32 tokens
64 tokens
128 tokens
256 tokens
FlexTok d12-d12
Original RGB
FlexTok d18-d18
FlexTok d18-d88
Figure 37. FlexTok reconstructions for different numbers of tokens and model sizes. We show FlexTok (trained on ImageNet-IKg Ik) reconstructions for different numbers of tokens and model sizes (d12-d12, d18-d18, dl d18-d28) for ImageNet-Ie Ik validation set samples.

---

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
K.3. Image reconstruction comparison between FlexTok, TiTok, and ALIT
token
2 tokens
4 tokens
tokens
16 tokens
32 tokens
64 tokens
128 tokens
256 tokens
TiTok L-32,
TiTck B-64,
X
X
X
X
Original RGB
TiTck S-128
ALIT
X
X
X
X
X
(semi-large)
FlexTok d18-d88
taken
takens
tokens
tokens
16 tokens
takens
64 tokens
128 tokens
256 tokens
TiTok L-32,
TiTkk B-64,
X
X
X
X
X
Original RGB
TiTok S-128
ALIT (semii-arre)
X
X
X
FlexTok d18-d28
token
2 tokens
4 tokens
tokens
18 tokens
32 tokens
64 tokens
128 tokens
256 tokens
TiTok L-32,
TiTok B-64,
X
X
X
X
X
Original RGB
TiTok S-128
ALIT (semi-large)
X
X
X
X
FlexTok d18-d28
taken
tokens
tokens
tokens
16 tokens
takens
tokens
128 tokens
256 tokens
TiTok L-32,
TiTok B-64,
X
X
x
x
X
Original RGB
TiTok S-128
ALIT (semi-large)
X
X
X
X
FlexTok d18-d28
Figure Reconstruction comparison between FlexTok and baselines. We show FlexTok d18-d28 (trained on ImageNet- 1k) reconstructions for different numbers of tokens and for ImageNet- 1k validation set samples, and compare against three different TiTok (Yu al., 2024a) models, and ALIT (Duggal al., 2024).
42

---

FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
token
tokens
4 tolens
tokens
16 tokens
tokens
64 tolens
128 tokens
256 tokens
TiTok L-32,
B-64,
x
X
X
X
X
Original RGB
TTTok S-128
ALIT (semi-large)
X
X
X
X
FlexTok d18-d88
1 token
2 tokens
4 tokens
tokens
16 tokens
32 tokens
64 tokens
128 tokens
256 tokens
TiTok L-32,
TiTok B-64,
X
X
X
X
X
Original RGB
TiTok S-128
ALIT (semi-large)
X
X
X
X
FlexTok d18-d28
token
tokens
tokens
tokens
16
tokens
64 tokens
128 tokens
256 tokens
TiTok L-32.
TiTok B-64,
X
X
X
X
X
Original RGB
TiTk S-128
ALIT (semi-large)
X
X
X
X
FlexTok d18-d28
taken
2 tokens
4
tokens
tokens
16 tokens
32 tokens
64 tokens
128 tokens
256 tokens
TiTok L-32,
TiTok B-64,
X
X
X
X
X
X
Original RGB
TiTok S-128
ALIT (semi-large)
X
X
X
X
Flex/okk d18-d28
Figure 39. Reconstruction comparison between FlexTok and baselines. We show FlexTok 418-d28 (trained on ImageNet Ik) reconstructions for different numbers of tokens and for ImageNet- Ik validation samples, and compare against three different TiTok (Yu et al., 2024a) models, and ALIT (Duggal et al., 2024).
43

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
K.4. Class-conditional image generation visualizations
tokens
tokkens
256
arctic wolf (270)
valcano
cliff drop of (972)
balloon
panda (388)
(8B)
loggerhead sea turtle (33)
golden retriever (207)
lion (291)
Figure 40. FlexTok class-conditional image generations with varying numbers of tokens. From left to right the number of tokens used increases in powers of from 1 up to 256 tokens. Each row represents a different ImageNet- lk class index supplicd to the auto-regressive image generator conditioning. Images generated using the FlexTok di 8 8-d28 tokenizer combined with 1.33B parameter AR Transformer.
44

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
token
tokens
tokens
Tokens
tokens
tokens
64 tokens
tokens
256 tokens
Figure 41, FlexTok class-conditional image generations with varying numbers of tokens. From left to right the number of tokens
used increases powers of from up to 256 tokens. Images are generated using FlexZok d1 d18-d28 tokenizer combined with 1.33B parameter AR Transformer. The model is conditioned on the class label from ImageNet- Ik for "golden retriever" (207), using different random seeds each row.
45

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
token
tokens
tokens
8 Tokens
tokens
tokens
64 tokens
tokens
256 tokens
Figure 42, FlexTok class-conditional image generations with varying numbers of tokens. From left to right the number of tokens
used increases in powers of from up to 256 tokens. Images are generated using the FlexZok d18-d28 tokenizer combined with 1.33B parameter AR Transformer. The model is conditioned on the class label from ImageNet-Ik N for "flamingo" (130), using different random seeds for each row.
46

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
token
tokens
tokens
3 tokens
tokens
32 tokens
64 takens
128 takens
256 tokens
Figure 43. FlexTok class-conditional image generations with varying numbers of tokens. From left to right the number of tokens
used increases in powers of 2 from up 256 tokens. Images are generated using the FlexTok di B B-d23 tokenizer combined with a 1.33B parameter AR Transformer. The model is conditioned on the class label from ImageNet- Ik for "Macaw" (88), using different random seeds for each row.
47

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
K.S. Text-conditional image generation visualizations
taken
tokens
tokens
takens
16 tokens
32 tokens
64 tokens
128 tokens
256 tokens
A cutting board lopped with bread, mest and vegelables.
8
kitchen that the
4.5
big purple bus parked in parking spot
furry,
standing in rocky, weedly
A plate on wooden table full bread.
resting opan laptop computer.
large truck city
A few bags laying around in a living room.
Two birds sitting marsh area.
Figure FlexTok text-conditional image generations with varying numbers of tokens. From left right the number of tokens
used increases in powers of 2 from I up to 256 tokens. Images are generated using the FlexTok d18-d28 8 tokenizer trained on DFN combined with 3.06B parameter AR Transformer also trained on DFN. For each row the Transformer conditioned text embeddings of the prompts.
48

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
token
takens
tokens
tokens
tokens
32 tokens
64 takens
tokens
256 takens
Figure 45. FlexTok text-conditional image generations with varying numbers tokens. From right the number of tokens
used increases in powers of from I up 256 tokens. Images generated using the FlexTok d18-d22 tokenizer trained DFN combined with 3.06B parameter AR Transformer also trained on DFN, For each row the AR Transformer is conditioned on the text embeddings of the prompt "A furry, black bear standing in rocky, weedy, in the wild". but uses different random seed.
49

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
A photo of an astronaut riding horsa the forest. There river in front of them with water lilies.
Parsche parked
yellow
Dogs sitting around a poker table with bear bottles and chips. Their hands holding cards.
large white yacht in a calm bay on a sunny day
the
photo of four- leaf clover made of water.
TOUR
photograph squirrel holding holdding longbow
corgi's head depicted an explosion nebula
Figure 46. FlexTok text-conditional image generations with varying numbers of tokens. From left right the number of tokens
used increases in powers of from up to 256 tokens. Images are generated using the FlexTok d18-d28 tokenizer trained on DFN combined with 3.06B parameter AR Transformer also trained DFN. For each row the AR Transformer is conditioned the text embeddings of a different PartiPrompt (Yu et al., 2022).
50

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
tokken
2 tokens
takens
8 tokens
18 takens
32 tokens
64 tokens
128 tokens
256 toliens
black and orange yin-yang symbol with tigar's heads instead airoles
wall with framed watercolor painting mountain
BAAL
a real llamingo reading a large open book. a big stack of books is piled up next to it. dsir photograph.
rocket
a cityscape at night with a full moon
professional grand canyon
Water tower next to a deserted road
mountain
stained glass window depicting calm yrannosaurus rex
Figure 47. FlexTok text-conditional image generations with varying numbers of tokens. From left right the number of tokens
used increases in powers of from up to 256 tokens. Images are generated using the FlexTok d18-d28 tokenizer trained on DFN combined with 3.06B parameter AR Transformer also trained DFN. For each row the AR Transformer is conditioned the text embeddings of a different PartiPrompt (Yu et al., 2022).
51

---

FlexTok: Resampling Images into ID Token Sequences of Flexible Length
token
tokens
tokens
tokens
tokens
32 tokens
64 takens
128 tokens
256 tokens
Figure 48. FlexTok text-conditional image generations with varying numbers of tokens. From left to right the number of tokens used increases in powers of from up to 256 tokens. Images are generated using the FlexTok d18-d28 tokenizer trained DFN combined with 3.06B parameter AR Transformer also trained DFN. For each row the AR Transformer is conditioned the text embeddings of the prompt "A blue Porsche 356 parked in front of a yellow brick wall", but uses a different random seed.
52


Reviews:
[{"review_id": "abj5zglYbe", "reviewer": "Reviewer_MMdz", "summary": "The paper proposes FlexTok, a method for improving the tokenizer (VAE compression) used in image generation frameworks. Like previous approaches (TiTok, ALIT), FlexTok compresses 2D images into 1D tokens initialized as learnable registers. These tokens interact with encoded image patch tokens via attention mechanisms. Unlike TiTok and ALIT, FlexTok’s decoder is trained with a flow matching loss rather than standard reconstruction loss, enabling multi-step denoising decoding. Additionally, FlexTok introduces a unique causal attention masking structure: encoded image patches attend only among themselves and not to registers, whereas register tokens attend to all patches but follow a causal pattern among themselves (the i-th register attends to j-th registers only if i ≥ j). The model further uses token-dropping techniques during training, similar to ElasticTok, allowing variable-length token representations with earlier tokens representing more general information and later tokens representing details. The second-stage generative model, based on an autoregressive approach, progressively generates finer-grained images as more tokens are used.\n\n## update after rebuttal\nI am willing to update my score to weak accept since ALIT and ElasticTok are concurrent works. However, my concerns about reconstruction quality, the inference cost, and the motivation are not fully addressed.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer MMdz for the thoughtful and constructive feedback. Below we address the main points raised:\n\n**1. Single-token generation claim**\n\nImage tokenization is commonly performed with lossy autoencoders that abstract away imperceptible information, meaning all tokenizer decoders (whether trained with diffusion/flow methods or GANs) are inherently generative to some extent. The degree to which they must be \"generative\" directly corresponds to the amount of compression. At the coarsest level (single FlexTok token = 2 bytes), our representation necessarily operates at a highly semantic level, conceptually similar to semantic tokenizers like SEED (Ge et al. 2023). At the finest level (256 tokens = 512 bytes), we achieve compression comparable to classical 2D grid tokenizers like VQ-GAN (also, see response to reviewer ekCv). FlexTok's unique strength is providing a single model that learns these different hierarchies, effectively offering an alternative way to describe images in a coarse-to-fine manner. The single-token scenario is indeed best interpreted as semantic conditioning rather than pixel-level compression.\n\n**2. Computational overhead (25-step flow matching decoder)**\nWe acknowledge that FlexTok's rectified flow decoder adds computational complexity during inference (though encoding remains efficient). We explicitly chose this architecture because it consistently maintains high reconstruction fidelity across a wide range of token counts (Fig. 4). Importantly, decoding happens after the autoregressive (AR) generation step, which is already computationally intensive, so the flow decoding step adds a constant overhead rather than introducing an entirely new computational bottleneck. We also anticipate that common distillation methods (e.g., consistency decoders, Reflow) can substantially lower inference complexity.\n\n**3. Token-level faithfulness (single-image reconstruction quality)**\nRegarding reconstruction quality at 256 tokens (512 bytes), it's important to recognize that this still represents extremely high compression: a full-color image compressed to just 512 bytes will naturally show some loss of detail. This token count was explicitly chosen to match standard tokenizers (e.g., VQ-GAN, LlamaGen), allowing direct and fair comparisons. At this standard token count, FlexTok achieves reconstruction fidelity comparable to these established methods (see the comparison in response to reviewer ekCv). The visible imperfections at this compression rate are expected, and our results show a clear log-linear trend (Fig. 4), strongly suggesting that higher token counts (e.g., 1024 tokens or more) would yield increasingly faithful reconstructions. This trade-off between compression and fidelity is fundamental to all image tokenization approaches, and exploring higher-token-count scenarios is a natural next step.\n\n**4. Technical novelty compared to TiTok, ALIT, ElasticTok, and DALL-E 3**\nRegarding FlexTok's novelty beyond causal attention masking: while ElasticTok and ALIT indeed share similar concepts, these methods were developed concurrently and independently. FlexTok directly builds on TiTok's 1D tokenizer framework and diffusion/flow-matching ideas, but introduces several critical new technical components: specifically, causal attention masking combined explicitly with nested token dropout and rectified flow decoding. Crucially, these innovations collectively enable hierarchical tokenization, smoothly transitioning from coarse, semantic-level representations to detailed pixel-level reconstructions within one tokenizer. This hierarchical tokenization was not shown by TiTok or other prior methods. Additionally, we provide a detailed analysis of this hierarchical behavior in generative settings, clearly showing its practical benefits and trade-offs.", "claims_and_evidence": "The authors claim significant benefits from their method, particularly highlighting their capability of generating images using even a single token. However, several of these claims are problematic:\n\n- **Single-token generation claim:**  \n  The authors assert that their model can encode an image into just one token. However, this claim is unfair or misleading, as the decoding process itself is multi-step flow matching (a generative process rather than a faithful reconstruction). Thus, the single-token encoding is effectively used as a conditioning input similar to class or text tokens, rather than a compressed latent representation.\n\n- **Reduction of compute or acceleration claim:**  \n  Due to the iterative, generative nature of the decoder (25-step flow matching), this method requires significantly more computational resources. This contradicts one key motivation for employing latent-based generative models, which typically aim at computational efficiency.\n\n- **Token-level faithfulness:**  \n  Reconstruction quality of individual images (crucial for downstream tasks such as editing or conditional generation) appears limited. For instance, Figure 3 shows significant discrepancies between generated images and ground truth, even with 256 tokens (e.g., misaligned dog tails), indicating poor pixel-level or even patch-level alignment.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are reasonable in the context of image generation research. However, the experimental evaluation has critical limitations:\n\n- Reconstruction quality at the single-image level is not sufficiently addressed or emphasized.\n- Claims of single-token generation and compression efficiency are misleading due to the generative, iterative decoder.", "theoretical_claims": "The paper does not present explicit theoretical claims or analyses.", "experimental_designs_or_analyses": "Key weaknesses identified in experimental design:\n\n- Lack of adequate evaluation of single-image reconstruction fidelity. Given the latent-based approach, individual reconstruction quality is crucial but is largely overlooked.\n- The experiments presented (particularly in Figure 3) clearly illustrate significant quality gaps even at relatively high token counts (256 tokens), undermining claims of efficient and faithful representation.", "relation_to_broader_scientific_literature": "FlexTok builds upon existing literature, notably TiTok, ALIT, and ElasticTok, which have previously introduced concepts of variable-length, learnable register tokens. The flow matching decoding strategy aligns closely with ideas presented earlier (e.g., OpenAI's DALL-E 3), where diffusion-based decoders was used (see Consistency Decoder). Thus, the technical novelty of FlexTok appears limited, with incremental advances primarily in the causal attention masking strategy.", "essential_references_not_discussed": "No essential missing references identified.", "other_strengths_and_weaknesses": "**Strengths:**\n\n- Causal attention mask is a novel modification.\n- Interesting concept of flexible token lengths with importance ordering.\n\n**Weaknesses:**\n\n- Misleading or unfair claims regarding single-token generation and compression effectiveness.\n- Decoder complexity and computational overhead contradict original VAE motivations (speed and efficiency).\n- Poor single-image reconstruction fidelity limiting applicability to editing or conditional generation tasks.\n- Limited overall technical novelty given strong reliance on prior works (TiTok, ALIT, DALL-E 3).", "other_comments_or_suggestions": "The authors should clarify their claims, explicitly distinguishing their method as employing an additional generative decoding process rather than true latent compression. More focus should be placed on improving reconstruction fidelity and clearly discussing limitations inherent to the proposed approach.", "questions_for_authors": "1. Could you clarify why you claim that encoding images to a single token is feasible or meaningful, given your decoder itself is a multi-step generative model rather than a direct reconstruction?\n\n2. Given the computational overhead introduced by iterative flow matching decoding (25 steps), how do you justify the additional complexity against typical VAE motivations (speed, compression)?\n\n3. Reconstruction fidelity appears severely limited even at relatively high token counts (e.g., 256 tokens). Could you provide deeper analysis or experimental insights into how your method might address pixel-level or patch-level misalignments, which are crucial for tasks like conditional generation or editing?\n\n4. Apart from the novel causal attention masking, could you clearly summarize the distinctive technical contributions of FlexTok beyond existing works such as TiTok, ALIT, and the diffusion-based decoding approaches found in DALL-E 3?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EBjmEFXj0B", "reviewer": "Reviewer_6zZ5", "summary": "In this paper, the authors introduce a tokenizer that maps 2D images into variable-length, ordered 1D token sequences. This tokenizer allows images to be represented with a flexible number of tokens based on their content. In addition, an autoregressive model leverages this approach to achieve high-quality generation results with fewer image tokens. The authors conduct extensive experiments to validate the effectiveness of the proposed method.\n\n## update after rebuttal\nI appreciate the authors' thorough response and the effort they put into the rebuttal. However, I still have concerns regarding the limitations of using a fixed number of register tokens, as well as the lack of a comprehensive system-level comparison for text-conditional image generation. Therefore, I am keeping my score unchanged.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 6zZ5 for the thoughtful and constructive feedback. Below we address the main points raised:\n\n**1. Higher computational cost of rectified flow decoder and token-count limitations**\n\nPlease see our response to reviewer MMdz. Our preliminary experiments suggested that higher token counts can significantly reduce reconstruction errors, and enable decoding with fewer steps, however, for simplicity of training subsequent autoregressive models on the token sequences, we decided to choose 256 tokens as the upper-bound for this submission.\n\n**2. System-level comparison for text-to-image generation**\n\nWe fully understand the motivation for requesting comparisons against external text-to-image baselines. However, unlike the somewhat standardized class-conditional ImageNet setting, proper comparison on text-to-image generation with external baselines is usually extremely difficult and nuanced due to differences in compute and dataset (size, caption quality, diversity, aesthetics, similarity to COCO, etc.) having significant impact on downstream evaluations (e.g. see \"On the Scalability of Diffusion-based Text-to-Image Generation\", Li et al. 2024). For that reason we decided to perform a controlled experiment in which we train a 2D grid tokenizer with the same data, compute, and rectified flow decoder objective, and perform autoregressive generation on its tokens. The results in Fig. 6 suggest that FlexTok performs comparably to classical 2D grid tokenizers at 256 tokens, but offers more flexibility overall.\n\n**3. Additional standard metrics for class-conditional generation**\n\nWe appreciate the suggestion and provide the requested metrics (Inception Score, Precision, Recall, and gFID) in the table below. Our results are comparable to state-of-the-art methods such as VAR-d30 (323.1 IS, 0.82 precision, 0.59 recall, 1.92 gFID) and TiTok (gFID between 1.97 and 2.77 depending on tokenizer choice) across a broad range of token counts. We will incorporate these results into the camera-ready version of the paper.\n\n| #Tokens | IS | Precision | Recall | gFID |\n| --- | --- | --- | --- | --- |\n| 1 | 236.47 | 0.83 | 0.53 | 3.14 |\n| 2 | 238.07 | 0.82 | 0.57 | 2.51 |\n| 4 | 226.77 | 0.80 | 0.60 | 2.00 |\n| 8 | 266.48 | 0.82 | 0.61 | 1.82 |\n| 16 | 277.45 | 0.82 | 0.61 | 1.75 |\n| 32 | 284.99 | 0.82 | 0.61 | 1.71 |\n| 64 | 286.40 | 0.82 | 0.61 | 1.76 |\n| 128 | 275.63 | 0.82 | 0.61 | 1.89 |\n| 256 | 258.33 | 0.80 | 0.61 | 2.45 |\n\n**4. Additional suggestion: summary of appendix content**\n\nThanks for the helpful suggestion. Adding a concise appendix summary at the start is indeed beneficial. We will add this to the camera-ready version.", "claims_and_evidence": "Strengths\n+ The claims regarding the limitations of existing generative models are correct and wildly recognized in the field of generative models. \n+ The proposed method is reasonable  and easy to understand. It encodes images with variable-length tokens based on image complexity. \n\nWeaknesses\n\nNone", "methods_and_evaluation_criteria": "Strengths\n+ The proposed method is both intuitive and reasonable. It is an effective technique to employing rectified flow decoder to alleviate the blurry reconstruction caused by fewer register tokens. The nested dropout and causal attention masks also benefit the learning of visual vocabulary and AR generation. \n+ The evaluation benchmark is reasonably appropriate for assessing the effectiveness of the proposed method.\n\nWeaknesses\n- Compared to the decoders used in existing tokenizers, the rectified flow decoder requires higher computation costs, due to its extensive denoising steps.\n- The pre-defined maximum number of register tokens may not suitable for extremely complex images, and determining an optimal setting remains an open and difficult problem.", "theoretical_claims": "This paper does not include any theoretical claims.", "experimental_designs_or_analyses": "Strengths\n+ The superior results on both class-conditonal and text-conditional image generation demonstrate the effectiveness of the proposed method. This paper also provides exhaustive ablation studies to assess each key component.\n+ The qualitative results across different numbers of tokens make the efficiency of FlexTok more apparent and intuitive. \n\nWeaknesses\n- It is better to show system-level comparison on text-conditional image generation, not only the ablation studies in the main text. \n- For class-conditional generation, Inception score (IS), Precision, and Recall are the primary evaluation metrics, which is widely used in the generative field. Thus, it is essential to include them in this paper.", "relation_to_broader_scientific_literature": "To my knowledge, the proposed method in this paper is new.", "essential_references_not_discussed": "To my knowledge, there is no other works that should be discussed.", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "It would be beneficial to provide a summary of the appendix content at the beginning.", "questions_for_authors": "I acknowledge the novelty of the proposed method, despite its limitations regarding the pre-defined number of register tokens and the computation cost of the decoder. Thus, I am inclined to rate this paper as accept.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WWf8N4iHuG", "reviewer": "Reviewer_ekCv", "summary": "This paper introduces FlexTok, a novel 1D tokenizer that can encode images with variable token lengths. It combines casual masking and nested dropout in training to force the tokenizer to learn to reconstruct an image with a varying number of tokens. This strategy further promotes the tokenizer to encode images in a coarse-to-fine order, where the initial tokens encapsulate semantic and geometric concepts, and the subsequent tokens progressively capture finer details. To ensure reconstruction quality at extreme compression rates (e.g., using only 1-2 tokens), FlexTok employs a rectified flow model as its decoder. The method achieves strong performance in both reconstruction and generative tasks on ImageNet and COCO.\n\n## update after rebuttal\n\nThank the authors for their detailed response. Considering that FlexTok shows diminishing improvements for long token sequences, which limits the method's upper bound, and considering the inefficiency issues raised by other reviewers, I will maintain my initial rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer ekCv for the thoughtful and constructive feedback. Below we address the main points raised:\n\n**1. Variable-length generation limitation (fixed token length in generator)**\n\nTo clarify: in our current setup, we train a single autoregressive (AR) model capable of generating a full 256-token sequence. During inference, shorter outputs are obtained simply by early stopping this sequence at the desired length. While effective, you're right that it could be even more flexible if the AR model itself determined when to halt generation, and this is an interesting future research direction. We expect that we can create such halting conditions by augmenting the training data with per-token-subsequence reconstruction metrics (e.g., MSE, DreamSim), and when training the stage 2 AR model, simply truncating the training sequences if the score reaches a certain pre-defined threshold.\n\n**2. Weakness at longer token sequences (limited improvement beyond ~32 tokens)**\n\nThe reviewer raises an insightful point about long-sequence performance. Indeed, our \"Pow2\" nested dropout schedule intentionally emphasizes shorter sequences during training. This design optimizes FlexTok’s ability to reconstruct images effectively at extreme compression (few tokens), a core contribution of our work. However, it is correct that this strategy results in diminishing improvements for longer sequences (>32 tokens). Adjusting dropout sampling strategies to more evenly balance short and long sequences could potentially mitigate this trade-off, and we appreciate the suggestion here.\n\n**3. Semantic-level vs. pixel-level reconstruction (PSNR/SSIM comparisons)**\n\nWe agree, rFID is not a good measure for pixel alignment. We measure rFID to demonstrate that the FlexTok decoder is capable of producing outputs that could \"plausibly\" come from the image distribution, no matter the number of tokens given. To measure pixel-level reconstruction alignment, we show MAE and DreamSim in Fig 4, and observe a roughly log-linear relationship between the scores number of tokens. We additionally show PSNR and SSIM in the table below for various number of tokens, and find that at 256 tokens used, FlexTok reaches comparable compression performance to common 2D-grid tokenizers that use 16x16 discrete tokens.\n\nFlexTok d18-d28 reconstruction metrics on IN1K validation set, resolution 256x256:\n\n| # Tokens | PSNR | SSIM |\n| --- | --- | --- |\n| 1 | 9.35 | 0.187 |\n| 2 | 10.25 | 0.222 |\n| 4 | 11.51 | 0.254 |\n| 8 | 11.90 | 0.269 |\n| 16 | 13.05 | 0.304 |\n| 32 | 13.96 | 0.330 |\n| 64 | 14.34 | 0.343 |\n| 128 | 15.90 | 0.407 |\n| 256 | 17.70 | 0.489 |\n\nComparison with common discrete tokenizer baselines (numbers from Cosmos paper):\n\n| Model | # Tokens | PSNR | SSIM |\n| --- | --- | --- | --- |\n| Open-MAGVIT2 | 16x16 | 17.00 | 0.398 |\n| LlamaGen | 16x16 | 18.38 | 0.338 |\n| Cosmos-0.1 | 16x16 | 20.49 | 0.518 |\n\n**4. Impact of REPA vs. longer training schedules**\n\nThis is a good question, however, the difference in convergence speed is so significant that we found it computationally too expensive to ablate this. We expect that the ~17x convergence speedup (in terms of FID) demonstrated in the original REPA paper may roughly translate to our setting too. Unfortunately we are unable to add qualitative examples to this text-only reply, but we find the non-REPA reconstructions (after training for the same number of steps) to be significantly worse in terms of fidelity, and overall less semantic. We will add a discussion of these points as well as qualitatives to the camera-ready.", "claims_and_evidence": "The claims in this paper supported by experimental results or prior studies.", "methods_and_evaluation_criteria": "The paper mainly uses rFID, MAE, and DreamSim for reconstruction quality evaluation, and leverages gFID, top-1 accuracy, and CLIP score for generation quality evaluation. These metrics are proper for the study of an visual tokenizer.", "theoretical_claims": "There is no proof or theoretical claim in this paper.", "experimental_designs_or_analyses": "The experimental designs are solid and fair.", "relation_to_broader_scientific_literature": "In recent literatures, there is a growing interest in 1D tokenizers which improve computational efficiency (e.g., encoding an image with fewer tokens) while maintaining competitive quality. Prior works like TiTok are fixed in token length. FlexTok takes a step forward to enable variable length in tokenization. Besides, by incorporating advancements from tokenizers with rectified flow decoding, FlexTok further reduces the minimum token length from 32 to 1.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths:**\n- This paper studies a very important problem in image tokenization. The idea of 1D and variable length tokenization in FlexTok is novel and interesting, which enables images to be modeled in a similar way to language sequences.\n- Compared to traditional 2D tokenizers and prior 1D tokenizers, FlexTok demonstrates improved reconstruction and generation performance, while using fewer number of tokens.\n- The paper is well-written. The experiments are solid and comprehensive.\n\n**Weaknesses:**\n- It seems that variable length in the tokenization stage does not generalize to the generation stage. That is, although a single FlexTok tokenizer can handle variable length of tokens, the generator is limited to a fixed token length. As a result, it basically needs multiple generators to support variable length generation. This constrains the practical use of FlexTok in generation. (Maybe I have misunderstandings here, the authors can correct me.)\n- FlexTok is relatively weak in long-sequence reconstruction and generation. In Figure 4, 6, 7, it can be seen that increasing the number of tokens beyond 32 has minor improvements in rFID, and even negatively impacts gFID. This may be attributed to the trade-off in performance between low number and high number of tokens, as shown in Appendix Table 4. FlexTok adopts the “Pow2” dropout strategy, which is favored for short token length but under-samples long sequences in training.", "other_comments_or_suggestions": "I think this is an overall good paper, but could benefit from further enhancing the flexibility in tokenization and generation. For example, the default setting of FlexTok only supports reconstruction for a predefined set of token lengths, rather than a truly random number of tokens. Besides, instead of having a predefined token length in generation, it would be more exciting to see the generator evolve to decide the number of tokens to generate, similar to how language models generate texts.", "questions_for_authors": "1. From my experience, 1D tokenizers like TiTok are more adept at 'semantic-level' reconstruction than 'pixel-level' reconstruction. Therefore, having a low rFID does not guarantee that the tokenizer faithfully reconstruct the images. I wonder whether FlexTok suffers from the same problem. Could the authors provide a comparison with other tokenizers on the PSNR and SSIM metrics?\n\n2. As shown in Appendix Figure 10, REPA greatly accelerates the convergency of FlexTok training. I am curious about whether FlexTok could achieve similar performance without REPA but longer training schedules. It would be better if the authors could provide some visualizations of the reconstruction results without REPA.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zrQJrXe9hO", "reviewer": "Reviewer_jx6L", "summary": "This paper presents a novel method for tokenizing an image into a one-dimensional token sequence, which allows for flexible image representation and processing. Most of the existing VAE/VQVAE methods employ quantization on 2D grids, thus the token size is proportional to the image size. This paper proposes a novel VQ method that resamples images of varying sizes into a fixed-size sequence. It can be combined with AR architecture for image generation. \nThis paper presents thorough experiments to analyze different modules and hyperparameters for the generation. From experiments, the paper achieves performance comparable to that of existing SOTA image generation methods.\n\n\n## update after rebuttal\nI thank the authors for their detailed response. Most of my concerns have been solved. However, as the reconstruction uncertainty and diffusion/AR sampling will affect the method's performance, the main paper should enclose a detailed discussion on them in the final version.  Considering several revisions are required, I will keep my initial rate.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer jx6L for the thoughtful and constructive feedback. Below we address the main points raised:\n\n**1. Relation between image size, complexity, and token count**\n\nThis is a good point, and we haven't explored this explicitly yet. We performed all our experiments at 256x256 resolution specifically to enable direct comparison with standard methods (e.g., VQ-GAN, LlamaGen). While FlexTok could support variable-resolution inputs (e.g. using architectures like NaViT), that's a separate research direction we leave for future work. We also agree the interplay between image complexity, resolution, and token counts is an interesting open question.\n\n**2. “Lossless” compression**\n\nWe appreciate this observation about reconstruction quality at low token counts. It's important to clarify that all image tokenizers are inherently lossy, just to different degrees. Some tokenizers (e.g., SEED (Ge et al. 2023)) focus purely on semantic features without pixel accuracy, while others (e.g., VQ-GAN) are more pixel-aligned. FlexTok sits in between, explicitly transitioning from semantic-level reconstructions at low token counts (<16 tokens) toward highly pixel-aligned reconstructions as token count increases (e.g. at 256 tokens). We explicitly measure pixel-level reconstruction fidelity using Mean Absolute Error (MAE), which steadily improves with more tokens (Fig. 4). Regarding the rFID metric, we agree that additional reconstruction metrics would strengthen our analysis; we'll include PSNR and SSIM results compared directly to baseline tokenizers like VQ-GAN (see the comparison in response to reviewer ekCv).\n\n**3. Why a diffusion-type decoder rather than VQGAN?**\n\nWe chose a diffusion-type decoder (rectified flow) exactly because it models conditional uncertainty. When using few tokens, the degree of compression is high and reconstructions naturally have uncertainty; the diffusion decoder handles this gracefully, producing plausible, semantically coherent outputs rather than blurry averages. As the number of tokens increases, the uncertainty naturally reduces, making reconstructions progressively more accurate and deterministic (see image reconstructions in Appendix J.1). A VQGAN-style decoder wouldn't offer this flexible control over reconstruction uncertainty.\n\n**4. AR sampling vs. diffusion sampling impact**\n\nWe found both AR sampling (top-k, top-p, temperature) and diffusion sampling parameters influence image quality and diversity. AR sampling tends to be stable across reasonable settings (see Appendix F), while diffusion decoding, particularly adaptive projected guidance (Sadat et al., 2024), had a significant impact on final image quality. In short: both matter, but diffusion guidance is particularly critical. We'll clarify this explicitly.", "claims_and_evidence": "1. The paper proposes the token numbers of previous methods are proportional to image size, but the method is independent of size and is affected by the complexity of the image. The claim is clear but it lacks experiments to support this. The method quantizes the image to 1-D sequence tokens, but all experiments are done on 256*256 size. The paper does not discuss: a) Does the image size affect the token sizes? For larger images, does it need more tokens for representation? b) How to define the complexity of the image? The paper lacks an experiment to analyze this. If an image' content is 'simple', does the method only require a fixed-size token to represent it no matter the size of the image?\n\n2. FlixTok is an image tokenizer, which should losslessly compress and reconstruct an image. However, in Figure 3, when the tokens are less than 16, the reconstructed image is different from the original. It is normal the reconstruction should be worse with limited tokens, but the content should be similar to the original images. From this result, could it be called a tokenizer? It is more like a generation model.", "methods_and_evaluation_criteria": "The problem is present clearly and the method can solve it.", "theoretical_claims": "The theoretical claims are clear and correct.", "experimental_designs_or_analyses": "1. The paper lacks the experiments present in the 'Claims And Evidence' part.\n\n2. The FlexTok is an image tokenizer. Ideally, it should losslessly compress an image to tokens and then reconstruct it. In experiments, it only presents two figures to analyze the reconstruction. It lacks enough experiments and comparisons with other methods, especially 2-D grid tokenization methods. Furthermore, Figure 4, only presents rFiD, which can only analyze the GT and prediction distribution. PSNR can better compare the compression losses. \n\n3. The decoder of FlexTok is a diffusion-type model. Does the initial noise affect the reconstruction quality and appearance during inference? Is this uncertainty in inference optimal for the reconstruction? Why not employ a VQGAN decoder and corresponding supervision? The paper should give more discussions.\n\n4. In generation, it employs an AR architecture. The sampling strategy of AR (such as TopK and TopP) and diffusion tricks both affect generation quality and diversity. Which one is more important? The paper lacks a discussion on this.", "relation_to_broader_scientific_literature": "No more specific contributions to a broader scientific literature.", "essential_references_not_discussed": "All important references have been enclosed in the paper.", "other_strengths_and_weaknesses": "All weaknesses have been present in'Experimental Designs Or Analyses' and 'Claims And Evidence'.", "other_comments_or_suggestions": "No more comments and suggestions.", "questions_for_authors": "No more questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for improving the tokenizer used in image generation frameworks. The decoder is trained with a flow matching loss, which enables multi-step denoising decoding.

The reviewers found the flexible token lengths interesting, and the evaluation reasonable despite the shortcomings in the reconstruction. There were some concerns raised during the rebuttal, but the authors' discussion resolved most of them. Minor concerns, such as the motivation for the work, remained open.

Despite the remaining concerns, the reviewers recommend the acceptance of the paper. Thus, I also recommend it.
Decision: Accept (poster)