Highly Compressed Tokenizer Can Generate Without Training
Authors: Lukas Lao Beyer
Tianhong Li
Xinlei Chen
Sertac Karaman
Kaiming He
Venue: ICML 2025 poster (2025)
Keywords: image tokenizer
1D tokenizer
autoencoder
generative model
text-to-image generation
image editing
training-free
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: Tokenizers with very high compression ratio possess an expressive latent space suitable for image generation via latent space manipulation.

Abstract: Commonly used image tokenizers produce a 2D grid of spatially arranged tokens. In contrast, so-called *1D* image tokenizers represent images as highly compressed one-dimensional sequences of as few as 32 discrete tokens. We find that the high degree of compression achieved by a 1D tokenizer with vector quantization enables image editing and generative capabilities through heuristic manipulation of tokens, demonstrating that even very crude manipulations -- such as copying and replacing tokens between latent representations of images -- enable fine-grained image editing by transferring appearance and semantic attributes. Motivated by the expressivity of the 1D tokenizer's latent space, we construct an image generation pipeline leveraging gradient-based test-time optimization of tokens with plug-and-play loss functions such as reconstruction or CLIP similarity. Our approach is demonstrated for inpainting and text-guided image editing use cases, and can generate diverse and realistic samples without requiring training of any generative model.

Highly Compressed Tokenizer Can Generate Without Training
Lukas Lao Beyer' Tianhong Li Xinlei Chen
Sertac Karaman
Kaiming He?
Abstract
seed
optimization
Commonly used image tokenizers produce a 2D grid of spatially arranged tokens. In contrast, so- called ID image tokenizers represent images as highly compressed one-dimensional sequences of few as 32 discrete tokens. We find that the high degree of compression achieved by ID to- kenizer with vector quantization enables image editing and generative capabilities through heuris- tic manipulation of tokens, demonstrating that even very crude manipulations - such as copy- ing and replacing tokens between latent represen- tations of images enable fine-grained image editing by transferring appearance and semantic attributes. Motivated by the expressivity of the ID tokenizer's latent space, we construct an image generation pipeline leveraging gradient- test- time optimization of tokens with plug-and-play loss functions such as reconstruction or CLIP sim- ilarity. Our approach demonstrated for inpaint- ing and text-guided image editing cases, and can generate diverse and realistic samples with- out requiring training of any generative model. Code is available at https://github. httpbbb.ttt.///c//gg///. com/ lukaslaose Mikkaslaobeyer/tokere
(a)
(b)
seed
2
objective
maximize
TiTok
32 discrete
TiTok
encoder
tokens
decoder
Figure 1. Without further training, pretrained tokenizer can perform generative tasks such as (a) text-guided editing and (b) inpainting. Because the highly compressed latent space of ID tokenizers enables generative capabilities via direct token manipu- lation, generate using test-time optimization of tokens with (a) CLIP similarity or (b) reconstruction objectives, without training any dedicated generative model.
While the generative models used in such an image genera- tion pipeline have received much attention, recently intro- duced /D) tokenizers highlight that highly compressed latent spaces provide significant leverage in improving generation performance. For example, the TTTok 11) tokenizer achieve order of magnitude runtime reduction without compromising sample quality by representing an image in as few as 32 discrete tokens (Yu et al., 2024b).
1. Introduction
We argue that scaling image tokenizers to higher compres- sion ratios requires powerful decoders that may themselves be viewed generative models. As thought experiment, consider tokenizer that encodes an image into just single discrete token from finite codebook. Then, the decoder for such a "tokenizer" is faced with generative modeling task, if its output is to be successful in capturing the data distri- bution. Supposing we had a decoder for this single-toke tokenizer at hand, using as a generative model would be straightforward: an image could be generated by sampling random token from the codebook and decoding it. Given the very high compression offered by ID tokenizers, we wish to find out to what extent these tokenizers can indeed be viewed as generative models.
standard modern image generation pipeline consists tokenizer an autoencoder which embeds an image into compact latent space operating this latent space. Commonly, this generative model leverages some form of iterative refinement such as diffusion (Rombach et al., 2022; Peebles & Xie, 2023), autoregression (Esser et al., 2021; Ramesh et al., 2021; Razavi et al., 2019), masked modeling (Chang et al., 2022), or combination of those approaches (Li al., 2024b; Zhou al., 2025; Xie et al., 2024).
well generative model
'MIT LIDS 'MIT CSAIL "Meta FAIR. Correspondence to: Lukas Lao Beyer <llbe mit.edu>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Through series of experiments, we demonstrate that sim- ple latent space manipulations of tokens can result in im-

---

Highly Compressed Tokenizer Can Generate Without Training
age editing capabilities typically associated with generative models. Building upon this insight, we find that the latent space of ID tokens is amenable to straightforward gradient- based optimization of a variety of objectives. By optimizing a reconstruction loss, a pretrained tokenizer can perform inpainting tasks; similarly, optimizing CLIP similarity (Rad- ford et al., 2021), the tokenizer can perform text-guided image editing (Figure 1).
2021) or gradient-based optimization (Patashnik et al., 2021) in the latent space of a pretrained GAN. Another approach is to represent visual information in the embedding space of frozen text-to-image models (Gal et al., 2023; Wang et al., 2025), enabling image generation with rich visual informa- tion as part of the input prompt. However, we focus on the setting where only a tokenizer, not a full-blown generative model like a GAN or text-to-image model. is available.
Combining the text-guided token optimization with retrieval from a small set of seed images, we find that it is possible to generate realistic and diverse samples without training any dedicated generative model.
Unconditional generation. Unconditional generative mod- eling is attractive due to the difficulty associated with large- scale annotation. Many generative models benefit greatly from class labels available at training time (Dhariwal & Nichol, 2021; Bao et al., 2022), although recently the gap in performance has been bridged with the help of self- supervised pretrained encoders (Li et al., 2024a). Alter- natively, geni-parametric generation assumes access to dataset of images at test time (Casanova et al., 2021; Bordes et al., 2022; Blattmann et al., 2022; Sheynin et al., 2023; Zhou et al., 2023). In our case, we view the pretrained tokenizer as an unconditional generative model in the semi- parametric setting, which we can guide with the desired conditioning signal at test-time.
2. Background
Visual tokenization. Many image generation methods rely on tokenization to compress input images into a lower- dimensional latent space for efficiency. A variational au- toencoder ( (VAE) (Kingma & Welling, 2014) can be used to produce continuous tokens to be modeled with diffusion (Rombach et al., 2022). To further regularize the tokenizer's latent space and ease the task of modeling distributions over tokens autoregressively (Sun al., 2024; Li et al., 2024b), VAE (van den Oord et al., 2017; Razavi et al., 2019; Mentzer et al., 2024; Yu et al., 2024a) applies vector quantization to produce discrete-value tokens. Perceptual (Zhang et al., 2018) and adversarial losses during training (Esser et al., 2021; Chang et al., 2022) allow for visually compelling reconstructions under strong lossy compression. Note that most image tokenizers produce 21) grid of la- tents due their convolutional (Chang et al., 2022) vision transformer (Yu et al., 2023; Cao et al., 2023) architectures.
3. Latent Space of 1D Tokens
In this section we perform a series of simple experiments us- ing a pretrained TiTok (Yu et al., 2024b) 1D tokenizer to bet- ter understand the latent space of ID tokens and determine whether it provides a useful representation for latent-space editing of images.
3.1. Token Position Is Key Token Semantics
The spatial structure imposed by standard tokenizers producing a 2D grid of latents can prevent each individual token from describing global attributes of the input image, instead containing information only about a limited spatial extent (Cao et al., 2023; Rao et al., 2021). In contrast, the TYTok (Yu et al., 2024b) ID tokenizer learns sequence of latents with no particular spatial arrangement. This allows each token to capture global information about the input image and enables extremely high compression ra- tios (e.g. 32 discrete tokens) because global redundancies in the input can be exploited. Further background is provided in Section A.
ID tokenization.
Since the TiTok latents receive a positional encoding, one might wonder whether the 1D tokens disentangle seman- tics in the sense that certain token positions correspond to certain interpretable, high-level attributes. To investigate this question, we propose to analyze the TiTok encodings of several different partitions of the ImageNet validation dataset. Since there is no commonly agreed upon disentan- glement metric (Locatello et al., 2019), in the following we introduce a tn-ttoke--pssitiin importance metric that we later verify to closely align with experimental results.
Start by tokenizing each image I in the dataset with the ID tokenizer, obtaining the tokenized representation x := Tok(Z)= = [x(1), ...., x(K),, where K denotes the number of image tokens produced by the tokenizer. Each token (*(k) E {1...., [D]} refers to a particular D-dimensional entry z(k) =- (1) RRD of the codebook D.
Image generation without training. Similar to our pro- posed text-guided image editing approach, VQGAN-CLIP (Crowson et al., 2022) performs test-time optimization of tokens, but is limited by its use of 2D latents and need for complex augmentation pipeline. is applied in style transfer or artistic image generation settings where these limitations are less apparent. To enable text conditioning, other methods attempt CLIP-guide search (Galatolo. et al.,
Next, create set of partitions {P1, ...} } of the dataset. Each partition P. splits the dataset into N, high-level classes C,,,, Cr,N, described by text prompts. The class assign- ment is computed automatically via CLIP similarity.
2

---

Highly Compressed Tokenizer Can Generate Without Training
Token Pos. (Attribute)
Classification Prompts
Token Importance
Token Perturbation Result
"animal or human" "
var
"inanimate object"
(subject type)
0
27
"abundant vegetation" "sparse or vegetation" "
var
(bg. material)
18 (bg. blur)
"large dominant subject"
vur
"small or subtle subject"
31 (scene lighting)
"daytime scene"
"nighttime scene"
0
1
12 (image quality)
"high definition" "blurry, low quality"
0
10
20
token index
Figure 2. Certain high-level attributes are represented by particular token positions. We verify this using two independent experiments. First, we partition the ImageNet validation set into diffcrent sets of classes automatically laheled according to CLJIP prompts (first column), observing for each partition that distinct token indices show the greatest variance when comparing averaged features across classes (second column; normalized). Second, we perturb token positions one-by-on (selected token positions shown in third column), visualizing the perturbation that maximizes L.I difference with respect to the unperturbed image (fourth column). The token positions identified through the token importance experiment are consistent with the attributes altered by the token perturbation experiment.
In order to determine which tokens contain the most informa- tion with respect to a particular partition, we will compute class-specific statistics for each token (1)) ...,, (K) Con- cretely, we fit a Gaussian to each set of features, yielding a distribution (( (.j. L.g describing feature statistics of a particular class Ci,j for each of the K tokens. Given some measure of similarity between these statistics we can now determine which tokens vary the most under a particu- lar partition. We choose to use the sample variance of the (k) mean pij computed across the classes of each partition,
3.2. Token Perturbations Can Achieve Meaningful Edits
The observation that a limited number of token positions vary for a given class partition suggests a crude way of editing images: is possible to modify a certain high level attribute by directly manipulating the token the most relevant position? We will attempt to answer this question by considering single-token edits of the following form.
(k) (())
arg max (Dec(Reeplae,, (Tok(I), x (+))), (2)
Here, Dec decodes sequence of ID tokens back into an image, and Replace, (applacex[("e ([x(1), ,..., (..,,,(k))),))
(k) gp. = cov (f (1) j j (3,
(1)
==
[z(11) "..,,..,""" (k-1),,,)), ,,, (((+1),,...... ,..., x(K)] returns its first argu- ment with the kth input token (*()) replaced by its second argument y. By maximizing the loss criterion E(---) we can therefore find the replacement token at position i which causes the greatest change in output according to f.
The leftmost two columns in Figure 2 show the result of this per-token inter-class variance computation for the pretrained TiTok-L-32 tokenizer (K = 32, [D] 2'2). Surprisingly, we find that for a number of different class partitions there exists significant disentanglement across token position in the sense that, for a given partition, a certain small number of token positions display a large difference in per-class mean statistics while most others do not.
Thanks to the modest codebook size of [D] = 4096 we can easily solve the above optimization via exhaustive search. An LI loss criterion is used so that the discovered single- token edits maximize visual difference.
3

---

Highly Compressed Tokenizer Can Generate Without Training
Attribute Tokens
background blur
scene lighting
sharpening 12
colorization
pose 21
18
31
12, 24, 27
Reference
Target
Edited
Figure 3. Image editing via "copy and paste" latent space manipulation. Edited image is obtained by replacing tokens in the target image with tokens copied from the reference image. Using the token positions previously identified to correspond desired attributes, we can reliably obtain fine-grained contral over appearance-related properties such as background blur, scene lighting conditions, and image quality. Class-specific semantic properties such as pose and gaze can also be controlled with some limited success (see last two columns for a successful and a failing example, respectively).
As shown in the rightmost two columns of Figure 2. many of the edits discovered with this single-token editing procedure produce valid images, and most of the edits are consistent with the results from the earlier class partition based token importance experiment.
arbitrary target images. Furthermore, the reference image used to adjust these properties need not be semantically or visually related to the target image except in the appearance property to be transferred. In many cases, we can even use abstract synthetic reference images demonstrating the prop- erty to transfer. For example, we may use an arbitrary image blurred to the desired extent in order to precisely adjust the amount of background blur. We do not observe the same level of token compositionality for semantic properties. For example, transferring the pose of a subject by copying and replacing tokens is only successful to a limited extent (see last columns in Figure 3).
3.3. "Copy& & Paste" in Latent Space for Image Editing
Having discovered that certain high-level image properties appear to be represented by just one token position, and hav- ing demonstrated that perturbing tokens at these positions can produce valid images while affecting the desired high level properties, we propose an extremely simple "copy and paste" approach for direct latent space editing of images. Given an input image to edit Zarget and reference image Tee that possesses an attribute that should be exhibited in the edited image Tedit; we tokenize both the target and reference images with the ID tokenizer and then simply copy the token lying at the position known correspond to the desired attribute from the reference to the target tokens:
4. Gradient-Based Latent Optimization
In the preceding sections we have demonstrated that ID tokenizers' highly compressed representation enables even extremely crude latent space manipulations to result in visu- ally coherent and interpretable edits to input images. While this enables surprising amount of control over certain pre- identified attributes (particularly those controlling global appearance), it is not particularly flexible, and cannot be reliably used for editing of more complex properties. In this section, we explore more systematic and powerful image editing procedure based on gradient- -hased optimization of ID tokens at test time.
Fearget Tok(Inrget).
Jeet Tok(Irr),
(k) Tedit Replace; (Fuuget) Fiet
(3)
Iedit
Dec(Feam).
Such a procedure cannot be expected to produce coherent results in the case of a 2D tokenizer, where single-token edits would result in spatially localized edits (Cao et al., 2023). Remarkably, in the case of 1D tokenization, it can produce interpretable, high-quality outputs such as those shown in Figure 3.
4.1. Text-Guided Image Editing
We present a simple text-based image editing procedure based on test-time maximization of CLIP similarity (Rad- ford et al., 2021) with respect to user-supplied text prompt. First, the tokens to optimize are initialized from the input or seed image to be edited. These tokens are then iteratively
We observe that global appearance related properties - es- pecially background blur, scene lighting and image qual- itylsharpness - can be accurately edited for essentially
4

---

Highly Compressed Tokenizer Can Generate Without Training
seed
optimization
imput image
edit
edit
edit
100 R00
engle hummingbird
Figure 4. Test-time optimization according to CLIP similarity objective. Selected intermediate optimization results shown in middle columns, with final result after 400 optimizer iterations shown in the rightmost column. CLIPP prompts are "a photo of class", with class shown the right.
updated by backpropagating the CLIP objective's gradient and performing gradient ascent as shown in Figure 1.
Specifically, we perform the following optimization:
arg max e (Dec (e 2013 (M))
(2
where the decision variables are the continuous feature vec- tors (R) output by the tokenizer's encoder just prior to the vector quantization step (A) VQp (*k).. The
straight-through estimator is used for gradient computation through the vector quantization step. CLIP similarity of the decoded image with respect to the text prompt is denoted by Note that it is essential to include the vector quantiza- tion step within the optimization, as optimizing (k) directly leads poor results. To solve the optimization in practice, use the Adam optimizer with learning rate of 0.1. A more detailed description of the algorithm can be found in Section C.I.
Figure 5. Flexible text-guided editing of an input image using test optimization with CLIP gradients. Key attributes of the imput image such as pose are preserved while aligning the image with the prompt. Prompts are listed in the appendix (Table A1).
4.2. Text-to-Image Generation
straightforward implementation this optimization with no further tricks can already produce text-guided edits which visually coherent and aligned with the prompt. However, do find that few additional tweaks to smooth the CLIP objective's 's gradient, to artificially in- troduce additional stochasticity, and to add a small amount of regularization, can help. We ablate these design choices in Section 5.4.
Consider now the case in which no seed image is available. In this case, we can apply the test-time optimization proce- dure starting with randomly selected tokens. particular, we initialize the feature vectors ((k) (prior to the vector quantization step) by randomly sampling their value from a normal distribution. We find that this works better than ini- tializing from tokens randomly selected from the codebook, and allows for successful text-to-image generation with- out the need for a seed image (see Figure 6 for example generations). A further discussion of tweaks and parameters used for this "from-scratch" text-to-image generation task can be found Section 5.5
Figure shows examples of using our CLIP- guided token optimization transform the main subject in an image into different class. However, the usage of CLIP guidance al- lows going beyond class-conditiona, generation. We demon- strate the capability for open- open-domain text-guided editing of a subject's 's context in Figure 5, noting that the optimiza- tion preserves key aspects of the subject while aligning the generated image with the prompt.
4.3. Inpainting
Test-time optimization tokens provides general frame- work for optimizing arbitrary cost functions, and is not
5

---

Highly Compressed Tokenizer Can Generate Without Training
5. Evaluation
For quantitative evaluation of editing and generation quality, we will consider class-conditional generation pipeline based on a small seed image dataset subsampled from the ImageNet training data, along with set of CLIP text prompts used to guide generation towards target classes, This setting is similar to semi parimparaettric generation such as instance conditioning (Casanova et al., 2021) or retrieval- augmented diffusion (Blattmann et al., 2022).
Figure 6. "From-scratce" generation starting from random to- kens. Instead of starting with an initial image, is also possible to apply our test-time latent token optimization starting from ran- domly sampled initialization. Selected samples for prompts "a dog running on a dirt trail", "majestic deer standing in a snowy forest", "a duck in pond," and "a wide landscape vista overlooking fields and a town".
Seed image set. A fixed number of ImageNet ILSVRC2012 (Deng al., 2009) training set images are randomly selected. For each ImageNet class, equal number of images is sampled at random without replacement. The 1D) tokens for images from this small seed image dataset are used to initialize the test-time token optimization.
CLIP prompts. Prompts of the form "a photo of class" are constructed. We generate 50k prompts, distributed ac- cording to the ImageNet validation set class statistics. In each prompt, class is the natural language label for an ImageNet synset (the first WordNet lemma name).
Class-to-seed association. We consider a number of dif- ferent strategies to associate a target prompt with a seed image. First, we consider random association. Second, we can use CLIP similarity associate each prompt with either the most one of the (randomly picked) top-k most similar seed images.
Models. Our evaluation uses the pretrained TiTok tokenizer checkpoints released by the TiTok authors. We refer to the models by name as VQ VAE-Sen-Sae-K where the prefix stands for the type of autoencoder (VQ: discrete tokens using vector quantization, VAE: continuous tokens with KL- regularization), Senc and Saec refer to the size (L, B, S) of encoder and decoder, respectively, and K denotes the number of tokens.
Figure 7. Inpainting via test-time optimization of tokens. For each pair of images, the left image is the input to the optimization procedure, which optimizes ID tokens according to a reconstruc- tion loss on the given part of the image. The final image is produced by blending the input with the reconstruction.
Computational cost. Running 300 iterations of the text-guided image editing optimization (with CLIP loss smoothed over 8 random crops) in half precision using the VQ-LL-32 tokenizer takes 7 seconds per image on an NVIDIA A100. Most experiments in this section use this configuration.
limited CLIP similarity scores. As additional use case, consider the inpainting problem. In this case, the test- time optimization is modified to use an LI reconstruction loss over the unmasked part of the image.
While this modified algorithm does perform the inpainting task successfully in some cases, we observe that the recon- struction loss tends favor blurry reconstructions. To solve this issue, we periodically reset the tokens during the opti- mization by (i) decoding them into image, (ii) replacing the parts of the image which are known with their given val- ues, and (iii) re-encoding this modified image to obtain the reset tokens. We also find it essential to inject noise during the optimization process and to apply L2 regularization on the tokens. Example results are visualized in Figure 7.
5.1. Evaluation Protocol
In order to evaluate quality of generations, we compute the Fréchet Inception Distance (FID) (Heusel et al., 2017) with respect to the ImageNet training statistics following the protocol from ADM (Dhariwal & Nichol, 2021). We also report Inception Score (IS) (Salimans et al., 2016).
6

---

Highly Compressed Tokenizer Can Generate Without Training
Table A limited number of seed images combined with diverse prompts can achieve good FID and prompt alignment. Baseline refers to directly using the seed images (reconstructed via TiTok VQ-LL-32) without any optimization. Adversarial refers to direct optimization of pixels to maximize a CLIP objective intended to demonstrate the immunity of the SigLIP similarity. Num Seeds denotes the size of the pool of randomly selected seed images, while Seed Assoc describes whether seeds are randomly assigned to target prompts or use -Lie- selection. Configuration marked (+) used for remaining experiments in this section.
Num Seeds
Seed Assoc
Method
FID-SkL
1st
CLIPT
SigLIPt
baseline
25.2
293
0.12
-16.2
2000
random
adversarial
262
(+241)
2 (-291)
0.59 (+0.47)
-14.4 (+1.8)
VQ-LI-32 w/ token opt.
20.7
(-4.5)
75 (-218)
0.36 (+0.24)
-0.3 (+15.9)
baseline
46.1
226
0.25
-6.9
2000
CLIP top-198
VQ-LL-32 w/ token opt.
14,6
(-31.5)
161 (-65)
0.39 (+0.15)
2.6 (49.6)
baseline
73.7
197
0.24
-7.0
1000
CLIP top-1%
VQ-LL-32 w/ token opt.
15.1
(-58.6)
160 (-37)
0.39 (+0.15)
2.5 (+9.5)
baseline
71.9
243
0.31
-1.3
1000
CLIP top-1
VQ-LL-32 w/ token opt.
21.2
(-50.7)
281 (+38)
0.40 (+0.09)
3.5 (+4.8)
baseline
119.2
135
0.25
-7.0
500
CLIP top-1%
VQ-LL-32 w/ token opt.
16.6
(-102.6)
155 (+20)
0.39 (+0.14)
2.5
(49.5)
To evaluate alignment with the target prompt, we consider CLIP as well as SigI.P (Zhai et al., 2023) similarity. It is critical to include SigLIP scores, as the CLIP-guided test- time optimization resembles an adversarial attack the CLIP model and high CLIP similarity therefore does not necessarily indicate successful alignment of the generated image with the prompt (see Table for an example of maxi- mizing CLIP score without significantly improving SigLIP score).
Table 2. Increased compression leads to improved generation.
Usage of tokenizers with increasing number of tokens or increasing codebook size |D] degrades generative performance. The number of optimization iterations used corresponds to the best FID-5k score achieved.
Tokenizer
PP
FID-5k]
1st
CLIPt
SigLIPt
VQ-LL--32
4096
15.1
160
0.39
2.53
VQ-BB-64
4096
16.3
135
0.40
2.77
VQ-BL-64
8192
18.6
106
0.40
2.07
VQ-BL-128
8192
22.3
92
0.42
2.39
5.2. Seed Image Set Size and Association
To assess the degree of the test-time optimization's 's reliance on seed images as a source of diversity, we vary the size of the seed image set. We further check the optimization's ability handle diverse seed images with varying degrees of similarity to the target prompt by ablating the CLIP- based seed association, and vary the degree of stochasticity in the prompt-to-seed association by selecting either the most similar seed image or picking uniformly at random between the top 1 5 of most similar seed images.
association leads to poor results, indicating that the test time optimization struggles to generate images when the seed image unrelated to the prompt. Table summarizes these results.
Note on determinism. The experiments in this section include a small amount of randomness in the loss. This is due to nondeterminism in some of the CUDA kernels used in our PyTorch implementation. Interestingly, this is sufficient to lead to diverse generations by the test-time optimization procedure. Note that in the case of at least 1000 seed images with top- 1 % selection, there is enough diversity in the seed- to- prompmp association to yield about 4000 unique inputs, such that the FID-5k results would be relatively unaffected by the use of a fully deterministic implementation.
We find that only small number of seed images (less than 1000) is needed to produce diverse generations, achieving an FID of 8.6 for 50k samples. We also find that the number of seed-to- tond-to-pue to-prompt combinations is an important source of diversity, observing that seed association schemes that favor more diversity in the association (i.e., allowing for associations that do not necessarily have the best similarity) outperform those favoring higher similarity of seed and prompt in terms of FID,, although those with less diversity can lead to improved IS. Varying the degree of stochasticity in the seed selection therefore provides way to control the tradeoff between diversity and quality. Performing random
5.3. Tokenizer Variations
Generative performance improves with increasing com- pression by the tokenizer. As shown in Table 2, a decreas- ing number of tokens as well as a decreasing codebook size lead to significant improvements generation quality. Out of
7

---

Highly Compressed Tokenizer Can Generate Without Training
Table Vector quantization and compression enabled by ID tokenization are key to generative performance. Test-time opti- mization using ID continuous tokens or 2D discrete tokens fails. In case of continuous tokens, we apply 1.2 regularization with weight chosen for best results from a parameter sweep. Note that when using the tokenizer trained with VQ but optimizing without VQ (sccond row), performance drops significantly.
Table S. Generating images from scratch. It is possible to gen- erate images even when starting from randomly initialized tokens as opposed to seed images. When using our vanilla token opti- mization algorithm, we observe decrease in generation quality, However, additional optimization iterations as well as the tweaks presented in Section 5.4 help to significantly decrease the gap in generation performance when comparing seed-image- image-based initial- ization and random token initialization.
Tokenizer
FID-5k4
ISt
CLIP
LigLIPt
Tweaks? Seed
Iter.
FID 5k,
1st
CLIPt SigLIPt
VQ-LL-32
15.1
160
0.39
2.53
VQ-LL-32 w/ VQ
17.1
127
0.43
3.05
CIIIP-to-- 1%
300
15.1
160
0.39
2.53
VAE-LL-32
33.2
93
0.46
3.05
no random tokens
300
17.0
114
0.38
1.68
MasKGIT-VQGAN
34.3
73
0.44
2.42
CLIP-top-115 1%
300
14.6
182
0.38
2.07
yes
random tokens
400
15.5
152
0.40
2.54
Table 4. Ablation study, All using the VQ-LL-32 tokenizer and 300 optimizer iterations. All metries other than FID-50k computed over 5000 samples. We use the configuration from the line marked (+) for the preceding experiments in this section, unless noted otherwise.
loss function, while the 2D tokenizers' spatially arranged tokens tend to lead to optimization results that are spatially incoherent.
FID 5k (50k) 4
ISt
CLIP
SigLIPt
baseline
14.8
140
0.39
1.81
Number of optimization iterations. It is possible to con- trol the tradeoff between FID and IS scores by varying the number of optimization iterations. In fact, the best FID and best IS scores are achieved at different numbers of iterations, and this number varies between different tokenizers. There- fore, we report values at the number of iterations yielding the best FID score in Table 2, and include additional results in Section C.2.
+random crops
15.3
157
0.40
2.55
+EMA
15.1
(8.6)
160
0.39
2.53
(-100 opt. iter)
15.2
149
0.38
1.93
(+100 opt. iter)
15.3
169
0.40
2,88
+tkeen noise
14.5
(8.2)
165
0.39
2.46
+token reg.
14.6
(8.2)
182
0.38
2.07
all models we tested, were only able to achieve qualitatively satisfactory performance in editing and generation tasks us- ing the TYTok-LL-32 model (see Figure A6 in the appendix for qualitative results).
5.4. Tweaking the Test Time Optimization Algorithm
We find that averaging the CLIP loss over multiple random crops (we use 8 crops encompassing 75% of the image area) significantly helps prompt alignment and boosts IS, while slightly degrading FID, likely due to smoother gradients of the objective function leading to reduced diversity in the generations. Returning an exponential moving average (EMA) of token iterates yields a small boost to FID and IS, Injecting additional noise to the tokens at the beginning of each optimization iteration can further improve FID and IS, achieving FID of 8.2 over 50k samples. Applying weak L2 regularization on the tokens yields a significant boost to IS, while slightly degrading FID. Finally, increasing the number of optimization iterations can improve IS as well as CLIP and SigLIP scores at the expense of degraded FID. Table 4 summarizes these findings. Additional results to establish the relationship between optimization iterations and FID/IS are presented in the appendix (Section C.2).
Discrete vs. continuous tokens. In comparing a version of the TYTok tokenizer trained as a VAE with continuous tokens with the standard VQ model, we find that the compression provided by the discrete latent space is essential in achiev- ing good generative performance (Table 3). Even when optimizing the continuous tokens of a tokenizer trained with VQ (i.e. optimizing (k) directly and skipping the VQ step), performance drops significantly. We hypothesize that VQ provides an essential form of regularization that prevents the optimization from behaving adversarially (Santurkar et al., 2019).
ID vs. 2D tokenizer. We further find that the large num- ber of tokens in standard VQGAN 2D tokenizer such as MaskGIT's VQGAN (Chang et al., 2022) prevent success- ful application of the test-time optimization approach for generation (Table 3). This is consistent with the finding that an increasing number of tokens is detrimental to genera- tion (Table 2). Note also that the 32-token ID tokenizer is able to successfully generate visually coherent images even without any tricks such as augmentations to stabilize the
While these tweaks do improve performance, we note that our very simple baseline is already strong. This highlights that the pretrained tokenizer on its own possesses strong generative capabilities and can easily be made to generate images with simple heuristic methods.
8

---

Highly Compressed Tokenizer Can Generate Without Training
Table System-level comparison. Approaches other than our own require training generative model (gray). We compare with models supporting unconditional generation, as our method supports arbitrary objective functions and the ID tokenizer is trained in a supf-riiee manner without class labels. We also compare with semi- parametric methods leveraging retrieval training set images. The second column specifies whether arbitrary objective functions can be used as guidance without training a new model. The third column refers to the need to access training set images lest time, where selected denotes that the entire training set has been considered in selection of the pool of training images available at test time, while random describes randomly subsampled pool of seed images.
Gen. Model Training
Plug & Play Guidance
Access Training Data
Method
FID
1St
ADDP (Tian et al., 2024)
7.6
105
Yes
No
No
RCG-G (Li et al., 2024a)
2.2
254
Full training set
RCDM (Bordes et al., 2022)
19.0
52
Yes
No
1000 selected images
IC-GAN (Casanova et al., 2021)
15.6
59
Full training set
RDM-IN (Blattmann et al., 2022)
5.9
159
No
Yes
1000 random images
Test-time optimization (VQ-LL-32)
8.2
182
5.5. Text-to-Image Generation Without Seed Image
Qualitative image generation results can be found in the appendix (Figure A9).
When no seed image is available, is still possible to ap- ply the test-time optimization approach for text-to-image generation. As reported in Table 5, FID and IS scores are weaker than when starting with a seed image. While this is expected, the tweaks from Section 5.4 as well as additional optimization iterations help significantly close the gap in per- formance. Note that image generation quality when starting from randomly selected tokens (that is, (()) NN00, mia) is higher than when starting from random but valid seed images (see third row in Table 1). In our experiments, we set mit = 0.3 as chosen for best performance from sweep including (0.05, 0.3, 1.03.
6. Discussion
ID tokenizers have recently shown impressive compression of images into small numbers of discrete tokens. Compared to generative models using 2D tokenizers, models trained in this highly compressed latent space produce high qual- ity images orders of magnitude faster without significantly compromising sample quality, In our paper, we show that this is facilitated by the strong generative capabilities of the tokenizer itself. We find that the latent space of ID tokens is highly semantic, to the point that certain image editing oper- ations can be performed in an accurate and interpretable way through targeted perturbations of specific tokens. Further- more, ID tokenizer can generate diverse and high quality images without the use of any dedicated generative model at all with assistance from simple test-time optimization procedure.
5.6. System-Level Comparison
Since tokenizers are trained unconditionally, we focus our system-level comparison in Table 6 on methods support- ing unconditional generation. We find that even though image generation via test-time optimization of tokens is training-free, it approaches or exceeds performance of pre- vious unconditional generation approaches evaluated on Im- ageNet (Tian et al., 2024), including comi-parametric ones which also require access to database of images (Bordes et al., 2022; Casanova et al., 2021; Blattmann et al., 2022). Compared to these generative models, our tokenizer-only approach can maintain high diversity of generations with even smaller seed dataset sizes.
Our experiments suggest that increasing compression - in the form of a small number of tokens, vector quantization, and small codebook size- is crucial in giving rise to gen- erative capabilities in the tokenizer. We therefore hope work encourages scaling of tokenizers to even higher com- pression ratios, larger datasets, or different domains.
Impact Statement
Recent unconditional generation methods such RCG (Li et al., 2024a) can match or exceed the performance of class- conditioned generators. However, we stress that RCG and all other approaches we compare against require training of a generative model. Overall, we conclude that the pre- trained TiTok VQ-LL-32 tokenizer is able to achieve rea- sonable generation quality without any further training through the use of straightforward test-time optimization approach.
This paper presents work that is intended improve under- standing of image tokenizers with very high compression ratios. It does not advance the state of the art in image gener- ation, although it may encourage work that does. Improved image understanding and generation have many potential societal consequences, none which we feel must be specifi- cally highlighted here.
9

---

Highly Compressed Tokenizer Can Generate Without Training
References
Galatolo., F., Cimino., M., and Vaglini, G. Generating im- ages from caption and vice versa via -LIP-guided genera- tive latent space search. Proceedings of the International Conference on Image Processing and Vision Engineering, 2021.
Bao, F., Li, C., Sun, J., and Zhu, J. Why are conditional generative models better than unconditional ones? In NeurIPS 2022 Workshop on score-Based Methods, 2022.
Blattmann, A., Rombach, R., Oktay. K., Müller, J., and Ommer, B. Retrieval-augmentte diffusion models. Ad- vances in Neural Information Processing Systems, 35: 15309,15324, 2022.
Heusel, M., Ramsauer, H.,, Unterthiner, T., Nessler, B., and Hochreiter, S. GANs trained by two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
Bordes, F., Balestriero, R., and Vincent, P. High fidelity visualization of what your self-supervised representation knows about. Transactions Machine Learning Re- search, 2022.
Kingma, D. P. and Welling, M. Auto-encoding variational Bayes. In International Conference on Learning Repre- sentations, 2014.
Li, T., Katabi, D., and He, K. Return of unconditional generation: A self-supervised representation generation method. In Advances in Neural Information Processing Systems, 2024a.
Cao, S., Yin, Y., Huang, L., Liu, Y., Zhao, X., Zhao, D., and Huang, K. Efficient- VQQGAN Towards rish-resolution image generation with efficient vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7368-7377, 2023.
Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37:56424- 56445, 2024b.
Casanova, A., Careil, M., Verbeek, J., Drozdzal, M., and Romero Soriano, A. Instance-conditioned GAN. Ad- vances in Neural Information Processing Systems, 34: 27517-27529, 2021.
Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O. Challenging common assumptions in the unsupervised learning of disentangled representations. International Conference on Machine Learning, pp. 41 14-4412. PMLR, 2019.
Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. MaskGIT: Masked generative image transformer. In The IEEE Conference on Computer Vision and Pattern Recognition, June 2022.
Mentzer, F., Minnen, D., Agustsson, E., and Tschannen, M. Finite scalar quantization: Vq-vae made simple. In Proceedings of the Eleventh International Conference on Learning Representations, 2024.
Crowson, K., Biderman, S., Kornis, D., Stander, D., Hal- lahan, E., Castricato, L., and Raff, E. VQGAN-CLIP: Open domain image generation and editing with natural language guidance. arXiv preprint arXiv:2204.08588 2022.
Patashník, O., Wu, Z., Shechtman, E., Cohen-Or, D., and Lischinski, D. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF inter- national conference on computer vision, pp. 2085-2094, 2021.
Deng, J., Dong, W., Socher, Li, L., Li, K., and Fei-Fei, L. ImageNet: large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255. Icee, 2009.
Peebles, W. and Xie, Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Dhariwal, P. and Nichol, A. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resoluttin image synthesis. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern June 2021.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8766 PMIR, 2021.
Recognition, pp.
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-Or, D. An image is worth one word: Personalizing text-to-image generation using textual inversion. In Proceedings of the Eleventh Inter- national Conference on Learning Representations, 1-5 May 2023.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine
10

---

Highly Compressed Tokenizer Can Generate Without Training
Learning Research, pp. 8821-8831. PMLR, 18-24 Jul 2021.
single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. /2528, 2024.
Rao, Y. Zhao, W., Liu, B., Lu, J., Zhou, J., and Hsich, C.-J. Dynamie ViT: Efficient vision transformers with dynamic token sparsification. In Beygelzimer, A., Dauphin. Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021.
Yu, J., Li, X., Koh, J. Y., Zhang, H.,, Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, J., and Wu, Y. sector-quantized image modeling with improved VQGAN. In Proceed- ings of the 10th International Conference on Learning Representations, 2023.
Razavi, A., Van den Oord, A., and Vinyals, O. Generating images with VQ- Advances in Neural Information Processing Systems, 32, 2019.
Yu, I.., Lezama, J., Gundavarapu, N. B., Versari, I.., Sohn, K., Minnen, D., Cheng, Y., Birodkar, V., Gupta, A., Gu, X., Hauptmann, A. G., Gong, B., Yang, M.-H., Essa, I. Ross, D. A., and Jiang, L. Language model beats diffusion tokenizer is key to visual generation. In Pro- ceedings of the Eleventh International Conference on Learning Representations, 2024a.
diverse
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Yu, Q.. Weber, M., Deng, X., Shen, X., Cremers, D., and Chen, L.-C. An image is worth 32 tokens for reconstruc- tion and generation. In Advances in Neural Information Processing Systems, volume 38, 2024b.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for train- ing GANs. Advances in Neural Information Processing Systems, 29, 2016.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision pp. 11975-1 1996, 2023.
Santurkar, S., Ilyas, A., Tsipras, D., Engstrom, L., Tran, B., and Madry, A. Image synthesis with a single (robust) classifier. Advances in Neural Information Processing Systems, 32, 2019.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Sheynin, S., Ashual, O., Polyak, A., Singer, U., Gafni, O., Nachmani, E., and Taigman, Y. KNN-Diffusion: Im- age generation via scage retrieval. In Proceedings of the Eleventh International Conference on Learning Representations, 1-5 May 2023.
Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma. X., Zettlemoyer. L., and Levy, O. Transfusion: Predict the next token and diffuse images with one multi-modal model. 2025.
Sun, P., Jiang, Y., Chen, S., Zhang, S., Peng, B., Luo, P., and Yuan, Z. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06 06500. 2024.
Zhou, Y., Liu, B., Zhu, Y., Yang, X., Chen, C., and Xu, J. Shifted diffusion for text-to-image generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10157-10166, 2023.
Tian, C., Tao, C., Dai, J., Li, H., Li, Z., Lu, L., Wang, X., Li, H., Huang, G., and Zhu, X. ADDP: Learning gen- eral representations for image recognition and generation with alternating denoising diffusion process. In Proceed- ings of the Twelfth International Conference on Learning Representations, 2024.
van den Oord, A., Vinyals, O., and Kavukcuoglu, K. Neural discrete representation learning. In Advances in Neural Information Processing Systems, volume 30, 2017.
Wang, X., Zhou, X., Fathi, A., Darrell, T., and Schmid, C. Visual lexicon: Rich image features language space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
Xie, J.. Mao, W., Bai, Z., Zhang, J.,. Wang, Lin, K. Q., Gu, Y., Chen, Z.,, Yang, Z., and Shou, M. Z.. Show-o: One

---

Highly Compressed Tokenizer Can Generate Without Training
A. Additional Background on TiTok Tokenizer
For completeness, we provide a graphical overview of the TiTok ID tokenizer Figure A1. 2D tokens are first obtained by patchifying the imput image and embedding the patches, which interact with the ID tokens via several layers bidirectional attention. TiTok relies on an off-the-shelf 2D tokenizer (MaskGIT's VQGAN) to reconstruct an image from the set of 2D tokens predicted by the decoder, and is trained in two stages. First, the objective is learn to reconstruct the (frozen) VQGAN's tokens, and later, the VQGAN itself is also fine-tuned (Yu et al., 2024b).
M
Reconstruction
2D Tokens:
Image
(discard)
2D
of latents decoded standard VQ-VAE
patch
M
tok
embed
dec
2D taken
M Mask token
VIT
M
VIT
1D Tokens:
Processed jointly with tokens via bidirectional attention
(discard)
VQ
Latent token
Discrete 1D taken
Figure AI. Architecture TiTok (Yu al., 2024b) ID tokenizer. An image encoded into short sequence (e.g. tokens) "ID" tokens, which unlike the grid-arranged latents from standard 2D tokenizer, do not follow any particular spatial structure and are therefore able to capture global attributes of the image.
B. Token Semantics
In Section 3.1 we introduce a metric for token importance based on the variance averaged features across different partitions of the ImageNet validation set. In the following, we first provide additional experiments to further support the effectiveness of this metric in capturing token importance (Section B.1). Then, present additional results showing token importance for different tokenizers (Section B.2).
B.1. Linear Probing
1.00
0.75
importance
0.50
0.00
1.00%
0.75%
accuracy
0.50%
0.25%
0.00%
10
15
20
25
30
token index
Figure A2. ImageNet classification accuracy using individual discrete tokens poor (bottom), but correlated with the proposed importance metric (top). Red line at 0.1 % corresponds to random guessing among the 1000 possible classes. Importance metric (top) is computed according to 3.1.
Classification using only one token position at a time. Figure A2 we attempt ImageNet- Ik classification using discrete tokens, one-at-a-time. Since we here only consider single token a time, we can directly learn a table mapping token
12

---

Highly Compressed Tokenizer Can Generate Without Training
value class logits for each token position. We observe strong correlation between the results of this experiment and the importance metric we propose in Section 3.1.
Linear probing with iterative token masking, Next, we perform linear probing on multiple tokens at time, and consider both the high-dimensional transformer features from the last layer of TiTok's encoder, as well as discrete tokens in the previous experiment. In contrast to the previous experiment, we begin by linear probing all 32 available tokens, and iteratively remove the "least important" ones. Our exact protocol for linear probing with iterative token removal as follows:
1. Start by training a linear head on top of the concatenation of all 32 features.
2. During training, perform dropout to randomly mask features corresponding to certain token positions.
3. After a fixed number of epochs, evaluate the model on the IN- IN validation data with no dropout.
4. Select the most "unimportant" by applying dropout one-by-one each token position. After running several validations with each remaining token position masked, select the token position which leads to the smallest drop in classification accuracy with respect to the previous validation results that do not apply masking.
5. Repeat this process from the start (1), until all token positions are masked.
In the case of using discrete token indices, learn new codebook of features to perform linear probing on. Results can be found in Figure A3.
58%
32 tokens
1123 vol dosshcatto acauncy
50%
45%
40%
1.
16
75
70
(420)
(44)
(6144
[422) (410)
(+17) 14:20) (+26) (431) (+5) (46) [83] (60)
(47)
(127)
(A18)
(4:20 (#244
(444)
(2112) (+11)
32 calens (VQ)
20%
15%
8%6
IN-1k
14 15 (192)
(193)
(99)
(915)
(1914) (1776)
(166
(17) (116)
(195) (1900)
(#97) (89) (851) (199) (1944) (011)) (1118) (111))
mosked (Rtokan Indax)
Figure A.3. Results from linear probing while iteratively removing tokens with the smallest impact on classification accuracy largely match our token importance metric. Top: probing of high dimensional features produced by last transformer layer TYTok encoder. Bottom: linear probing after vector quantization, using discrete token indices. In both cases, we plot IN- 1k classification accuracy as function of number masked tokens. Token index remove after each masking iteration shown parentheses, so that most important token positions can be read off right-to-left. For example, the top plot based high-dimensional features suggests tokens 11, 12, 4, 2,24... as most important, while the discrete-token based plot on the bottom indicates most important tokens are 4, 11, 18, 12,
Results from these experiments are largely aligned with those from the token importance shown in Figure A2, indicating that the approach suggested in Section 3.1 indeed an informative token importance computation.
13

---

Highly Compressed Tokenizer Can Generate Without Training
B.2. Per-Token Inter-Class Variance and Linear Probing for VAE-TÝTok
In Figure A4 we repeat the token importance experiment from Section 1. for additional class partitions. Furthermore, find that the VAE version of TiTok does not exhibit the same high degree of correspondence between semantics and token position.
VQ
VAE
20
20
(a)
ImageNet classes
"animal human", 'inanimate object"
(c)
"abundant vegetation" / 'sparse vegetation"
(d)
'large dominant subject", 'small subtle subject"
"daytime
"nighttime scone'
(1)
"high definition"
"blurry, low quality"
(x)
"vibrant color" / "faded black and white"
"containing green items" "containing items" "containing blue items"
(h)
20
20
token index
token index
Figure A4. VAE-TYTok does not show semantic disentanglement across token positions. We plot token importance computed according to Section 3.1 for both the VQ (left) and VAE (right) versions of the LL-32 TYTok tokenizer for various class partitions of the ImageNet validation set. Normalized token importance shown on the y axis. Abbreviated prompts used as labels CLIP-based class assignment shown in the middle.
14

---

Highly Compressed Tokenizer Can Generate Without Training
C. Tradient-Based Latent Editing
C.1. Detailed Deseription of Test-Time Optimization Algorithm
The following algorithm describes the baseline algorithm for text-guided image editing.
Algorithm AI Time Optimization For CLIP-Guided Latent Editing.
Input:
img the seed image, and prompt text prompt
Output: recons the optimized image
tokens TYTokEnc(immg
2: prompt.ent <- CLIPToxtEEne(rooppl
3: loop
4: recons + (TTTokDec o TYTokQuant (token)
5: rec_enc <- CIIPPmaageFnc(reconss
prompt.enc 6: Gradient ascent step on tokens |ree pronpt.anc]
7: end loop
In Algorithm A1, we refer to TYTok's transformer-baseee image encoder taking images and producing K D-dimensional continuous features as TiTokEne [o, 1]Cx + IR KxD The continuous features are then quantized particular entry in the codebook via the vector quantization module TiTokQuant R - R using straight-through gradient estimates for the backward pass. Quantized features are decoded to image via the decoder TiTokDec: IR XD
[0, 1 In practice, we use the Adam optimizer with learning rate of 0.1, B1 = 0.9 and Bg = 0.999.
Additional changes we use for the inpainting task as well as in the improved versions from Table can be found below.
Algorithm A2 Time Optimization with Optional Tweaks.
Input: img the seed image, an objective function taking an image
Output:
recons the optimized image
tokens +- TiTokEnccingg
2: for + Niier do
3: sample noise of
4: tokens <- tokens + noise
5: recons + (TYTokDec o (TTokQuant(tokens)))
6: Gradient ascent step on tokkens (((recons) +X E gtokens
124122)
7: tokens +
8: end for
Algorithm A2 shows addition of token noise in blue. We use a cosine schedule to ramp the noise from of = 0.3 to 2200 0. Token regularization is highlighted in green. We obtain best results with = 0.02. Token EMA (not shown) uses a decay factor of 0.98.
Inpainting. The token reset procedure highlighted in red only used for the inpainting task. Here, Reset takes the decoded image and blends with the given parts of the image according the inpainting mask. For inpainting, we find important to use masks with soft transitions from fully masked to fully unmasked. Such "soft" mask can be obtained, for example, by applying Gaussian blur to binary mask. The reasons we use soft masks are twofold: first, it allows for graceful blending of the unmasked (given) part of the image with the generated image, which may appear discontinuous if blended with a binary mask that uses sharp edges. Second, we find that the optimization can be reliably initialized with masked images when the mask has a soft transition (such as those in Figure 7), however, the optimization struggles to succeed when initialized with images containing sharp edges.
15

---

Highly Compressed Tokenizer Can Generate Without Training
C.2. Choosing Optimization Iterations
The number of optimization iterations directly impacts the quality of generations. Intuitively, the longer the optimization is run, the further the image will deviate from the seed image. This be desirable necessary when the seed image needs significant modification in order match the target prompt, but for "smaller" edits, lesser number of iterations may be chosen. Therefore, the number of iterations should in general be picked on an ceaaple-b--example basis, and one could design adaptive stopping criteria based on monitoring the objective function's 's value and change across iterations. We leave this to future work.
For the evaluations performed in Section 5, we report either scores for fixed number of optimization iterations, or scores corresponding to the number of iterations yielding the best FID. For completeness, provide FID and IS for sweep of optimization iterations in Figure A5 below.
500
VQ-LL-32
VQ-BB-64
400
VQ-BL-64
VQ-BL-128
300
160
250
200
250
140
300
150
350
100
120
250
200
150
100
100
200
250
300
150
100
16
18
20
22
24
FID-5k
Figure A5. The number of optimization iterations controls the tradcoff between FID and IS. The small number alongside each point denotes the number of optimization iterations used. The optimal number of iterations for either FID, IS or given FID/IS tradeoff dependent the tokenizer used, and models with higher compression ratio shown require larger number of iterations to achieve peak performance.
16

---

Highly Compressed Tokenizer Can Generate Without Training
C.3. Image Editing Example Prompts
Table Al. Prompts used for Figure
Prompt Prefix
Edit 1
Edit 2
Edit 3
photo of a blue jay...
...in forest
with the ocean in the background
perched on fence
photo of lighthouse...
.n the desert
...t sunset
a golf course on the coast .. the beach road
photo of wallaby...
... the highway
in.in a town
photo of turtle...
...on the beach
in green field
a photo of a black swan...
...at sunset running in a field
among wildflowers
.n.in the rain
a photo of tibetan terrier...
wading in a river
...nn the beach
D. Additional Qualitative Results
We include additional qualitative results on the importance of compression (Figure A6) and on the out-of-domain tasks of generating images of classes not present ImageNet (Figure A7) as well as style transfer to non-natural-image styles (Figure A8). Finally, we show selected generations of the best configuration (32-token tokenizer with VQ) alongside their seed images and prompts in Figure A9.
Seed Image
VQ-LL-32
VQ-BB-64
MaskGIT VQGAN
Figure A6. Qualitatively, the gap in generation quality is very significant between the TiTok 32-token, 64-token, and VQGAN models. Prompts are "a photo of an eagle" and "a photo of a tiger" for the first and second rows, respectively.
17

---

Highly Compressed Tokenizer Can Generate Without Training
Cardinal
Capybara
Rhinoceros
A photo of an astronaut riding horse in the forest. There is river in front of them with water lilies.
Figure A7. Generating images of animals not included in the ImageNet dataset (top) open-domain scenarios including subjects not included in Net (hottom) can produce inaccurate results with varying degrees of plausibility, since the tokenizer has only been trained ImageNet.
input image
watercolor painting of lighthouse
comic-style drawing of lighthouse
8-bit pixel art featuring lighthouse
cartoon drawing of lighthouse
an abstract painting of lighthouse in the style of miro
Figure A8. Text-guided style editing to styles underrepresented in ImageNet.
18

---

Highly Compressed Tokenizer Can Generate Without Training
volcano
mashed potatoes
joystick honeycomb
castle
collie
green mamba
lakeside pomerian
bald eagle
black widow leopard
measuring cup
mountain bike
horned viper
Figure A9, Selected generation results (rows 2, 4, 6) alongside their seeds (rows 1, 3, 5). Corresponding classes shown in table.
19


Reviews:
[{"review_id": "sNZsgXmMWr", "reviewer": "Reviewer_mQYF", "summary": "The submission explores training-free image generation on TiTok's 1D tokenizer. It builds upon the observation that a heavily compressed tokenizer, like TiTok-L-32, is somewhat amenable to interpretable manipulation and editing of latents. The authors first demonstrate that by varying individual tokens, and by copy pasting latent manipulations from a reference example to a target image. They then build upon those insights and tune the tokens for an objective like CLIP score (between an image and prompt) with gradient descent through the TiTok decoder; i.e. without having to train a dedicated text-to-image model. Similarly, they are able to perform in-painting without training any model for that task.\n\n## update after rebuttal\nI thank the authors for their rebuttal. The conditional generation results from random initializations are quite interesting and I expect these results will strengthen the final paper. I will maintain my vote to accept this paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are happy to hear you enjoyed our paper, and would like to thank you for the great questions, which we hope to answer below.\n\n* **Q1.** 2D Tokenizer Results\n\nWe have produced visualizations of the optimization process using MaskGIT’s VQGAN, alongside the 32-token TiTok tokenizer, which we will include in the revised submission.\n\n**Examples from Figure A5: https://i.ibb.co/YB72k9Fh/2d-tok-vis-A5.png**\n\n**Additional example: https://i.ibb.co/WZnHmLx/2d-tok-vis.png**\n\nIn this visualization, we observe that the highly compressed tokenizer allows the optimization procedure to perform globally consistent edits to the image. On the other hand, the VQGAN optimization is easily able to change local texture and color to align with the prompt, but fails to perform \"global\" edits (for example, when changing the species of an animal, the shape of the head and position of the ears are relatively unchanged in the VQGAN case, but attributes such as color can be adapted more successfully).\n\n**Regarding VQGAN-CLIP:** To the best of our knowledge, there is no existing evaluation of VQGAN-CLIP for ImageNet generation. We hope that our comparison of MaskGIT's VQGAN which is already included in Table 3 can provide such a comparison, as it should be very similar to VQGAN-CLIP while being fair in the sense that the same seed image selection and cost function are used as for the 1D tokenizer experiments.\n\n* **Q2.** Token Regularization\n\n**VAE Tokens.** The experiment in the paper did not include regularization on the VAE tokens. We agree this is crucial, and have repeated this experiment. Results are provided in the table below.\n\n|  |FID-5k|IS|CLIP|SigLIP|\n|:-|:-|:-|:-|:-|\n|**without L2 reg.**|39.6|66|0.48|2.66|\n|**with L2 reg. (weight=0.2)\\*** |33.2|93|0.46|3.05|\n\n\\*weight chosen for best results from sweep including {0.02, 0.1, 0.2, 0.5}\n\nWhile the results for VAE tokens with regularization are improved over the numbers reported in the original submission, FID and IS are still poor compared to the VQ model, so we do not change our conclusion regarding the importance of VQ.\n\n**2D Tokens.** For MaskGIT’s VQGAN, we do not use regularization in the experiment from Table 3. We do not find that it makes a substantial difference.\n\n* **Q3.** Additional OOD Examples\n\nWe have run an additional out-of-domain text guided style-transfer task, which we describe in our response to Reviewer apSr.\n\n* **Initializing with Random Tokens**\n\nWe decided to run some additional experiments and find that it is indeed possible to generate images \"from scratch\", starting from randomly sampled tokens!\n\nWhile results are worse than in the case of CLIP-based prompt-to-seed association, FID and IS scores are still reasonable. Qualitatively, we find that it is still possible to generate relatively high quality samples, especially with more detailed CLIP prompts and longer optimization times (400-500 iterations). We plan to include uncurated generations starting from randomly sampled tokens (with **no** seed image) in the revised submission.\n\n**Link to uncurated generations starting from random tokens: https://i.ibb.co/20L8xk1q/rand-tok.png**\n\nQuantitative results (using the same settings as the line marked with (\\*) from Table 4 in the paper) can be found in the table below:\n\n|  |iters|Seed Assoc|FID-5k|IS|CLIP|SigLIP|\n|:-|:-|:-|:-|:-|:-|:-|\n| **start from seed image** | 300 | CLIP-top-1% | 15.1 | 160 | 0.39 | 2.53 |\n| **start from random tokens\\*** | 300 | n/a | 17.0 | 114 | 0.38 | 1.68 |\n\n**With** additional tweaks from Table 4 (token L2 regularization, extra token noise in the case of starting from a seed image, extra 100 iterations in the case of starting from random tokens), the gap in performance between the seed-image-initialized and random-token-initialized optimizations diminishes further:\n\n|  | iters | Seed Assoc | FID-5k | IS | CLIP | SigLIP |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| **start from seed image** | 300 | CLIP-top-1% | 14.6 | 182 | 0.38 | 2.07 |\n| **start from random tokens\\*** | 400 | n/a | 15.5 | 152 | 0.40 | 2.54 |\n\n*\\*For both tables, tokens are randomly initialized from normal distribution with std=0.3, chosen as the best performing value from a sweep including {0.05, 0.3, 1.0}.*\n\n**Note:** Starting from random tokens leads to **better performance** than starting from random seed images (i.e., without CLIP-based seed-to-prompt association) – cf. Table 1 row 3 (line 284).\n\n* **Initializing from Single Best Image**\n\nIn Table 1 (L290), we show the results of an experiment where only the best seed image from the seed image pool is picked for each prompt. This allows achieving high IS at the expense of degraded FID.\n\nIf you are referring to picking only one overall best image, and using that same image for every prompt, we suspect performance may not be great, since our previous experiment initializing with random tokens can achieve better results than initializing with random images.", "claims_and_evidence": "The claims in the paper are sufficiently supported by evidence. There are, however, some areas of uncertainty, which I list in the \"Methods and Evaluation Criteria\" section.", "methods_and_evaluation_criteria": "I would say that most of the proposed evaluations make sense, and the visuals help motivate that the method can in fact generate images in a \"training-free\" manner. That said, there is a range of ablations that would be important to see:\n1. How well does the proposed optimization procedure work when initializing with random tokens (instead of other images)? In other words, is this method limited to generating images from some existing seed image, or is it possible to perform the generation purely using the tokenizer's inductive biases and the optimization objective?\n2. The paper shows results for 500-2k seed images, but how about choosing only a single (best) one? This goes into the direction of the question above, regarding the limitations of the optimization procedure.\n3. Showing FID, IS, CLIP, and SigLIP scores is helpful, but I would be interested to see an analysis of how well a specific class can be generated, through the lens of a pre-trained classification model. CLIP and SigLIP scores show alignment, but the resulting scores are very close to each other after optimization, while there are much larger differences in FID and IS.\n4. In Table 3, during optimization of the VAE model, did the authors apply the same KL-divergence, or KL-divergence between the optimized latents and real image latents? With discrete tokens, no token can individually be OOD, but in the VAE case it seems crucial to keep the soft constraints in mind. That then also goes into the analysis B.2.\n5. How well does the proposed method compare to VQGAN-CLIP? This would be especially interesting as a comparison in Table 3.", "theoretical_claims": "The paper does not make any theoretical claims.", "experimental_designs_or_analyses": "The paper is not that detailed in terms of implementation details, but the methods are somewhat straight-forward and based on the descriptions I would feel comfortable reimplementing them to a good degree. Overall, the experimental design and analyses appear sound.", "relation_to_broader_scientific_literature": "The proposed method mostly builds upon the recent 1D tokenizer model TiTok, and first analyzes its latents and then proposes a training-free way to generate images using that tokenizer. For now it seems that the method is somewhat limited to such highly compressed latent spaces and does not work with 2D tokenizers, VAE variants, nor less compressed 1D tokenizers.", "essential_references_not_discussed": "There are a few works that could be discussed in the larger context of \"training-free\" generation. One area is the \"textual inversion\" [1,2,3] line of research that optimizes one or multiple tokens to capture a concept, which can then be used to generate said concept. There is also a range of works that use CLIP as an optimization objective that could be discussed, e.g. see the ones listed in surveys like [4].\n\n[1] An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion, Gal et al., 2022\n\n[2] Visual Lexicon: Rich Image Features in Language Space, Wang et al., 2024\n\n[3] Training-Free Consistent Text-to-Image Generation, Tewel et al., 2024\n\n[4] A Survey on CLIP-Guided Vision-Language Tasks, Yu et al., 2022", "other_strengths_and_weaknesses": "Overall, this is a very creative paper and a fun read! The paper is a callback to \"early\" CLIP-optimization-based image generation attempts. The fact that image generation can be performed in such a controlled and high-quality manner without any generative model training on top of the tokens is quite interesting, and the ablations show that such extreme compression ratios are necessary to achieve that (with the given optimization algorithm here). I also appreciate the attention of using SigLIP as an evaluation criteria and running the \"adversarial\" baseline.", "other_comments_or_suggestions": "The paper is easy to read and well motivated. I would suggest the authors to add more supporting visuals in the appendix.", "questions_for_authors": "1. Visually speaking, how do the generations using the 2D tokenizer in Table 3 look?\n2. For the ablation of continuous and 2D tokenizers, were they optimized using the vanilla objective, or did the authors try to regularize the latent space as well? (I.e. apply techniques such as presented in Table 4).\n3. The additional qualitative results in Appendix D are interesting, and I would be glad to see more such OOD examples. Even though TiTok was trained on ImageNet-1k, it seems to generalize slightly for out-of-distribution images, which begs the question of how well this works on text-conditional generation in a larger domain.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NnlDWHYLq2", "reviewer": "Reviewer_dszF", "summary": "This work shows that a highly compressed 1D token set can learn different attributes in tokens, and perform generaion tasks such as inpainting and text-guided image editing with only a tokenizer, without any extra generative model training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and helpful comments\\!\n\n* **Per-Token Attributes as Emergent Properties**\n\nThe direct token editing examples do indeed rely on emergent properties that are not controllable using the standard autoencoder-style/VQGAN training scheme used by TiTok.\n\nAs such, we agree that practical applications of token copy-paste for image editing is restricted to a few tasks (such as brightness change, background blur, etc.), and these tasks have to be manually discovered using methods such as those described in Sections 3.1 and 3.2.\n\nHowever, we find the fact that such editing is possible at all very surprising. In particular, perturbation of individual tokens does not lead to semantically meaningful and globally coherent edits in the case of 2D tokenizers due to the spatial relationship of tokens and regions of the input image (see, for example, Figure 1 in \\[1\\]). We therefore intended to present this finding to draw attention to underexplored emergent properties of highly compressed latent spaces, as well as to motivate our more scalable gradient-based image editing approach from Section 4, which can be directly applied for practical applications such as text-guided editing or generation, or inpainting.\n\n* **Notation: Top-1/Top-1%**\n\nRegarding L290 (top-1 vs top-1%): Thank you for pointing out the confusing notation. \"Top-1\" is used to denote a seed image association procedure by which only the single most aligned seed image is chosen. In contrast, top-1% associates a random seed image from the subset of the top 1% most similar seed images (for example, with 1000 seed images, top-1% would correspond to random association to one of the top-10 images from the seed image pool). We will try to clarify this in the final version.\n\n* **Stability of Optimization Process**\n\nWe do observe that the optimization process is sensitive to noise, such that even small perturbations to the objective and its gradients can lead to quite different generations. However, this does not mean that the generations are generally low quality – instead, we observe that this leads to generations that are diverse (even when adding only small amounts of noise), while still being reasonably high quality (as evidenced by our FID and IS scores).\n\n### References\n\n\\[1\\] Cao, S., Yin, Y., Huang, L., Liu, Y., Zhao, X., Zhao, D., and Huang, K. Efficient-VQGAN: Towards high-resolution image generation with efficient vision transformers. ICCV 2023, https://arxiv.org/abs/2310.05400", "claims_and_evidence": "The claims are supported by experiments. The authors show disentangled attributes in tokens as well as the image editing and generation applications.", "methods_and_evaluation_criteria": "The method is simple and the findings are interesting. Evaluation is with standard metrics (FID / IS).", "theoretical_claims": "Theoretical proofs are not the focus in this work.", "experimental_designs_or_analyses": "In experiments, the authors show several interesting applications and visualizations for analysis.", "relation_to_broader_scientific_literature": "This work is related to the recent work TiTok which converts images to a 1D set of tokens, and explores several interesting properties and applications with this idea of compact 1D tokenizer.", "essential_references_not_discussed": "I did not find any missing key related works.", "other_strengths_and_weaknesses": "The attributes decomposition findings and training-free generation or editing are interesting.\n\nHowever, the proposed method seems to be not directly scalable for the proposed applications. It relies on the properties of the compact tokens, which are more like emergent properties and not directly controllable.", "other_comments_or_suggestions": "L290: top-1 (%) ?", "questions_for_authors": "Is the gradient-based latent editing stable and can get high quality results with high probability?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DwLp8UXvcB", "reviewer": "Reviewer_apSr", "summary": "This paper introduces a generative pipeline leveraging a 1D image tokenizer (e.g., TiTok) to enable image editing and generation without training a dedicated generative model. By compressing images into highly compact 1D token sequences (e.g., 32 tokens), the authors demonstrate that simple token manipulations (e.g., copy-paste, gradient-based optimization) can achieve text-guided editing, inpainting, and unconditional generation. The approach relies on test-time optimization of tokens using objectives like CLIP similarity or reconstruction loss, bypassing the need for iterative generative models like diffusion. Experiments on ImageNet show competitive FID and IS scores compared to state-of-the-art methods, though qualitative results reveal limitations in handling complex scenes or novel concepts.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your review! As you point out, the idea of latent space optimization is indeed not new. However, we do believe that its application in the case of highly compressed latent spaces is noteworthy for a few reasons:\n\n1. **Previous attempts to use test time latent space optimization for image editing, i.e. VQGAN-CLIP, have not demonstrated high quality generation of photorealistic scenes, and rely on \"tricks\"**, such as the usage of a large number of different augmentations (various crops of the image being optimized, color-space corruptions, flips, etc.), in order to be successful. While these additional tricks can also be used to improve image generation in the case of highly compressed latent spaces, we demonstrate that they are not necessary. Remarkably, even an extremely straightforward application of test time optimization can produce reasonable results when operating in the highly compressed latent space (*baseline* in Table 4).\n\n2. Since our baseline test time optimization algorithm is very simple (7 lines of code, see Algorithm A1 in the appendix), **we do not claim any significant engineering contribution**. Instead, we view our findings as strong motivation to view tokenizers with increasingly high compression ratios as generative models. In particular, we find it surprising that tokenizers trained with a standard VQGAN-like objective can be used to perform a variety of generative tasks such as text-guided image editing or inpainting. We would also like to point to our response to Reviewer mQYF, in which we provide experimental evidence that *the VQ-LL-32 tokenizer can even generate “from scratch”, starting from pure noise*.\n\n---\n\n* **Comments on Out-of-Domain Generalization**\n\nSince TiTok tokenizers are trained only on ImageNet, we expect limited ability to generate complex out-of-distribution scenes, such as those involving classes that are not part of the ImageNet-1k dataset or requiring composition of multiple subjects (since ImageNet images often feature a single prominent subject). Unfortunately, no highly compressed 1D tokenizer trained on larger scale datasets was available at the time of submission, so our experiments are confined to ImageNet.\n\nHowever, as tokenizers with even higher compression ratios or trained on larger datasets become available, we expect application to more diverse domains to become possible. Further, we hope that the view presented in our paper – that **the lossy autoencoding task performed by tokenizers with very high compression ratios can be thought of as a generative modeling problem** – can provide an insightful perspective in scaling such tokenizers to these larger datasets.\n\n* **Additional Out-of-Domain Examples**\n\nAs this was also requested by Reviewer mQYF, we generated additional qualitative results in the out-of-domain setting of text-guided style transfer. In particular, we use CLIP prompts like \"a watercolor/pixel art/abstract/cartoon painting of a \\<subject\\>\", and find that the model can produce qualitatively very compelling looking results for text-guided style transfer, even for styles which we expect to be mostly absent from the ImageNet-1k dataset. We will include the generations in the revised submission.\n\n**Link to visualization: https://i.ibb.co/Bd1HkmX/style.png**", "claims_and_evidence": "N/A", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths**\n\n1. Novel Compression Approach: The use of 1D tokenization with extreme compression (32 tokens) is innovative, enabling efficient latent space editing and generation.\n\n2. Training-Free Generation: The framework avoids training generative models, reducing computational overhead and enabling rapid adaptation to new tasks.\n\n3. Empirical Validation: Comprehensive experiments (e.g., text-guided editing, inpainting) provide evidence of the method’s effectiveness, with competitive FID/IS scores on ImageNet.\n\n4. Practical Applications: The approach’s flexibility could facilitate real-world use cases like content moderation, image editing, or low-resource generation.\n\n**Weaknesses**\n\n1. While the 1D tokenizer is novel, the core idea of latent space optimization (e.g., VQGAN-CLIP) is not groundbreaking. The contribution lies more in engineering than foundational advance.\n\n2. Results are confined to ImageNet, leaving applicability to diverse domains (e.g., medical imaging, abstract art) unproven. Qualitative failures (e.g., Figure A6) highlight limitations with novel concepts or complex scenes.\n\n3. FID/IS scores lag behind modern generative models (e.g., ADM, RCG-G), and qualitative outputs exhibit artifacts (e.g., blurriness in inpainting).\n\n4. The paper lacks a rigorous analysis of why 1D tokenization enables generative capabilities, limiting its contribution to empirical observations.\n\n5. Test-time optimization (300+ iterations) is computationally intensive, undermining claims of efficiency compared to trained generative models.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LzNlHZ7jt1", "reviewer": "Reviewer_N1ti", "summary": "The paper finds that tokens in the latent space of 1D tokenizers are strongly correlated with specific attributes of images (e.g. captured subject, lighting, background).\nThe authors build on this finding and propose gradient-based text-guided image editing and inpainting algorithms that optimize these 1D tokens.\nThis approach enables the use of pre-trained 1D image tokenizers for editing and inpainting without requiring further training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thorough and detailed comments and suggestions, as well as the thoughtful questions. We hope the following answers can address your concerns.\n\n* **mec.q.1.** Optimizer Iterations\n\nThe assessment that the number of optimizer iterations will result in the input image deviating further and further from the input matches our results. In fact, the FID and IS scores are relatively sensitive to the number of iterations, and the choice of iterations can provide control over the FID/IS score tradeoff.\n\nWe have therefore computed FID and IS for different number of optimizer iterations (between 50 and 500 in 50 iteration increments). We report FID and IS at the number of iterations corresponding to the best FID, and separately, to the best IS. *Note that these differ from the results in Table 2 because Table 2 reports results using the same number of optimizer iterations for all models.*\n\n|@ best FID|iter|FID|IS|\n|:-| :- | :- | :- |\n|VQ-LL-32|300|15.1|160|\n|VQ-BB-64|150|16.4|134|\n|VQ-BL-64|150|18.6|105|\n|VQ-BL-128|150|22.4|92|\n\n|@ best IS|iter|FID|IS|\n|:-|:-|:-|:-|\n|VQ-LL-32|500|15.5|175|\n|VQ-BB-64|250|17.1|144|\n|VQ-BL-64|250|19.1|118|\n|VQ-BL-128|250|24.5|95|\n\nFor best results, the number of iterations should be picked on an example-by-example basis: \"smaller\" edits may require less iterations than more significant ones. One could also design adaptive stopping criteria based on monitoring the objective function's value and change across iterations.\n\n* **mec.q.2.** Optimizing without VQ\n\nWe find that the vector quantization step serves as regularization that prevents the test-time optimization procedure from behaving akin to an adversarial attack on the CLIP objective. In particular, the CLIP objective, when provided with the output of the 1D tokenizer with VQ, appears to be very adversarially robust (see also [1]). We observe weaker robustness in the case without VQ.\n\nThe best FID-5k achieved by the no-VQ optimization is 15.8 (at 100 optimization iterations, with an IS of 118) and the best IS is around 130 (with 250 optimization iterations, at an FID of 17.1). Even with L2 regularization applied on the tokens, we are not able to improve FID and see a very small improvement in IS. In all cases, the optimization with VQ significantly outperforms the no-VQ one with a top FID of 15.1 and IS of >160.\n\nWe have generated examples of how this manifests itself qualitatively, which we will include in the revised submission. We observe that the no-VQ optimization sometimes leads to more artifacts and more exaggerated and larger areas of repetitive texture.\n\n**Link to visualization: https://i.ibb.co/3yYd9vF6/no-vq-vis.png**\n\n* **eda.q.1.** FID 8.6 Experiment\n\nWe report this metric (FID-50k of 8.6) in Table 4, alongside the FID-5k and IS, CLIP and SigLIP scores. As you point out, IS, CLIP and SigLIP over 50k samples are not reported. We find that the metrics other than FID are very similar between the 5k and 50k sample evaluations, which is why we omit them.\n\n* **eda.q.2.** Decoder Size vs. Number of Tokens\n\nWe agree that this is a shortcoming in our evaluation, so we have run additional experiments comparing the VQ-BL-64 and VQ-BL-128 tokenizers, which share the same model and codebook sizes, differing only in the number of 1D tokens. The results are included in the tables from our response to question **mec.q.1**, and show that the VQ-BL-64 tokenizer always outperforms the VQ-BL-128 tokenizer.\n\n* **eda.q.3a.** Role of 2D Tokens\n\nOur main reason for including the 2D VQGAN in this table was to demonstrate the poor performance of simple test-time optimization with widely used “traditional” tokenizers. Since we are not aware of any 2D tokenizers with VQ that achieve a similarly high compression ratio as TiTok, the conclusion from this experiment may be better reworded as *the high degree of compression enabled by 1D tokenization is key to generative performance*, rather than 1D tokenization itself being key.\n\n* **eda.q.3b.** 2D Tokens Qualitative Results\n\nPlease see our response to Reviewer mQYF’s **Question 1** for some qualitative examples of 2D tokenizer generations.\n\n* **sm.q.1.** Code\n\nWe will make the code publicly available.\n\n* **Preserving Input Image Features**\n\nThere is indeed no guarantee on preservation of aspects of the input image.\n\nThis could be mitigated with further engineering effort, e.g., the difference in tokens compared to the tokenized version of the input image could be slightly penalized. One could also combine the text-guided objective with the inpainting one to explicitly preserve user-defined regions.\n\nCertain differences wrt. the input are introduced by the tokenizer itself, due to imperfect reconstruction (VQ-LL-32 example: https://i.ibb.co/qMnp23fq/recons.png).\n\n### References\n\n[1] S. Santurkar et al. Image synthesis with a single (robust) classifier. NeurIPS 2019. https://arxiv.org/abs/1906.09453", "claims_and_evidence": "The proposition of this paper is built on two main claims, as discussed below.\n\n## Correlation between token positions and semantics\n\nThe authors claim that particular token positions in 1D tokenizers correlate with certain high-level attributes of images. \nThis is confirmed (for the pre-trained TiTok 1D tokenizer) through experiments in section 3, which analyze the encodings of various partitions of ImageNet. Specifically, the authors divide ImageNet into different classes (by utilizing CLIP similarity with a given prompt), and compute the variance (across classes) of the mean of tokens within classes (grouped by index). \nTheir results show that some token positions have a particularly high variance within a partition.\n\n## Editing through manipulation of tokenizers\n\nThe authors show that meaningful image edits can be achieved through the manipulation of tokens at certain positions.\nThey experiment with perturbing tokens at certain positions (by replacing them with the tokens leading to the highest visual difference), which qualitatively lead to edits that \"make sense\" given the prompt with which the token index was chosen.\nThey also show that certain image attributes can be *transferred* from a reference to a target image by replacing a token of the encoded target image with that of the reference image at the desired token index.", "methods_and_evaluation_criteria": "## Methods\n\nBuilding on the findings regarding the TiTok 1D tokenizer, the authors propose an image editing procedure that optimizes the encoded tokens of an image.\nSpecifically, they optimize encoded tokens to maximize a CLIP objective given a prompt.\nHowever, key parts of the seed image (that are not related to the editing prompt) seem to change after the optimization (e.g. in Figure 5, the lighthouse itself seems to change between edits, although the prompt suggest changing its surrounding context). \nThis is expected as the suggested CLIP objective does not guarantee the conservation of elements unrelated to the editing.\n\nNote that the authors claim in lines 271-274 that \"the optimization preserves key aspects of the subject while aligning the generated image with the prompt\". I would advise to reformulate this claim as key aspects of the subject **do change** between edits (e.g. the dog face in Figure 5, last row).\n\nThe authors also propose an inpainting procedure that utilizes a modified algorithm which periodically replaces the known parts of the image with their original counterparts, and encodes the resulting image back into the latent space.\n\n- **mec.q.1.** How is the number of optimizer iterations chosen? In Figure 4, it seems that the higher the number of iterations, the further the resulting image deviates from the seed (and fewer elements of the seed image are conserved). Does running the optimization with a higher number of iterations result in the final image being completely different from the seed image? Could the authors provide the results of such an experiment?\n\n- **mec.q.2.** In section 4.1, the authors say that optimizing $\\mathbf{z}^{(k)}$ directly leads to poor results. Could the authors provide experiments that support this?\n\n## Evaluation Criteria\n\nThe authors construct a \"seed image\" dataset that is subsampled from ImageNet.\nThey associate a target prompt to a seed image in various ways (random, top image in CLIP similarity, or randomly picked among top-k most similar images with CLIP).\nThey utilize various TiTok checkpoints, which allows them to compare results among different autoencoder types (with discrete or continuous tokens), different sizes, and different number of tokens.\nThey also compare with a 2D tokenizer (MaskGIT VQGAN).\n\nTo evaluate the diversity and quality of generation, the authors utilize the FID and IS metrics.\nTo evaluate alignment with target prompts, the authors utilize CLIP and SigLIP similarity.", "theoretical_claims": "The authors do not make any theoretical claims worth discussing in this section.", "experimental_designs_or_analyses": "The authors present a series of experiments to compare seed sizes and seed association strategies.\nThey build on these results and claim that around 1000 seed images are enough to produce diverse generations. They also conclude that adding some stochasticity in the seed-to-prompt associations leads to better FID (as it increases diversity) but worse IS (compared to taking the associations having the best similarity). \n\n- **eda.q.1** In line 370, \"achieving an FID of 8.6 for 50k samples\". Could the authors also provide the other metrics (IS, CLIP, SigLIP) for this experiment (and ideally add it to Table 1)?\n\nThe authors also present experiments that compare performance across different latent space dimensions, discrete and continuous tokens, as well as 1D and 2D tokenizers.\n\nThe authors conclude from Table 2 that a decreasing number of tokens leads to \"significant improvements in generation quality\". Upon further investigation, the authors compare VQ-LL-32, a **large size** variant of the tokenizer with 32 tokens, and VQ-BB-64, a **base size** variant of the tokenizer with 64 tokens (while keeping a constant codebook size). \n\n- **eda.q.2** Could the authors provide additional evidence to confirm that the improvements indeed stem from the smaller number of tokens and not the larger model?\n\nThe authors conclude from Table 3 (first two rows) that a discrete latent space is essential in achieving good generative performance. \nThe authors also conclude from Table 3 (first and last row) that the large number of tokens in MaskGIT's VQGAN \"prevent the successful application of the test-time optimization for generation\" and that \"the 2D tokenizers' spatially arranged tokens lead to optimization results that are spatially incoherent\".\n\n- **eda.q.3a** Is the degraded generative quality caused by the large number of tokens of the 2D tokenizer, the spatially arranged 2D grid, or both? It seems that further experiments are needed to support this claim, especially that eda.q.2 is not a given.\n\n- **eda.q.3b** Regarding the spatially incoherent results, could the authors provide visuals that illustrate these observations?\n\nThe authors also provide some tweaks of their algorithm along with an ablation study, which justifies their design choices.", "relation_to_broader_scientific_literature": "This paper explores the latent space of one-dimensional image tokenizers which are relatively recent.\nThe authors highlight an interesting correlation between token positions and image semantics, and utilize it to perform editing and inpainting. \nAs such, this paper relates to other works involving 1D image tokenizers, and works tackling text-guided image editing and inpainting, specifically in a test-time optimization (training-free) context.", "essential_references_not_discussed": "The authors utilize test-time optimization on recent 1D image tokenizers for image editing and inpainting. \nWhile the paper discusses other test-time optimization approaches for image editing, I recommend including a discussion on text-guided image editing approaches in a broader context, which would help clarify how the work relates to existing techniques.", "other_strengths_and_weaknesses": "## Strengths\n1. The paper is well written and follows a clear and logical structure, making it an enjoyable and insightful read.\n2. The concepts presented regarding 1D tokenizers in the paper are original, interesting, and insightful. \n3. Additionally, the concepts presented hold potential for future research in this area.\n\n## Weaknesses\n1. My main concern is that some claims and conclusions that are important for the context of this paper require additional experimental support (see related questions). \n2. Moreover, the test-time optimization text-guided editing algorithm seems to affect aspects of the image that are not related to the prompt (or that should be conserved). Could the authors elaborate on this issue?\n3. Providing additional explanation and interpretation of the results, particularly in Section 5.3, would strengthen the discussion and findings of the paper (e.g. visualizations and insights on why a 2D latent space leads worse results).\n4. For the sake of reproducibility, it would be valuable if the authors provided the code used to conduct the experiments in this paper as part of the supplementary material.", "other_comments_or_suggestions": "## Writing suggestions\n\n1. It would be helpful to provide equation numbers for future referencing.\n2. In the equation of section 4.1, the text prompt is not represented and not given to the loss function (which might lead to confusion).\n3. Typo in line 362 (second column).\n4. Typo in line 375 (prevent**s**)", "questions_for_authors": "Questions for the authors are listed in bullet points in their relevant context within the review.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A1AvaoK88w", "reviewer": "Reviewer_G4pF", "summary": "This paper proposes an optimization-based method to tweak the latent space of tokenizer for image editing tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review\\! We hope to address some of your concerns in this response.\n\n* **Image Editing vs. Generation**\n\nWe decided to choose the \"image generation\" nomenclature in line with previous work on so-called retrieval-augmented generative models, such as RDM \\[1\\].\n\n**Furthermore, as requested by Reviewer mQYF (see \"Initializing with Random Tokens\" for more details and quantitative results), we find that it is indeed possible to generate images \"from scratch\", by starting with random tokens.**\n\nLink to uncurated generations starting from random tokens: https://i.ibb.co/20L8xk1q/rand-tok.png\n\n* **Additional Failure Cases**\n\nThe \"from scratch\" generations referenced above are uncurated, and we plan to include additional visualizations in the appendix of our revised submission.\n\nYou may also be interested in our response to Reviewer apSr on additional out-of-domain examples.\n\nFinally, you can find a discussion of some limitations of the token optimization-based editing approach in our response to Reviewer N1ti's question regarding limited control over the editing process. In particular, there are no guarantees that text-guided editing will not result in unintended modifications to the input images, although we believe that this issue could be alleviated with further engineering effort (e.g., a more advanced objective function combining reconstruction and CLIP components).\n\n* **Determinism and Duplicated Seed/Prompt Inputs**\n\nAll experiments in Section 5 do in fact include some small amount of randomness in the loss due to nondeterminism in some of the CUDA kernels used in our implementation \\[2\\].\n\nInterestingly, even the small amount of nondeterminism caused by implementation details of the CUDA kernels used in PyTorch is sufficient to lead to diverse generations by the test-time optimization procedure.\n\nWe have experimentally verified that using deterministic implementations of these algorithms leads to deterministic results, in turn leading to degraded FID in the case of smaller number of seed images or deterministic seed selection (e.g. 500 seed images or 1000 seed images with top-1 selection). In the case of at least 1000 seed images with top-1% selection, there is enough diversity in the seed-to-prompt association to yield about 4000 unique inputs, such that the FID-5k results would be relatively unaffected by the use of a fully deterministic implementation.\n\nThank you for pointing out this issue. We will address this point explicitly in the main text of the revised submission.\n\n### References\n\n\\[1\\] Blattmann, A., Rombach, R., Oktay, K., Müller, J., and Ommer, B. Retrieval-augmented diffusion models. NeurIPS 2022\\.\n\n\\[2\\] https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html", "claims_and_evidence": "- I think this paper highlights the generation ability of the 1D tokenizer, specifically TiTok. However, I find using the word of \"image editing\" or \"image variation\" is better, because it is not like the standard generation model like diffusion model, that can generate from very cheap random noise, it needs to start from a seed image\n- And on a high level, I feel this paper is telling two stories, i.e. section 3; section 4-5. It firstly shows the position 1D tokenizer has semantic meaning, and later presents the 1D tokenizer can be used to search the latent space. However, I don't see a clear connection between these two stories, or they are kind of broken. I think the reason of 1D tokenizer works better than 2D tokenizer for gradient-based method is its latent space is smaller and easier to perform optimization. It looks to me that even without the section 3, section 4-5 can already explain itself.\n- I hope to see more failure cases and limitation discussed in the paper", "methods_and_evaluation_criteria": ".", "theoretical_claims": ".", "experimental_designs_or_analyses": "- Regarding section 5, I wonder does 50k text prompts “a photo of a class” have duplicated prompts? Asking this because I think the optimization process is deterministic if you loss is deterministic and the number of iterations and optimizer are fixed. Then same text prompts should result in same image variation? Then I believe FID in this case won't be good? So I think if you want the FID low, you need to have non-duplicated promtps or make the optimization process have some randomness?", "relation_to_broader_scientific_literature": ".", "essential_references_not_discussed": "Some other open-sourced 2D discrete image tokenizers:\n\n- Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation\n- Cosmos World Foundation Model Platform for Physical AI", "other_strengths_and_weaknesses": "I think the design in section 3 is interesting, but I find it's a bit impractical to use because you always need to do a classification problem first to find the token that controls some concept.", "other_comments_or_suggestions": ".", "questions_for_authors": ".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: - Reasons to Accept:
Novel application of 1D tokenizers for generation and editing (agreed by all reviewers).
Training-free approach is simple, elegant, and surprisingly effective (noted by mQYF, N1ti).
Strong empirical results with competitive FID/IS scores and compelling ablations (mQYF, apSr).
Rebuttal provided new experiments on generation from random tokens and clarified key limitations (N1ti upgraded his score).

- Reasons to Reject / Concerns:
Limited generalization beyond ImageNet and reliance on 1D tokenizer properties (apSr, G4pF).
Editing instability and lack of control over preserved image regions (dszF, N1ti).
Disconnection between token analysis (Sec. 3) and generation pipeline (Sec. 4–5) (G4pF).
Contribution is mostly empirical (apSr).

Overall even if simple the observation has been considered as interesting by most reviewer and is worth to be discussed.
Decision: Accept (poster)