Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models
Authors: Bowei Chen
Sai Bi
Hao Tan
He Zhang
Tianyuan Zhang
Zhengqi Li
Yuanjun Xiong
Jianming Zhang
Kai Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Visual Tokenizer
Latent Diffusion Model
Foundation Encoder
Primary area: generative models
TL;DR: We propose aligning pretrained visual encoders to serve as tokenizers for latent diffusion models in image generation.

Abstract: In this work, we propose aligning pretrained visual encoders to serve as tokenizers for latent diffusion models in image generation. Unlike training a variational autoencoder (VAE) from scratch, which primarily emphasizes low-level details, our approach leverages the rich semantic structure of foundation encoders.
We introduce a three-stage alignment strategy called AlignTok: (1) freeze the encoder and train an adapter and a decoder to establish a semantic latent space; (2) jointly optimize all components with an additional semantic preservation loss, enabling the encoder to capture perceptual details while retaining high-level semantics; and (3) refine the decoder for improved reconstruction quality.
This alignment yields semantically rich image tokenizers that benefit diffusion models.
On ImageNet 256$\times$256, our tokenizer accelerates the convergence of diffusion models, reaching a gFID of 1.90 within just 64 epochs, and improves generation both with and without classifier-free guidance. Scaling to LAION, text-to-image models trained with our tokenizer consistently outperforms FLUX VAE and VA-VAE under the same training steps. Overall, our method is simple, scalable, and establishes a semantically grounded paradigm for continuous tokenizer design.

Published as a conference paper at ICLR 2026
ALIGNING VISUAL FOUNDATION ENCODERS TO TOK- ENIZERS FOR DIFFUSION MODELS
Bowei Chen'
Sai Bi
Hao Tan He Zhang² Tianyuan Zhang² Zhengqi Li
Yuanjun Xiong? Jianming Zhang Kai Zhang
'University Washington 'Adob Research
https://a /att:///algntok... www.llggnnt..t....ii.oeo. github...h........ 10
ABSTRACT
In this work, we propose aligning pretrained visual encoders to serve as tokeniz- ers for latent diffusion models in image generation. Unlike training variational autoencoder (VAE) from scratch, which primarily emphasizes low-level details, our approach leverages the rich semantic structure of foundation encoders. We introduce a three-stage alignment strategy called Align'Tok: (1) freeze the encoder and train an adapter and decoder to establish a semantic latent space; (2) jointly optimize all components with an additional semantic preservation loss, enabling the encoder to capture perceptual details while retaining high-level semanties; and (3) refine the decoder for improved reconstruction quality. This alignment yields semantically rich image tokenizers that benefit diffusion models. On ImageNet x56x256. our tokenizer accelerates the convergence of diffusion models, reach- ing a gFID of 1.90 within just 64 epochs, and improves generation both with and without classifier-free guidance. Scaling to LAION, text-to-image models trained with our tokenizer consistently outperforms FLUX VAE and VA-VAE under the same training steps. Overall, our method is simple, scalable, and establishes a semantically grounded paradigm for continuous tokenizer design.
INTRODUCTION
Diffusion models have recently emerged as the leading method for high-fidelity image generation. crucial component of training image diffusion models is the continuous visual tokenizer, which defines the latent space where diffusion operates (Rombach et al. 2022). Training such a tokenizer involves two tasks: (1) the encoder must learn a diffusion- friendly latent space, often referred to as the diffusability of the latent space (Skorokhodov et al., 2025); and (2) the decoder must learn to reconstruct the input signal. A common practice for training continuous visual tokenizer is to adopt variational autoencoder (VAE), optimized with reconstruction loss and lightly weighted KL regularization term. Since the KL term typically has only a small weight, training is dominated by reconstruction loss, making the two tasks asymmetric: the decoder's reconstruction learning is direct and well-supervised, while the encoder's representation learning is indirect - the latent space shaped largely as byproduct of reconstruction and only weakly regularized by the KL prior. As a result, the latent space often develops an unpredictable structure dominated by low-level details, limiting its diffusability (Yao et al. 2025). Samantic Regudarization (e.g,, VA-VAE)
Pretrained Alignment (Ours)
Recent work on tokenizers for image dif- fusion models has explored adding con- straints such as semantic regularization (Fig. [ left), which adds loss term to encourage the latent space to align with representations from large pretrained en- coders (Yao et al.) 2025 Chen et al.) 2025a). These studies demonstrate that se- mantically grounded latents exhibit better diffusability, leading to improved genera- tion quality in diffusion models. However,
Input
Input
Encoder
Pretrained Encoder
Encoder ?
Pretrained
Latant
Regularization loss
Decoder
Decoder
Output
Trainable
Figure 1: Regularization vs. Alignment.

---

Published as a conference paper at ICLR 2026
these methods still fall short, as the encoder must still learn semantic structure from scratch via the regularization loss while simultaneously managing the competing reconstruction objective.
Our goal is to design image tokenizer with stronger semantic grounding hence better diffus- ability- - with competitive reconstruction ability. Our intuition is that learning semantic is inherently more difficult than learning reconstruction. Thus, we take different perspective: rather than forcing the encoder to learn semantics from scratch, we align a pretrained visual foundation encoder (e.g., visual tokenizer (Fig. right). Since the encoder already cap- tures rich semantic structure, our alignment makes the first task - learning a diffusion-friendd latent space (i.e.. achieving strong diffusability) - much easier. Training can then focus on equipping the tokenizer with reconstruction ability, avoiding the difficulties of learning semantic from scratch.
DINOv2 (Oquab al. 2023
We implement this idea through a three-stage alignment procedure. First, we freeze the pretrained encoder and train a lightweight adapter and decoder with reconstruction loss, establishing a semanti- cally grounded latent space. Second, we jointly optimize all components with an additional semantic preservation loss, enabling the encoder to capture fine-grained perceptual details essential for both reconstruction and generation, while maintaining its underlying semantic structure. Finally, we fine- tune only the decoder to enhance reconstruction fidelity while keeping the latent space fixed. This progressive alignment preserves a semantically rich, diffusion-frient friendll latent space that also retains the details necessary for accurate reconstruction and high-quality generation.
We demonstrate the effectiveness of our method on both the ImageNet 256 dataset (Deng et al,) 2009) and the LAION dataset (Schuhmann et al. 2022) by training diffusion models using our tinuous tokenizers. On ImageNet, our semantic tokenizer accelerates the convergence of diffusion models and improves generation quality over previous methods, both with and without classifier- free guidance (CFG), as well as in unconditional generation settings. To further validate scalability, we train text-to-image diffusion models on LAION and show that they converge significantly faster than the FLUX VAE (Labs) 2024) and VA-VAE (Yao et al. (025). We conjecture that these benefits arise from a well groundedd semantic latent space, which provides more structured representation.
In summary, we propose new paradigm for training visual tokenizers by aligning pretrained visual encoders. Our approach is simple, scalable, and effective, and we believe it can open new directions in tokenizer design while inspiring future research in generative modeling.
RELATED WORK
2.1 FOUNDATION ENCODER FOR REPRESENTATION LEARNING
Large-scale foundation visual encoders enable the extraction of rich, transferable representations from diverse visual data. Models such as CLIP RRafford et al. 2021). SigI.IP (Zhai et al. 2023)
SigL.P 2 (Tschannen et al. 2025). MAE (He al. 20022), Perception Encoder (Bolya al. 2025
DINOv2 (Quub et al. 2023), and DINOv3 (Siméoni et al. 2025) demonstrate that pretraining on massive datasets enables encoders to capture meaningtul visual features that generalize effectively across downstream visual understanding tasks. In this paper, we adopt DINOv2 as our visual foun- dation encoder by default, since we empirically find that it is more effective for diffusion modeling than alternatives such as SigLIP 2 and MAE.
2.2
IMAGE TOKENIZERS FOR GENERATIVE MODELS
Image tokenizers are essential for scaling visual generation. They use an encoder-decoder frame- work to map images into compact latent spaces where generative models can operate more effec- tively and efficiently (Rombach et al. 2022). A common practice in training tokenizers is rely on reconstruction losses such as I.1 loss, perceptual loss, and adversarial loss (Rombach et al. 2022
Kouzelis et al., 2025 Wen et al. 2025 Wu et al. 2025; Yu et al.) 2024, Skorokhodov et al. 2025 Hansen-Estruch et al. 2025 Silvestri & Ambrogioni 2025 Van Den Oord et al.) 2017). Depending on the design, tokenizers are either continuous, where the latent distribution is regularized by a KL loss, or discrete, where quantization with a codebook is enforced via VQ loss. While these methods achieve faithful reconstruction fidelity, the resulting latent space is often dominated by fine- -nn-grri details because they mainly rely on reconstruction loss, which can hinder generative performance.
Semantic Regularization. Recent methods introduce semantic regularization strategies to enrich the image tokenizers with higher-level semantics (Xiong al. 2025 Chen et al.) 2025a Yao al.
2

---

Published as a conference paper at ICLR 2026
2025
Chen et al. 2025b Lee et al. 2025 Yang al. 2025 Chu et al. 2025 Lu et al. 2025
VA- VAE Yao et al. 2025) introduces a continuous tokenizer for diffusion models by regularizing its latent space to be close to pretrained encoder, whereas GigaTok (Xiong et al. 2025) proposes discrete tokenizer for autoregressive models by imposing semantic constraints on decoder features. Instead of using semantic regularization, we align a pretrained encoder that is already capable of extracting high-level semantic representations, and show that this leads to more hiffussion-friedde latent space. We conduct extensive comparisons with VA-VAE, as both methods target diffusion models, and demonstrate that our alignment strategy outperforms semantic regularization.
Encoders as Discrete Tokenizers. The use of pretrained encoders in tokenizers has also been studied, but primarily in the context of discrete tokenizers for autoregressive models. One line of work treats the pretrained encoder as a fixed feature extractor without fine-tuning to encode perceptual details (Zheng et al.; 2025a Chen et al., 2025c Xie et al. 2024; Wang et al.) 2024). Another line of work fine-tunes pretrained encoders with additional architectural designs, such as introducing extra encoders (Qu et al.
Pretrained
2025 Jiao et al. 2025) or decoders (Han et al. (025). or
splitting encoder features into separate vocabularies (Song et al. 2025). A third direction leverages contrastive learning, fine-tuning the pretrained encoder with image text supervision to capture
mantic alignment (Wu et al. (2044) [M et al. 2025: ZZhao et al. 2025 Lin et al.. 2025). However, this strategy primarily assumes that the encoder language-alignet model (e.g.. SigL.P 2).
In contrast to these works on discrete tokenizers, we focus on continuous tokenizers for diffusion models. Rather than introducing additional architectural design or requiring image- text supervision, we keep the simple architecture of autoencoder and directly align a pretrained encoder with a self- supervised semantic preservation loss. Our method can generalize any visual encoders, offering a simple and scalable path toward semantically rich tokenizers for generative modeling.
Concurrent Work. (Tang et al. 2025) explores enabling pretrained CLIP with reconstruction ability. The key distinction is that we target diffusion- based generation, providing extensive exper- iments showing that aligning a pretrained encoder yields latent space with better diffusability than using semantic regularization. In contrast, their work focuses on unified multimodal understand- ing, generation, and editing within a hybrid architecture that combines multimodal large language models (MLLMs) with diffusion. Another difference is that we study different foundation visual encoders and identify DINOv2 as particularly well suited for latent diffusion models, whereas their focus remains on CLIP in the context of unified modeling.
Relationship to RAE. Zheng et al. 2025b) proposes to directly adopt a pretrained foundation encoder, without fine-tuning, as the tokenizer encoder. While this preserves strong semantic repre- sentations, it results in high-dimensional latent space with a large channel dimension. Diffusion models are known to struggle in such high-channel latent spaces, optimization becomes unstable and noise scheduling becomes less effective. To mitigate this issue, RAE introduces techniques such as dimeensionddeeeddent noise shifting and widened diffusion head to stabilize training in high- dimensional latents. Although RAE demonstrates strong performance for downstream generative model training, its reconstruction quality is limited by the frozen encoder, which is not optimized for faithful pixel-level reconstruction. In contrast, our approach fine-tunes the encoder, significantly improving reconstruction fidelity while maintaining semantic alignment. We believe that combining RAE'S high-channel semantic latent space with our alignment and fine-tuning strategy could offer the best of both worlds: strong semantic representations together with high-quality reconstruction.
Relationship to REPA-E. While both REPA-E et al. (2025) and our method leverage a vision foundation model, it is not straightforward to claim an overall advantage of one approach over the other because the two methods follow different setups and can in fact be combined. Specifically, REPA-E trains the tokenizer and the diffusion model jointly in an end-to-end manner. Even under this end-to-end regime, the authors show that initializing the tokenizer with existing tokenizer works better than random initialization; specifically, their best model (E2E-VAH) initializes the tokenizer from VA-VAR. In contrast, our method focuses solety on training the tokenizer itself. This makes our tokenizer a drop-in replacement for the VA-VAE component inside REPA-E: since REPA-E already benefits from initializing its tokenizer with VA-VAE, could similarly initialize with our tokenizer as a potentially stronger starting point for its end-to-end training.
3

---

Published as a conference paper at ICLR 2026
Stage Latent Alignment
Trainable
Reconstruction loss
Foundation Encoder
Adapter A
Decoder
Input
Sg
Output
f
Stage 2: Perceptual Alignment
Stage 3: Decoder Refinement
Preservation Loss
B
Reconstruction loss
Reconstruction loss
Figure 2: Method Overview. Stage 1: Latent Alignment (top). The pretrained encoder is kept frozen, while the adapter and decoder trained with reconstruction loss align its output into semantically grounded latent space for generation. Stage Perceptual Alignment (bottom left). All components are optimized jointly enrich the latent space with low-level details, while semantic preservation loss ensures that it retains high-level semantics. Stage 3: Decoder Refinement (bottom right). Only the decoder fine-tuned with reconstruction loss enhance reconstruction fidelity.
3 METHOD
We aim to build semantic, diffusion--fienndy visual tokenizer by aligning pretrained visual en- coder. We begin with a review of latent diffusion models, followed by an introduction of our method.
3.1 BACKGROUND OF LATENT DIFFUSION MODELS IN IMAGE GENERATION
Latent diffusion models (LDMs) (Rombach et al. 2022) operate by learning a denoising process in the compressed latent space produced by continuous visual tokenizer. The tokenizer consists of an encoder E and decoder D. The encoder maps an input image x E R [f <ixW>x' x to latent
Hf x where is image height, is image width, f is the downsampling factor and d is the channel dimension. The decoder reconstructs & D(2)). The tokenizer is trained with a reconstruction objective combining pixel-level L1, perceptual, and adversarial losses:
code 2g = Exx)
free = Lr, (x. () + Eperceptual (I, .)) + wy (x,
(1)
where Wp and Wo are the weights for the perceptual and adversarial loss, respectively. In addition, KL regularization term is often included alongside the reconstruction loss encourage a well- structured latent space. After training the tokenizer, a diffusion model learns to reverse forward noising process in this learned latent space. A common formulation is flow matching, where we define an interpolating path:
= (1 f1 tt 21 ~ I). t [0.
(2)
where is the diffusion timestep. The velocity field is given by u, $4 - 2g. The diffusion model Up is trained to predict this velocity, with the loss LFM = Ezo. [Ilvo(ze,t) uell
3.2 ALIGNING PRETRAINED ENCODER TO VISUAL TOKENIZER
Our method leverages pretrained visual encoder, which offers rich semantic representations, and progressively adapts into a diffusion-friennt friendly visual tokenizer. This implemented through three alignment stages, as illustrated in Fig. 2 First, we align the encoder's semantic space to generative latent space by training a lightweight adapter and decoder (Latens Aligument). Second, we jointly optimize all components enhance generation and reconstruction fidelity while preserving semantic structure (Perceptual Alignment). Finally, we fine-tune the decoder to further improve reconstruction quality while keeping the latent space untouched (Decoder Refinemen)..
4

---

Published as a conference paper at ICLR 2026
ITD to Receriant Thaning Stops
Unser hhittggccsssyy "elerrer Patinn Steza
14
Stipe Proeeaate Low Stage
70-
14
1004 thines
12
2
1.8
5A-
10-
0.2 2006
1,50
2006
2006
250K
45cK 156k 200K
2006
200C
230
453K
2aza
Figure 3: Reconstruction vs. Semantic Preservation in Tokenizer Training. Left: reconstruction FID (rFID) across training steps. Right: linear probing accuracy across training steps. Linear prob- ing accuracy is evaluated on the latent code (32 channels), except for All Stages, Pre-Adapter (1024 channels), which is reported only for reference. In this case, linear probing accuracy is measured on the feature before the adapter, using the same checkpoint as Stage w/ Semantic Preservation Loss.
Stage I:
Latent Alignment. As the first stage, the goal is adapt the pretrained encoder create latent space suitable for generation (Fig. ttop)) This requires that the latent representations contain semantic information and can be mapped back the image domain for reconstruction.
Given an input image T, we extract its embedding using a frozen pretrained encoder Ei,, Since em- beddings from encoders trained for representation learning tasks are typically very high-dimensional (e.g., 1024 channels for DINOv2-L/14), they are not directly suitable for diffusion models, which usually operate in lower dimensions (e.g.. 32 or 64). To address this, we introduce an adapter that projects the high-dimensional features into a compact latent code of dimension d (32 by default):
2g = (xE)
To complete the tokenizer, a decoder D is then introduced to reconstruct the input image from 2g. During this stage, only the adapter and decoder are trained with the reconstruction loss in
Eq. [ while the pretrained encoder F.p. remains frozen to ensure a semantically-rice latent space. We omit the KI. term because we found that does not provide benefits and only imposes unnecessary distributional constraints, which can distort the encoder's semantics in the latent space.
Although this stage yields a semantically grounded latent space, it does not achieve high-fidelity reconstruction (see noticeable color shift in top left), as the frozen encoder cannot capture the fine- meaiggai perceptual details required for precise image reconstruction and high-quality generation.
right image of Fig. and the orange point in Fig.
Stage 2: Perceptual Alignment. The goal of this stage is to adapt the pretrained encoder so that can capture fine-grained, low-level image details while still preserving semantic features, as shown in Fig. _ (bottom left). Starting from the checkpoints of the previous stage, we jointly optimize Ep. A, and D using the same reconstruction loss as in Eq. [T. This encourages Ep to encode richer details, thereby improving reconstruction quality, as indicated by the green curve in Fig. 3 (left). However, while reconstruction improves rapidly, this optimization simultaneously causes the latent space to catastrophically lose its semantic structure, as reflected by the sharp drop in linear probing accuracy (green curve in Fig. 3 right). To address this issue, we introduce a simple yet effective semantic preservation loss, which constrains the latent codes produced in the current stage to remain close to those obtained in the previous stage. Formally, we define this loss as an L.2 loss:
Cap = Leg (2ū,z0)
(4)
where =ū and zo are the latent codes produced by E, and A in the current stage (being updated) and the previous stage (kept frozen), respectively. An alternative is to apply this loss directly on the output of Ep. providing more flexibility by leaving the adapter A unregularized. However, we found this variant degrades generation quality (see ablation study). The overall loss for this stage is:
Epa = Lree + Wap
(5)
where wsp = 1 balances the semantic preservation loss. As shown in 3 (blue curve), this strategy maintains semantically rich latent space while achieving comparable reconstruction performance.
Stage 3: Decoder Refinement. The previous two stages already align the pre- mre-trained encoder into a visual tokenizer that significantly boosts generation performance. The goal of this stage is to further
5

---

Published as a conference paper at ICLR 2026
Table I: Ablation study. Evaluated on ImageNet 256x256 at 80K training steps with 30 sampling steps, using the CFG scale that yields the lowest generation FID (gFID). Our full stree-stage model achieves the best balance between reconstruction and generation quality.
Conbguration
rFID
PSNRI
LPIPS
L. P. Ace.
gFID
187
Preet
Recall
Semantic Preservation Loss
Weight
0.33
26.29
0.110
9.50%
3.05
215.1
0.832
Weight 5
0.49
23.70
0.163
40.55%
2.48
244.4
0.836
0.566
Weight =
0.59
22.63
0.189
40.89%
2.59
243.5
0.839
0.562
Applied Pre-Adupter
0.34
25.78
0.118
15.61%
2.83
226.9
0.835
0.553
Cosine
0.37
25.23
0.129
37.99%
2.23
248.6
0.819
0.585
Training Strategy for Stage
LoRA Fine-Tuning
1.35
26.
0.121
18.56%
2.97
243.3
0.815
0.557
w/o EMA
0.33
25.70
0.122
35.04%
2.24
246.0
0.809
0.587
Ligh-LLvel Design
Larger Decoder
0.36
26.27
0.121
27.24%
2.52
248.3
0.808
0.571
Stage only
1.63
17.34
0.323
41.53%
3.00
246.4
0.843
0.529
Stage + Stage 2
0.36
25.62
0.121
35.09%
2.19
248.6
0.811
0.591
Full Model
0.26
25.83
0.117
35.09%
2.17
249.3
0.811
0.599
refine the decoder improve reconstruction quality, as shown in Fig. 2 (bottom right). The key motivation is that, although the latent space is semantically aligned, the decoder may still underfit because the latent space kept changing throughout the previous two stages. Fine-tuning the decoder alone allows it to better exploit the existing latent representation without disturbing its semantic structure. Specifically, we continue training from the second stage but only update the decoder. This preserves the learned latent space and can even be applied after training the downstream generative model. As shown in Fig. 3 left (purple curve), this stage improves reconstruction performance.
EXPERIMENTS
We evaluate on two datasets: ImageNet 256x256 (Deng et al. 2009) and a subset of LAION-
2B (Schuhmann et al., 2022). Most ablation studies and baseline comparisons are conducted on ImageNet (Sec. 4.3) followed by larger scale rext-to-image experiments on LAION (Sec. 4.4) For ImageNet, we set the downsampling ratio to f 16, the latent channel dimension to d = 32, the semantic preservation loss weight to W'sp 1, and the sampling step to 30, unless otherwise speci- fied. Same VA-VAE (Yao et al.) 2025), we use LightningDiT (~673M parameters) as generative model for ImageNet experiments. See more implementation details in Appendix (Sec.
Metrics. For ImageNet, we use reconstruction FID (rFID), PSNR, and LPIPS to evaluate reconstruc- tion quality; linear probing accuracy to assess the semantic structure of latent space; and generation FID (gFID), Inception Score (IS), Precision (Prec), and Recall to measure generative performance.
4.1 ABLATION STUDY
We test different variants of our design and summarize them in Tab. I All variants we test do not employ the third stage, thus we mainly compare them to Stage 1 Stage 2 for analysis.
Semantic Preservation Loss. We first analyze the effect of varying the weight of the semantic preservation loss in Eq. 5 Without this loss (weight = O), the model achieves slightly better recon- struction quality, but the linear probing accuracy and generative metrics degrade, indicating that the latent space collapses toward low-level details the expense of semantic structure. Increasing the weight (5 or 10) enforces stronger preservation, which improves generative performance over the one with weight = O. but comes with a notable drop in reconstruction fidelity. These results reveal a clear trade-off: too little preservation leads to semantic drift, whereas too much preservation overly constrains the encoder and harms pixel-level fidelity. In the Stage 1 + Stage variant, a moderate weight of achieves the best balance between generation and reconstruction.
We test a variant Applied Pre-Adapter where the semantic preservation loss is applied directly on the outputs of the pretrained encoder (before adapter). This approach partially preserves semantics, and the generation quality lags behind Stage 1 + Stage 2 variant. This may be because leaving the adapter unregularized gives it too much fiexibility, leading also test a variant Cosine Loss that replaces the L2 loss with a cosine similarity loss, which applies the semantic constraint with explicit normalization. This variant achieves performance comparable
loss of semantic structure. In addition,

---

Published as a conference paper at ICLR 2026
SoAce Lightieg Go
515
Figure 4: Comparison of Sampling Steps, CFG Scales, and Convergence Speed. Evaluated on ImageNet x56x Left: effect of sampling steps versus gFID at 80K training steps. Middle: effect of CFG scale versus gFID at 80K training steps with 30 sampling steps. Right: effect of training steps versus gFID with 30 sampling steps. QKNorm is enabled during extended training to ensure stability. All gFIDs in the left and right figures are reported using the best-searched CFG scale.
to L2 loss. This suggests that, least in current setup, both losses lead to similar behavior, indicating that the performance differences are not dominated by the specific choice of L2 versus cosine loss.
Training Strategy. We compare different optimization strategies for the perceptual alignment stage against the Stage + Stage 2 variant. LoRA Fine-Tuning performs noticeably worse, showing that restricting updates to low-rank adapters is insufficient for balancing semantics and reconstruction. Removing EMA (Karras et al., 2024) slightly decreases generation performance. The Stage Stage 2 variant uses EMA to stabilize training and thereby produce a more stable latent space.
High-Level Design. We also evaluate alternative architectural and training-stage designs. Employ- ing a larger decoder (~355IM parameters), implemented as a hybrid Cunsformer-CNe architecture following (Xiong et al., (025)). yields slight improvements in reconstruction while degrading gener- ative performance. This suggests that increasing decoder capacity may not yield additional benefits when training on a dataset of ImageNet's scale. Using only the latent alignment stage (stage 1) pre- serves high-level semantics but causes poor reconstruction, rendering this configuration unsuitable for generative modeling. Introducing the perceptual alignment stage (stage 2) resolves this issue, substantially improving both reconstruction and generation, thereby validating the need to fine-tune the encoder. Extending to the stage design with decoder refinement further strengthens recon- struction and slightly improves generative quality, showing the importance of progressive alignment.
Other Pretrained Encoders. We compare different pretrained encoders as the backbone of our tokenizer. As shown in Tab. [ MAE achieves the strongest re- construction fidelity but performs the worst in gener- ation, likely due to its reconstruction objective. DI- NOv2 achieves the best balance, delivering superior generation quality while maintaining competitive re- construction performance.
Table 2: Comparison of Various Pre- trained Encoders. ImageNet 256x256 at 80K steps; 30 sampling steps; CFG tuned for best gFID Stage 3 is not applied.
Config. MAE
rFID
gF
18t
Prect
Recall?
0.29
3.12
216.5
0.834
0.543
SigL.P 2
0.35
2.22
246.1
0.816
0.576
DINOv2
0.36
2.19
248.6
0.811
0.591
4.2 COMPARISON WITH OTHER TOKENIZERS
We provide comprehensive comparison with two baselines: the vanilla VAE and VA-VAE (Yao al. 2025), which is a representative method that applies semantic regularization using a pretrained DINOv2 model to the latent space. Unless otherwise noted, all baseline checkpoints are taken from the official VA-VAE implementation.
4.2.1 CLASS-CONDITIONAL GENERATION
Sampling Step. In Fig. left, we show that our method achieves better performance than VA-VAE with fewer sampling steps. While VA-VAE requires more than 120 steps to approach its best FID, our tokenizer reaches near-optimal performance with 80 steps. Remarkably, our 50-step generations even surpass the quality of VA-VAE's outputs at 250 steps the default in its official implementa- tion. We attribute this to the smoother latent space, where discretization errors cause only minor variations, rather than drastic shifts in color, semantics, or overall composition. This robustness allows the model maintain higher fidelity with fewer sampling steps.
7

---

Published as a conference paper at ICLR 2026
Table 3: Comparison with Other Tokenizers. Evaluated on ImageNet 256x256 80K training steps with 30 sampling steps. The checkpoints for both the Vanilla VAE and VA-VAE with CNN encoders are taken from the official VA- VAEVAE repository. VA-VAE denotes the VA-VA VA-VEE model we trained, using VIT encoder that matches our architecture but initialized from scratch.
Tokenizer
Enc. Arch.
rFID
L. A.c.""
gFID (uncond). gFID w/o CFGL
BEID w/ CFG.
fl6d32 (downsampling factor 16, latent dimension 32)
Vanilla VAE
CNN
0.26
6.04%
29.12
10.17
3.31
VA-VAE
CNN
0.28
22.96%
19.12
7.79
3.13
VA-VAE
VIT
0.37
33.57%
18.27
8.21
3.16
Ours
VIT
0.26
35.09%
11.80
4.05
2.17
f16d64 (downsampling factor 16, latent dimension 64)
Vanilla VAE
CNN
0.17
5.09%
36.41
16.99
4.03
VA-VAE
CNN
0.14
19.72%
26.70
12.23
3.20
VA-VAET
VIr
0.18
43.53%
19.81
7.92
3.19
Ours
VIT
0.17
46.99%
14.40
5.24
2.34
CFG Seale. Fig. middle, we plot gFID versus CFG scale, where our tokenizer consistently outperforms VA-VAE across the entire range. Notably, our method achieves strong performance even at low CFG values, whereas VA-VAE relies on larger guidance scales to reach comparable fidelity. This shows that our latent space already encodes well-separated class semantics, reducing the dependence on aggressive guidance and yielding better generations with smaller CFG.
Convergence Speed. We compare the convergence speed of generative model training using tokenizer against VA-VAE. As shown in Fig. righht our approach consistently achieves better gFID across training iterations. By aligning with pretrained encoder rather than relying regularization, our method provides a more semantically structured latent space. This leads to roughly 5x faster training. requiring only ~60K steps compared VA-VAE's 300K steps for comparable quality, when evaluated with 30 sampling steps. See Appendix for qualitative comparisons (Sec. C.3
Different Channel Dimensions. In Tab. we compare Vanilla VAE, VA-VAE, and our method using latent spaces with 32 and 64 channels. Key observations include: (1) Vanilla VAE performs worst in terms of gFID. This is because its latent space primarily encodes low-level details, which diffusion models struggle to exploit effectively. (2) Our method consistently outperforms VA-VAE terms of gFID, regardless of whether CFG is used. (3) When VA-VAE employs the same ViT encoder and is trained for the same number of steps as ours, it achieves comparable linear probing accuracy but still falls short in generative performance. This suggests that our semantic structure provides advantages beyond class separation, yielding more semantically organized latent space that boosts the generative performance of diffusion models.
4.2.2 UNCONDITIONAL GENERATION
We also evaluate method in the unconditional setting (class number 1), shown in Tab. Our semantic latent space consistently outperforms baseline tokenizers, producing higher-quality generations without relying on class information. It is worth noting that all models are trained for only 80K steps, so the results may not reflect fully optimized performance.
4.2.3 RECONSTRUCTION
As shown in Tab. our method achieves competitive reconstruction with 32 channels but lags behind VA-VAE (CNN encoder) at 64 channels. Lowering the semantic preservation loss weight and increasing learning rate in stage 2 improves reconstruction match VA- MA-VE, with only a slight
drop in generation performance (still surpassing VA-VAE). See Appendix (Sec. C.2 for details.
4.3 COMPARISON WITH OTHER SYSTEMS
We conduct a systematic comparison with other systems, as shown Tab. Both VA-VAE and method are sampled using 250 sampling steps. When comparing method to VA-VAE 64 epochs (80K training steps), we surpass it in both reconstruction and generation quality, high- lighting superior convergence speed. For the 800-epoch setting (1M training steps), retrain
8

---

Published as a conference paper at ICLR 2026
Table 4: System-Level Comparison. We compare with VAR TTan al.
2024 Mag V2 Yu
et al. 2023), MAR (i et al. 2024), I-DeTTok YYng et al. 2025).
MaskDiT (Zheng et al. 2023
DiT (Peebles & Xie ).23). SiT (Ma et al. 2024), FasterDi Yao et al., 2024), MDT Gao al.
2023a), MDTv2 (Gao et al. 2023b), REPA (Yu et al.
2025 l CausalFusion (Deng al. 2024).
MAETok (Chen et al., 2025a), and VA-VAE (Yao et al. 2025). Gray and purple regions refer to
LightningDiT trained for 64 epochs (80K training steps, no QKNorm) and 800 (IM training steps, with QKNorm) epochs, respectively. Bold numbers indicate the best result in each color block.
Method
Tokenizer
token
Training Epochs
Gen w/o CFG
Gen w/ CFG
g AgteRegree (AR)
ISt
2PID
I8
Pree
Recall
VAR-d30
32
350
1.92
823.1
0.82
0.39
VTT-2
256 x 5
1080
3.65
200.5
1.78
319.4
LDM L-eTok
2.35
227.8
0.79
0.62
303.7
MAR-L
258 x 16
0.68
800
1,86
238.6
0.82
0.61
1.35
304,
0.81
0.62
Dilluzion
Prongjormeer
Using
S7>-AE
MaskDiT DIT
1600
5.69
177.9
0.74
0.60
2.28
276.6
0.80
0.61
1400
9.62
121.5
0.67
0.67
2.27
278.2
0.85
0.57
1400
8.61
131.7
0.68
0.67
2.06
270.3
0.82
0.59
FisterDiT
SD-VAE
1024 x
0.61
400
7.91
131.3
0.67
0.69
2.03
264.0
0.81
0.60
143.0
0.71
0.65
MDTV2
1080
1.58
314.7
0.79
0.65
REPA
800
5.90
1.42
305.7
0.80
0.65
CausalFusiio
800
3,61
180.9
0.75
0.66
1.77
282.3
0.82
0.61
Liglaning
(KNorm
LightingDDi LightingDiT
MALTOK
128 x
0.48
320
2.21
208.3
1.73
308.4
VA-VAE
256 322
0.28
800
2.17
205.6
0.77
0.65
1.35
295.3
0.79
0.65
Lightming wilh QKNorw
LighningDiT
VA-VAE AlignTok
256 32
0.28
800
2.50
206.2
0.76
0.65
1.52
286.6
0.79
0.65
LightningDiT
256 32
0.26
800
2,04
206.2
0.76
0.67
1.37
293.6
0.79
0.65
Liglaning withour QKNorm
Dig
VA-VAE
256 32
0.28
64
5.14
130.2
0.76
0.62
2.11
252.3
0.81
0.58
LightingDDiT
Tok
256 32
0.26
64
3,71
148.9
0.77
0.62
1.90
260.9
0.81
0.61
Table 5: Quantitative Comparison on Text-to-lmage (T2I) Generation with FLUX VAE. Com- pared on COCO Prompr 6K, which has 6K captions sampled from the COCO validation set. Each 2B-parameter T2I model is trained for 100K steps and evaluated at 256x256 resolution with CFG. rFID is computed using 200K randomly sampled images from the COYO-700M dataset (Minwoo)
et al. 2022).
Tokenizer
rFID
gFll)
KID
HPSv2
PickScore
ImageReward
Aesthetic Scores
CLIIP Scores
VQA Scores
FLUX VAE
0.102
35.78
0,018
0.242
0.397
0,162
5,411
31.21
0,775
AlignTok
0.443
30.27
0.015
0.249
0.603
0.564
5,573
32,21
0.849
LightningDiT of VA-VAE using the official repository with QKNorm enabled - necessary to avoid loss spikes, but slightly degrade generative performance, as noted by the authors. Our method (with QKNorm) achieves a gFID of 1.37, outperforming VA-VAE's 1.52 (with QKNorm) and comparable to the original VA-VAE result of 1.35 (without QKNorm) reported in their paper.
4.4 SP EXPERIMENTS ON TEXT-TO-IMAGE GENERATION
We conduct a levell comparison of our tokenizer with the widely adopted FLUX VAE on the exxt-to-image generation task. We train our tokenizer on images resized so that their shortest edge is 256 pixels, preserving aspect ratios. The diffusion model is trained for 200K steps at 256 resolution and fine-tuned for 90K steps at 512 resolution, both with variable aspect ratios.
For comparison, Fig. presents results from generative models trained with our tokenizer and with FLUX VAE for 100K steps 256x256 resolution. Our method produces images with better coher- ence, stronger text alignment, and competitive visual quality. Quantitative results (Tab. 5 further confirm the advantage of our approach. Fig. presents our generated results at 512 resolution. No- tably, tokenizer is trained only on 256-resolution images, demonstrating its ability to generalize to unseen resolutions. These results suggest that our approach scales effectively and can potentially serve strong alternative FL.UX VAE in large-scale training. See Appendix (Sec. C.3 for additional quantitative results on two more prompt sets, as well as qualitative results, including convergence speed comparisons and generated images across different resolutions and aspect ratios.

---

Published as a conference paper at ICLR 2026
Align FLUX
book
malk pouring into
luge
helicopter flies over Grand Canyou
an illustationn of tespet
red blocks
todky bear skatchoard
sliees of manga
Figure 5: Qualitative Comparison on Text-to-Image Generation with FLUX VAE. Input text prompts shown below the images and results (256x resolution) generated from generative models trained for 100K steps. Our method (bottom row) produces images with better coherence and prompt alignment compared the one using FLUX VAE (top row).
DONT PANiC!
poctraie o stane of waaiing steampunk glosses, white babist and leather jacker. deir photogragh
words Panic!" written on
large clock body
three friends magical bakery
photo a palm made water
Figure 6: Qualitative Results of Our Method on Text-to-Image Generation at 512 Resolution. The input prompts are shown below the images. Results are obtained from diffusion models trained for 290K steps. The first four are square images, and the final one has 4:5 aspect ratio.
LIMITATIONS AND DISCUSSIONS
Our method, while effective, has several limitations. First, although the semantic latent space im- proves generative quality, its reconstruction ability still lags behind FLUX VAE. This gap could be narrowed with stronger decoders, larger channel dimensions, longer training, or scaling to larger models and compute budgets. Second, evaluation is limited to images up 512 resolution. Exploring higher resolutions is an interesting future direction, and the recently introduced DINOv3 showing strong capability for variable resolutions could be leveraged for this purpose.
Our study highlights a key insight: aligning a pretrained semantic encoder yields a more generation- friendly latent space than learning semantic from scratch. Although work focuses tokenizers for image diffusion, extending this approach to video tokenization, discrete tokenizers for autore- gressive generation, and unified representations for multi-modal models promising direction for future work. We hope our findings inspire a rethinking of tokenizer design in generative modeling.
Acknowledgements. This work was done while Bowei Chen and Tianyuan Zhang interning at Adobe. We would like to thank Hailin Jin and Jingwan (Cynthia) Lu for their valuable support throughout the internship.
Reproducibility Statement. We provide the implementation details and hyperparameters in Sec. of the Appendix, which are sufficient reproduce the results of our method.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Mikotaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv: 01 401, 2018.
Daniel Bolya, Po- Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi. Jathushan Rajasegaran, Hanoona Rasheed, al. Perception encoder: The best visual embeddings are not at the output of the network. arXiv preprint arXiv:2504. 10044,1311,, 2025.
Hao Chen, Yujin Han, Fangyi Chen, Xiang Li, Yidong Wang, Jindong Wang. Ze Wang, Zicheng Liu, Difan Zou, and Bhiksha Raj. Masked autoencoders are effective tokenizers for diffusion models. In Forty-second International Conference Machine Learning. 2025a.
Junyu Chen, Dongyun Zou, Wenkun He, Junsong Chen, Enze Xie, Song Han, and Han Cai. De- ac 1.5: Accelerating diffusion model convergence with structured latent space. arXiv preprint wwXiv225080044.3 2025b.
Zisheng Chen, Chunwei Wang, Xiuwei Chen, Hongbin Xu, Runhui Huang, Jun Zhou, Jianhua Han, Hang Xu, and Xiaodan Liang. Semhitok: unified image tokenizer via semantic-guided hierar- chical codebook for multimodal understanding and generation. arXiv preprint arXiv:2503.0676: 2025c.
Xiangxiang Chu, Renda Li, and Yong Wang. Usp: Unified self-supervised pretraining for image generation and understanding. arXiv preprint arXiv:2503.06/33 2025.
Chaorui Deng, Deyao Zhu, Kunchang Li, Shi Guang, and Haoqi Fan. Causal diffusion transformers for generative modeling. arXiv preprint arXiv:2412.12095 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is a strong image synthesizer, 2023a.
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Mdtv2: Masked diffusion trans- former a strong image synthesizer. arXiv preprint arXiv:22303. 14389, 2023b.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances Neural Information Processing Systems, 36: 52132-52152, 2023.
Jiaming Han, Hao Chen, Yang Zhao, Hanyu Wang, Qi Zhao, Ziyan Yang. Hao He, Xiangyu Yue, and Lu Jiang. Vision as a dialect: Unifying visual understanding and generation via text-aligned representations. arXiv preprint arXiv:2506. 18898, 2025.
Philippe Cunsen-EEstruch David Yan, Ching Yao Chung, Orr Zohar, Jialiang Wang, Tingbo Hou, Tao Xu, Sriram Vishwanath, Peter Vajda, and Xinlei Chen. Learnings from scaling visual tok- enizers for reconstruction and generation. arXiv preprint arXiv:2501.09755, 2025.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.087/8 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.079887 2024.

---

Published as a conference paper at ICLR 2026
Yang Jiao, Haibo Qiu, Zequn Jie, Shaoxiang Chen, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Unitoken: Harmonizing multimodal understanding and generation through unified visual encod- ing. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 3600-3610, 2025.
Zhang Kai, Wang Peng, Bi Sai, Zhang Jianming, and Xiong Yuanjun. Knapformer. // wiithu................. com 2eeeaai446KKppppp /ttp////npFrrrrrrre 2025a.
https:
Zhang Kai, Wang Peng, Bi Sai, Zhang Jianming, and Xiong Yuanjun. minfm. https://githtn com/ Kai 18a1-46/minFM) 2025b.
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyz- ing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24174-24184 24184, 2024.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for eext-to-mmage generation. 2023.
Theodoros Kouzelis, loannis Kakogeorgiou, Spyros Gidaris, and Nikos Komodakis. EQ-VAE: Equivariance regularized latent space for improved generative image modeling. In Forty-second International Conference on Machine Learning, 2025. URL /ttp:::::///eenneeeee...ee
Sorum?id=0WhW5YY
Black Forest Labs. Flux.
forest- labs/flus 2024.
Junho Lee, Jeongwoo Shin, Hyungwook Choi, and Joonseok Lee. Latent diffusion models with masked autoencoders. arXiv preprint arXiv:2507.0998/ 2025.
Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng- Repa-e: Unlocking vae for end-to-end tuming of latent diffusion transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 18262-18272, 2025.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37: 56424-56445 2024.
Haokun Lin, Teng Wang, Yixiao Ge, Yuying Ge, Zhichao Lu, Ying Wei, Qingfu Zhang, Zhenan Sun, and Ying Shan. Toklip: Marry visual tokens to clip for multimodal comprehension and generation. arXiv preprint arX1v:2505.05422. 2025.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. arXiv preprint arXiv:2404.0 2024.
Jiasen Lu, Liangchen Song, Mingze Xu, Byeongjoo Ahn, Yanjun Wang. Chen Chen, Afshin Dehghan, and Yinfei Yang. Atoken: A unified tokenizer for vision, 2025. URL. https:
7/arxiv. wwwxii..rrg/ass...eo www.oorr/abbaaaae 14476
Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiao- juan Qi. Unitok: A unified tokenizer for visual generation and understanding. arXiv preprint
arXiv:2502.20321. 2025.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Sain- ing Xie. Sit: Exploring flow and diffusion- based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pp. 23-40. Springer. 2024.
Byeon Minwoo, Park Beomhee, Kim Haccheon, Lee Sungjun, Back Woonhyuk, and Kim Sae- hoon. Coyo-700m: Image-text pair dataset. https: htt///tttuuu com/kakacbrain)
coyo-dataset. 2022.
12

---

Published as a conference paper at ICLR 2026
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Russell Howes, Yao Huang. Hu Xu, Vasu Sharma, Shang- Shang-Wen Li, Wojeiech Galuba. Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEF/CVF international conference on computer vision, pp. 1195542000 2023.
Liao Qu, Huichao Zhang. Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Ze- huan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp-
2545-2555 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PmLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural formation processing systems, 35:25278-285 -22994, 2022.
Gianluigi Silvestri and Luca Ambrogioni. Covae: Consistency training of variational autoencoders. arXiv preprint 25X7:22507.0103, 2025.
Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang. Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. DINOv3, 2025. URL htLLhttps://// /ttp:ss///axvv..oo
2bs/2508..m 10104
Ivan Skorokhodov, Sharath Girish, Benran Hu, Willi Menapace, Yanyu Li, Rameen Abdal, Sergey Tulyakov, and Aliaksandr Siarohin. Improving the diffusability of autoencoders. arXiv preprint 248.. 2025.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
Wei Song, Yuran Wang, Zijia Song, Yadong Li, Haoze Sun, Weipeng Chen, Zenan Zhou, Jian- hua Xu, Jiaqi Wang, and Kaicheng Yu. Dualtoken: Towards unifying visual understanding and generation with dual visual vocabularies. arXiv preprint arXiv:2503. 14324, 2025.
Hao Tang, Chenwei Xie, Xiaoyi Bao, Tingyu Weng, Pandeng Li, Yun Zheng, and Liwei Wang- Unilip: Adapting clip for unified multimodal understanding, generation and editing. arXiv preprint arXiv:2507.23277 2025.
Keyu Tian, Yi Jiang. Zehuan Yuan, Bingyue Peng. and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 37:84839-84865 2024.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vasion-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502. 2025.
13

---

Published as a conference paper at ICLR 2026
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Chunwei Wang, Guansong Lu, Junwei Yang, Runhui Huang, Jianhua Han, Lu Hou, Wei Zhang, and Hang Xu. Illume: Illuminating your Ilms see, draw, and self-enhance. arXiv preprint arXiv:2412.06672 2024.
Xin Wen, Bingchen Zhao, Ismail Elezi, Jiankang Deng, and Xiaojuan Qi. principal components" enable new language of images. arXiv preprint arXiv:2503. 208608 2025.
Pingyu Wu, Kai Zhu, Yu Liu, Longxiang Tang, Jian Yang, Yansong Peng, Wei Zhai, Yang Cao, and Zheng-Jun Zha, Alitok: Towards sequence modeling alignment between tokenizer and autore- gressive model. arXiv preprint arXiv:2506.05289 2025.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to- image synthesis. arXiv preprint arXiv:2306. 09344, 2023.
Yecheng Wu, Zhuoyang Zhang. Junyu Chen, Haotian Tang. Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, al. Vila-u: unified foundation model integrating visual understanding and generation. arXiv preprint ar wxvv:2409.044/2 2024.
Rongchang Xie, Chen Du, Ping Song, and Chang Liu. Muse-vl: Modeling unified vim through semantic discrete encoding. arXiv preprint arXiv:2411.17760 2024.
Tianwei Xiong, Jun Hao Liew, Zilong Huang, Jiashi Feng, and Xihui Liu. Gigatok: Scaling visual tokenizers billion parameters for autoregressive image generation. arXiv preprint arXiv:2504.08736, 2025.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li. Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: learning and evaluating human preferences for text-to-image generation. In Proceedings of the 37th International Conference Neural Information Processing Systems, pp- 15903-15923 2023.
Jiawei Yang, Tianhong Li, Lijie Fan, Yonglong Tian, and Yue Wang. Latent denoising makes good visual tokenizers. arXiv preprint vrXiv:2507.1582 2025.
Jingfeng Yao, Cheng Wang, Wenyu Liu, and Xinggang Wang. Fasterdit: Towards faster diffusion transformers training without architecture modification. Advances in Neural Information Pro- cessing Systems, 17756166-5618, 2024.
Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction generation: Taming optimization dilemma in latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2(3):5, 2022.
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXiv:2310.0573: 2023.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems, 37:128940-128966 2024.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong. Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is casier than you think. International Conference on Learning Representations, 2025.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, 2023.
14

---

Published as a conference paper at ICLR 2026
Kai Zhang, Fujun Luan, Sai Bi, and Jianming Zhang. Ep-cfg: guidance. arXiv preprint arXiv:2412.09966 2024.
classifier-frs free
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018
Yue Zhao, Fuzhao Xue, Scott Reed, Linxi Fan, Yuke Zhu, Jan Kautz, Zhiding Yu, Philipp Krähenbühl, and De-An Huang. Qlip: Text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation. arXiv preprint arXiv:2502.05178 2025.
Anlin Zheng, Xin Wen, Xuanyang Zhang, Chuofan Ma, Tiancai Wang. Gang Yu, Xiangyu Zhang, and Xiaojuan Qi. Vision foundation models effective visual tokenizers for autoregressive image generation. arXiv preprint NXiv:2257700441,, 2025a.
Boyang Zheng, Nanye Ma, Shengbang Tong. and Saining Xie. Diffusion transformers with repre- sentation autoencoders. arXiv preprint arXiv:2510.11690, 2025b.
Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar, Fast training of diffusion models with masked transformers. arXiv preprint arXiv:2306.09R0 09305, 2023.
15

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS
Tokenizer. For ImageNet, we set the downsampling ratio to f- 16, the latent channel dimension to d = =32. We use the pretrained DINOv2 checkpoint vit_ pallaaeeppacch/ 1.44inoo22.vdd4444.4 parameters) (Oquab et al., ) as encoder. The adapter is implemented as a two-layer MLP that projects the (024-dimensional encoder output into 32-dimensional latent space. The decoder is convolutional network with m.42M parameters, same as VA-VAE (Yao et al. 2025). The training images first resized so that the shorter edge is 256 pixels while preserving the aspect ratio, and then randomly cropped to 256x256. Since the DINOv2 encoder uses a patch size of 14, we first resize the input image to 224x222 resolution before feeding it into the encoder. This produces 16x 16 latent feature map, matching the downsampling ratio of f /= 16 for 256-resolution images. For training, we use a batch size of 64 and train on NVIDIA H100 GPUs. The learning rate is set to lc-4 for the first and third stages, and lc-5 for the second stage. Following (Yao et al. 2025), the generator loss in LGAN is rescaled to match the magnitude of Le, + T based on the ratio of their gradient norms at the last layer of the decoder, where wPp = 1.0. Similarly, we rescale Lop according to the gradient ratio at the last layer of the encoder. We set w/ = 0.5 and wap = 1, which are applied after loss rescaling. We enable LGAN after 5K training steps in the first stage, and keep it active throughout the second and third stages. We enable Exponential Moving Average (EMA) (Karras et al. 2024) during training and apply them in the second and third stages during inference. Training runs for epochs in the first stage (~ steps, ~0 hours), 11 epochs in the second stage (~~200 steps, 24 hours), and 5 epochs in the third stage (~100K steps, ~66 hours).
(~304M
For LAION dataset, we use the pretrained DINOv2 checkpoint vit large prrrge-paccl4..eee 2dnovv2 (~304M parameters) (Schuhmann al., 2022) encoder. The tokenizer is trained with batch size of 256 and a learning rate of le-4 for all stages, using 32 NVIDIA H100 GPUs. We set Wap to 3. All other settings follow those of the ImageNet experiments. Training images are resized such that the shortest edge is 256 pixels while preserving the aspect ratio, without restricting them to square shapes. Similar before. the input images are resized to multiples of 14 so that the encoder output aligns with the downsampling ratio of f 16. We train for 300K steps in the first stage (~30 hours), 100K steps in the second stage (~20 hours), and 50K steps in the last stage (~55 hours).
Generative Models. For ImageNet experiments, we use LightningDiT (aoo et al. 2025) as the generative model (~673M parameters) and adopt the same hyperparameters as in its official im- plementation. We set the batch size to 1024, the learning rate to 2e-4, and the transformer patch size to 1. Unless otherwise specified, all experiments are trained for 80K steps (64 epochs), which takes approximately 12 hours 8 NVIDIA H100 GPUs We do not enable QKNorm for models trained with fewer than 200K steps. following the official implementation. When training models for more than 200K steps, we enable QKNorm from the start of the training to stabilize optimization and prevent loss spikes. We use the Euler sampler with 30 sampling steps unless otherwise spec- ified. Following LightningDiT, we apply CFG only the first three latent channels for fair and consistent comparison, unless otherwise specified.
For LAION experiments, we employ a diffusion transformer (2B parameters) following the FLUX implementation (Labs) 2024 Zhang et al. 2024| Kai et al. 2025a b). We use a learning rate of lc-4 with weight decay and a transformer patch size of I. Training begins on 256-resolution images with varying aspect ratios for 200K steps using a batch size of 2048, which requires roughly 140 hours on 32 H100 GPUs We then continue on 512-resolution images with varying aspect ratios for an additional 90K steps using a batch size of 512, requiring about 135 hours on 32 H100 GPUs. For inference, we apply EMA checkpoints and use a DDIM sampler (Song et al., 2020) with 50 steps, setting the classifier-free guidance (CFG) scale to 5.
Datasets. For our text-to-image training dataset LAION-2B, we apply a series of pre-filters to remove low-quality or undesired samples: exclude images that NSFW, watermarked, low- aesthetic, invalid, or from blocked domains, resulting in a subset of about 616M images.
Linear Probing. We follow the standard ImageNet protocol widely used to assess the semantic quality of latent representations. Specifically, we freeze both the VAE encoder and the adapter, and train a single linear classifier on top of the adapter's output features to predict ImageNet- -tt1KK 1K classes. We use linear probing because it directly measures how much semantic information is linearly accessible in the latent space, making it widely adopted proxy for semantic quality. The
16

---

Published as a conference paper at ICLR 2026
classifier is trained for epochs using Adam optimizer with learning rate of 0.001 and a batch size of 256 on the ImageNet-lk IK training set, and evaluated on the ImageNet validation set.
Ablation Study. All methods, except for the Full Model, are trained without the third stage (i.e., decoder refinement). For the Larger Decoder variant, we replace the CNN decoder with a larger decoder composed of ViT architecture followed by a CNN module (totaling ~35 M parameters), adapted from (Xiong et al., 2025). For the variant LoRA Fine-Tuning, we apply LoRA to both the attention and MLP layers, using rank of 16, an alpha of 32, and dropout of 0.1. All variants (except for LoRA Fine-Tuning) are trained with the same hyperparameters and training iterations for fair comparison: 6 epochs in the first stage and 11 epochs in the second stage. The Full Model is additionally trained with a third stage of 6 epochs. These settings follow the default configuration described in the main paper.
Comparison with Other Pretrained Encoders. All methods are evaluated without training Stage larrsggliip2-arge-aaahhlll
We use lait_ 2aggee_pccch6_22..mee m_222.maee" and encoders for the MAE and SigLIP 2 variants, respectively. For the SigLIP 2 variant, we set Wap 2. as this yields better results. Apart from this adjustment, all methods are trained with identical hyperparameters and iterations, following the default configuration outlined in the main paper.
as the pretrained
Comparison with Other Tokenizers. Our method with 32 channels is trained following the default configuration described in the main paper. For the 64-channel version, train for 6 epochs (~ 120K steps) in the first stage, 17 epochs (~340K steps) in the second stage, and 11 epochs (~~220K steps) in the third stage. For the Vanilla VAE and VAE with CNN encoder (~28MM parameters), the pretrained checkpoints from (ao et al. 2025) for both the 32- and 64-channel experiments. All pretrained checkpoints were trained with batch size of 256 for 50 epochs, except the 32-channel VA-VAE model, which was trained for 125 epochs. For VA-VAÆ with a ViT encoder, we adopt the same architecture as our method but without initializing from DINOv2 weights, and use the learning rate of Ic-4. The 32-channel version is trained for 22 epochs, whereas the 64-channel version is trained for 34 epochs, matching the number of epochs used in our model training.
Scale-Up Experiments on Text-to-Iman Image Generation. To compute rFID, we use 200K im- ages randomly sampled from the COY0-700M (Minwoo et al., 2022) dataset. The gFID and KID (Binkowski et al.] (0118) on COCO Prompr 6K are computed using 6K generated images and 6K corresponding real images from the sampled captions.
For completeness, we provide the reference for the metrics we used for evaluation: FID (Heusel al.] (017). KID (Binkowski al.] (018), HPSv2 (Wu al.] 2023), PickScore (Kirstain al., 2023). ImageReward (Xu et al. 2023), Aesthetic Scores (Schuhmann et al. 2022). CLIP
Scores (Hessel et al. 2021), VQA Scores (Lin et al. 2024). LPIPS (Zhang et al. 2018).
B TRAINING AND INFERENCE COST
We show cost comparison of training in Tab, 6 Our main observations are as follows:
(1) Total GPU memory. The vanilla VAE consumes the least memory because does not load any pretrained encoder. Our Stage and Stage 3 use less memory than VA-VAE since our encoder is frozen during these stages, whereas VAE loads CNN encoder (trainable) + DINOv2 encoder (frozen). Our Stage 2 consumes the most memory because fine-tunes the DINOv2 encoder.
(2) Time per training step. The pattern mirrors memory usage for the same reason: VA-VAE incurs higher cost than our Stage and 3, and our Stage 2 incurs the highest cost when the encoder is trainable.
(3) Total A100 GPU hours. The cumulative GPU training hours of our method (107.2 377.4 91.55 576.15) are lower than those of both Vanilla VAE and VA-VAE, demonstrating the overall training efficiency of our pipeline.
We also show cost comparison of inference in Tab. and Tab. follows:
Our main observations are as
(1) Peak GPU Memory, At lower resolutions and smaller batch sizes, our encoder consumes more peak GPU memory than VA-VAE (ours: 1.372 GB, VA-VAE: 0.838 GB at 256 resolution with
17

---

Published as a conference paper at ICLR 2026
Table 6: Training Cost Comparison. All measurements are obtained an imput resolution of x on 8 NVIDIA A100 GPUs (batch size per GPU). Values marked with denote GPU- hour estimates computed from the training epochs reported in the official implementation of (Yao et al. 2025).
Trainable Paroms
GPU Memory
Time Training Step
Vanilla 28.41
72.60
14.11 M
148.2 GiB
0.360
8000"
VA-VAE
72.63 M
319.0 M
203.0 GB
0,534
2966'
Our Siago
14.25 M
319.0
157.5 GB
107.2 0.402
Stage
319.
238.2 GRR
Our Stage
44.18 M
319.1 M
159.0 GB
0.412
91.55
Table 7: Inference Memory Consumption Comparison. We evaluate the peak GPU memory consumption under different image resolutions and batch sizes.
Imaaee Resolation
Encoder #Params
Encoder Peak GPU
batch size=1
size
Decoder #Params
Decoder Peak GPL Memory (GB)
size=l
size
VA-VAE
Ours
256 256
256
28.41 M 304.4
0.301
1,363
0.838
1.572
1.385
1.543
41.42M
0.310
0.713
1,250
Durs
512
1,284
1
1.674
0.713
VA-VAE
Ours
1024
1024 1024
1024
28.41 M
304.4 M
2998
1.450
2.121
11.45
22.78
3.015
41.42 M
2.324
8,768
17.36
Table 8: Inference Compute Cost Comparison. We measure the GFLOPS and Latency under different image resolutions, using single NVIDIA A100 GPU with a batch size of 1.
Encoder
Tokenizer
Image Resolution
#Params
GFLOPS
Latency
@Params
GFLOPS
Latency
VA-VAE
256 256
28.41 M
69.21
6.548
41.42
126.5
13.91
Ours
256 256
304.4 M
77.85
13.42 ms
VAE
41.42 M
509.3
42.31 ms
Ours
512 512
304.4 M
310.4
18.92 ms
1024
41.42 M
2089
173.9 ms
Ours
1024 1024
304.4 M
1241
121.2 ms
batch size of 4). However, at higher resolutions and larger batch sizes, our encoder uses substantially less memory (ours: 3.015 GB, VA-VA VAE 22.78 GB at 1024 resolution with a batch size 8).
(2) Latency and GFLOPS. Overall, tokenizer has higher encoding latency and GFLOPS than VA- VAE. The only exception at 512 resolution, where our tokenizer achieves lower encoding latency.
C MORE EXPERIMENTS
C.1 OTHER PRETRAINED ENCODERS
Tab. reports additional reconstruction metrics comparing different pretrained encoders. The one using DINOv2 achieves the best balance, delivering superior generation quality while maintaining competitive reconstruction performance.
C.2 RECONSTRUCTION
Fig. 9 shows qualitative comparison of reconstruction quality the ImageNet 256x256 dataset. The variant Ours w/o Stage 2 + (fourth column) fails to reconstruct the input accurately, while the other methods show similar reconstruction quality.
Tab 10 reports additional reconstruction metrics for different tokenizers. While Owrs (the version presented in the main paper) outperforms all baselines in generative performance, it remains quanti- tatively weaker in reconstruction quality. To address this, we report a variant of our method trained with larger learning rate (increased from le-5 to le-4) and smaller semantic preservation weight
18

---

Published as a conference paper at ICLR 2026
Table Comparison with Other Pretrained Encoders. Evaluated on ImageNet 256x256 at 80K training steps with 30 sampling steps, using the CFG scale that yields the lowest gFID. Decoder refinements (stage 3) are not applied. While the model with MAE yields the strongest reconstruction performance, it performs the worst in generation quality. The model with DINOv2 achieves the best overall generation quality, with a slight drop reconstruction quality compared to MAE.
Configuration
rFID.
PSNR
LPIPS!
2FID
1ST
Prect
Recall"
MAE
0.29
26.12
0.113
3.12
216.5
0.834
0.543
SigL.P 2
0.35
25.29
0.129
2.22
246.1
0.816
0.576
DINOv2
0.36
25.62
0.121
2.19
248.6
0.811
0.591
Table 10: Comparison of Other Tokenizers with Different Configurations. Evaluated on Ima- geNet 256x256 at 80K training steps with 30 sampling steps. The checkpoints for both the Vanilla VAE and VA-VAE with CNN encoders are taken from the official VA-VAE repository. VA-VAE! denotes the VA-VAR model trained, using VIT encoder that matches architecture but ini- tialized from scratch. Ours refers to the version of our method presented in the main paper. Ours* indicates variant trained with larger learning rate and smaller semantic preservation weight, which achieves reconstruction quality comparable to VA-VAE but with slightly reduced generation performance.
Tokenizer
Enc. Arch.
FFID
PSNR
LPIPS
L. Acc.
gFID w/ CFG.
f16d32 (downsampling factor 16, latent dimension 32)
Vanilla VAE
CNN
0.26
27.14
0.097
6.04%
3.31
VA-VAE
0.28
26.31
0.104
22.96%
3.13
VA-VAE
VIT
0.37
25.66
0.130
33.57%
3.16
Ours
VIT
0.26
25.83
0.117
35.09%
2.17
f16d64 (downsampling factor 16, latent dimension 64)
Vanilla VAE
CNN
0.17
29.38
0.061
5.09%
4.03
VA-VAE
CNN
0.14
29.13
0.062
19.72%
3.20
VA-VAE VAB
VIT
0.18
29.12
0.075
43.53%
3.19
Ours
VIT
0.17
27.41
0.089
46.99%
2.34
Ours
VIT
0.14
28.91
0.070
45.22%
2.50
(reduced from 1.0 to 0.5) in stage 2. This hyperparameter setting pushes the tokenizer to learn perceptual details more aggressively. For this variant, we train the first stage for 6 epochs, second stage for 11 epochs, the final stage for 10 epochs. The resulting variant achieves competitive re- construction performance (on par with VA- VA-VAE using ViT encoder) while still surpassing VA-V VA-VAE in generation. This demonstrates that method can improve reconstruction with only minor trade-off in generative quality through simple hyperparameter adjustments. Other strategies for im- provement include extending stage 2 stage 3 with additional training iterations, using larger batch size. A more aggressive approach to replace the decoder with a stronger architecture dur- ing stage 3 and train the new decoder. While this requires additional training resources, it offers flexibility without compromising the learned semantic space.
C.3 CONVERGENCE SPEED
Figs. Fig. and Fig. 12 present additional comparisons between our method and VA- VAE on the ImageNet 256x 256 dataset. Our method converges faster, indicating that aligning and preserving semantics the pretrained encoder is more effective than semantic regularization.
Fig. 13 Fig. 14 Fig 15 and Fig. [o]come the convergence speed of our method with FLUX VAE.
Our method converges significantly faster, demonstrating the advantage of the learned semantically rich latent space.
C.4 MORE QUANTITATIVE RESULTS
Tab. [ reports quantitative results on two additional prompt sets. Tab. and Tab. 15
present com-
parisons against FLUX VAE and VA-VAE on GenEval Ghosh al. 2023 Tab. 12 and Tab.
19

---

Published as a conference paper at ICLR 2026
Table 11: Quantitative Compartson on Text-to-Image (T2I) Generation with FLUX VAE. We report metrics on two additional prompt sets for T2I models trained with our VAE and FLUX VAE, each for 100K steps, evaluated at x resolution. The rFID is computed using 200K randomly sampled images from the COYO-700M dataset (Minwoo et al. 2022
Tokenizer rFID
HPSv2
PickScore
ImageReward
Aesthetic Scores
CLIP Scores
VQA Scores
Parti Prompi
Yi of al., 2022
(with CFG)
FLUX VAE
0.102
0.239
0.391
0.235
5.292
31.49
0.705
Ours
0.443
0.246
0.609
0.594
5,389
32.56
0.782
Prompr WW et
2023
(with CPG)
FLUX VAE
0.102
0.224
0.373
0.090
5,478
31.59
0.737
Ours
0.443
0.231
0.626
0.366
5.604
33.12
0.792
Quantitative Comparison on Text-to-lmage (T2I) Generation with FLUX VAE (Without CFG). We report metries on diverse prompt sets for 2B-parameter T2I models trained with our tokenizer and the FLUX VAE. Each T2I model is trained for 100K steps and evaluated at 256 x 256 resolution. Our tokenizer trained on LAION dataset. The rFID computed using 200K randomly sampled images from the COYO-700M dataset (Minwoo et al. 2022).
Table 12:
Tokenizer
rFID
gFID
KID
HPSv2
PickScore
ImageReward
Aesthetic Scores
CLIP Scores
Scores Scores
Coco Prompt 6K Lin al. 2014
(withou CFG)
FLUX VAE
0.102
48.76
0.025
0.183
0.345
-1.127
4.205
27.05
0.534
Ours
0.443
33.46
0.014
0.210
0,655
-0,581
4.963
28.87
0.658
Porti Prompt (Yu et al. 2022 (without CFG)
FLUX VAE
0.102
0.186
0.378
.1.17
4.003
26.85
0.538
Ours
0.443
0.206
0.622
-0.554
4.751
28.59
0.618
HPSv2 Promps Wir al. 2023 (without CFG)
FLUX VAE
0.102
0.170
0,372
-1.206
4.142
26.59
0.567
Ours
0.443
0.192
0.628
-0.764
4.955
28.20
0.645
Table 13: GenEval Quanlitative Comparison on Text-to-Image (T21) Generation with FLUX VAE (with and without CFG). We report metrics on GenEval (Ghosh al. 2023 for 2B-parameter T2I models trained with our tokenizer and FLUX VAE. Each T2I model is trained for 100K steps and evaluated 256x256 resolution. Our tokenizer trained on LAION dataset. The rFID is computed using 200K randomly sampled images from the COYO-700M dataset MMinwoo et al. 2022).
Tokenizer
rFID
Overall
Single object
Two object
Counting
Colors
Position
Color attribution
wäth CFG
FLUX VAE
0.102
0.476
0.978
0.477
0.346
0.813
0.107
0.135
Ours
0.443
0.556
0.984
0.702
0.446
0.824
0.120
0.257
without
FLUX VAB
0.102
0.230
0.746
0.101
0.090
0.406
0.020
0.015
Ours
0.443
0.329
0.850
0.303
0.234
0.508
0.037
0.040
further compare the main metrics under settings with and without CFG. Across all configurations, our tokenizer consistently outperforms both FLUX VAE and VA-VAE in generation quality, regard- less of whether CFG is applied.
C.5
MORE QUALITATIVE RESULTS
Fig. 17 and Fig.
18 show the comparison with FLUX-VAB (withou CFG) and VA-VALE VAE (with CFG).
Fig. 19 shows our sampled results of class-conditioned image generation ImageNet 256x256 dataset, trained with 800 epochs.
Fig. 20 shows our sampled results of text-to-image generation at resolution 256x256, trained with 200K steps.
Fig. and Fig. 22 show sampled results of text-to-image generation resolution 512x512. Fig. 23 and Fig. further show results at different aspect ratios, including portrait mode and landscape mode. All these results generated using the model trained with 290K steps.
20

---

Published as a conference paper at ICLR 2026
Table 14: Quantitative Comparison on Text-to-Image (T21) Generation with VA-VAE (with and without CFG). We report metrics on diverse prompt sets for 1B- 1B-parameter T2I models trained with our tokenizer and the VAE, Each T2I model is trained for 50K steps and evaluated at x 256 resolution. Both our tokenizer and VA-VAE are trained on ImageNet. The rFID is com- puted using 200K randomly sampled images from the COYO-700M dataset (Minwoo et al. 2022).
Tokenizer
rFID
gFID
KID
HPSv2
PickScore
Rewae
Aesthttic Scores
CLIP Scores
VQA Scores
Coco Prompt 6K (Lin et al. 2014]
(winh CFG)
VA VAE
0.241
34.13
0.016
0.221
0.426
0.129
5.088
31.50
0,740
Ours
0.191
31.19
0.015
0.230
0,574
0.088
5.215
31.68
0.781
Parti Prompt Yu et ml.
2022
(wüt CFG)
VAE
0.241
0.220
0.444
0.146
4.893
31.40
0.677
Ours
0.191
0.228
0.556
0.062
5,015
31.73
0.706
MPS Prompt [Wa et
2023
(with CFG)
VA-VAE
0.241
0.201
0.435
-0.287
5.042
31.63
0.708
Ours
0.191
0,210
0,565
40.11
5.120
32.05
0.737
Coco P Prompi 6K Lin ol.
2014
(without CFG)
VA VAE
0.241
42.35
0.019
0.179
0.448
-1.292
4.352
26.89
0.497
Ours
0.191
39.86
0.017
0.187
0.552
-1.121
4.493
27.53
0,548
Prompt Di et
2022
(without CFG)
VA-VAE
0.241
0.179
0.457
-1.396
4.163
26,23
0.493
Ours
0.191
0.186
0.543
-1.213
4.320
26.99
0.524
HPSr2 Prompr Wir et al.,
2023
(without CFG)
VA-VAE
0.241
0,162
0.455
-1.378
4.256
26.05
0.521
Ours
0.191
0,170
0.545
-1.251
4,451
26.48
0.551
Table 15: GenEval Quanlitative Comparison on Text-to-Image (T2I) Generation with VA-VAE (with and without CFG). We report metrics on GenEval (Ghosh et al. 2023) for 1B-parameter T21 models trained with our tokenizer and VA-VAF. Each T21 model is trained for 50K steps and evaluated at 256x 256 resolution. Both our tokenizer and VA-VAE are trained ImageNet. The rFID is computed using 200K randomly sampled images from the COYO-700M dataset (Minwoo)
al. 2022).
Tokenizer
rFID
Overall
Single object
Two object
Counting
Colors
Position
Color attribution
with CPG
VA-VAR
0.411
0,944
0.338
0.275
0.782
0.050
0.078
Ours
0.191
0.454
0.978
0.434
0.338
0.795
0.073
0.110
without CFG
VA-
0.241
0.194
0.653
0.071
0.088
0.335
0.010
0.005
Ours
0.191
0.243
0.734
0.136
0.125
0.418
0.033
0.013
Table 16: Quantitative Analysis of Latent Space. We report metrics for latent space of three tokenizers (Vanilla VAE, VA-VAE, and Ours), using the output space of the pretrained DINOv2 encoder reference. CKNNA Huh al. (2024) measures the similarity between each tokenizer's latent space and the DINOv2 output. For the other metrics, we additionally report the percentage indicating how close each tokenizer's results are to those of DINOv2. All metries are computed using 10K samples from the ImageNet validation set.
Tokenbzer
CKNNA
Spatial Variance
Total Variation
Density
Gini Coefficient
Normalized Entropy
gFlD
Vanilla VAE
0.023
11.41
(4432.9%)
957.6 (496.3%)
0,310 (+24.0%
0.173 (+24.4%%
0.994 (-0.2%)
3.31
13.74
(4541.7%)
(+65.1%)
0.215 (-14.0%)
(-12.2%)
0.997 (+0.1%)
3.13
Ours
1.205
(-43.7%)
302.3 (-38.0%)
0.295 (+18.0%)
0.160 (+15.19))
0.994 (-0.2%)
2.17
DINOv2
1.000
2,141
487.9
0.250
0.139
0.996
C.6 ANALYSIS OF THE SEMANTICS OF THE LATENT SPACE
Fig. 7 shows PCA visualizations of the latent space. The figure compares the latent spaces produced by three tokenizers (Vanilla VAE, VA- VAE, and Ours) with the output of the pretrained DINOv2 encoder semantic reference. Vanilla VAE tends to produce latents that are either overly smooth or overly sharp, while VA-VAE exhibits oversmoothing. In contrast, our latent space most closely resembles the structure of DINOv2 features, reflecting its stronger semantic structure.
21

---

Published as a conference paper at ICLR 2026
(a) Input
(b) Vanilla VAE
(c) VA-VAE
(d) Ours
(c) DINOv2
Figure 7: PCA Visualization of Latent Space. Vanilla VAE can produce latents that are either over- smooth or overly sharp, and VA-VAE tends to generate consistently over-smooth representations. In contrast, our latent space most closely matches the characteristics of DINOv2 outputs, demonstrating that our tokenizer preserves richer semantics.
Besides, Tab. 16 shows the quantitative analysis of the latent space. We evaluate several distri- butional metries such CKNNA Huh al. (2024), Total Variation, and Gini Coefficient. While our latent space aligns more closely with the DINOv2 space than other tokenizers in metrics like CKNNA and Total Variation, VA-VAN appears closest in terms of metrics like Gini Coefficient. We note, however, that each of these metrics captures only one aspect of the latent distribution, and none of them individually correlates perfectly with semantic quality. This is why we relied primarily on linear probing to evaluate the semantic of the latent space.
C.7 FAILURE CASE
Fig. shows the failure case our method. Issues include inaccurate rendering clock numerals (e.g., the 12), incorrect object counts, incorrect long text generation, and difficulties in rendering fine details such as hands.
D LLM USAGE
In preparing this manuscript, we used large language models (LLMs) as general-purpose writing assistants for grammar corrections, rephrasing, and clarity/concision edits. All LLM-suggested edits were reviewed and verified by the authors, who take full responsibility for the final manuscript.
22

---

Published as a conference paper at ICLR 2026
KEEP OFF THE THE GRRASS
clack brick bailding wall
the words 'KEEP OFF THE GRASS' brick
grandmotherr raaling book to her grandson gandddughter
apples
Figure 8: Failure Case of Our Method on Text-to-lmage Generation at 512x512 Resolution. The input text prompts are shown below the images. Results are obtained from generative models trained for 290K steps. Common issues include inaccurate rendering of clock numerals (e.g., the 12), errors in object counting, inconsistencies generating longer text, and difficulties in producing fine details such as hands.
23

---

Published as a conference paper at ICLR 2026
Input
Vanilla
VA-VAE
w/o Stage +
Ours
POTHERT
Input
Vanilla
VA-VAE
Ours w/o Stage 2 + 3
Ours
Figure 9: Qualitative Comparison of Reconstruction Quality on ImageNet 256x Validation Set. The variant Ours w/o Stage + 3 (fourth column) fails to accurately reconstruct the input, as the pretrained visual encoder does not capture sufficient perceptual details in the latent space. All the other methods achieve comparable reconstruction quality qualitatively.
24

---

Published as a conference paper at ICLR 2026
20K steps
40K steps
60K steps
80K steps
100K steps
VA-VAE
Ours
Class Name: "tennis ball"
VA-VAE
Ours
Class Name: "espresso"
VA-VAE
Ours
Class Name: "toilet tissue, toilet paper, bathroom tissue" "
20K steps
40K steps
60K steps
80K steps
100K steps
Figure 10: Qualitative Comparison of Convergence Speed ImageNet 256 256. We compare with VA-VAE Results are reported with the best CFG scale, using EMA for sampling except at 20K and 40K steps.
25

---

Published as a conference paper at ICLR 2026
20K steps
40K steps
60K steps
80K steps
100K steps
VA-VAT
MAC
Oun
A01%) A.
Class Name: "projector"
VA-VAF
Ours
Class Name: "dial telephone, dial phone" T
VA-YAR
Ours
Class Name: "border collie"
20K steps
40K steps
60K steps
80K steps
100K steps
Figure 11: Qualitative Comparison of Convergence Speed on ImageNet 256x We compare with VA-VAR Results are reported with the best CFG scale, using EMA for sampling except at 20K and 40K steps.
26

---

Published as a conference paper at ICLR 2026
20K steps
40K steps
60K steps
80K steps
100K steps
VA-VAE
Ours
Class Name: "toucan"
VA-VAR
Ours
Class Name: "ruffed grouse. partridge. Bonasa umbellus"
VA-VAE
Ours
Class Name: "brambling, fringilla montifringilla" "
20K steps
40K steps
60K steps
80K steps
100K steps
Figure 12: Qualitative Comparison of Convergence Speed ImageNet 256x256 We compare with VA-VA VAE. Results are reported with the best CFG scale, using EMA for sampling except at 20K
and 40K steps.
27

---

Published as a conference paper at ICLR 2026
20K steps
40K steps
60K steps
80K steps
100K steps
FLUX YAE
Ours
Prompt: "two red balls on table"
FLUX VAF
Ours
Prompt: tree with leaves that look like purple balloons"
FLUX YAR
Ours
Prompt: "slices of mango on piece of toast"
20K steps
40K steps
60K steps
80K steps
100K steps
Figure 13: Qualitative Comparison of Convergence Speed on Text-to-Image Generation. We compare with FLUX VAE. Results are reported with CFG scale 5, using images generated 256x256 resolution.
28

---

Published as a conference paper at ICLR 2026
20K steps
40K steps
60K steps
80K steps
100K steps
FLUX VAE
Ours
Prompt: "a zebra"
FLUX VAF
Ours
Prompt: "The Statue of Liberty with the Manhattan skyline the background"
FLUX YAR
Ours
Prompt: "a thumbnail image of an ice cream cone
20K steps
40K steps
60K steps
80K steps
100K steps
Figure 14: Qualitative Comparison of Convergence Speed on Text-to-Image Generation. We compare with FLUX VAE. Results are reported with CFG scale 5, using images generated 256 resolution.
29

---

Published as a conference paper at ICLR 2026
20K steps
40K steps
60K steps
80K steps
100K steps
FLUX VAE
Ours
Prompt: "a teddy bear on a skateboard"
FLUX VAF
Ours
Prompt: summer tree without any leaves" "
FLUX YAR
OurS
Prompt: "a snail"
20K steps
40K steps
60K steps
80K steps
100K steps
Figure 15: Qualitative Comparison of Convergence Speed on Text-to-Image Generation. We compare with FLUX VAE. Results are reported with CFG scale 5, using images generated 256x256 resolution.
30

---

Published as a conference paper at ICLR 2026
20K steps
40K steps
60K steps
80K steps
100K steps
FLUU YAE
Ours
Prompt: "a squirrel"
FLUX VAE
Ours
Prompt: helicopter flies over the Arches National Park." "
FLUX YAR
Ours
Prompt:
"three red lego blocks"
20K steps
40K steps
60K steps
80K steps
100K steps
Figure 16: Qualitative Comparison of Convergence Speed on Text-to-Image Generation. We compare with FLUX VAE. Results are reported with CFG scale 5, using images generated 256x256 resolution.
31

---

Published as a conference paper at ICLR 2026
FLUX VAE
Ours
voloring page next to stream
giraile standing sand at z0o
puinting style of starry night
photo
FLUX VAE
Oars
a photo of two snowboards
figer
woodes deck overlooking mounttin valley
yellow sticky mote
abstract painting of Empire State Building
FLUX VAT
Ours
an old dairy rack is rotting field
Face of an orange frog style
Golden Giate bridge en the surface Mars
the kifflel Tower winter
the Taj Mahal sunsise
Figure 17: Qualitative Comparison on Text-to-Image Generation with FLUX VAE (without CFG). Input text prompts shown below the images and results (256x256 resolution) gener- ated from 2B- 2B-parameter diffusion models trained for 100K steps, without using CFG. Our method (bottom row) produces images with better coherence and prompt alignment compared the one using FLUX VAE (top row).
32

---

Published as a conference paper at ICLR 2026
VA-VAE
Ours
high Sydney Opera sining next to Eiffel tower. under blue night sky of roiling exploding yellow
watermelon chair
bundle blue and yellow
a jumbo jes plane in night with city the background
green box drop moil into covered with grathi
Blowess vase
radiiiing
VA-VAE
Ours
photo of and hear
photo twn hackpacks
train crussing bridge over shallow
triangle with smiling face
wine botile with red ribbon wrapped around
VA-VAE
selephone
Sailhosts
slices mango toast
beach
tromed oil painting
Figure 18: Qualitative Comparison on Text-to-Image Generation with VA-VAE, Input text prompts are shown below the images and results (256x256 resolution) are generated from 1B- parameter diffusion models trained for 50K steps, using CFG scale of 5. Note that, both our tok- enizer and VA- VAE are trained on ImageNet. Our method (bottom row) produces images with better coherence and prompt alignment compared to the one using VA-VAE (top row).
33

---

Published as a conference paper at ICLR 2026
great while shack, white shark,
mun eater man eaiinging shark. Carcharooon carcharias
cock
estich. Siruthoo camelus
jay
lorkket
abcaa trshi garoage can,
wastehin, aeb-hin. dusthin, harrel, trsch bin
interpres
spaniel
ridgeback
macaque
M
AF
INPTURSR
backpack. back pack. knapsock, parksack, rucksack, haversack
container ship, containenship. container vessel
crash helmet
electic locamotive
hamper
microphone,
pickelhoube
ship
printer
refies camero
school bes
sxpping basket
stadio couch, day bed
Figure 19: Qualitative Results of Our Method on ImageNet Class- Conditional Generation. ImageNet class names are shown below the images and results are from diffusion models trained with 800 epochs. We set the CFG to and apply all latent channels.
34

---

Published as a conference paper at ICLR 2026
cogy living painting congi on wall shove a cruch and round cuffee table in front couch vase codfee table
capybara
carioon of heppy cur on the road
chimpanzes
ovegi weiring red bowiie purple party hut
crowa with raby
flig with dinosaur
kachina doil
painting of fox the style of stury night
photo of san trancisco's polden gate
A smiling sloth wearing leather jacket, comboy hut,
pottril of statee of
pickup
with ropped by vegetables
wearing stampunk glasses, leather jucket. delr photograh
green
bolding quatterstalf and big
book
5
ne anime illustmtion Sydney Opera House sitting next under sky roiling energy. exploding yellow stars, and meliating swirts of hiu
Jhumbrail image gingerbread mon
wooden post with pointed on top
a yellow
night
painting of punti strickes creatire, simultanennly corpsctike cemunisceat sperm fetus, whose are ochoed in the swirling lines of the blood-red sky
elderly woman
Snow moussain and
yin-yang
Figure 20: Qualitative Results of Our Method on Text-to-Image Generation at 256x 256 Reso- lution. The input text prompts are shown below the images. Results are obtained from generative models trained for 200K steps.
35

---

Published as a conference paper at ICLR 2026
M.
painting pokemon resembling phune booth wearing clothes walking on two legs cobblesione street in magical
a Lemon characier wearing sunglasses on the beach
flying through space cating pizza and cheese among candy planets in comic book style drawing
a plazt with orange flwwrss shaped like
stars
plushy tired antique ilsttttioon
SpongeHob Dragon style
glass
elophant standing dint
vase on table
HUG ME!
young boy wearing smiling
yellow wall with frumed West Highland white teniier holding painting
"Hug sign
spray point
Welcome
kangaroo sa orange hoodie and blue sunglasses stands on the grass front of Sydney Opera House holding
staffed panda bear sitting aext baddlaa
sailbout emoji with rainoow colored sail
teddy bear sisting wund
Fricads" siga
Figure 21: Qualitative Results of Our Method on Text-to-Image Generation 512x512 Reso- lution. The input text prompts shown below the images. Results are obtained from generative models trained for 290K steps.
36

---

Published as a conference paper at ICLR 2026
stares directly at the camera in an extreme closz-ap Instigram piettome
girl sott festures
cune unthoopnccppic knight wearing a cape and crows in pale blue
deg wearing busaness suit smoking cigur cinematic style
A fox wearing yellow dress
OPENAI
START SMILING
carrynng yellow container tilled with lemons
store that 'openai" writlen on it
Multiple baskets recently picked grapefruiis display
'START" wasten word 'SMILING'
with bike rider traveling down crowd people watching fireworks rupo
toy
by a park.
pormit of Pilachu with an amy minions, surrouneed dramatic lighning clecnicity
mountain and reflection in lake
plane flying low over the Great Wall
smiling man with wavy brown hair and trimmed beard
Figure 22: Qualitative Results of Our Method on Text-to-Image Generation at 512x512 Reso- lution. The input text prompts shown below the images. Results are obtained from generative models trained for 290K steps.
37

---

Published as a conference paper at ICLR 2026
corgi
the Kremlin abstract drewing Chest Wall
oil poining and building
START
ornate jondleecnedl key
the word 'START' written on a street
photo of son francisco's gollen gate bridge
surfice
landscape small cotlage
shiny with ciyysap painted and parked grass
suleaarin flosting post
holding shield in lighting stance. The wombut stands tront of the Are de Tnoouhh stmouded with the high
the Gresn Pyramid of Giza simsted in front of Moust Everest
steam locomorive speading through
un cogle
desert
illustation
sitting the Partheson under blue night sky millng energy. exploding yellow stars, and chromatic blue
painting of sport car in the style of Monet
sunser over the KCa
green train coming down the trocks
Figure 23: Qualitative Results of Our Method on Eex-to-Image Generation Various Aspect Ratios with the Shortest Edge Equal to 512. The input text prompts are shown below the images. Results are obtained from generative models trained for 290K steps. Rows and show 512 640 (3:4), row 3 and 4 show 512 x 768 (2:3), and row 5 shows 512 910 (9:16).
38

---

Published as a conference paper at ICLR 2026
KEEP OFF THE GRASS
photo golden gale
boy and a tiger
bridge
writlen un brick wall
postrait Egyptian god Anabis
people watching by citty
tan blue jeans and yellow shirt
a yellow & tohirt with a dog on
A wearing fommal clothes, wearing bolling raccuon holling gurbuge bug. Oil painting the style Rembrandi
the Liberty with owl
giri
puncupple surting wave
Figure 24: Qualitative Results of Our Method on Text-to-Iman Image Generation at Various Aspect Ratios with the Shortest Edge Equal to 512. The input text prompts are shown below the images. Results obtained from generative models trained for 290K steps. Rows I and show 640 512 (4:3), rows 3 and 4 show 768 x 512 (3:2), and row shows 910 x 512 (16:9).
39


Reviews:
[{"review_id": "QPwz6NKOwm", "reviewer": "Reviewer_XEQ8", "summary": "This paper introduces a new way of image tokenizer training for image generation. It targets the weak semantics of the latent space of traditional VAE tokenizers, which can be caused by the reconstruction-only training supervision. This method builds an image tokenizer based on pre-trained discriminative vision foundation models such as DINO. It introduces a progressive three-stage training method that trains an adaptor and a decoder to reconstruct the images, while maintaining the semantically rich latent space. Experiments on ImageNet and text-to-image generation prove the effectiveness. Extensive ablations justify the design of each training stage.", "questions": "- I didn't find detailed comparisons for the costs. Will using the pretrained vision encoder introduce increased memory consumption and encoding latency? \n\n- Could the authors provide GenEval score comparisons in Table 5? It will provide more comprehensive evaluations on alignment and better justify the core contribution of this paper. \n\n- Figure 4 right. The baseline VA-VAE fails to train in full training epochs in both setting due to numerical issues. Would it be possible to add another baseline for full-scale comparisons? This is optional as it might require prohibitive time during the rebuttal period.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper targets an important problem of semantic-rich image tokenizer traning. The solution shown is simple and effective. The idea is well presented and illustrated. \n\n- The extensive ablation and detailed reconstruction / semantic capacity tracking along the training process are very helpful and insightful. \n\n- The final quantivative evaluation results are strong.", "weaknesses": "- Not technically a weakness, but the final linear probing performance is only ~35%, dropping by a lot compared to the original DINO latent space. What could be the potential ways of further bridging the gap? \n\n- In the stage 2 training, applying the regularization to the output of $E_p$ instead of $A$ seems to be a more intuitive way and can better preserve the semantics. Could the authors provide some insights on why that does not work well?\n\n- In summary, my main concern is that, although the idea is motivated by leveraging a semantically rich encoder to obtain a more diffusable latent space, the substantial linear-probing performance drop and the observed trade-offs between reconstruction fidelity and semantic preservation make it unclear whether the improved results stem from a genuinely richer semantic space or merely from initializing the tokenizer with a well-pretrained encoder that can also be some other models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "039k1PwHy9", "reviewer": "Reviewer_svcV", "summary": "The authors propose a three stage framework to transform a pre-trained semantic encoder into an image tokenizer. In the first stage, they employ an adaptor that maps features from the frozen encoder into a lower-dimensional latent space (32 dimensions in their experiments). The adaptor and decoder are jointly trained using a reconstruction loss. In the second stage, the encoder is fine-tuned to improve reconstruction quality. However, since this process can degrade semantic capacity, an L2 regularization term is applied on the low-dimensional latent to keep the updated encoder close to its previous representation. Experiments conducted mainly on ImageNet 256 show a 0.46 improvement in FID when using the LightningDiT framework.", "questions": "What's the performance before CFG?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The presentation, though sometimes not self-contained, is overall clear.\n\nThe empirical performance, evaluated on ImageNet, shows 3.4% improvements on linear probing, on ImageNet 256.", "weaknesses": "1. It's not clear what the overall computational cost is for all stages, when compared to other methods.\n\n2. I am not sure how sensitive the dimensionality of the adaptor is. If we want to generalize to ImageNet 512, does it mean that we only need to double the latent?\n\n3. The use of an L2 loss to preserve semantic consistency may not be robust, especially when the latent space lacks proper normalization.\n\n4. While semantic preservation appears improved, the drop in linear probing accuracy during training raises concerns about the trade-off between reconstruction fidelity and semantic capacity.\n\n5. The decrease with rFID shows that the constraint on the adaptor and the stage-wise framework might impact the reconstruction accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FDM6twWynB", "reviewer": "Reviewer_GRL2", "summary": "- Generation models (e.g., Diffusion models, flow matching model) employ an autoencoder to reduce the dimension of the image dataset for computational efficiency and generation ability.\n- The authors claim that the latent space of the encoder should be semantically meaningful.\n- By introducing DINO-v2 initialized encoder and semantic preservation regularization, they achieve faster convergence while training and better fid with the same NFE while inference.", "questions": "- What is the author's thought about the “generative-friendly latent space”? According to the claim of the paper, it seems that preservation of the latent space of DINOv2 is helpful for training generative models. If so, would it be an optimal and promising direction of constructing latent space to only get closer to the DINO space? Are there any other components that should be considered?\n- What  ‘linear probing’ exactly did you use for evaluation, and why? Do all the measurements for linear probing give the same correlation and conclusion?\n- The vision foundation model could be huge and computationally expensive. Can you compare the size of the proposed tokneizer with the others?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The author first proposed to directly employ the pretrained vision foundation model, followed by an adaptor as an encoder for training a generative model.\n- The proposed architecture is simple but effective.\n- The experiments in Table 1 support the effects of each configuration. Specifically, in the Semantic preservation loss table, there is a trade-off relationship between the reconstruction and generation performance\n- Compared to the VA-VAE, the proposed method achieves better gFID along various sampling steps and CFG scale. In addition, the proposed latent space makes the generative models converge faster than the latent space of VA-VAE\n- Compared to FLUX VAE, the proposed methods achieve better generation performance qualitatively and quantitatively.", "weaknesses": "- The overall training strategy seems heuristic, falling short of scalability. Combining multiple stages into a simple pipeline or providing principled methods to measure when to stop each stage would be helpful for applicability.\n- Why DINO-v2 works better than other foundation models? An analysis or theoretical explanation would be helpful\n- Lack of analysis of the semantics of the latent space.\n- [REPA-E] also incorporates the vision foundation model for training VAE. What is the advantage of the proposed method over the [REPA-E]? \n\n[REPA-E]: REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sTLFsCssRi", "reviewer": "Reviewer_LVhM", "summary": "This paper proposes replacing the tokenizer VAE in LDM with pretrained visual encoders, specifically DINOv2, and designs a three-stage training framework. Extensive experiments demonstrate the effectiveness of the proposed method.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed approach is simple yet effective which leverages a pretrained DINO model as the encoder and introduces a three-stage training scheme that balances semantic representation and reconstruction quality.\n2. Comprehensive experiments verify the effectiveness of the method.", "weaknesses": "1. The motivation, *“Our intuition is that learning semantics is inherently more difficult than learning reconstruction”*,  requires further clarification and supporting evidence.\n\n2. My main concern is that all training and evaluation are conducted on ImageNet without any OOD testing. It is possible that the low gFID results stem from overfitting rather than genuine improvements in generalization, since introducing semantic latent distributions may inherently simplify the latent structure compared to pixel-level ones [1,2].\n\n3. Moreover, the authors use DINOv2 as a frozen encoder, which is pretrained on hundreds of millions of images[3]. This model may already include exposure to datasets similar to ImageNet, making it unclear whether the observed gFID improvement truly arises from the proposed architecture itself, or simply from the pretrained encoder’s prior knowledge of similar data.\n\n4. Although the proposed three-stage training improves reconstruction performance as much as possible, the reconstruction FID still degrades (see Table 5), which limits the applicability of the method as a general-purpose encoder for tasks requiring accurate reconstruction, such as image editing and inpainting.\n\n5. The code and evaluation scripts should be released to ensure reproducibility.\n\n[1] Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models\n\n[2] Masked Autoencoders Are Effective Tokenizers for Diffusion Models\n\n[3] DINOv2: Learning Robust Visual Features without Supervision", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)