Latent Denoising Makes Good Tokenizers
Authors: Jiawei Yang
Tianhong Li
Lijie Fan
Yonglong Tian
Yue Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Image Tokenizer
Image Generative Models
Representation Learning
Primary area: generative models
TL;DR: We find training tokenizers with latent denoising objectives significantly improve their generative performance across different and diverse generative models.

Abstract: Despite their fundamental role, it remains unclear what properties could make tokenizers more effective for generative modeling. We observe that modern generative models share a conceptually similar training objective---reconstructing clean signals from corrupted inputs, such as signals degraded by Gaussian noise or masking---a process we term \emph{denoising}. Motivated by this insight, we propose aligning tokenizer embeddings directly with the downstream denoising objective, encouraging latent embeddings that remain reconstructable even under significant corruption. To achieve this, we introduce the Latent Denoising Tokenizer (\method), a simple yet highly effective tokenizer trained to reconstruct clean images from latent embeddings corrupted via interpolative noise or random masking. Extensive experiments on class-conditioned (ImageNet $256\times256$ and $512\times512$) and text-conditioned (MSCOCO) image generation benchmarks demonstrate that our \method consistently improves generation quality across \textit{six} representative generative models compared to prior tokenizers. Our findings highlight denoising as a fundamental design principle for tokenizer development, and we hope it could motivate new perspectives for future tokenizer design. Code is available at: https://github.com/Jiawei-Yang/DeTok

Published as a conference paper at ICLR 2026
LATENT DENOISING MAKES GOOD TOKENIZERS
Jiawei Yang'
Tianhong Li
Lijie Fan
Yonglong Tian 4f
Yue Wang!
USC
MIT CSAIL
3 Google DeepMind
OpenAA
ABSTRACT
Despite their fundamental role, it remains unclear what properties could make to- kenizers more effective for generative modeling. We observe that modern genera- tive models share a conceptually similar training objective- reconstructing clean signals from corrupted inputs, such as signals degraded by Gaussian noise or masking-a process we term denoising. Motivated by this insight, we propose aligning tokenizer embeddings directly with the downstream denoising objective, encouraging latent embeddings that remain reconstructable even under signifi- cant corruption. To achieve this, introduce the Latent Denoising Tokenizer (I-DeTokk), a simple yet highly effective tokenizer trained to reconstruct clean im- ages from latent embeddings corrupted via interpolative noise random masking. Extensive experiments on cans-coonditonee (ImageNet 256 x 256 and 512 x 512) and text-conditioned (MSCOCO) image generation benchmarks demonstrate that our I-De'Tok consistently improves generation quality across six representative generative models compared to prior tokenizers. Our findings highlight denois- ing as fundamental design principle for tokenizer development, and we hope it could motivate new perspectives for future tokenizer design. Code available at:
INTRODUCTION
Modern visual generative models commonly operate on compact latent embeddings, produced by tokenizers, circumvent the prohibitive complexity of pixel-level modeling (Rombach et al., 2022; Peebles & Xic, 2023; Chang ct al., 2022; et al., 2024a). Current tokenizers are typically trained as standard variational autoencoders (Kingma & Welling, 2014), primarily optimizing for pixel-level reconstruction. Despite their critical influence on downstream generative quality. it remains unclear what properties enable more effective tokenizers for generation. As result, tokenizer development has lagged behind recent rapid advances in generative model architectures.
In this work, we ask: What properties can make visual tokenizers more effective for generative modeling? We observe that modern generative models, despite methodological differences, share a conceptually similar training
original signals from corrupted ones. For instance, diffusion models remove difflusionindduced noise to recover clean signals (Ho et al., 2020; Peebles Xie, 2023), while autoregressive models reconstruct complete sequences from partially observed contexts (Li et al., 2024a; Chang et al., 2022), analogous to removing "masking noise" (He et al., 2022; Chen et al., 2025c; Devlin et al., 2019). We collectively refer to these reconstruction- from-deconstructtonn processes as denoising.
This unified denoising perspective of modern generative models suggests that effective visual tok- enizers for these models should produce latent embeddings that are reconstructable even under sig- nificant corruption. Such embeddings naturally align with the denoising objectives downstream generative models, facilitating their training and subsequently enhancing their generation quality.
Motivated by this insight, we propose to train tokenizers as latent denoising autoencoders, termed as I-DeTok. Specifically, corrupt latent embeddings via interpolative noise, generated by interpo- lating original embeddings with Gaussian noise. The tokenizer decoder is then trained to reconstruct
Advisory-only
Work done prior to joining OpenAl
We use the terms "deconstruction" and "comuption" interchangeably throughout the paper.

---

Published as a conference paper at ICLR 2026
clean images from these heavily noised latent embeddings. Additionally, we explore random mask- ing, akin to masked autoencoders (MAE) (He et al., 2022), as an alternative and optional form of deconstruction and find it similarly effective.
Conceptually, these bust, stable, and easily reconstructable under strong corruption, aligning with the downstream de- noising tasks central to generative models. Indeed, our experiments show that stronger noise used in our Tok training typically yields better downstream generative performance.
strategies encourage latent embeddings to be ro-
We demonstrate the effectiveness and generalizability of (-DeTook across six representative gen- erative models, including non-autoregresssvve (DiT (Peebles & Xie, 2023), SiT (Ma et al., 2024), LightningDiT (Yao & Wang, 2025)) and autoregressive models (MAR (Li et al., 2024a), RasterAR, RandomAR (Li al., 2024a; Pang et al.. 2025; Yu et al., 2024)) on the ImageNet generation bench- mark. By adopting our aokenizer-withowd MAR models (Li al., 2024a), improving FID from 2.31 to 1.55 for MAR-B, matching the per- formance of the original huge-sized MAR (I.55), For MAR-L, FID improves from 1.78 to 1.35. Importantly, these gains come without semanties distillation, thus avoiding reliance on visual en- coders pretrained at far larger scale (Oquab et al., 2024; Radford et al., 2021).
modifying model architectures- we push the limits for
In summary, our work demonstrates simple yet crucial insight: explicitly incorporating denoising objectives into tokenizer training significantly enhances their effectiveness for generative modeling since it is downstream task-aligned, We hope this perspective will stimulate new research directions in tokenizer design and accelerate future advances in generative modeling.
2 RELATED WORK
Representation learning in visual recognition. Representation learning has been a decades-long pursuit in visual recognition, aiming to obtain transferable embeddings that generalize across di- verse downstream tasks (Bengio et al., 2013). At the core of these approaches lies foundational principle: pre-training should encourage representations to encode the information most relevant to downstream tasks. This principle has inspired the design of diverse and effective pretext tasks, such as instance discrimination (He et al., 2020; Chen al., 2020b), self-distillation (Grill et al., 2020; Caron et al., 2021; Oquab et al., 2024), and masked-image reconstruction (He et al., 2022; Oquab et al., 2024; Zhou et al., 2022), each aligning representations with downstream utility. These insights motivate us to explore tokenizer embeddings that align with downstream generative tasks.
Visual tokenizers for generative modeling. Modern generative models rely on tokenizers to en- code images into compact latent embeddings, significantly reducing computational complexity com- pared to pixel-level modeling. While conventional tokenizers optimize pixel reconstruction with KL-regularization, recent approaches (Chen et al., 2025a; Yao & Wang, 2025; Chen et al., 2025b; Li et al., 2025) have advocated semantics distillation from powerful pretrained vision models (Oquab al., 2024; Radford et al., 2021). Yet, in many domains (e.g., video, audio, 3D/4D), such encoders may not exist. In contrast, our 1-De'Tok learns good tokenizers without these dependencies, and it could be complementary to approaches such as VAE, which replaces the deterministic VAE de- coder with denoising diffusion process guided by encoder latents (Zhao et al., 2025), and residual vector-quantized tokens, which support high-fidelity yet efficient discrete diffusion generation via collective token prediction (Kim et al., 2025).
Generative modeling frameworks. Generative modeling frameworks broadly include autoregres- sive (AR) (Chen et al., 2020a; Esser et al., 2021; Chang et al., 2022; Li et al., 2024a; Tian et al., 2025; Pang et al., 2025; Yu et al., 2024; Tschannen et al., 2024) and diffusion-based non-autoreeressiive (non-AR) methods (Ho et al., 2020; Rombach et al., 2022; Peebles & Xie, 2023; Ma et al., 2024; Yao & Wang, 2025). AR models predict latent tokens sequentially conditioned on partial contexts, while non-AR models jointly generate tokens via iterative refinement, typically using diffusion (Ho et al., 2020; Sohl-Dickstein et al., 2015) or bow-based processes (Lipman et al., 2023; Esser et al., 2024). Despite methodological differences. both paradigms critically rely high-quality latent embeddings from upstream tokenizers (Hansen- Maansen-Estruch et al., 2025). Thus, improving tokenizer embeddings is fundamental for enhancing generative performance across diverse frameworks.
2

---

Published as a conference paper at ICLR 2026
Masked patches
Gaussian Noise
Mask token
Input
Target
Encoder
(1 - r) +
Decoder
Masking noise
Interpolative latent noise
Figure 1: Our latent denoising tokenizers (I-DeTok) framework. During tokenizer training, we randomly mask imput patches (masking noise) and interpolate encoder-produced latent embeddings with Gaussian noise (interpolative latent noise). The decoder processes these deconstructed latents and mask tokens to reconstruct the original images pixels. We refer this process as denoising. When serving tokenizer for downstream generative models, both noises are disabled.
3 METHOD
Our goal is to design visual tokenizers that are more effective for generative modeling compared to standard autoencoders trained for pixel reconstruction. This section first revisits the training objective shared by all modern generative models, i.e., denoising, to motivate our design, and then details our latent denoising tokenizers (I-DeToo).
3.1 PRELIMINARIES OF GENERATIVE MODELING
Modern generative frameworks can be primarily divided into non-autoregressive (non-AR) and toregressive (AR) paradigms. Despite their methodological differences, both paradigms aim to grad- ually reconstruct the original representations from deconstructed ones.
Non-autoregressive generative models. Non-autoregresssive models, exemplified by diffusion (Ho al., 2020; Peebles Xie, 2023) and flow-matching methods (Lipman et al., 2023), learn itera- tively refine latent representations deconstructed by controlled noise. Given latent representation of input Xo, the forward noising process progressively corrupts these latents into X,;
X a(t) X. + b(t) E,, E, ~ ~0,11,
where a(t) and b(t) are noise schedules. Generative models are trained to revert this deconstruction:
(()) = Ex.e,( [[lco ccll?
cell).
(2)
where co is a learnable noise estimator parameterized by 0. Essentially, non-AR diffusion models learn to reconstruct original latents from intermediate latents deconstructed by noise.
Autoregressive generative models. Autoregressive approaches factorize image generation into sequential prediction problem. Given an ordered sequence of latent tokens {x" "x!,...,x* xN),, AR meth- ods factorize the joint distribution as:
(x
I Po (x'
(3)
where x" denotes the latent tokens generated at step i. Recent generalized AR variants extend this framework to arbitrary generation orders (Li et al., 2024a; Yu et al., 2024; Pang et al., 2025) or set-wise prediction strategies (Tian et al., 2025; Ren et al., 2025). Nonetheless, the fundamental training objective remains consistent: reconstructing full sequences from partially observed or cquivalently, partially
other words, AR models learn to reconstruct original latents from intermediate latents deconstructed by masking.
3.2 LATENT DENOISING TOKENIZERS
Motivated by the discussions above, we propose latent denoising tokenizer (I-DeTok), a simple to- kenizer trained by reconstructing original images from deconstructed latent representations. This cooonstrction-rrcoostttucce design aligns with the denoising tasks employed by modern genera- tive models. Figure 1 shows overview of method. We detail each component next.

---

Published as a conference paper at ICLR 2026
Overview. Our tokenizer follows encoder-decoder architecture based on Vision Transformers (ViT) (Dosovitskiy et al., 2021). Input images are divided into -veeeapplg patches, linearly projected into embedding vectors, and added with positional embeddings. During training, de- construct these embeddings using two complementary strategies: (i) injecting noise in latent em- beddings and (ii) randomly masking image patches. The decoder reconstructs original images from these deconstructed embeddings. This strategy encourages coscoos-rconttruett latent embeddings under heavy corruption, aiming to simplify downstream denoising tasks in generative models.
Noising as deconstruction. Our core idea is to deconstruct latent embeddings by noise. Specifically, given latent embeddings x from the encoder, we interpolate them with Gaussian noise as follows:
x' (1 x)x + TE(?); where e(>) ~ 7 I),,,), T (0010,
(4)
Here, the scalar y controls noise standard deviation, and the factor T specifies noise level. Critically, this interpolative strategy differs from the conventional additive noise, i.e., x' by standard VAEs (Pu et al., 2016), as it ensures latents can be effectively and heavily corrupted when the noise level 7 is high. Moreover, random sampling of encourages latents to remain robust across diverse corruption levels. At inference time, latent noising is disabled (T = 0).
TE, employed
Masking deconstruction. We further generalize our denoising perspective by interpreting mask- ing as another form of latent deconstruction (Chen et al., 2025c). Inspired by masked autoencoders (MAE) (He et al., 2022), we randomly mask subset of image patches. Different from MAE, we use random masking ratio. Concretely, given an input image partitioned into patches, we mask a random subset, where the masking ratio m is sampled from slightly biased uniform distribution:
m max (-.1,
(5)
where 4(-0.1, M.) .) denotes a uniform distribution on [-0.1, The slight bias towards zero re- duces the distribution gap between training and inference (no masking). The encoder processes only the visible patches, and masked positions are represented by shared learnable [MASK] tokens at the decoder input. At inference time, all patches are visible (m = 0).
Training objectives. Our tokenizer is trained to reconstruct the original images from corrupted latent embeddings. The training objective follows established practice (Rombach et al., 2022; Esser et al., 2021; Yu et al., 2025a), combining pixel-wise mean-squared-erro (MSE), latent-space KL-regularization (Pu al., 2016). perceptual losses (VGG- and ConvNeXt-based as in Yu et al.
(2025a)), and an adversarial GAN objective (Goodfellow et al., 2014):
Listal LASE + λKLCKI + Apereep + AGAN
(6)
where each À controls the contribution of the corresponding loss component.
IMPLEMENTATION
We briefly summarize implementation details here, including datasets, evaluation metrics, and model training procedures. Due to limited space, we defer comprehensive details to Sections A. and A.2.
Dataset and metrics. We primarily experiment with class-conditioned image generation the ImageNet dataset (Russakovsky et al., 2015) at 256 x 256 and 512 x 512 resolutions, and further evaluate text-to-image generation on the MS-COCO dataset (Lin et al., 2014). For evaluation, we report Fréchet Inception Distance (FID), Inception Score (IS), precision, and recall.
Tokenizer baselines. We benchmark our tokenizer against diverse set of tokenizers: (i) MAR-VAE from MAR (Li et al., 2024a), trained on ImageNet using the implementation from Esser et al. (2021);
(ii VA-VAE (Yao & Wang, 2025), aligning latent embeddings with DINOv2 features: (iii) MAE- Tok (Chen et al., 2025a), distilling HOG, DINOv2 (Oquab et al., 2024), and CL.IP (Radford et al., 2021) features through auxiliary decoders; (iv) SD-VAE and SD3-VAH from Stable-Diffusion (Rom- bach et al., 2022; Esser et al., 2024), trained on significantly larger datasets. Additionally, for con- trolled comparisons, we also train our ViT-based tokenizer without denoising objective as a baseline.
Our tokenizer implementation. We implement our tokenizer using ViTs for both encoder and de- coder (Vaswani et al., 2017; Dosovitskiy et al., 2021). We adopt recent architectural advances from LLaMA (Touvron et al., 2023), including RoPE (Su et al., 2024) (together with learned positional embeddings following Fang et al. (2023)), RMSNorm (Zhang & Sennrich, 2019), and SwiGLU- FFN (Shazeer, 2020). We patch size of 16, and the latent dimension is set 16.
4

---

Published as a conference paper at ICLR 2026
FID@110K vs CFG scale for MAR-U
FID@10% VS CFC Scale for MAR-S
15810k CFG ScB MAR-B
Nose Stratlgy
Variance
X
Bapeiin-
FD
200
y-0.5
2-
y=1.
V=1.5
y=2.
200
20
20
25
20
4.0
S-a
CT6 scale
CTG scale
CPG scate
FID@10k vs CFG scale for STT-B
FIDg10k vs CFG scale for SIT-B
15@10k vs CFG scale for SIT-B
11.0
11.5
Noice Strategy
xeex
33.5
240
Base 1ow0..0
Veriance
11.0
0.55%
33.0
Variance
232-
20.5
1:04f
1y=0.0)
F10e
Kex i 1.ttc
yel
340
yw2
yoll
yw1s
yuz5
V-20
yw3
0.5
1.5
y=2
342
338
2.
2.4 13
26
11
2.0
2.4
1.
1.
2.0
12
CFO sca
scals
(a) Additive w.. interpolative
(b) Latent noise variance
Figure 2: Ablation on latent noise design. (a) Additive vs. interpolative noise. Interpolative noise clearly outperforms additive noise for both MAR (Li et al., 2024a) and SiT (Ma et al., 2024). Both interpolative and additive latent noise lead to improved performance for MAR. (b) Latent noise standard deviation (r). Our 1-DeTok remains robust across various noise standard deviations. Generally, increasing improves generation quality, with best results achieved around y = 3.0.
Tokenizer training. We set the default weights in Eq. 6 to ÅKI. = 10-6, Apencep = 1.0, and AGAN 0.1. In ablation studies, we use VIT-S for the encoder and ViT-B for the decoder, disable the GAN loss, and train for 50 epochs. For final experiments, we use ViT-B for both encoder and decoder, train for 200 epochs, and activate the GAN loss starting from epoch 100.
Generative models. To evaluate the broad effectiveness of a tokenizer, we experiment with six representative generative models, including three non-autoregressive models: DiT (Peebles & Xie, 2023), SiT (Ma et al., 2024), and LightningDiT (Yao & Wang, 2025); and three autoregressive models: MAR (Li et al., 2024a), causal RandomAR, and RasterAR based RAR (Yu et al., 2024) and diffloss (Li et al., 2024a). We follow the officially released implementations to reimplement all methods within a unified codebase, standardizing training and evaluation across methods. Previous works on visual tokenizers often exclusively evaluate on non-AR models (e.g. DiT, SiT), yet we find improvements from non-AR models do not necessarily generalize AR models (more on this later). Our experiments aim to provide useful data points toward a more universal tokenizer.
Generative model training. We use standardized training recipe for all generative models. Specif- ically, we follow the hyperparameters from Yao & Wang (2025), training generative models with global batch size of 1024, using AdamW (Loshchilov Hutter, 2019) with a constant learning rate of 2 x 10-4, without warm-up, gradient clipping, or weight decay. Autoregressive models adopt the three-layer 1024-channel diffloss MLP from Li et al. (2024a). For ablation studies, we train genera- tive models for 100 epochs. For larger-scale experiments on MAR models, we train for 800 epochs.
5 EXPERIMENTS
5.1
MAIN PROPERTIES
We use SiT-B (Ma et al., 2024) and MAR-B (Li et al., 2024a) as representative non-autoregreesive (non-AR) and autoregressive (AR) generative models to study the generalizability of a tokenizer. Quantitative reconstruction results for tokenizers trained under different noises are in Sec. B.1.
5.1.1 PROPERTIES OF LATENT NOISING
We first study latent noising the sole form of deconstruction, without any masking.
Interpolative vs. additive noise. An important design choice in our I-DeTok is the use of inter- polative latent noise instead of additive noise, which we ablate here. Specifically, we compare two
5

---

Published as a conference paper at ICLR 2026
F Dog CFG scale for MAR-B
15@ vs CFG scale for MAR-B
FID@10k vs CFG scale for MAR-B
Max Masking Ratio
240
Mox Masking Rabio
Condtant Masking Ratto
200
22
const
const M-2
Ne
N=0.8
const M
MeO
150
K-0.5
const M=29
100
CPG
CFG scale
CPG scate
FID@10K
FiDg10kk OFG SIT-6
220
Mac Masking Hatio
12
200
Baseline (Mi0
K=03
K-
Max Nasking Ratio
5
N=0.7
ribe1ok
Constant Macking Ratio lef Meq.7 Baye the [Mad o1
Beselin (K=QLD)
100
K-D.8
12
M-2.6
Mu6.3
L40
N-0.5
cerist H-2.5
M--0
const H-2.6
120
M-28
3.5
1.8
2.
2.0
2.2
3.6
1.
2.5
212
scate
scale
(a) Random masking ratio
(b) Constant masking ratio
Figure 3: Ablation on masking ratio. We show generative performance (FID.) with varying mask- ing ratios for tokenizers trained with (a) random and (b) constant masking ratio. Both MAR (Li al., 2024b) (top) and SiT (Ma et al., 2024) (bottom) benefit from masking- tokenizers, favor- ing heavy masking (70% to 90%). Randomized masking consistently outperforms constant masking.
latent noising variants: interpolative noise, ie.,, x' = (1 -)x + TE (Eq. 4), and additive noise, ie., x + TE. We set the noise standard deviation to ? = 1.0 here. Figure 2-(a) presents the results: Interpolative noise clearly outperforms additive noise for both SiT (Ma et al., 2024) and MAR (Li et al., 2024a). This aligns with our expectation: interpolative noise ensures latent embed- dings heavily corrupted high noise levels. In contrast, additive noise potentially create shortcuts by making original signals remain dominant, reducing the effectiveness of the additive noise. Nonetheless, observe the additive latent noise still improves MAR but SiT.
Noise standard deviation. Figure 2-(b) studies the effect of noise standard deviation (y in Eq. 4). Both SiT and MAR consistently improve with interpolative latent noise across all tested levels. Per- formance peaks at moderately high standard deviation, indicating that stronger corruption generally yields more effective latents. This result confirms our key hypothesis: challenging denoising tasks naturally produce robust, Nownstream-aligned latents that benefit generative modeling.
Noise schedule. Inspired by Hoogeboom et al. (2023); Esser et al. (2024), we study how the sampling distribu- tion of the noise level 7 in Eq. affects generation quality, Following Esser et al. (2024), we draw 7 from a logit- normal distribution with mean 4 (and fix 7 = 3.0). As shown in Table 1, all uniform and non-uniform schedules improve FID over the = 0 baseline, and concentrating more probability mass higher noise levels (μ+ 0.8) yields the relatively best results for both MAR-B and SiT- B. This indicates that I-DeTok is robust to the choice of noise schedule and can still benefit from further tuning of the noise distribution.
FIDJ
Sampling of
MAR-B
SiT-B
(hascline)
3.31
6.97
T U(0, 1) (default)
2.77
5.56
logit(r) ~ (-0008, 1)
2.79
6.04
logit(r) ~ ~VO0.
2.77
5.93
logit(r) (0..8,
2.58
5.44
Table 1: Impact of noise level sampling.
5.1.2 PROPERTIES OF MASKING
We next investigate masking noise independently, without any latent noise.
Masking ratio. We examine how varying the maximal masking ratio M (Eq. 5) influences gener- ation quality. As shown in Figure 3-(a), both SiT and MAR benefit from masking- tokenizer training in generation quality. Masking ratios between 70% and 90% consistently yield stronger performance compared low masking ratios (e.g., 30%), favoring high degrees of masking. This

---

Published as a conference paper at ICLR 2026
Seswdre
Masking (M-0.))
Nolsing 1y-3.0)
Joint (M-077, y-3.01
Dl0kkkk (MAR-B)
15@10k (MAR-B)
Figure 4: Impact of tokenizer train- ing strategies on generative perfor- mance. We compare our baseline tokenizer with our 1-DeTTok variants: masking-only (M=0077), latent noising- only (==3.0), and their combination (joint noising). Both masking and latent noising independently improve genera- tion quality, with latent noising showing a stronger effect. Joint noising further improves performance for MAR, partic- ularly in inception scores (IS), but pro- vides limited additional benefit for SiT when latent noising is already applied, FID @ 50k scores are detailed in Tab.
260
240
220
FI03810
1.
262
140-
120
300
CFG scale
CFG scale
FiD@@10k (SIT-B)
15@ 10k (SiT-8)
260
12-
240
1.2
200-
H108
180
160
340
100
3.4
2.4
2.8
2/2
24
Table 2:
Effectiveness of denoising. We report FID and IS evaluated on 50,000 images here. Compared to base- lines, we see substantial gains in gener- ative models when using our I-DeTok.
MAR-B
SiT-B
Setup
w/ CFG
w/o CFG
w/ CFG
weo CFG
FIDJ
1S1
FIDJ
FID
1S
FIDJ
Baseline
3.31
247.63
16.70
6.97
181.61
26.82
Masking only
2.90
243.00
12.57
6.43
189.24
22.50
Latent noise only
2.77
249.02
8.82
5.56
193.52
16.11
Extended: larger encoder, longer train-
Joint noise
2.65
263.03
7.48
5.50
195.07
16.03
ing. GAN enabled midway through training.
+Extended
2.43
266.50
6.72
5.13
207.35
14.81
behavior mirrors observations from latent denoising, i.e., challenging denoising more beneficial, indicating a common underlying principle under the deconstractinn-rcoossiiiioo strategy.
Constant vs. randomized masking ratio. Figure 3-(b) compares randomized masking ratios against constant ones. For constant ratios, we further fine-tune the tokenizer decoder for 10 epochs using villyv latents to alleviate the distribution shift between training and inference since fully- visible inputs are absent in constant-ratio training. This adjustment improves rPSNR by 1-3 and rFID by 0.6 (details in Sec. 1. Figure 3-(b) shows randomized masking consistently outperforms constant masking, as it encourages latent embeddings to be robust to varying corruption levels. This aligns naturally with downstream tasks that require denoising across diverse corruption levels.
5.1.3 JOINT DENOISING
Prior ablations indicate that both latent noising and masking can independently improve generation quality, with latent noising showing stronger effect. Here, We investigate the effect of joint denois- ing. Based on prior results, we fix the noise standard deviation to = 3.0 and the masking ratio to M 0.7. Figure and Table summarize the results. With joint denoising, our (-DeTok achieves FID scores of 5.50 (SiT-B) and 2.65 (MAR-B) with CFG (Table 2). In comparison, our baseline tokenizer trained with identical settings but without any noise obtains worse results: 6.97 (SiT- B) and 3.31 (MAR-B). We observe that joint denoising is more effective for MAR, but provides limited additional benefit for SiT when latent noising is already applied. This indicates that latent denoising is essential, while the masking-based denoising is optional.
Lastly, with joint denoising, we increase the encoder to base size, train for 200 epochs, and enable the GAN loss starting from epoch 100 (+Extended in Table 2). Under this setting, FID improves to 5.13 (SiT) and 2.43 (MAR). We adopt this improved tokenizer for all subsequent evaluations.
FID4 (w/ CFG)
5.1.4 SCALABILITY
Model
Baseline
Ours
We study whether the gains from [-DeTok persist under generative model scaling. We train SiT-B/L/XL and MAR-B/L for 100 epochs, comparing the baseline tokenizer and ours under identical settings. As shown in Table 3, I-DeTok consistently improves FID at all model sizes for both SiT and MAR. The advantage remains when we extend training to 800 epochs the system-level comparison of Table
SiT-B
7.08
5.13
SiT-I.
4.66
3.49
SiT-XL
4.47
3.14
MAR-B
3.65
2.43
MAR-L
2.44
2.08
Table 3: Scalability.
7

---

Published as a conference paper at ICLR 2026
Table 4: Generalizability comparison of tokenizers across differrnt generative models. We compare various tokenizers on representative generative models. Our 1-DeTook tokenizer outper- forms other tokenizers for AR models, and also surpasses standard tokenizers trained without se- manties distillation for non-AR models. All results are obtained with optimal CFG scales.
Autoregressive Models
Non-antoregresiive Models
Tokenizer
#FDL
MAR
RandomAR
RasterAR
SIT
DiT
Light.DT
FIDL
1ST
FID. 1St
FID 1S1
FID
181
FID4
1S1
FID,
IST
Tokenizers trained with semantics distillation from external pretrained models
VA-VAE (Yao & Wang. 2025)
0.28
16.66
144.5
38.13
68.3
15.88
160.5
4.33
221.1
4.91
213.9
2.86
275.1
MAETok (Chen et al., 2025a)
0.48
6.99
201.8
24.83
97.6
15.92
127.2
4.77
243.2
5.24
224.7
3.92
273.3
Our I-DeTok Distillation
0.85
2.52
254.1
5.57
180.3
11.99
158.8
3.40
232.6
3.91
221.7
2.18
243.0
Tokenizers truined withou semandics distillation
AD-VEE(( (Rombach al., 2022)
0.61
4.64
259.8
13.11
141.8
8.26
179.3
7.66
187.5
8.33
179.8
4.24
223.7
SD3 E (Esser al., 2024)
0.21
6.46
250.0
40.65
70.89
21.03
101.7
9.57
170.8
13.63
126.8
5.52
222.2
MAR VAE (L. et al., 2024a)
0.53
3,71
265.3
11.78
147.9
7.99
189.7
6.26
177.5
8.20
171.8
3.98
218.7
Our I-DeTok
0.68
2.43
266.5
5.22
248.9
4.46
257.7
5.13
207.3
6.58
173.9
3.63
225.4
5.2
GENERALIZATION EXPERIMENTS
To comprehensively evaluate tokenizer generalizability, we compare performance across six repre- sentative generative models: three non-autoregressive (DiT (Peebles & Xie, 2023), SiT (Ma et al., 2024), LightningDiT (Yao & Wang, 2025)) and three autoregressive (MAR (Li et al., 2024a), Ran- domAR, RasterAR). RandomAR and RasterAR are adapted from RAR (Yu et al., 2024). We modify them to support decoding continuous tokens via diffloss MLPs (Li et al., 2024a). RandomAR gen- erates tokens in a random order while RasterAR uses a raster-scan order. We use base-sized models and train them for 100 epochs for experiments here.
Comparisons with standard convolutional tokenizers. Our (-DeToo tokenizer consistently out- performs conventional tokenizers (Rombach et al., 2022; Esser et al., 2021; 2024), by large margins across both AR and non-AR models. Table presents the results. Compared to the best existing to- kenizer (MAR-VAN VAE), our method significantly improves FID (with CFG) from 3.71 to 2.43 (~34%) for MAR, from 11.78 to 5.22 (~56%) for RandomAR, and from 7.99 to 4.46 (~44%% for RasterAR. Improvements for non-autoregresssie models are also consistent.
Comparisons with semtntics-distilled tokenizers. Our 1-DeToo generalizes significantly better than prior semantiss-distilled tokenizers. Table 4 compares our method with recent approaches such as VA- VA-VAE (Yao & Wang. 2025) and MAETok (Chen et al., 2025a), which distill semantics from pretrained encoders. Surprisingly, we empirically find these tokenizers, despite promising perfor- mance in non-AR models, do not generalize well to AR models. Previous studies implicitly assume that tokenizer improvements from one generative paradigm naturally transfer others. However, our experiments challenge this assumption, revealing a previously unrecognized gap: tokenizer ef- fectiveness in one paradigm does not necessarily transfer to others.
In sharp contrast, our method generalizes robustly across both non-AR and AR models. Critically, our 1-De'Tok achieves this without any semantics distillation from pretrained encoders. To further investigate whether our (-DeTok can benefit from semantics distillation, we incorporate an auxiliary cemantics-distillaioon loss (details in Sec. A.4). Remarkably, this privileged version of our tokenizer achieves the best FID scores for non-AR models, surpassing all previous semantics-distillee tok- enizers. Although AR models benefit relatively less from distillation and may even slightly degrade, their performance still remains substantially superior to previous methods.
1-DeTok with convolutional tokenizers. Our I-DeTok generalizes across Transformer- and convolution-basee tokenizers, as the idea of latent denoising is architecture-agnostic. To demon- strate this, we train CNN-based tokenizers for 50 epochs with and without our denoising objective (details in Sec. A.5). With the default CNN baseline, MAR-B achieves 3.32 FID and SiT-B 7.11 FID at best CFGs. With our denoising- CNN tokenizers, MAR-B improves to 2.82 FID and SiT-B to 5.62 FID, demonstrating that [-DeTTok provides consistent gains also in convolutional networks.
5.3 BENCHMARKING WITH PREVIOUS SYSTEMS
We compare against leading generative systems in Table 5. For this experiment, we train MAR-B and MAR-L for 800 epochs. Simply adopting our tokenizer substantially improves the generative performance: MAR-B achieves an FID of 1.55 (from 2.31), and MAR-L further improves to 1.35 (from 1.78). Notably, our MAR-B and MAR-L both match or surpass the previously rest-performing
8

---

Published as a conference paper at ICLR 2026
Table 5: System-level comparison ImageNet 256x256 plass-conditiones generation. Our ap- proach enables MAR models (Li et al., 2024a) to achieve leading results without relying on seman- ties distillation. t; With additional decoder fine-tuning (see Sec. A.3 for details).
wio CPG
w/ CRG
#params
FID1
1st
Pre.t
Rec.t
FID
1st
Pre.
Rec.
With semantics distillation from external pretruined models
SIT-XL + REPA (Yu et al., 2025b)
675M
5,90
157.8
0.70
0.69
1.42
305.7
0.80
0.64
STT-XL + MAETok (Chen et al., 2025a)
675M
2.31
216.5
1.67
311.2
LightningDiT + MAETok (Chen et al., 2025a)
675M
2.21
208.3
1.73
308.4 4
LightningDiT + VAVAE (Yao & Wang, 2025)
675M
2.17
205.6
0.77
0.65
1.35
295.3
0.79
0.65
DDT-XL (Wang et al., 2025)
675M
6,27
154.7
0.68
0.69
1.26
310.6
0.79
0.65
Without semantics distillation from external pretrained models
DIT-XL/2 (Peebles & Xie, 2023)
675M
9.62
121.5
0.67
0.67
2.27
278.2
0.83
0.57
STT-XL/2 (Ma ct al., 2024)
675M
8.30
2.06
270.3
0.82
0.59
VAR-d30 (Tian ct al., 2025)
2.0B
1.92
323.1
0.82
0.59
LlamaGen-3B (Sun et al., 2024)
3.1B
2.18
263.3
0.81
0.58
RandAR-XXL (Pang et al., 2025)
1.4B
2.15
322.0
0.79
0.62
CausalFusion (Deng el al., 2024)
676M
3.61
180.9
0.75
0.66
1.77
282.3
0.82
0.61
MAR-B- + MAR- VAE (Li et ul., 2024a)
208M
3.48
192.4
0.78
0.58
2.31
281.7
0.82
0.57
MAR-L + MAR-VAE (Li et al., 2024a)
479M
2.60
221.4
0.79
0.60
1.78
296.0
0.81
0.60
MAR-H MAR- VAF (Ii et al., 2024a)
943M
2.35
227.8
0.79
0.62
1.55
303.7
0.81
0.62
MAR-B (La et al., 2024a) + our I-DeTTok
208M
2.79
195.9
0.80
0.60
1.61
289.7
0.81
0.62
MAR-B (Li ct al., 2024a) + our /-DeTok!
208M
2.94
195.5
0.80
0.59
1.55
291.0
0.81
0.62
MAR-L (Li ct al., 2024a) + our T-DeTok
479M
1.84
238.4
0.82
0.60
1.43
303.5
0.82
0.61
MAR-L (Li et al., 2024a) + our I-DeTok"
479M
1.86
238.6
0.82
0.61
1.35
304.1
0.81
0.62
Table 6: System-level comparison on
Model
#params
FID.
1ST
ImageNet 512x512 class-conditioned generation. Our tokenizer is fine-tuned from ImageNet 256x256 checkpoint
ADM (Dhariwal & Nichol, 2021)
554M
7.72
172.7
DIT-XL/2 (Peebles & Xie, 2023)
675M
3.04
240.8
SIT-XL/2 (Ma et al., 2024)
675M
2.62
252.2
SIT-XL/2 REPA (Yu al., 2025b)
675M
2.08
274.6
for 25 epochs. Scratch vs. fine-tune:
MAR-L MAR-VAE (Li et al., 2024a)
479M
1.73
279.9
Generative models trained from scratch
MAR-B Ours (scratch, 400ep)
208M
1.83
279.6
or initialized from ImageNet 256x256 checkpoints (see Sec. A.3 for details).
MAR-L L + Ours (fine-tune, 200ep)
479M
1.61
315.7
huge-size MAR model (1.35 vs. 1.55). Qualitative results are provided in Figure 5. We also report results ImageNet 512 x 512 in Table 6. Our MAR-B variant already matches MAR-L, 2.3x larger model, when trained from scratch. Our MAR-L achieves remarkable 1.61 FID and 315.7 IS.
5.4 TEXT-TO-IMAGE GENERATION
We further validate I-DeTok on text-to- to-ie (T2I) generation. Following the protocol in Bao et al. (2022); Yu et al. (2025b), we train models from scratch on the MS-COCO train split (Lin et al., 2014) and evaluate and SiT-B under identical conditions, varying only the tokenizer (details Sec. C). Table 7 sum- marizes the results, demonstrating that (-DeTok substantially improves not only sample quality and diversity (FID), but also text-image conditioning alignment (CLIP score). Qualitatively (see Fig- ures. C.I, C.2.C.3, and C.4), other tokenizers frequently produce the "spot artifacts" reported in previous works (Fan et al., 2025; Team et al., 2025) under the text-to-image setting, whereas such artifacts are notably absent with [-DeTok.
the validation split using FID-30k. We compare T2I variants of MAR-B
Table 7: MS-COCO text-to-image
T21 MAR-B
T21 SiT-B
generation. Comparison of tokenizers
Tokenizer
FIDI
CLIP
FID,
CLIP
on T2I-variants MAR-B and SiT-B, re- ported with best CFG scales. Our I-
VA-VAE ( (Yao Wang. 2025)
34.64
21.98
5.83
25.07
SD- DD-V Rombach
(2022)
19.75
22.86
6.63
24.00
DeTok achieves both lower FID (better
MAR- VAE (Li et al., 2024a)
12.49
22.07
5.74
23.50
diversity) and higher CLIP scores (bet- ter alignment), outperforming all others.
Ours (I-DeTok)
4.97
24.82
4.31
24.61
5.5 BEYOND STANDARD CONTINUOUS TOKENIZERS
To assess the generality of our approach beyond standard 2D continuous tokenizers, we study both 1D continuous tokenizers and discrete vector-qunatized (VQ) tokenizers.

---

Published as a conference paper at ICLR 2026
Figure 5: Qualitative Results. We show selected examples of class-conditional generation on Ima- geNet 256 using MAR-L (Li et al., 2024a) trained with our tokenizer.
1D Tokenizers. We first test I-DeTTok with recent ID tokenizers that map images to compact 1D token sequences (Yu al., 2025a; Yan et al., 2025; Duggal al., 2025; Bachmann et al., 2025). These tokenizers substantially reduce the sequence length compared to standard 2D latent grids. Sec. A.6 provides implementation details and an interesting pitfall coupled with the "grokking" phenomenon we observe in these ID tokenizers. As shown in Table 8, I-DeTok improves generation performance over the corresponding baselines, with gains becoming more pronounced at higher token budgets. This suggests that the benefit of denoising is not tied to particular latent topology, and remains effective even when the representation is compressed and less-structured.
Vector- secttr-quantized Tokenizers. We next evaluate I-DeTok on VQ tokenizers, where the latents are discrete code indices instead of continuous features. In this setting, the idea of removing masking noise applies directly. Section A.7 presents implementation details. We consider both random- order (RandomAR) and standard raster-scan auto-regressive models (RastcrAR). As summarized in Table 9, (-DeTok improves both variants: for RandomAR, FID is improved from 8.48 to 7.03; for RasterAR, FID improves from 5.29 to 4.91, with IS also increasing in both cases. These gains indicate the broader applicability of our denoising principle beyond continuous tokenizers.
Table 8: 1D continuous tokenizers on ImageNet 256.256. Comparison MAR and SiT with dif- ferent numbers of ID tokens, with and without our denoising training, evaluated by FID 50k.
Table 9: Vector-quantized tokenizers on ImageNet 256 256. Comparison of VQ tokenizers with and without our denoising training in RandomAR and RasterAR mod- els, evaluated by FID 50k and IS @ 50k with CFGs. (-DeTok improves generative quality across generation orders.
MAR-B
SiT-B
#Tokens
Setting
CFG
wlo CPG
w/ CFG
CFG
Baseline
5.22
17.63
6.99
18.91
32
+Ours
5.51
14.87
7.02
16.46
RandomAR-B
RasterAR-B
Bascline
4.58
17.73
7.05
22.82
Setting
64
FIDL
1St
FID
ISt
+Ours
4.12
14.02
6.40
18.59
Baseline
3.92
18.63
7.95
28.36
Baseline VQ
8.48
144.93
5.29
204.15
128
+Ours
3.14
12.05
6.20
20.22
+Ours
7.03
162.68
4.91
208.61
6 DISCUSSION AND CONCLUSION
Simple principles that scale well are at the core of generative modeling. this work, we have shown that surprisingly simple principle, i.e., denoising basedd tokenizer, already provides consistent ben- efits across a broad design space, including con-autoregressive and autoregressive generative mod- els, 2D/1D/VQ tokenizers, CNN and Transformer-based architectures, and both raster and random generation orders, while adding almost no system complexity. Several questions remain open.
First, experiments conducted within moderate-scale regime (O(10%) samples), where I- DeTTk improves performance when we scale up within this envelope. A natural next step is to test it in the frontier regime of web-scale models and datasets (0(10))0(104)) samples), more challenging settings such as video generation, and to understand how deenoising-alinned tokenizers behave those cases. Second, our results also parallel the recent trend of semantics distillation from pre-trained encoders such as DINOv2. Distillation is effective when strong teachers exist and align well with the target domain, but may become a bottleneck as we move to larger data and broader distributions that surpass the teacher's coverage, or to domains without reliable teachers. In contrast, I-DeTok is self-contained and does not depend on external teachers. At scale, we hope such task- aligned tokenizers may provide more flexible path forward. Finally, reconstructing clean inputs from noisy latents makes our tokenizer objective reminiscent of zp Zo-prediction modern generative models. Understanding the boundary and eventual unification-aal aotercationammnnng reconstruction denoising and generation is an intriguing direction, e.g., Li & He (2025). We hope this perspective will help guide future work scalable generative models and the tokenizers that support them.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Roman Bachmann, Jesse Allardice, David Mizrahi, Enrico Fini, Oguzhan Fatih Kar, Elmira Amir- loo, Alaaeldin El-Nouby, Amir Zamir, and Afshin Dehghan. Flextok: Resampling images into Id token sequences of flexible length. In ICML, 2025.
Fan Bao, Chongxuan Li, Yue Cao, and Jun Zhu. All are worth words: a backbone for score-based diffusion models. In NeurlPS 2022 Workshop on Score-Based Methods, 2022.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 3598)11988-1828, 2013.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. MaskGIT: Masked genera- tive image Transformer. In CVPR, 2022.
Hao Chen, Yujin Han, Fangyi Chen, Xiang Li, Yidong Wang, Jindong Wang. Ze Wang, Zicheng Liu, Difan Zou, and Bhiksha Raj. Masked autoencoders effective tokenizers for diffusion models. arXiv preprint arXiv:2502.03344 033444, 2025a.
Hao Chen, Ze Wang, Xiang Li, Ximeng Sun, Fangyi Chen, Jiang Liu, Jindong Wang, Bhiksha Raj. Zicheng Liu, and Emad Barsoum. Softvq-vae: Efficient 1-dimensional continuous tokenizer. In CVPR, 2025b.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and lyaa Sutskever. Generative pretraining from pixels. ICML., 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020b.
Xinlci Chen, Zhuang Liu, Saining Xie, and Kaiming He. Deconstructing denoising diffusion models for self-supervised learning. In ICLR, 2025c
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers 22 billion parameters. In ICML, 2023.
Chaorui Deng, Deyao Zhu, Kunchang Li, Shi Guang, and Haoqi Fan. Causal diffusion transformers for generative modeling. arXiv preprint arXiv:2412.12099 2024.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HILT, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs image synthesis. In NeurlPS, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Shivam Duggal, Phillip Isola, Antonio Torralba, and William T Freeman. Adaptive length image tokenization via recurrent allocation. ICLR, 2025.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederie Boesel, et Scaling rectified flow transformers for high-resolution image synthesis. In /CML, 2024.

---

Published as a conference paper at ICLR 2026
Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling autoregressive so-ttoo-mmge generative models with continuous tokens. In ICLR, 2025.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023.
lan J Goodfellow, Jean Pouget-Abadie. Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Neur/PS, 2014.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangging Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in hour. arXiv: 706. 02677, 2017.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. In NeurlPS, 2020.
Philippe Hansen-Estruch, David Yan, Ching- Yao Chung, Orr Zohar, Jialiang Wang, Tingbo Hou, Tao Xu, Sriram Vishwanath, Peter Vajda, and Xinlei Chen. Learnings from scaling visual tok- enizers for reconstruction and generation. arXiv preprint arXiv:2501.09755, 2025.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. ICMI., 2023.
Jaehyeon Kim, Tachong Moon, Keon Lee, and Jaewoong Cho. Efficient generative modeling with residual vector quantization-bart based tokens. In ICML, 2025.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise. arXiv preprint arXiv:2511. 2025.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. In NeurlPS, 2024a.
Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Jiuxiang Gu, Bhiksha Raj, and Zhe Lin. Imagefolder: Autoregressive image generation with folded tokens. In ICLR, 2025.
Yazhe Li, Jorg Bornschein, and Ting Chen. Denoising autoregressive representation learning. In ICML, 2024b.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In ICLR, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden- Eijnden, and Sain- ing Xie. Sit: Exploring flow and diffusion-basee generative models with scalable interpolant transformers. In ECCV, 2024.
12

---

Published as a conference paper at ICLR 2026
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, pp. 1-31, 2024.
Ziqi Pang, Tianyuan Zhang, Fujun Luan, Yunze Man, Hao Tan, Kai Zhang, William T Freeman, and Yu-Xiong Wang. Randar: Decoder-only autoregressive visual generation in random orders. In CVPR, 2025.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
Yunchen Pu, Zhe Gan, Ricardo Henao, Xin Yuan, Chunyuan Li, Andrew Stevens, and Lawrence Carin. Variational autoencoder for deep learning of images, labels and captions. In NeurIPS, 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Sucheng Ren, Qihang Yu, Ju He, Xiaohui Shen, Alan Yuille, and Liang-Chiin Chich Chen. Flowar: Scale- wise autoregressive image generation meets flow matching. In ICML, 2025.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 15(3))211-25,, 2015.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202. 2020.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 0662022006.06525, 2024.
NextStep Team, Chunrui Han, Guopeng Li, Jingwei Wu, Quan Sun, Yan Cai, Yuang Peng, Zheng Ge, Deyu Zhou, Haomiao Tang, et al. Nextstep-I: Toward autorgrressive image generation with continuous tokens at scale. arXiv preprint arXiv:2508. 10711, 2025.
Keyu Tian, Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. In NeurIPS, 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint Xii:22307.09288. 2023.
Michael Tschannen, Cian Eastwood, and Fabian Mentzer. transformers. In ECCV, 2024.
Givt: Generative infinite-vocabulary
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
Shuai Wang, Zhi Tian, Weilin Huang, and Limin Wang. Ddt: Decoupled diffusion transformer. arXiv preprint arX1v:2504.05741, 2025.
Wilson Yan, Volodymyr Mnih, Aleksandra Faust, Matei Zaharia, Pieter Abbeel, and Hao Liu. Elas- tictok: Adaptive tokenization for image and video. In ICLR, 2025.
13

---

Published as a conference paper at ICLR 2026
Jingfeng Yao and Xinggang Wang. Reconstruction generation: Taming optimization dilemma in latent diffusion models. In CVPR, 2025.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint 0rXiv22110.0422. 2021.
Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang- Ching Chen. Randomized autoregres- sive visual generation. arXiv preprint (xXiv:2411.00770 2024.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chich Chen. An image is worth 32 tokens for reconstruction and generation. In NeurIPS, 2025a.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is casier than you think. In ICLR, 2025b.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. NeurIPS, 2019.
Long Zhao, Sanghyun Woo, Ziyu Wan, Yandong Li, Han Zhang, Boqing Gong, Hartwig Adam, Xuhui Jia, and Ting Liu. epsilon-vae: Denoising visual decoding. 2025.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In ICLR, 2022.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A TRAINING AND INFERENCE DETAILS
A.I TOKENIZER
Model. We implement our tokenizer using ViTs for both encoder and decoder (Vaswani et al., 2017: Dosovitskiy et al., 2021). We provide detailed model parameters in Table A.1. We adopt recent architectural advances from LLaMA (Touvron et al., 2023), including RoPE (Su et al., 2024) (together with learnable positional embeddings following Fang al. (2023)), RMSNorm (Zhang & Sennrich, 2019), and SwiGLU-FFN (Shazeer, 2020). The encoder operates at a patch size of 16, yielding 256 latent tokens for each 256 256 image, while the decoder uses patch size 1 as there is no resolution change. The latent dimension is set to 16. We omit the [CLS] token for simplicity.
Table A: Model Size. We report the model configurations and parameter counts of tokenizer encoders and decoders. The total tokenizer size is the sum of encoder and decoder parameters. For example, tokenizer combining ViT-S encoder and ViT-B decoder (S-B) has 111.6M parameters, while a tokenizer with both ViT-B encoder and decoder (B-B) has 171.7M parameters.
Size
Hidden Size
Blocks
Heads
Parameters
Small (S)
512
8
8
25.75M
Base (B)
768
12
12
85,85M
Large (L)
1024
24
16
303.33M
Training. Our tokenizer is trained using the weighted loss defined in Eq. 6, with default weights a ViT-S encoder and ViT-
AKL 10-6, Axpecep = 1.0, and AGAN 0.1. For ablation studies, decoder, disable GAN loss, and train for 50 epochs. We observe that including a GAN loss (Yu et al., 2021; Goodfellow et al., 2014; Esser et al., 2021) sharpens reconstructions but roughly doubles training time, without altering result trends. For our final experiments, we adopt ViT-B for both encoder and decoder, enable GAN loss from epoch 100, and train for 200 epochs. In both settings, we use a global batch size of 1024. and the peak learning rate is set to 4.0 x 10 (scaled linearly batch size of 256 (Goyal et al., 2017)). We apply linear warm-up for 25% of the total epochs (12 epochs for 50-epoch training, and 50 epochs for 200-epoch training), followed by cosine learning rate decay. We use the Adam W optimizer (Loshchilov & Hutter, 2019) with parameters (0.9, 0.95) and a weight decay of 1.0 x 10-4, The only data augmentation employed is horizontal flipping. Our reconstruction loss closely follows the implementation in Yu et al. (2025a).
from 1.0 x 10
Training budget. Training an S-B tokenizer (see Table A.1) without GAN loss for 50 epochs takes roughly 1 160 NVIDIA A A100 GPU hours (enabling GAN loss from epoch 20 extends training to about 288 A100 GPU hours). Training B-B tokenizer with GAN loss enabled from epoch 100 for total of 200 epochs takes approximately 1,150 A100 GPU hours.
We present a detailed comparison of training cost between our baseline and (-DeTok tokenizers in Table A.2. All configurations use the same architecture, batch size (1024), and total number of epochs; the only difference is whether the approximate A 100 hours of our method are essentially identical to, or slightly lower than, the baseline: S-B goes from 180 (baseline) to 160-180 hours with T-DeTTok, B-B from 1150 1140 hours, and the ID and VQ variants remain within the same 170-190 hour range. This indicates that the denoising mechanism does not introduce any meaningful extra computational overhead beyond the standard tokenizer training; in practice, the variation is within normal run-to-run fluctuation.
use the baseline objective or 1-DeTok. As shown,
Pseudo-code of latent denoising. See Algorithm
A.2 GENERATIVE MODELS
Model. To evaluate the broad effectiveness of a tokenizer, we experiment with six representative
generative models, including three non-autoregressive models: DiT (Peebles & Xie, 2023), SiT (Ma et al., 2024), and LightningDiT (Yao & Wang. 2025); and three autoregressive models: MAR (Li
al., 2024a), causal RandomAR, and RasterAR based RAR (Yu et al., 2024) and diffloss (Li
15

---

Published as a conference paper at ICLR 2026
Table A.2: Training Cost. Comparison of training schedules and approximate compute for different tokenizer and setting combinations. Our models are trained across mixed compute and storage environments (e.g. A100s, H100s, and A6000s). We report approximate A100-equivalent hours over the full training run. Size: Encoder size Decoder size. S: Small, B: Base.
Size
Setting
Epochs
Batch size
GAN start epoch
Approx. A100 Hours
S-B
Baseline
50
1024
disabled disabled
180
S-B
I-DeTTok
50
1024
160-180
B-B
Baseline
200
1024
100
1150
B-B
I-DeTok
200
1024
100
1140
B-B
I-DeTok + Distill.
200
1024
100
1900
S-B
ID-baseline
50
1024
disabled
170-190
S-B
ID-I-DeTok
50
1024
disahled
170-190
S-B
VQ-baseline
50
1024
disabled
190
S-B
VQ-I-DeTok
50
disahled
170
Algorithm Latent Denoising: PyTorch-like Pseudo-code
def denoise (x, encoder, decoder, max mask ratio-0.7, gamma-3.0) encode input image latent embeddings under (optional) masking ids_restore = encoder (x, max mask ratio=max wwt.otma...aaa.kaaaaaaaaa.... catiaa__mak_rriiiiio ratio)
variational latent embeddings posteriors diagonal_gaussi an_dist (z)
s_sampled
sample interpolation factor uniformly from [0, 1] bsz, h_tokens, chans 2_sampled. shape
10
m
device = 2_sampled. cemmpledddevice... noise level torch, (bsz, 1, 1, device-device). exxand 1, a a_tokens, chans)
12
13
14
generate Gaussian noise noise gamma torch, randn (baz, R_tekens, chans, device=device)
15
17
# interpolate latent embeddings with noise
18
noised = (1 noise_levell * u quanttized + novee_level + noise
19
reconstruct the inputs recon = decoder (z_noised ids_restore)
22
return recon
al., 2024a). We follow their officially released code re-implement generative models within our codebase to standardize training and evaluation, ensuring fair comparisons.
We introduce some modifications: for DiT (Peebles & Xie, 2023), SiT (Ma et al., 2024), and Light- ningDiT (Yao & Wang, 2025), we apply classifier- free guidance (CFG) to all latent channels, rather than only the first three channels used in their original implementations. For training on 1D tokens (e.g., MAETok (Chen et al., 2025a)), we use simple ID learnable positional embeddings and disable RoPE used in LightningDiT. We adopt the default samplers from the original implementations, us- ing 250 denoising steps during inference. Due to the frequently observed training instability in SiT and LightningDiT in our early experiments, we apply QK-Norm (Dehghani et al., 2023) to stabilize training. These modifications are consistent across all tokenizers for fairness.
For autoregressive (AR) methods, we use the default MAR model and implement RandomAR and RasterAR following RAR (Yu al., 2024). To produce continuous tokens, employ the diffloss (Li et al., 2024a) with 3-layer, 1024-channel MLP head. We disable dropout in all MLP layers within Transformer blocks for autoregressive models. For inference, we use 64 autoregressive steps and 100 denoising steps for all experiments, except those in system-level comparison (Tables 5 and 6), where we use 256 autoregressive steps or 512 autoregressive steps. Sampling is performed with the default MAR sampler across all AR models. Following MAR (Li et al., 2024a), we set the sampling temperature to 1.0 when using CFG, and sweep temperatures when CFG disabled (i.e., CFG=1.0).
"ee original implementations DIT, SiT, and LightingDiT.
16

---

Published as a conference paper at ICLR 2026
Training. We use standardized training recipe for all generative models. Specifically, we follow the hyperparameters from Yao & Wang (2025), training generative models with a global batch size of 1024, using AdamW optimizer (Loshchilov & Hutter, 2019) with constant learning rate of 2 x 10-2, without warm-up, gradient clipping, or weight decay. We do not tune these hyperparameters. For ablation studies, we train generative models for 100 epochs. For larger-scale experiments on MAR models, we train them for 800 epochs. All models utilize exponential moving average (EMA) with a decay rate of 0.9999.
Standardization. We always standardize tokenizer outputs by subtracting the channel-wise mean and dividing by the chamnel-wise standard deviation, both computed from the ImageNet training set. For publicly available tokenizers, we use their official standardization steps.
Metries. Unless noted otherwise. we report FID @ 50k scores with classifier-free guidance (CFG), with optimal CFG scales searched from FID @ 10k results. We abbreviate FID@50k as FID.
Training budget. Training DiT-B, SYT-B, and LightningDiT-B for 100 epochs takes approxi- mately 128 to 200 A100 hours using locally cached tokens. Training MAR-B, RandomAR B, and RasterAR-B for 100 epochs takes approximately 220 to 250 A100 hours. For the final MAR-B model used in the system-level comparison (Table 5), training for 800 epochs takes roughly 2,450 A 100 hours; training MAR-L. for the same duration takes about 3,850 A 110 hours (online evaluation time included).
A.3 TOKENIZER TUNING
Decoder fine-tuning, We observe an interesting discrepancy between training and inference in our I-DeTok. Our decoder is trained predominantly with noise-corrupedd latent embeddings but en- counters nearly clean embeddings during inference. To address this gap. we fine-tune the decoder on clean latent embeddings, i.e., masking and latent noising are disabled, for additional 100 epochs. This adjustment partially alleviates the discrepancy, improving the FID of the 800-epoch MAR-L model from 1.43 to 1.35 and MAR-B model from 1.61 to 1.55 (Table 5). Figure A.I and Figure A.2 compare the denoising capability of different tokenizers. Although fine-tuning decoder reduces the decoder's 's denoising strength, it enhances the image quality (FID and IS) when decoding from generated latent embeddings. Crucially, this demonstrates that the quality of latent represen- tations produced by the tokenizer encoder, rather than the decoder's s error-tolerance, driver of generative model improvements. Therefore, our l-DeTTok does more than improving the the decoder's robustness to sampling errors.
the primary
To further verify that performance gains primarily stem from improved latent representations rather than enhanced decoder-side error tolerance, we conduct an additional experiment by fine-tuning only the MAR- AAE decoder the denoising task while keeping the encoder frozen. In this way, we can reuse pre-trained models for comparison since their modeling space, encoder latent space, remains fixed. Contrary expectations, this setup-improving decoder robustness without alter- ing latent mmbeddings-actuallle leads to slightly worse FIDs (around 0.1 to 0.3 degradation). This indicates that merely boosting decoder denoising capability exacerbates the traiing-inferenee dis- crepancy, as generative models are already proficient at denoising. These results reinforce that the primary advantage of our approach lies explicitly in the improved quality of latent embeddings, not in decoder-side denoising enhancements.
Adopting tokenizers 512 512 resolution. To save computation, we fine-tune tokenizer check- points from ImageNet-255, 256 x 256 for the ImageNet-512 x 512 generation task. Specifically, we halve the learning rate to 2.0 x 10-4 and fine-tune tokenizers initialized from the 200-epoch 256 256 checkpoints for an additional 25 epochs, enabling GAN loss from the start. We compare full fine- tuning to decoder-only fine-tuning: decoder-only fine-tuning achieves better PSNR (26.3 vs. 24.9) and comparable rFID (0.92 vs. 0.86), yet fully fine-tuned tokenizers consistently provide supe- rior downstream generative performance. For instance, using fully fine-tuned tokenizers, MAR-B achieves 2.66 FID and SiT achieves 5.98 FID, compared to decoder-only fine-tuned tokenizers at 3.63 and 6.32 FID, respectively. All generative models are trained from scratch for 100 epochs, and report FID 50K at optimal CFG scales. Consequently, adopt fully fine-tuned tokenizers.
17

---

Published as a conference paper at ICLR 2026
00
twO
2-0
TwO
TwO
T-O
DwO
TwO 50
Original
E-000
L-010
E-020
E-000
E-040
E-000
L-300
1-0.0
T-0
Decorbe-i
I-0.10
1-0.0
PH0300
1-0.0
1-0.30
t-2.02
1-0.00
I-0.00
1-0.00
Basaire
Figure A.I: Visualization of latent denoising. Images generated from latent embeddings corrupted with varying noise levels (f) by the original decoder (top), fine-tuned decoder (middle), and baseline decoder (bottom). The fine-tuned decoder shows a reduced ability to recover images from noisy embeddings compared the original decoder. The baseline tokenizer trained without the denoising objective fails to reconstruct original images from noisy latents.
m=0.10
m=0220
m=0
m=0-40
m=0.50
m=0.0
m-6070
m-0..0
0=0000
1t-0002
m=0.00
m=0.20
m-0 m=0.40
m=0.50
m-0.0 m=0.70
m=0.89 mi00.90
Decodeefft
Figure A.2: Visualization of "mask" denoising. Images generated from masked inputs with vary- ing masking ratios (m) by the original decoder (top) and fine-tuned decoder (bottom). The fine-tuned decoder exhibits diminished capability in reconstructing masked regions compared to the original decoder.
These results reinforce motivation: encoder-produced tokenizer embeddings primarily drive generative quality, while decoder fine-tuning provides incremental improvements. Therefore, im- proving encoder more fundamental than improving the decoder.
Adapting generative models to 512 x 512 resolution. Similar to tokenizer fine-tuning, we fine- tune MAR-L checkpoint from ImageNet-256 256 for the ImageNet-512 512 generation task to reduce compute. Additionally, we train MAR-B model from scratch to verify the compatibility of our tokenizer for end-to-ent training. Note that we do aim to hill climb the performance metrics in Table 6, since our tokenizer is only fine-tuned for 25 epochs rather than trained from scratch for many epochs. Instead, our intention for these results demonstrate the broad applicability of our approach.
A.4 (-DETOK WITH SEMANTICS DISTILLATION
Our [-DeTook already holds great promise for generative models across both autoregressive (AR) and non-autoregressive (non-AR) methods without semantics distillation. It is natural question whether incorporating semanties distillation could further improve its performance.
To investigate this, we introduce an auxiliary cemantics-distilatton loss into our tokenizer training pipeline. Specifically, we implement the latent representation projector as three-layer MLP, similar to REPA (Yu et al., 2025b). The noised latent embeddings fz.noise sed at line in Algorithm 1) are projected through the following layers: Linear (16, 2048) + SiLU () - Linear (2048, 2048) - SiLU () + Linear (2048, 768), where 16 represents the latent embedding mension and 2048 the intermediate projection dimension. This projector is trained to maximize the cosine similarity between the projected embeddings and semantic features extracted from the pretrained DINOv2-Base model (Oquab et al., 2024).
18

---

Published as a conference paper at ICLR 2026
We have also experimented with decoder-basee semantics-dsstilationn approach, i.e., using iliary Transformer- based decoder to predict pretrained features. After training both implementations for 50 epochs, we found their results were very similar, though the decoder-basee implementation required more computation because of the decoder overhead. Thus, we adopted the simpler MIP- based approach for efficiency.
The loss coefficient for minimizing the cosine distance is set to 1.0 for simplicity, which we do not sweep or tune. We train this tokenizer using the identical settings as the "+Extended" protocol described in Table 2: 200 epochs in total, activating GAN loss from epoch 100.
The results from this experiment (summarized in Table 4) demonstrate substantial improvements, particularly for biffusion-basee non-AR models (c.g.. SiT). With semantics distillation, our I-DeTTok achieves theee-of-the-art FID scores for non-AR models, clearly surpassing previous semantics- distilled tokenizers such as MAETok (Chen et al., 2025a) and VA- (Yao & Wang, 2025). While AR models benefit relatively less from distillation and performance may even slightly degrade compared to the non-distilled (-DeTok- generative quality remains notably better than previ- ous approaches.
Nonetheless, we emphasize the practical significance of our original I-DeTok formulation with- out semantics distillation, especially for domains like video, audio, proteins, poses, or trajectories, where strong pretrained semantic models comparable to DINOv2 (Oquab et al., 2024) may be un- available or significantly weaker. Thus, our proposed tokenizer remains broadly valuable both with and without semantic distillation.
A.S CONVOLUTIONAL TOKENIZER
Model. We reuse MAR- VAE and SD-VAE architectures with a downsampling ratio of 16 and latent dimension 16, producing latent representations of shape (16, 16, 16) for (C, H, W). We add only a a.few lines latent-denoising code (similar to Algorithm 1) to CNN-based tokenizers. Tokenizers are trained for 50 epochs, both with and without latent-denoising, using hyperparameters identical to our Transformer- bbasdd tokenizers, potentially disadvantaging CNN models. 3 We enable GAN loss from epoch 10 for CNN tokenizers. We do not sweep the noise standard deviation and only run with y = 3.0 and y = 0.0 (baseline). Further hyperparameters and noise strength tuming and longer training may yield additional improvements for CNN-based tokenizers, which leave for future work.
Results. We report FID@ 50k under best CFG scales. With the default CNN baseline, MAR-B achieves 3.32 FID and SiT-B 7.11 FID. With our denoising-based CNN tokenizers, MAR-B im- proves to 2.82 FID and SiT-B to 5.62 FID, demonstrating that L-DeTok provides consistent gains also in convolutional networks. Our results clearly demonstrate the general, architecture-agnoste nature of our latent-denoisin idea.
A.6 ID TOKENIZERS
Implementations. We construct 1D continuous tokenizers by minimally modifying the 2D contin- tokenizers used in main experiments. Concretely, keep the image encoder and decoder backbones unchanged and only alter the latent parameterization and positional embeddings. Fol- lowing Yu et al. (2025a), we append K E {32, 64, 128) learnable latent tokens to the encoder input sequence, Icading to 256+K K tokens for a 256 x 256 image with patch size 16 (i.e., 256 patch tokens plus K latent tokens). After encoding, only the K latent tokens are passed the decoder. The de- coder then uses learnable mask tokens to reconstruct the original images from these K latents. All Transformer blocks are based on self-attention. For simplicity, we drop RoPE and use only learnable absolute positional embeddings.
1D tokenizers with i-DeTok. When applying [-DeTook to ID tokenizers, we use interpolative latent noising with 2.0 and omit masking entirely for simplicity. All other training hyperparameters (optimizer, schedule, batch size, and loss) are kept identical to those used for our 2D continuous tokenizers, which is convenient for comparison but may be suboptimal for the ID tokenizers.
In our experience, Transformer- and CNN-hased tokenizers typically require distinct hyperparameter sets.
19

---

Published as a conference paper at ICLR 2026
psnr
22
18
16
Grokking
12
(a) Baseline ID tokenizers.
psnr
Delayed "Grokking"
Stop
10k (b) Naive denoising with y = 2.0 (delayed "grokking").
30k
psnr
22.0 32 withh
42 54 warh
g2.0 withumarnup
"Grokking"
16
Enable denoising after "Grokking"
Stop
10k
20k
30k
40k
(c) Denoising with 2.0 enabled after warm-up.
Figure A.3: Training dynamics of 1D tokenizers. PSNR curves for different token counts K. (a) Baseline 1D) tokenizers exhibit "grokking" -like phase where PSNR suddenly improves after a plateau. (b) Directly adding denoising with 7 2.0 delays this transition, especially for larger K. (c) Enabling denoising only after the baseline grokking phase (10-epoch warm-up) yields stable training across all K. X-axis: calibrated global step, lk-step denotes ImageNet epoch.
Grokking. We observe a practical "grokking" -lie phase when training ID tokenizers: AA (Fig. A.3a). This behavior already appears in the baseline 1D tokenizers without I-DeTTok. A naive application of denoising with & 2.0 (without any warm-up) further delays this transition, espe- cially for larger K (Fig. A.3b), so under a fixed training budget the tokenizers remain undertrained and yield poor downstream FID. To mitigate this, we warm up the ID tokenizer for 10 epochs by disabling the denoising task, and then enable it from epoch 10 onward with milder noising level ? = 2.0 to reduce task difficulty." 4 This staged schedule makes all runs stable and successful (Fig. A.3c), and is analogous common tokenizer training practices that warm up the model before
We initially used y 3.0 and observed poor downstream performance. We first attributed this failure to the large 7, but later found that the delayed grokking the primary cause.
20

---

Published as a conference paper at ICLR 2026
turning GAN perceptual losses. We hope this full transparency will serve useful reference and provide concrete data points for future research.
A.7 VECTOR-QUANTIZER (VQ) TOKENIZERS
Implementations. For the VQ tokenizer, we start from the same continuous tokenizer architecture and only replace the bottleneck head. In the continuous variant, a linear layer maps Transformer tokens from the model width to the latent channels (e.g., 16 or 2 16 for the variational autoencoder). In the VQ variant, this head is replaced by a VQ bottleneck: a linear projection first maps each token to 64-dimensional latent vector, which is then quantized using codebook of size 4096, Le., a learnable matrix of shape (4096, 64). Following prior work (Yu et al., 2021; 2025a), we f2- normalize the codebook vectors and train with the standard VQ objective (reconstruction loss plus codebook loss) and a commitment loss with weight 0.25; we refer readers to these works for further formulation details. All other components of the encoder and decoder are unchanged. As with our ID and continuous tokenizers, we reuse the same training hyperparameters (optimizer, schedule, and loss weights apart from the VQ terms), which simplifies comparison but may be suboptimal for the VQ setting.
Generators. On top of the learned VQ tokenizer, we train discrete auto-regressive (AR) generators in two settings: (i) RandomAR, where the 2D grid of discrete tokens is visited in random permu- tation at both training and sampling time; and (ii) RasterAR, where the same architecture trained with the standard raster-scan ordering. We directly use the implementation from RAR (Yu et al., 2024) without modification and fix the generation order (random or raster-scan) consistently during both training and inference. These models are decoder-only, causal aatsal-attention Transformers, trained with cross entropy loss in a teacher-forcing manner.
VQ tokenizers with (oDeTok. Our (-DeTok variant for VQ tokenizers is intentionally minimal: we only leverage masking as deconstruction during training. Starting from the baseline RasterAR setup, we sweep masking ratios of 0.7 and 0.95. Both ratios yield better downstream FID than the baseline VQ tokenizer, with 0.95 performing best. We therefore adopt a masking ratio of 0.95 for RandomAR runs.
B ADDITIONAL RESULTS
B. EFFECT OF NOISING ON RECONSTRUCTION PERFORMANCE
Figure B.1 reports tokenizer reconstruction quality (measured by rPSNR and rFID) under various noising strategies discussed in Section 5. All tokenizers are trained for 50 epochs without GAN loss. As expected, reconstruction performance degrades with increased noise strength (noise stan- dard deviation or masking ratio). Nevertheless, even under substantial noise levels, our tokenizers achieve reasonably strong reconstruction quality, consistently exceeding 24.0 rPSNR and remaining below 1.2 rFID.
Figure B.b) highlights the importance of the additional 10-epoch decoder fine-tuning step for constant-masking tokenizers, which effectively mitigates the visual degradation and photorealism loss stemming from the training-inferene distribution mismatch. This fine-tuning significantly im- proves decoder reconstruction, particularly higher masking ratios. Training with randomized masking consistently outperforms constant masking, indicating that variable masking ratios lead to more robust reconstruction and superior generative outcomes.
B.2 FID vs. CFG CURVES.
Figures B.2 and B.3 compare how classifier-free guidance (CFG) scales influence generative perfor- mance (FID and IS) across different tokenizers and generative models. We use warm colors to denote samantics-distilled tokenizers (marked by "*"), and cool colors for tokenizers without distillation. Our ToDe consistently achieves stronger performance across varying CFG scales, improving tably over standard tokenizers and matching or even surpassing momaantcs-distillee ones. Final FID scores computed over 50,000 generated images optimal CFG scales summarized in Table 4.
21

---

Published as a conference paper at ICLR 2026
interpolerive Noise
Addibve Noise
Random Masking
Constant Masking
Constant Masking
13
(PSNR
(PSNRR 11:
(DD())
25.0
10
23.0
1.0
0.8
213
1.5 2.0 255
3.5
10
0.5
Noise standard deviation
Masking ratio M
(al Resonstruction performance vs. Latent Noise
(b) Reconstruction performance vs. Masking
Figure B.1: Tokenizer reconstruction under various noising strategies. We report rPSNR and rFID for tokenizers studied in Section 5.1, trained for 50 epochs without GAN loss. With masking ratio M = 0.9, constant masking unexpectedly collapses performance, yielding an unexpected rFID of 13.95 (out of plot range).
FIDg Ok scale for MAR
FIDg 10k vs CFG scale for RasterAR
FID 10k vs CFG scale RandomAR
MAmaz SOAAE
SEAKE
FDB-0K
8
10
16
20
10
CPG orale
CPG scale
CFG scale
vs
15@900 scale RasterfR
vs CFG RandomAR
225
vokke>
SCAME
15gL
125
500
50-
10 scale
1.1 16
CFG seade
CFC scale
Figure B.2: Tokenizer comparison for autoregressive models. We report FID @ 10k and IS@ @ 10k scores at different classifier-free guidance (CFG) scales. Semantics-distiledt tokenizers (denoted by "w") are shown in warm colors, while those without distillation are shown in cool colors. Our i- De'Tok consistently achieves superior FID and IS metrics compared to other tokenizers. See Table for results on 50,000 images evaluated at the optimal CFG scales.
B.3 TRAINING CURVES
Convergence. We visualize the training loss of our tokenizers under different latent denoising strate- gies, ie., tokenizers we use in Sections 5.1.1 and 5.1.2. Figures B.5 and B.4 vary only the masking ratio and the noise standard deviation 7, respectively, while keeping all other hyperparameters fixed. In both cases the curves decrease smoothly and follow the same convergence pattern, with heavier corruption leading to higher final loss but no signs of instability under this setting.
Baseline vs. (-DeTok. Figure B.6 compares full training curves of the baseline tokenizer and I- DeTok under the same 200-epoch schedule, with GAN enabled from epoch 100, leading to sudden jumps of loss. The per-step compute is effectively unchanged, and both models converge similar speed in both the autoencoding and adversarial stages. Although /-DeTok optimizes a harder denois- ing problem and therefore attains higher loss, this does not slow optimization, while consistently
22

---

Published as a conference paper at ICLR 2026
FID scale
FID&110k CFG
ccale for Lightn inghiT
WIKE MANUKE
20.0
16
20
15.0
Villsk
FIDgJOK
0um
8
10.0
10-
5.
12
2.5
2.0
2,5
3.0
35
5.5
25
20
23
35
10
CFC mrae
15th 10k CFG scale for DIT
15g9 vs CFG scale SIT
15/5 10k CTG scale for LightningDiT
200
100
373
MAttcc" MAT
PAC-l- (Epoch 593)
SONAE
100
VATAA:
100
Cart
75
Gurt
1.0
2.0
255
3.5
2.0
2.5
3.5
5.6
Scale
scals
Figure B.3: Tokenizer comparison for non-autoregressive models. We report FID@ 10k and @ 10k scores at different classifier-free guidance (CFG) scales. Semantics-distie tokenizers (denoted by "*") are shown in warm colors, while those without distillation are shown in cool colors. Our 1-DeTok consistently outperforms standard (non-semantics-disitee performance of the best semantics-distiless tokenizer (VA- VAE (Yao & Wang, 2025)) and surpassing MAETok (Chen et al., 2025a) in IS. See Table 4 for results on 50,000 images evaluated at the optimal CFG scales.
tokenizers, matching the
loss
l65s WIDLES WIRAS mEni))
gerea2 press
PH PHARO PONLZ PIOLET
0.3
0.4
Figure B.4: Training loss under different noise amplitudes y. Larger y increases the reconstruc- tion difficulty and thus the final loss, but all set- tings converge monotonically without instabil- ity under this setting. We adopt 3.0 in final experiments, which slightly increases loss but yields the best downstream generative quality. X-axis: calibrated global step, Ik-step denotes ImageNet epoch.
Figure B.5: Training loss under different mask- ing ratios, with all other hyperparameters fixed. Heavier masking (e.g.. 0.7-0.95) makes recon- struction harder and yields slightly higher final loss. All curves smooth and stable, indicat- ing that adding the denoising branch does not introduce optimization issues under this setting. X-axis: calibrated global step, 1k-step denotes ImageNet epoch.
improving downstream FID/IS. Together with Table A.2, this confirms that the denoising objective introduces negligible training-time overhead relative to vanilla tokenizer.
Scalability. Finally, Figure B.7 examines how the tokenizer training loss scales with model size. Larger tokenizers achieve lower absolute loss, as expected. More importantly, after normalizing each curve by its initial loss, the baseline and [-DeTok trajectories remain closely aligned at all scales, with only minor deviations for the smallest (SS) variant, indicating similar convergence behavior. These curves suggest that the proposed denoising principle extends smoothly to larger tokenizers without introducing additional optimization barriers compute bottlenecks.
23

---

Published as a conference paper at ICLR 2026
Training Curves
Normalized Training
2.00
Seseine
Besetie
Dc d6 m0.7gg.. (Durs]
Detilmmo (0uns)
1.5
Normalized LoGs
0.5
1055
0.4
3.5
325
1.2
2310 200
Epoch
Epoch
Figure B.6: Full training curves of the baseline tokenizer and I-DeTTok under the same schedule (200 epochs, with GAN enabled starting at epoch 100). The per-step wall-clock time is essentially identical. Both models converge quickly in the autoencoding stage and remain stable throughout adversarial training. I-DeTTok shows a higher loss due to the harder denoising task, but reaches its plateau in similar number of epochs while enabling better downstream FID/IS.
Training
Normalized Training
Belta-L
1.0
VIM-- LL
De
0.8
Loss
2.5
244
02
01
0.0
10 Epoch
70
40
Epoch
Figure B.7: Scaling behavior of baseline and I-DeTok tokenizers actoss model sizes (ViTok- SS/BB/LL). Left: absolute loss, where larger models achieve lower final loss as expected. Right: loss normalized into [0, 1]. At each scale, the normalized trajectories of baseline and [-DeTTok nearly resemble each other, indicating similar convergence speed and optimization difficulty. This shows that denoising objective scales smoothly without introducing optimization compute bottle- necks under this setting. ViTok: baselines. DeTTok: ours with y = 3.0 and M = 0.7.
C TEXT-TO-IMAGE GENERATION
We further validate I-DeTok on text-to-image (T2I) generation. Following the protocol in (Bao et al., 2022; Yu et al., 2025b), we train models from scratch on the MS-COCO train split (Lin al., 2014) and evaluate on the validation split. To adapt SiT and MAR for T2I, we make minimal modifications: (1) For SiT, we concatenate text and image tokens, computing the loss only on image tokens. The AdaLN modulation input is simplified from (timestep embedding class embedding) to only timestep embedding. (2) For MAR, we fill the buffer tokens, which originally stores class embeddings, with text embeddings. In both cases, text embeddings are extracted using CLIP-L as in Bao et al. (2022); Yu et al. (2025b).
We train T2I variants of MAR-B and SiT-B under identical settings with different tokenizers. Ex- cept for SDVAE, all tokenizers are pretrained on ImageNet 256 x 256 without any tuning the MSCOCO dataset. MAR models are trained for 800 epochs while SiT models are trained for 1000 epochs.
24

---

Published as a conference paper at ICLR 2026
Figure C.I: Text-to-image generation using MAR-VAL VAE tokenizer. Uncurated text-conditional generations on MS-COCO with T2I-MAR-B trained using MAR-VAF tokenizer (Li et al., 2024a). Red boxes highlight the "spot artifacts" reported in previous works (Fan al., 2025; Team et al., 2025), which notably do not appear when using our tokenizer (see Figure C.4 for comparison).
Qualitative Text-to- Text-to-Image Comparison. Figures C.I, C.2, C.3, and C.4 visualize uncurated text- to-image samples generated by T2I-MAR with various tokenizers. Other tokenizers frequently pro- duce "spot artifacts" as previously reported (Fan al., 2025; Team al., 2025), whereas such artifacts are notably absent with our I-DeTook.
Text prompts for each column (from left right) are: green train coming down the tracks.", "A group of skiers are preparing to ski down mountain.", "A small kitchen with low ceiling.". "A group of elephants walking muddy water.", "A living area with a television and a table.", "A road with traffic lights, street lights and cars.", "A bus driving in city area with traffic signs.", "A bus pulls over to the curb close to an intersection.", "A group of people are walking and one holding umbrella." ", "A baseball player taking swing at incoming ball.". "A city street lined with brick buildings and trees.", and "A close up of plate of broccoli and sauce."
D LIMITATION
Limitations. On the theoretical side, our understanding of why (-DeTok works as well as does is still limited: its success supported by consistent empirical gains, but a principled and theo- retical analysis of the underlying optimization dynamics and representation properties is left open. On the technical side, we observe a training/inference discrepancy in our tokenizer: the decoder is primarily trained on noise-injected latent embeddings, whereas it operates on almost noise-free em- beddings during inference. Fine-tuning the decoder on clean latent embeddings partially addresses this discrepancy. Further investigation into mitigating this discrepancy could yield additional im- provements. Nonetheless, the core motivation and insights of our work remain robust.
25

---

Published as a conference paper at ICLR 2026
Figure C.2: Text-to-image generation using SD-VAE tokenizer. Uncurated text-conditional gen- erations on MS-COCO with T2I-MAR-B trained using SD-VAE tokenizer (Rombach al., 2022). Red boxes highlight the "spot artifacts" reported in previous works (Fan et al., 2025; Team et al., 2025), which notably do not appear when using our tokenizer (see Figure C.4 for comparison).
Figure C.3: Text-to-image generation using VA-VAE tokenizer. Uncurated text-conditional gen- erations MS-COCO with T2I-MAR-B trained using VA-VAE tokenizer (Yao & Wang, 2025). Red boxes highlight the "spot artifacts" reported in previous works (Fan et al., 2025; Team et al., 2025), which notably do not appear when using our tokenizer (see Figure C.4 for comparison).
26

---

Published as a conference paper at ICLR 2026
Figure C.4: Text-to-image generation using our 1-DeTok tokenizer. Uncurated text-conditional generations on MS-COCO with T2I-MAR-B trained using our I-DeTTok tokenizer (Yao & Wang, 2025).
27


Reviews:
[{"review_id": "eziOAaqCBR", "reviewer": "Reviewer_71cM", "summary": "This paper introduces the Latent Denoising Tokenizer, a visual tokenizer framework designed to explicitly align with the denoising objectives common to modern generative models. The authors propose training tokenizers to reconstruct images from latent representations subjected to strong interpolative noise and/or random masking, departing from traditional pixel-reconstruction VAE objectives. By doing so, $l$-DeTok aims to produce latent embeddings that are robust and reconstructable under significant corruption, which theoretically matches the requirements of downstream denoising-centric generative models. Empirical evaluation covers six prominent generative models—both autoregressive and non-autoregressive—across ImageNet and MS-COCO text-to-image settings, demonstrating that $l$-DeTok yields consistent improvements over standard, semantics-distilled, and convolutional tokenizers.", "questions": "Please refer to Weaknesses", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is motivated by an accurate and under-discussed observation: modern generative models, regardless of architecture, are fundamentally denoising systems. Training tokenizers with explicit latent corruption (interpolative noise, masking) is a clean conceptual shift that breaks with the tradition of mere pixel-wise autoencoding. This alignment is theoretically meaningful and empirically justified.\n\n- The authors benchmark $l$-DeTok across a broad spectrum of generative models, on both class-conditional (ImageNet) and text-conditional (COCO) tasks. Baselines include state-of-the-art semantics-distilled tokenizers (e.g., VA-VAE, MAETok), standard VAE-style tokenizers, and convolutional tokenizers. These comparisons are fair, with well-matched training recipes and strong ablations.\n\n- The authors provide clear, actionable analysis of the key components of $l$-DeTok. For instance, Figure 2 and Figure 3 dissect the importance of interpolative versus additive noise and the effects of constant versus randomized masking ratio on FID/IS. This isolates the impact of each design choice, giving both insight and reproducibility.\n\n- Strong Generalization and Architectural-Agnosticism: The improvements hold for both Transformer-based and CNN-based tokenizers, as shown in Section A.5, and do not depend on external semantics distillation resources—important for domains lacking large vision encoders.", "weaknesses": "- Lack of Theoretical Analysis Regarding Optimality or Limitations: The empirical link between denoising-aligned tokenizers and improved downstream performance is clear, but the theoretical rationale is underdeveloped. For example, there is no formal analysis or proof of why interpolative over additive noise leads to strictly more robust or generative-friendly latents (as claimed in Section 5.1.1). While Figure 2 empirically demonstrates this, a mathematical discussion (e.g., in terms of mutual information or denoising risk) would strengthen the scientific value.\n\n- Training/Test Distribution Discrepancy and Impact: Section A.3 (in APPENDIX) raises the training/inference mismatch issue: since the decoder is trained primarily on heavily corrupted inputs but, in deployment, must reconstruct from nearly clean latents, there is a risk of distribution shift. While decoder fine-tuning helps (see Figure A.1, Figure A.2), the paper might understate the downside: performance gains appear to hinge in part on decoder adaptability, not just the quality of the encoder's latents. The extent to which this is a fundamental limitation (versus an optimization artifact) is not fully explored.\n\n- Potential Over-Claims on \"Semantics Distillation Independence\": The abstract and body at times appear to overemphasize $l$-DeTok’s superiority or independence compared to semantics distillation. However, Section A.4 shows that adding semantics distillation to $l$-DeTok further improves performance—especially for non-AR models—sometimes even surpassing pure denoising. This suggests the two are complementary rather than exclusive. The paper should be more circumspect in presenting $l$-DeTok as a replacement, rather than a supplement, to semantics-based approaches.\n\n- The evaluation should be benchmarked against more powerful and contemporary VAEs, such as the SD3-VAE or Flux-VAE. The currently used SD-VAE is an outdated and underperforming baseline due to its 4-channel latent space. A more meaningful and fair comparison would be against a modern 16-channel VAE. This is a crucial point, as we have observed a concerning trend in recent tokenizer research where comparisons are made against the old SD-VAE to inflate perceived performance gains. Such a comparison is not a fair assessment of the proposed method's true capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f8rVKJILrS", "reviewer": "Reviewer_BHYG", "summary": "This paper proposes a continuous VAE tokenizer that incorporates a denoising prior. The core idea is to corrupt the latent representations by injecting Gaussian noise or applying masking, and then training the tokenizer to reconstruct the original, uncorrupted latents. This design aims to address the discrepancy between the objectives of tokenizer training and subsequent image generative modeling, thereby improving the \"denoisability\" of the VAE latent space. The authors provide extensive experiments that effectively validate the proposed denoising tokenizer. Additionally, the paper presents several interesting empirical findings, including comparisons between random and fixed masking, as well as interpolative versus additive noise.", "questions": "see weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-motivated, addressing the critical challenge of aligning the training objectives of visual tokenizers and generative models.\n- The proposed method is simple yet effective. The strategy of injecting interpolative or masking noise is conceptually sound and well-justified.\n- The methodology and implementation details are presented with clarity, making the work easy to understand and reproduce.\n- The experimental evaluation is extensive and well-structured, providing strong empirical support for the paper's claims.", "weaknesses": "- Convergence and scalability: A potential concern is the training convergence. While the denoising objective complements the pixel-reconstruction loss, it is plausible that learning to reconstruct from corrupted latents could slow down convergence compared to a vanilla baseline. It would be beneficial for the authors to provide an analysis of the training speed and computational overhead. Furthermore, a discussion on the scalability of the proposed method to larger models and datasets would strengthen the paper.\n\n- Distribution of noise level: The paper appears to use a uniform distribution for the noise level factor \\tau. Drawing inspiration from recent diffusion-based methods (e.g., SD3), which have shown that non-uniform timestep sampling can improve performance, it would be interesting to investigate whether a non-uniform sampling strategy for \\tau could offer similar benefits for the proposed tokenizer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8evOX6l0vS", "reviewer": "Reviewer_gSib", "summary": "The paper proposes Latent Denoising Tokenizer (l-DeTok), a visual tokenizer trained to reconstruct clean images from corrupted latent embeddings using interpolative Gaussian noise and random masking during tokenizer training. The authors demonstrate that it leads to consistent gains for both non-autoregressive and autoregressive models. Extensive experiments on ImageNet and MS-COCO across two resolutions support the claim that a simple, intuition-driven denoising objective can yield broadly useful tokenizer embeddings that transfer across generative paradigms.", "questions": "1. Discrete extension of the method.\n- Given that the corruption scheme includes random masking, which is naturally defined in discrete token spaces (masking or replacing code indices), how well would the approach extend to discrete tokenizers such as VQ or RVQ? A brief discussion of potential pitfalls, such as codebook collapse, corruption schedule design and any small-scale evidence may extend the scope of the contribution.\n\n\n2. Orthogonality to aggressive compression and compact regimes.\n- Recent work indicates that aggressive latent compression, such as approximately 32 tokens or adaptive-length tokenization, can improve both quality and efficiency. Is the proposed denoising-aligned training orthogonal to these compression strategies, and does combining them yield further gains? In particular, beyond the reported configuration with patch size 16 and latent dimension 16, are there results at more compact representation?\n\nReferences\n- Yu et al., 2024. An Image is Worth 32 Tokens for Reconstruction and Generation\n- Duggal et al., 2025. Adaptive Length Image Tokenization via Recurrent Allocation", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Generalizes across AR and non-AR generators.\n- The same tokenizer improves both diffusion-based (non-AR) and autoregressive models without architectural changes, indicating that the denoising-aligned latent space is broadly compatible with diverse generation mechanisms.\n\n2. Simple and practical method such that no external encoder alignment or semantic distillation required.\n- While recent approaches emphasize semantics distillation from powerful pretrained vision models, l-DeTok shows that a lightweight, corruption-robust training objective can match (or surpass) such methods without explicitly aligning to external encoders.\n\n3. Drop-in usability and clear training signal.\n- The method is easy to implement, and the paper provides a clear heuristic that practitioners can readily apply.", "weaknesses": "Regarding with Related work, please add the following references.\n- Zhao et al., ε-VAE: Denoising as Visual Decoding.\n- Tschannen et al., Generative Infinite-Vocabulary Transformers.\n- Kim et al., Efficient Generative Modeling with Residual Vector Quantization-Based Tokens.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "77Fgq7VXyp", "reviewer": "Reviewer_pDBk", "summary": "This paper focuses on training more effective visual tokenizers for subsequent generative modeling. Motivated by the shared denoising nature of current generative models, it incorporates the denoising objective into tokenizer training to align its embedding with downstream generative models. Specifically, this work introduces Latent Denoising Tokenizer (l-DeTok), a simple yet highly effective tokenizer trained to reconstruct clean images from latent embeddings corrupted via interpolative noise or random masking. Comprehensive experiments on various settings and types of generative models validate the effectiveness and generalization of the trained tokenizers.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper is well-motivated and easy to follow. It aligns tokenizer embeddings with the downstream denoising objective in generative modeling.\n2. The writing of this paper is clear, and the expressions are polished and elegant. I appreciate this point.\n3. The proposed method and spirit are general and robust, which is applicable to a wide spectrum of generative models.\n4. The paper provides comprehensive experiments and ablations to validate the superiority of the proposed method.", "weaknesses": "1. How much additional computational overhead does the proposed denoising mechanism introduce compared with conventional tokenizer training? Detailed comparisons should be presented.\n2. How do the performance gains vary across various tokenizer and generative model sizes? (results with different sizes of both tokenizer and generators)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)