Diffusion Transformers with Representation Autoencoders
Authors: Boyang Zheng
Nanye Ma
Shengbang Tong
Saining Xie
Venue: ICLR 2026 Poster (2026)
Keywords: Generative Models
Diffusion Models
Representation Learning
High-dimensional Diffusion
Primary area: generative models
TL;DR: Pretrained representation encoder as autoencoder for diffusion models

Abstract: Latent generative modeling has become the standard strategy for Diffusion Transformers (DiTs), but the autoencoder has barely evolved. Most DiTs still use the legacy VAE encoder, which introduces several limitations: large convolutional backbones that compromise architectural simplicity, low-dimensional latent spaces that restrict information capacity, and weak representations resulting from purely reconstruction-based training. In this work, we investigate replacing the VAE encoder–decoder with pretrained representation encoders (e.g., DINO, SigLIP, MAE) combined with trained decoders, forming what we call \emph{Representation Autoencoders} (RAEs). These models provide both high-quality reconstructions and semantically rich latent spaces, while allowing for a scalable transformer-based architecture. A key challenge arises in enabling diffusion transformers to operate effectively within these high-dimensional representations. We analyze the sources of this difficulty, propose theoretically motivated solutions, and validate them empirically. Our approach achieves faster convergence without auxiliary representation alignment losses. Using a DiT variant with a lightweight wide DDT-head, we demonstrate state-of-the-art image generation performance, reaching FIDs of 1.18 @256 resolution and 1.13 @512 on ImageNet.

Preprint
DIFFUSION TRANSFORMERS WITH REPRESENTATION AUTOENCODERS
Boyang Zheng Nanye Ma New York University
Shenghang Tong Saining Xie
ABSTRACT
Latent generative modeling, where a pretrained autoencoder maps pixels into a la- tent space for the diffusion process, has become the standard strategy for Diffusion Transformers (DiT); however, the autoencoder component has barely evolved. Most DiT's continue to rely on the original VAE encoder, which introduces several limitations: outdated backbones that compromise architectural simplicity, low- dimensional latent spaces that restrict information capacity, and weak represen- tations that result from purely reconstruction-bee based training and ultimately limit generative quality. In this work, we explore replacing the VAE with pretrained representation encoders (e.g.. DINO, SigLIP, MAE) paired with trained decoders, forming what we term Representation Autoencoders (RAFs). These models pro- vide both high-quality reconstructions and semantically rich latent spaces, while allowing for scalable transformer-basee architecture. Since these latent spaces are typically high-dimensional, key challenge is enabling diffusion transform- ers to operate effectively within them. We analyze the sources of this difficulty, propose theoretically motivated solutions, and validate them empirically. Our approach achieves faster convergence without auxiliary representation alignment losses. Using a DiT variant equipped with a lightweight, wide DDT head, we achieve strong image generation results on ImageNet: 1.51 FID at 256 x 256 (no guidance) and 1.13 at both 256 256 and 512 x 512 (with guidance). RAE offers clear advantages and should be the new default for diffusion transformer training.
30
VVQ-VA12
VAF-looad (prist work)
RAF-cool (nurs)
20-
VgGGAN
I00PM
10.
DT
8:
FID
MalDDT
nPT
RUTA-I
DCT
DTP
IN
2022
2023
2025
Figure 1: Representation Autoencoder (RAH) uses frozen pretrained representations as the encoder with a lightweight decoder to reconstruct input images without compression. RAE enables faster convergence and higher-quality samples in latent diffusion training compared to VAE-based models.
INTRODUCTION
The evolution of generative modeling has been driven by continual redefinition of where and how models learn to represent data. Early pixel-space models sought to directly capture image statistics, but the emergence of latent diffusiin (Vahdat et al., 2021; Rombach et al., 2022) reframed generation as a process operating within learned, compact representation space. By diffusing in this space rather than in raw pixels, models such Latent Diffusion Models (LDM) (Rombach et al., 2022) and Diffusion Transformers (DiT) (Peebles & Xie, 2023; Ma et al., 2024) achieve higher visual fidelity and efficiency, powering the most capable image and video generators of today.
Despite progress in diffusion backbones, the autoencoder defining the latent space remains largely unchanged. The widely used SD-VAE (Rombach et al., 2022) still relies on heavy channel- compression and a reconstruction-only objective, producing low-capacity latents that capture local appearance but lack global semantic structure crucial for generalization and generative performance of diffusion models (Song et al., 2025). In addition, SD- VAE, built on a legacy convolutional design, remains computationally inefficient (see Fig. 2).
Meanwhile, visual representation learning has undergone a rapid transformation. Self-supervised and multimodal encoders such as DINO (Oquab al., 2023), MAE (He et al., 2021). JEPA (Assran

---

Preprint
DiT w/ VAE
DiT w/ RAE
L-Net B1 Flock
D
M||
BLoCx 11A
Diffusion Trans former
Diffusion Transformes
Decoder
Encodex 22orrope
Decodex Lops
Encoder
310 Loss
135sttops
Figure 2: Comparison of SD-VAE and RAE (DINOv2-B). The VAE relies on convolutional backbones with aggressive down- and up-sampling, while the RAE uses a ViT architecture without compres- sion. SD-VAY ID-VAE is also more computationally expensive, requiring about 6x and 3x more GFLOPs than RAE for the encoder and decoder, respectively. GFlops are evaluated on one 256 256 image.
et al.. 2023) and CLIP SigLIP (Radford et al., 2021; Tschannen et al., 2025) learn semantically structured visual features that generalize across tasks and scales and provide natural basis for visual understanding. However, latent diffusion remains largely isolated from this progress, continuing to diffuse in reconstruction-traine
VAE spaces rather than semantically meaningful representational ones. Recent work attempts to improve latent quality indirectly through REPA-style (Yu et al., 2025; Yao et al., 2025; Leng et al., 2025) alignment with external encoders, but these methods introduce extra training stages, auxiliary losses, and tuning complexity.
This separation stems from long-standing assumptions about the incompatibility between seman- tic and generative objectives. It is widely believed that encoders trained to capture semantics are unsuited for faithful reconstruction, since they "focus on high-level information and can only construct an image with high-level semantic similarities." " (Yu et al., 2024b) In addition, diffusion models are believed to perform poorly in nigh-dimensional latent spaces (Skorokhodov al., 2025; Yao et al., 2025; Esser et al., 2024; Liu et al., 2024), leading practitioners to favor low-dimensional VAE latents over the typically much higher- digee--dimensional representations of semantic encoders. In this work, we show that both assumptions might be wrong. We demonstrate that frozen representa- tion encoders, even those explicitly optimized for semantics over reconstruction, can be repurposed into powerful autoencoders for generation, yielding reconstructions superior to VAR without ar- chitectural complexity or auxiliary losses. Furthermore, we find that diffusion transformer training can be stable and efficient in these dimee-diieesiinall latent spaces. With the right architectural adjustments, highee-dimensional representations are not a liability but an advantage, offering richer structure, faster convergence, and better generation quality. Notably, higher- lighen-dimensionall latents in- troduce effectively no extra compute or memory costs since the token count is fixed (determined by the patch size) and the channels are projected to the DiT hidden dimension in the first layer.
We formalize this insight through Representation Autoencoders (RAFs), a new class of autoencoders that replace the VAE with pretrained representation encoder (e.g., DINO) paired with trained decoder. RAEs produce latent spaces that are semantically rich, structurally coherent, and diffission- friendly, linking semantic and generative modeling through shared latent representation.
While feasible in principle, adapting diffusion transformers to these high-dimensional semantic la- tents requires careful design. Original DiTs, designed for compact SD-VAEs, struggle with the increased dimensionality due to: (1) Transformer design: DiTs cannot fit even a single image un- less their width exceeds the token dimension, implying width must scale with latent dimensionality; (2) Noise scheduling: resolution based schedule shifts (Chen, 2023; Hoogeboom et al., 2023; Esser et al., 2024), derived from pixel- and VAE-based inputs, neglect token dimensionality, motivating a cimeesionnaliy-deeeddee
shift; (3) Decoder robustness: unlike VAEs trained on continuous latent distributions (Kingma & Welling, 2014), RAE decoders learn from discretely supported latents but must reconstruct samples from diffusion model that follow a continuous distribution, which address by noise-augmented decoder training. Finally, we introduce a new DiT variant, DIT in- spired by DDT ( (Wang et al., 2025c) but motivated by a different design perspective. It augments the standard DiT architecture with a lightweight, shallow yet wide head, enabling the diffusion model to scale in width without incurring quadratic computational costs. Empirically, this design further enhances diffusion transformer training in high-dimensional RAE spaces.
Empirically, RAEs demonstrate strong visual generation performance. On ImageNet, our RAE- based DH achieves FIDs of 1.51 at 256x256 without any guidance, and 1.13 at both 256x256 and 512x512 with AutoGuidance (Karras et al., 2025), showing the efffcttivneess of RAEs an alter- native to conventional VAES in diffusion transformer training. More broadly, these results reframes autoencoding from compression mechanism into representation foundation, one that enables diffusion transformers to train more efficiently and generate more effectively.
2

---

Preprint
Model
IFDD
Decoder
#FDD
GFLOP
Encoder #F
Model
Top-1 Acc.
DINOv2-B
0.49
ViT-B
0.58
22.2
DINOv2-S 0.52
DINOv2-B
84,5
SigLIP2-B
0.53
VIH
0.50
78.1
DINOv2-B
0.49
SigLIP2-B
79.1
MAF-B
0.16
WTFLL
0.49
106.7
DINOv2-L
0.52
MAF-B
68.0
SD-VAL VAE
0.62
SD-VAB
0.62
310.4
SD-VAE
8.0
Encoder choice. All
(h) Larger decoders improve
(c) Encoder scaling. IFI is stable across RAE sizes.
Representation quality. RATS have much higher linear probing accuracy than VAEs.
coders outperform VAE.
(FID while remaining much
more eflicient than VAEs.
Table 1:
RAEs consistently outperform SD-VAE VAE in reconstruction (rFID) and representation quality (linear probing accuracy) on ImageNet-IK, while being more efficient. If not specified, we use ViT- XI. as the decoder and DINOv2-B as the encoder for RAE. Default settings in this paper are in gray.
2 HIGH FIDELITY RECONSTRUCTION FROM FROZEN ENCODERS
In this section, we challenge the common assumption that pretrained representation encoders, such as DINOv2 (Oquab et al., 2023) and SigLIP2 (Tschannen et al., 2025). are unsuitable for the re- construction task because they "emphasize high-level semantics while downplaying low-level de- tails" (Tang et al., 2025; Yu et al., 2024b). We show that, with a properly trained decoder, frozen representation encoders can in fact serve strong encoders for the diffusion latent space. Our Representation Autoencoders (RAE) pair frozen, pretrained representation encoders with a ViT- based decoder, yielding reconstructions on par with or even better than SD-VAE. More importantly, RAES alleviate the fundamental limitations of VAEs (Kingma & Welling, 2014), whose heavily compressed latent space (e.g. SD-VAE maps 256² images to 32² (Esser et al., 2021; Rombach et al., 2022) latents) restricts reconstruction fidelity and more importantly, representation quality.
Our training recipe for the RAE decoder is as follows. Given an input x E Rx3*H* and the frozen representation encoder E with patch size Pe and hidden size d, we obtain N = HW/p2 tokens with channel d. A ViT decoder D) with patch size pa maps them back to pixels with shape 3 x H Rs x W Rs. pe By default we use Pa Pe. the reconstruction matches the input resolution. For Be.
all experiments on 256 x 256 images, the encoder produces 256 tokens, matching the token count of most prior DiT based models trained with SD-VAE latents (Peebles Xie, 2023; Yu et al., 2025; Ma et al., 2024). The decoder D is trained with combination of L1, LPIPS (Zhang et al., 2018), and adversarial losses (Goodfellow et al., 2014), 4), following common practice in VAFs:
Exx), f = D(2)
Crec(x) WL LPIPS( LL x)) + WG GGN(AN(E,
We provide implementation details about the decoder architecture, hyperparameters, coefficients, and GAN training details in Appendix C.
We select three representative encoders from different pretraining paradigms: DINOv2-B (Oquab et al., 2023) (p.=14, d=768), self-supervised self-distillation model; SigLIP2-B (Tschannen et al., 2025) (p.=16, d=768), sunguage-supervise masked autoencoder. For DINOv2, we also study different model sizes S.B.L. (d=384,,68.1024. Un- less otherwise specified, we use an VIT-XL. decoder for all RAES. We use FID score (Heusel et al., 2017) computed on the reconstructed ImageNet (Russakovsky et al., 2015) validation set our main metric for reconstruction quality, denoted as rFID.
model; and MAE-B (He et al., 2021) (p.=16, d=768), a
Reconstruction, scaling, and representation. As shown in Table Ia, RAEs with frozen encoders achieve consistently better reconstruction quality (rFID) than SD-VAE. For instance, RAE with MAE-B/16 reaches an rFID of 0.16, clearly outperforming SD-VAE and challenging the assumption that representation encoders cannot recover pixel-level detail.
We next study the scaling behavior of both encoders and decoders. As shown in Table Ic, reconstruc- tion quality remains stable across DINOv2-S, B, and I., indicating that even small representation encoders models preserve sufficient low-level detail for decoding. On the decoder side (Table lb), increasing capacity consistently improves rFID; from 0.58 with ViT-B to 0.49 with ViT-XL. Impor- tantly, ViT-B already outperforms SD-VAE while being 14x more efficient in GFLOPs, and VIT-XL further improves quality at only one-third of SD- VAE's cost.
We also evaluate representation quality via linear probing ImageNet- 1K in Table 1d. Because RAES use frozen pretrained encoders, they directly inherit the representation of the underlying rep- resentation encoders. In contrast, SD-VAE achieves only ~8% accuracy.
3

---

Preprint
Nay Tated Dae 766
Seby Lutel Dee
0.3
rust
Tassection
0.5
Olarrved
0.3-
diffision succesatilly overitt
0,3
0.2-
fayers imaze!
0.1
0.1
still can't oveniit ove'iveriia
0.0
differion can't even evartis engle ampilc
0.0
Obbrrved
354
512
640
768
800
16 DTT Depth (4)
24
DIT Width (d)
Figure 3: Overfitting a single sample. Left: increasing model width lead to lower loss and better sample quality; Right: changing model depth has marginal effect on overfitting results.
TAMING DIFFUSION TRANSFORMERS FOR RAE
With RAE demonstrating good reconstruction quality, we now proceed to investigate the diffusabil- ity (Skorokhodov al., 2025) of its latent space; that is, how easily its latent distribution can be modeled by diffusion model, and how good the generation performance can be.
Before turning to generation, we fix the encoder to study generation capabilities. Table la shows that MAE, SigI.I22, and DINOv2 all achieve lower reconstruction rFID than SD-VAE, with MAE the best among them. However, reconstruction alone does not determine generation quality. Em- pirically, DINOv2 produces the strongest generation results, so unless otherwise noted, adopt DINOv2 as our default encoder and defer the full comparison to Appendix G.1.
Following standard practice, adopt the flow matching objective (Lipman et al., 2023; Liu al., 2023) with linear interpolation X2 = (1 - t)x + tE, where x ~ p(x) and & ~ -N I), and train the model to predict the velocity vxx,, t) (see Appendix J). We use LightningDiT (Yao et al., 2025), a variant of DiT (Peebles & Xie, 2023), as our model backbone. We adopt a patch size of 1, which results in sequence length of 256 for all RAEs on 256 256 images, matching the token length used by VAE-based DiTs (Peebles & Xie, 2023; Yu et al., 2025; Yao et al., 2025). Since the computational cost of DiT depends primarily on the sequence length, using DiT on RAE therefore effectively incurs no additional overhead compared to its VAE-based counterparts. We evaluate our models using FID computed on 50K samples generated with 50 steps with the Euler sampler (denoted as gFID), and all quantitative results are trained for 80 training epochs ImageNet at 256 256 unless otherwise specified. More training details are included in Appendix D.
DIT does not work out of the box. To our surprise, the stan- dard diffusion recipe fails with RAE (see Table 2). Training directly on RAE latents causes small backbone such as DiT-S to completely fail, while a larger backbone like DiT-XL signifi- cantly underperforms it's counterpart with the SD-VAE latents.
RAE
SD- VAE
DiT-S DIT-XL
215.76
51.74
23.08
7.13
To investigate this observation, we raise several hypotheses de- tailed below, which we will discuss in the following sections:
Table 2: Standard DIT struggles to model RAE's latent distribution.
Suboptimal design for diffusion transformers. When modeling high-dimensional RAE tokens, the optimal design choices for diffusion transformers can diverge from those of the standard DiT, which was originally tailored for low- www-dmensiioal VAE tokens.
Suboptimal noise scheduling. Prior noise scheduling and loss re-weighting tricks are derived for image-based or VAE-based input, and remains unclear if they transfer well to high-dimension semantic tokens.
Diffusson generates noisy latents. VAE decoders are trained to reconstruct images from noisy latents, making them more tolerant to small noises in diffusion outputs. In contrast, RAE decoders are trained on only clean latents and may therefore struggle to generalize.
3.1 ScaLiNG DIT WIDTH TO MATCH TOKEN DIMENSIONALITY
To better understand the training dynamics of diffusion transformers with RAE latents, first construct a simplified experiment. Rather than training on the entire ImageNet, we randomly select single image, encode it by RAE, and test whether the diffusion model can reconstruct it.
4

---

Preprint
Table 2 shows that although RAE underperforms SD-VAE, DiT performance improves with in- creased capacity. To dissect this effect, we vary model width while fixing depth. Using a DiT-S, we increase the hidden dimension from 384 to 784. As shown in Figure 3, sample quality is poor when the model width d < token dimension = 768, but improves sharply and reproduces the input almost perfectly once d 2 Training losses exhibit the same trend, converging only when d 2
One might suspect that this improvement still arises from the larger model capacity. To disentangle this effect, fix the width at d 384 and vary the depth of the SiT-S model. As shown in Figure 3, even when doubling the depth from 12 to 24, the generated images remain artifact-heavy, and the training losses shown in Figure 3 fail to converge to similar level of d 768.
Together, the results indicate that for generation in RAE's latent space to succeed, the diffusion model's width must match or exceed the RAE's token dimension. At first glance, this appears to contradict the common belief that data manifolds have low intrinsic dimensionality (Pope et al., 2021), allowing generative models such as GANs operate effectively within that manifold without scaling to the full data dimension (Sauer et al., 2022). We argue that this contradiction arises from the formulation diffusion models (Appendix J): injecting Gaussian noise directly to the data (e.g., in the construction of xx) throughout training effectively extends the data distribution's 's support to the entire space, thereby "diffusing" the data manifold into a full-rank one, requiring model capacity that scales with the full data dimensionality.
In the following, we provide a theoretical justification for this conjecture:
Theorem 1. Assuming p(x) IR" (N(O, I,), 1 j0, 1]. Let x, =1 1)x + le, consider the function family
Go
A A B E B ff | 10,1]
(1)
where d < n,, J refers to stack of standard DiT blocks whose width is smaller than the token dimension from the representation encoder, and A, denote the imput and output linear projections, respectively. Then any Gd.
C(g.0) 0) = Ex.pptshew [1lg(x,,) - ---xx)ll]ad
Σ i=d+1
A
(2)
where X are the eigenvalues of the covariance matrix of the random variable W = = Notably, when d n,, Ga contains the unique minimizer Lig, 0).
Proof. See Appendix B.1.
In our toy setting where p(x) = 5(x - -x)), we have W ~ (- In) and X = 1 for all i. Thus by Theorem 1, the lower bound of the average loss becomes L(g, 0) 2 1 >x 2ind41 1 n-d As shown
in Figure 3, this theoretical bound is consistent with our empirical results.
We further extend our investigation to a more practi- cal setting by examining three models of varying width, (DiT-S, DiT-B, DiT-L}. Each model is overfit on single image encoded by (DINOv2-S. DINOv2-B, DINOv2-L), respectively, corresponding to different shown in Section 3.1, conver-
DiT-S
DIT-B
DIT-L
DINOv2-S 3.6e-2 / 1.0e-
9.7e-4V
DINOv2-B 5.2e-1 X 2.4e
1.3e-34
DINOv2-L. 6.5e-1 2.7e 1 2.2e-2 22-24
token dimensions. gence occurs only when the model width is at least as large as the token dimension (e.g., DiT-B with DINOv2-B), while the loss fails to converge otherwise (e.g.. DiT-S with DINOv2-B).
Table 3: Overfitting losses. Compared be- tween different combinations of model width and token dimension.
Suboptimal design for diffusion transformers. We now fix the width of DiT to be at least as large as the RAE tkkn dimension. For RAE with the DINOv2-B encoder, we pair it with DiT-XL. in our following experiments.
3.2 DIMENSION-DEPENDERR NOISE SCHEDULE SHIFT
Many prior works (Teng et al., 2023: Chen, 2023; Hoogeboom et al., 2023; Esser et al., 2024) have observed that, for inputs z E mation corruption at the same noise level, impairing diffusion training. These findings, however,
increasing the spatial resolution (H x W) reduces infor-
5

---

Preprint
based mainly on pixel- VAE-encoded inputs with few channels (e.g.. C 16). In practice, the Gaussian noise applied to both spatial and channel dimensions; as the number of channels in- creases, the effective "resolution" per token also grows, reducing information corruption further. We therefore argue that proposed resolution-depeedent strategies these prior works should be gener- alized to the effective data dimension, defined the number of tokens times their dimensionality.
We adopt the shifting strategy of Esser et al. (2024): for a sched- ule E [0, 1] and input dimensions nI, m, the shifted timestep is defined as fm T- where a / /m/n is a dimension- dependent scaling factor. We follow (Esser et al., 2024) in using 4096 the base dimension and set to the effective data dimension of RAE. As shown in Table 4, this yields significant performance gains, showing its importance for training diffusion models in the high-dimensional RAE latent space.
gFID
w/o shift
23. 4.81
w/ shift
Table Impact schedule shift.
Suboptimal noise scheduling. We default the noise schedule be dependent the effective data dimension for all our following experiments.
3.3
NOGSE-AUGMENTED DECODING
Unlike VAEs, where latent tokens encoded continuous distribution ([,,, o²1) (Kingma & Welling, 2014), the RAE decoder is trained to reconstruct images from the discrete distribution p(z) Σ, 8(x zz), where {z,} denotes the training set processed by the RAE encoder E. At inference time, however, the diffusion model may generate latents that are noisy or deviate slightly from the training distribution due to imperfect training and sampling Abuduweili al. (2024). This could introduce notable out-of-distribution challenge for D, which degrades sampling quality.
To mitigate this issue, inspired by prior works on Normalizing Flows (Dinh al., 2017; Ho et al., 2019; Zhai et al., 2025), augment the RAE decoder training with an additive noise (0, 2T). Concretely, rather than decoding directly from the clean latent distribution p(z), we enhance the decoder's generalization to the denser output space of diffusion models. We further introduce stochasticity into by sampling from [V 73) which helps regularize training and improve robustness.
train D smoothed distribution Pa(z) p(z (0, x (n)
We analyze how Pn (z) affects reconstruction and generation. As shown in Table 5, it improves gFID but slightly worsens rFID. This trade-off is expected: adding noise smooths the latent distribution and, therefore, helps reduce OOD issues for the decoder, but also removes fine details, lowering reconstruction quality. We conduct more ablation experiments on т and different encoders in Appendix G.2.
gFID
~ p(z)
4.81
0.49
Pa(z)
Table 5:
Impact of pa(z).
Diffussion generates noisy latents. We now adopt the noise-augmented decoding for all our following experiments.
We combine all the above techniques to train
FID-50K ImageNet @ 256
40
SIT-XL REPA-XL
DiT-XL model RAE latents. Our improved
diffusion recipe achieves a gFID of 4.28 (Fig- 4) after only 80 epochs and 2.39 after 720
20
(RAE: DINOv2-B)
epochs RAE's latent space. With same model size, this not only surpasses prior diffusion base-
8.26
Taster
lines (c.g., SiT-XL (Ma et al., 2024)) trained on VAE latents (achieving a 47x training speedup), but also outperforms the convergence speed of re- cent methods based on representation alignment (e.g., REPA-XL (Yu et al., 2025)), achieving a 16x training speedup. In the following sections, investigate ways to make RAE generation more efficient and effective, pushing toward state-of-the-ar -hrt---rtttttttte performance.
5.90
16x faster
4.28
2.39
20
80
400
800 1400
Training Epochs
Figure DIT RAE reaches much faster con- vergence and better FID than SIT or REPA.

---

Preprint
DiTCILsS
6.0
RAE: DINOV2-8
DITRM
DIT-E
REPA
DDT
BDD -maggNet @ 238
5.5
DITXL
FID-SOK - Ineughhi CAA
20
REG
5.0
nEnWAL
FID (DD-50K ImageNal @
4.5
DITXL
HDTVVOL
10
DI
MAL OPTAAIA
3.5
DITPLLB
3.0
DITPM
2.5
2.0
x 10
10tt
5x
Companisee ison of Sealing Different Diffusion Model
Diffuion GFLOPs
Difusioo GFLOPs
(a)
DIT scales much better DIT with RAE latents.
DIT converges faster than VAE-based methods.
better FID VAE-bused at all model seales. Bubble area indicates the flops of the model.
DiT
Figure 6: Scalability of DiT With RAE latents, DiT scales more efficiently in both training compute and model size than RAE-based DiT and VAE-based methods.
IMPROVING THE MODEL SCALABILITY WITH WIDE DIFFUSION HEAD
As discussed in Section within the standard DiT framework, handling higher-dimensional RAE latents requires scaling up the width of the entire backbone, which quickly becomes computation- ally expensive. To overcome this limitation, we draw inspiration from DDT (Wang al., 2025c) and introduce the DDT head-a shallow yet wide transformer module dedicated to denoising. By at- taching this head to a standard DIT, we effectively increase model width without incurring quadratic growth in FLOPs. We refer to this augmented architecture DiT throughout the remainder of the paper. We also conduct experiment of the design choice of DDT head in Appendix G.3
Wide DDT head. Formally, DiT model consists of a base DiT M and an additional wide, shallow transformer head H. Given noisy input zz, timestep and an optional class label y. the combined model predicts the velocity 82 as
DDT Head
DiT
Vt
M (xg v), H
Figure 5: The Wide DDT Head.
0t
DH converges faster than DIT. We train a series of DIT models with varying backbone sizes DH -s. B. L. and XL) on RAE latents. We use a 2-layer, 2048-dim DDT head for all DIT models. Performance is compared against the standard DÍT-XI. baseline. As shown in Figure 6a, DIT is substantially more FLOP-efficient than DIT. For example, DiT requires only ~40% of the training FLOPs yet outperforms DiT by large margin; when scaled to DiTPH_ -XL under comparable training budget, DH achieves an FID of 2.16 2.16-nearly half that of DIT-XL.
DH maintains its advantage across RAE scales. We com- pare DH_ and DIT-XI. on three RAE encoders-DINOv2.. S, DINOv2-B, and DINOv2-L. As shown in Section 4, DH consistently outperforms DIT, and the advantage grows with coder size. For example, with DINOv2-L, DH improves FID from 6.09 to 2.73. We attribute this robustness to the DDT head. Larger encoders produce higher-dimensional latents, which am- plify the width bottleneck of DiT. DII addresses this by satis- fying the width requirement discussed Section 3 while keeping features compact. It also filters out noisy information that be- comes more prevalent in high-dimensional RAE latents.
DINOv2
Model
S
DIT-XL
3.50
4.28
6.09
DH
XL
2.42
2.16
2,73
Table 6: DH outperforms DiT across RAE encoder sizes.
4.1 STATE-OF-THE-ARR DIFFUSION TRANSFORMERS
Convergence. We compare the convergence behavior of DH _XL with previous state-of-the. art diffusion models (Peebles & Xie, 2023; Ma al., 2024; Yu al., 2025; Gao al,, 2023; Yao

---

Preprint
Generation guidance
Generation w/ guidunce
Method
Epochs
#arams
gFID!
18
Prec."
Rec.
gFID,
18-
Prec.t
Rec.
Audoregressive
VAR (Tian al., 2024)
350
2.0B
1.92
323.1
0.82
0.59
1.73
350.2
0.82
0.60
2024b)
800
943M
2.35
227.8
0.79
0.62
1.55
0.81
0.62
xAR (Ren al., 2025)
800
1.1B
1.24
301.6
0.83
0.64
Pixel Diffusion
ADM (Dhariwal & Nichol, 2021)
400
554M
10.94
101.0
0.69
0.63
3.94
215.8
0.83
0.53
RIN (Jahri et al., 2023)
480
410M
3.42
182.0
Fixe (Chen et al., 2025e)
320
677M
1.98
282.1
0.81
0.60
PixNerd (Wang et al., 2025b)
160
700M
2.15
297.0
0.79
0.59
SID2 (Hoogehoom et al., 2025)
1280
1.38
Latent Diffusion with VAE
DiT (Peebles & Xie, 2023)
1400
675M
9.62
121.5
0.67
0.67
2.27
278.2
0.83
0.57
MaskDiT (Zheng et al.)
1600
675M
5.69
177.9
0.74
0.60
2.28
276.6
0.80
0.61
(Ma al., 2024)
1400
675M
0.67
2.06
0.59
MDTv2 (Gan al., 2023)
1080
675M
1.58
314.7
0.79
0.65
80
4.29
VA-VAE (Yao al., 2025)
675M
800
2.17
205.6
0.77
0.65
1.35
295.3
0.79
0.65
80
7.90
122.6
0.70
0.65
REPA (Yu ul., 2025)
675M
800
5.78
158.3
0,70
0.68
1.29
306.3
0.79
0.64
80
6.62
135.2
0.69
0.67
1.52
263.7
0.78
0.63
DDT (Wang et al., 2025c)
675M
400
6.27
154.7
0.68
0.69
1.26
310.6
0.79
0.65
80
3.46
159.8
0.77
0.63
1.67
266.3
0.80
0.63
REPA-E (Leng et al., 2025)
675M
800
1.70
217.3
0.77
0.66
1.15
304.0
0.79
0.66
Latent Diffusion with RAE (Ours)
DiT-XL (DINOv2-S)
800
676M
1.87
209.7
0.80
0.63
1.41
309.4
0.80
0.63
20
3.71
198.7
0.86
0.50
DIT (DINOv2-B)
80
839M
2.16
214.8
0.82
0.59
800
0.79
0.63
1.13
262.6
0.78
0.67
Table 8: Class-conditional performance on ImageNet 256x256. RAE reaches an FID of 1.51 without guidance, outperforming all prior methods by large margin. It also achieves an FID of 1.13 with AutoGuidance (Karras et al., 2025). We identified an inconsistency in the FID evaluation protocol in prior literature and re-ran the sampling process for several baselines. This resulted in higher baseline numbers than those originally reported. Further details are discussed in Section 4.1.
et al., 2025) in terms of FID without guidance. In Figure 6b, we show the convergence curve of DH -XL with training epuchs/GFLOOs, while baseline models are plotted at their reported final performance. Dir already surpasses REPA-XL, MDTv2-XL, and SiT-XL around x 1000 GFLOPs, and by x " GFLOPs it achieves the best FID overall, requiring over 40x less compute. Scaling. We compare DiT with recent methods of different model at different scales. As shown in Figure 6c, inereasing the size of DiT con- sistently improves the FID scores. The small- est model, DS reaches a competitive FID of 6.07, already outperforming the much larger REPA-XI. When scaling from DiTDH-S to OH B. the FID improves significantly from 6.07 to 3.38, surpassing all prior works of similar even larger scale. The performance continues to im- prove with DiT setting new state-of-the- art result of 2.16 at 80 training epochs.
Generation
Method
gEID..
18
Prec.
Rec."
BigGANNddepp (Brock et 2019)
8.43
177.9
0.88
0.29
StyleGAN XL (Smee et 2022)
2.41
267.8
0.77
0.52
VAR (Tian et al., 2024)
2.63
303.2
MAGVIT-22
2024a)
XAR (Ren 2025)
1.70
281.5
ADM
3.85
221.7
0.84
0.53
SiD2
1,50
DIT
3.04
240.8
0.84
0.54
SIT
2.62
252.2
0.84
0.57
DifiT (Hfaraizoeee al., 2024)
2.67
252.1
0.83
0.55
REPA
Performance. Finally, we provide a quantitative comparison between DII XLL, our most perfor- mant model, with recent state-of-the-art diffusion models on ImageNet 256 x 256 and 512 x 512 in Table 8 and Table 7. Our method outperforms all prior diffusion models by a large margin, set- ting new state-of-the-art FID scores of 1.51 with-
DDT
1.28
305.1
0.80
0.63
EDM2 (Karms et al., 2024)
1.25
Di XI. (DINOv2-B)
1.13
259.6
0.80
0.63
Table Class-conditional performance on Ima- geNet 512x512 with guidance. DiT with 400- epoch training achicves an strong FID score of 1.13.
8

---

Preprint
Figure 7: Qualitative samples from our model trained at 512 x 512 resolution with AutoGuidance. The RAE-based DiT demonstrates strong diversity, fine-grained detail, and high visual quality.
out guidance and 1.13 with guidance at 256 x 256. On 512 x 512, with 400-epoch training, DiT further achieves an FID 1.13 with guidance, surpassing the previous best performance achieved by EDM-2 (1.25).
Following the quantitative results, we present qualitative samples from our best models Figure 7. These visualizations exhibit both high semantic diversity and fine-grained details comparable to ground-truth ImageNet samples, consistent with the achieved state-of-the-ar FID, We provide addi- tional visualization samples in Appendix N and unconditional generation results in Appendix L.
Remarks on FID evaluation. To construct the 50,000 samples used for conditional FID evaluation, note that previous works such as DDT (Wang al., 2025c), VAR (Tian et al., 2024), MAR (Li et al., 2024b), and xAR (Ren et al., 2025) typically evaluate using exactly 50 images per class, while others employ uniform random sampling across the 1,000 class labels. Although uniform random sampling asymptotically approaches the balanced sampling case, we surprisingly observed that blaanebalanced sampling consistently achieves around 0.1 lower FID scores. To ensure fair com- parison, we re-evaluate several recent methods with accessible checkpoints, such as SiT (Peebles & Xie, 2023), REPA (Yu al., 2025), REPA-E (Leng al., 2025) using bass-balanced sampling and update their reported scores accordingly. A more detailed comparison is provided in Appendix E.
DISCUSSIONS
5.1 How CAN RAE EXTEND TO HIGH-RESOLUTION SYNTHESIS EFFICIENTLY?
A central challenge in generating high-resolution images that resolution scales with the number of tokens: doubling image size in each dimension requires roughly four times as many tokens. To address this, let the decoder handle resolution scaling by allowing its patch size patch size Pa to differ from the encoder patch size Pe- When Pa Pe, the output matches the input resolution; setting Pa 2p. produces 2x upsampled image, reconstructing a 512 512 image from the same tokens used at 256 x 256.
Since the decoder is decoupled from both the en- coder and the diffusion process, we can reuse dif- fusion models trained at 256 256 resolution, sim- ply swapping in an upsampling decoder to produce 512 x 512 outputs without retraining. As shown in Table 9, this approach slightly increases rFID but achieves competitive gFID, while being 4x more ef- ficient than quadrupling the number of tokens.
Method
#Tokens
gFID 4
rFID
Direct
1024
1.13
0.53
Upsample
0.97
Table 9: Comparison on ImageNet 512 x 512. Decoder upsampling achieves competitive FID compared to direct 512-resolution training. Both models trained for 400 epochs.

---

Preprint
5.2 DOES DIT WORK WITHOUT RAE?
In this work, we propose and study RAE and DiT In Section 3, we showed that RAE with DiT already brings substantial benefits, even in the absence of DiT Here, we turn the question around: can DiT still provide improvements, without the latent space of RAE?
To investigate, we train both DiT-XL and DiT on SD-VAE latents with patch size of 2, alongside DINOv2-B for comparison, for 80 epochs, and report un- guided FID. As shown Table 10, DiT -LL performs even worse than DÍT-XL on SD-VAE, despite the addi- tional computation introduced by the diffusion head. This indicates that the DDT head provides little benefit in low- dimensional latent spaces, and its primary strength arises in high-dimensione diffusion tasks introduced by RAE.
VAE
DINOv2-B
DiT-XL
7.13
4.28
DIT
XI.
11.70
2.16
Table 10: Performance on VAE. DiT yields worse FID than DiT, despite using ex- tra compute for the wide DDT head.
5.3 THE ROLE OF STRUCTURED REPRESENTATION IN HIGH- DIN--EIIINSINN DIFFUSION?
Di achieves strong performance when paired with the high-dimensional latent space of RAE. This raises a key question: is the structured representation of RAE essential, or would DIT work equally well on unstructured high-dimensional inputs such as raw pixels?
To evaluate this, train DiT-XL and DiT _XI. directly on raw pixels. For 256 x 256 images with a patch size of 16, the resulting DiT input token dimensionality is 16 x 16 x 3 = 768, matching that of the DINOv2-B la- tents. We report unguided FID after 80 epochs. As shown in Table 11. DiT outperforms DiT on pixels, but both models perform far worse than their counterparts trained on RAE latents. These results demonstrate that high di- mensionality alone is not sufficient: the structured representation provided by RAE is crucial for achieving strong performance gains.
Pixel
DINOv2-B
DIT-XL DiT -XL
51.09
4.28
2.16
Table 11: Comparison on pixel diffusion. Pixel Diffusion has much worse FID than diffusion on DINOv2-B.
RELATED WORKS
Here, we discuss previous work on the line of representation learning and reconstruction/ggnerattoo.. We present a more detailed related work discussion in Appendix A.
Representation for Generation and Revonstruction. Recent work enhances generative mod- eling with semantic representations for both reconstruction and generation. For reconstruction, VA- VAE (Yao et al., 2025) aligns VAE latents with a pretrained representation encoder, while MAETok (Chen et al., 2025a), DC-AE 1.5 (Chen et al., 2025d), and 1-DEtok (Yang et al., 2025) incorporate MAE- or DAE (Vincent et al., 2008)-inspired objectives into VAE training, substan- tially improving reconstruction and generation quality, yet still relying on heavily compressed, low- dimensional latents. For generation, REPA (Yu et al., 2025) aligns intermediate DiT features with representation encoders representations, DDT (Wang et al., 2025c) decouples DiT into encoder- decoder with REPA-style supervision, REG (Wu et al., 2025) introduces learnable token aligned representation encoders representation, and ReDi (Kouzelis et al., 2025b) jointly generates VAE latents and PCA components of DINOV2 features within diffusion models. In contrast, we eliminate latent compression and auxiliary alignment: we reconstruct directly from representation encoders features using simple ViT decoder and train diffusion models directly in the representation en- coders space, achieving reconstruction quality comparable or better than SD-VAE (Rombach et al., 2022) while preserving stronger representations and enabling faster convergence.
ACKNOWLEGMENTS
The authors would like to thank Sihyun Yu, Jaskirat Singh, Shusheng Yang. Xichen Pan, Chenyu Li, Bingda Tang, Fred Lu, David Fan, Amir Bar, Shangyuan Tong for insightful discussions and feedback on the manuscript. This work was mainly supported by the Google TPU Research Cloud (TRC) program and the Opend Path AI Foundation. ST is supported by Meta AI Mentorship Pro- gram. SX also acknowledges support from the MSIT ITTP grant (RS-20244 0024-00457882) and the NSF award 11S-2443404.
10

---

Preprint
REFERENCES
Abulikemu Abuduweili, Chenyang Yuan, Changliu Liu, and Frank Permenter. Enhancing sample generation of diffusion models using noise level correction. TMLR, 2024.
Michael Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden, Stochastic interpolants: unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08779, 2023, 18, 25, 26
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with coint-embedding predictive architecture. In CVPR, 2023.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. ICLR, 2019.
Shiyue Cao, Yueqin Yin, Lianghua Huang, Yu Liu, Xin Zhao, Deli Zhao, and Kaigi Huang. Efficient- vqgan: Towards high-resolution image generation with efficient vision transformers. In ICCV, 2023. 17
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021. 20
Hao Chen, Yujin Han, Fangyi Chen, Xiang Li, Yidong Wang, Jindong Wang, Ze Wang, Zicheng Liu, Difan Zou, and Bhiksha Raj. Masked autoencoders are effective tokenizers for diffusion models. In ICML, 2025a. 10
Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, and Ran Xu. Blip3-o: A family of fully open unified multimodal arrhts-ar training and dataset, 2025b. 17
Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang. Haotian Tang, Muyang Li, Yao Lu, and Song Han. Deep compression autoencoder for efficient high-resolution diffusion models. In ICLR, 2025c. 17
Junyu Chen, Dongyun Zou, Wenkun He, Junsong Chen, Enze Xie, Song Han, and Han Cai. De-ae 1.5: Accelerating diffusion model convergence with structured latent space, 2025d. 10
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020a. 17
Shoufa Chen, Chongjian Ge, Shilong Zhang, Peize Sun, and Ping Luo. Pixelflow: Pixel-space generative models with flow. arXiv preprint (xXv:25544.07633, 2025e. 8
Ting Chen. On the importance of noise scheduling for diffusion models. arXiv preprint
arXiv:2301.10972. 2023. 2,
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020b. 17
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- lingussii tasks. In CVPR, 2024. 17
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In ICLR, 2025. 20
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In NeurIPS, 2021. 8, 22, 25, 26
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In ICLR, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas
Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. 17, 20

---

Preprint
Patrick Esser, Robin Rombach, and Bjöm Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021. 3, 17
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion En- glish, Kyle Lacey. Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In /CML, 2024. 2, 6,
David Fan, Shengbang Tong, Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rab- bat, Nicolas Ballas, Yann LeCun, Amir Bar, et al. Scaling language- visual representation learning. In /CCV, 2025a. 17
Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. In ICLR, 2025b. 17
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Mdtv2: Masked diffusion trans- former is a strong image synthesizer. arXiv preprint arXiv:2303. 14389, 2023. 7, 8
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Farde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014. 3,20
Cean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. In NeurlPS, 2020.
17
Philippe Hansen-Estruch, David Yan, Ching- Yao Chung, Orr Zohar, Jialiang Wang, Tingho Hou, Tao Xu, Sriram Vishwanath, Peter Vajda, and Xinlei Chen. Learnings from scaling visual tok- enizers for reconstruction and generation. In ICML, 2025. 17
Ali Hatamizadeh, Jiaming Song, Guilin Liu, Jan Kautz, and Arash Vahdat. DiffIT: Diffusion vision transformers for image generation. In ECCV, 2024. 8
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arxiv e-prints, art. In CVPR, 2019. 17
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2021. 1, 17,20 20
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to local nash equilibrium. In NeurlPS, 2017. 3, 26
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786), 2006. 17
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. 25
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. In ICML., 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020. 17, 25
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. Simple diffusion: End-to-end diffusion for high resolution images. In ICML, 2023. 2,5
Emiel Hoogeboom, Thomas Mensink, Jonathan Heek, Kay Lamerigts, Ruiqi Gao, and Tim Sali- mans. Simpler diffusion (sid2): 1.5 fid imagenet512 with pixel-space diffusion. In CVPR, 2025. 8
Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. In ICMI., 2023. 8
12

---

Preprint
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In NeurlPS, 2022. 25
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyz- ing and improving the training dynamics of diffusion models. In CVPR, 2024.
Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding diffusion model with bad version of itself. In NeurlPS, 2025. 2, 8, 25
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014. 2, 17
Theodoros Kouzelis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Komodakis, Eq-vae: Equiv- ariance regularized latent space for improved generative image modeling. In ICMI,, 2025a. 17
Theodoros Kouzelis, Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Ko- modakis. Boosting generative image modeling via joint image-feature synthesis. In NeurlPS, 2025b. 10
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In NewrIPS, 2019. 26
Tuomas Kynkäänniemi, Miia Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. In NeurlPS, 2024. 25
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In CVPR, 2022. 17
Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng- Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers. In ICCV, 2025. 2. 9, 23
Tianhong Li, Dina Katabi, and Kaiming He. Return of unconditional generation: A self-supervised representation generation method. In NeurIPS, 2024a. 26
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. In NeurlPS, 2024b. 8, 9, 17, 22, 23
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In ICLR, 2023. 4, 25
Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Chase Lambert, Joao Souza, Suhail Doshi, and Daiqing Li. Playground v3: Improving text-to- image alignment with deep-fusion large language models, 2024. 2
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In ICLR, 2023. 4, 25
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden- Eijnden,, and Sain- ing Xie. Sit: Exploring flow and diffusion- based generative models with scalable interpolant transformers. In ECCV, 2024. 3, 6, 7, 8, 17, 22, 23, 25, 26
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantiza- tion: VQ-VAE made simple. In ICLR, 2024. 17
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In JCMI., 2021. 17
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation leam- ing. In NewrIPS, 2017. 17
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. TMLR, 2023. 1,3
13

---

Preprint
Xichen Pan, Satya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, Ji Hou, and Saining Xie. Transfer between modalities with metaqueries, 2025. 17
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In ICML., 2018. 17
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023. 1, 3, 4, 8, 9, 17, 22
Phillip Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact learning. In ICLR, 2021. 5
Kai Qiu, Xiang Li, Hao Chen, Jason Kuen, Xiaohao Xu, Jiuxiang Gu, Yinyi Luo, Bhiksha Raj. Zhe Lin, and Marios Savvides. Image tokenizer needs post-training, 2025. URL htttt://aaxix org/ abs 2509. 12474. 17
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICMI., 2021. 2,
Vivek Ramanujan, Kushal Tirumala, Armen Aghajanyan, Luke Zettlemoyer, and Ali Farhadi. When worse better: Navigating the caappeession-generati tradeoff in visual tokenization. In NeurIPS, 2025. 17
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
17
Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In NeurlPS, 2019. 17
Sucheng Ren, Qihang Yu, Ju He, Xiaohui Shen, Alan Yuille, and Liang- Chieh Chen. Beyond next- token: Next-x prediction for autoregressive visual generation. arXiv preprint arXiv:2502.20388, 2025. 8, 22, 23
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022. 3, 10, 17
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255) 2015. 3, 21, 22
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In NeurlPS, 2016. 26
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In SIGGRAPH, 2022. 8,8
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis. In ICML, 2023. 20, 21
Ivan Skorokhodov, Sharath Girish, Benran Hu, Willi Menapace, Yanyu Li, Rameen Abdal, Sergey Tulyakov, and Aliaksandr Siarohin. Improving the diffusability of autoencoders. In ICML, 2025.
2,
Kiwhan Song, Jaeyeon Kim, Sitan Chen, Yilun Du, Sham Kakade, and Vincent Sitzmann. Selective underfitting in diffusion models. arXiv preprint arXiv:2510.01378, 2025.
Yang Song, Jascha Sohl-Dicksen Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ICLR, 2021.
21
14

---

Preprint
Quan Sun, Yufeng Cui. Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models in-context learners. In CVPR, 2024. 17
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the Inception architecture for computer vision. In CVPR, 2016. 26
Hao Tang. Chenwei Xie, Xiaoyi Bao, Tingyu Weng, Pandeng Li, Yun Zheng, and Liwei Wang. Unilip: Adapting clip for unified multimodal understanding, generation and editing, 2025. 3, 17
Jiayan Teng, Wendi Zheng, Ming Ding, Wenyi Hong, Jianqiao Wangni, Zhuoyi Yang, and Jie Tang- Relay diffusion: Unifying diffusion process across resolutions for image synthesis. In ICLR, 2023. 5
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng. and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. In NeurIPS, 2024. 8, 9, 22, 23
Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning. In ICCV, 2025. 17
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features, 2025. 2, 3, 17
Arash Vahdat, Karsten Kreis, and Jan Kautz. NeurlPS, 2021.
generative modeling in latent space. In
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML, 2008 10, 17
Hanyu Wang, Saksham Suri, Yixuan Ren, Hao Chen, and Abhinav Shrivastava. Larp: Tokenizing videos with a learned autoregressive generative prior. In ICLR, 2025a. 17
Shuai Wang, Ziteng Gao, Chenhui Zhu, Weilin Huang, and Limin Wang. Pixnerd: Pixel neural field diffusion. arXiv preprint arXiv:2507.23226, 2025b.
Shuai Wang. Zhi Tian, Weilin Huang, and Limin Wang. Ddt: Decoupled diffusion transformer, 2025c. 2, 7, 8, 9, 10, 22, 23
Xiao Wang, Ibrahim Alabdulmohsin, Daniel Salz, Zhe Li, Keran Rong, and Xiaohua Zhai. Scaling pre-training one hundred billion data for vision language models. arXiv preprint arXiv:2502.07617, 2025d. 17
Ge Wu, Shen Zhang. Ruijing Shi, Shanghua Gao, Zhenyuan Chen, Lei Wang, Zhaowei Chen, Hongcheng Gao, Yao Tang, Jian Yang, Ming-Ming Cheng, and Xiang Li. Representation en- tanglement for generation:training diffusion transformers is much easier than you think, 2025.
10
Jiawei Yang, Tianhong Li. Lijie Fan, Yonglong Tian, and Yue Wang. Latent denoising makes good visual tokenizers, 2025. 10, 17
Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. In CVPR, 2025. 2, 4, 7, 8, 10, 21, 22
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. TMLR, 2022. 17
Lijun Yu, José Lezama. Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion- tokenizer is key to visual generation. In ICL.R 2024a. 8
15

---

Preprint
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang- Ching Chen. An image is worth 32 tokens for reconstruction and generation. In NeurIPS, 2024b. 3, 17
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. In ICLR, 2025. 2, 3, 4, 6, 7, 8, 9, 10, 17, 22, 23
Shuangfei Zhai, Ruixiang Zhang, Preetum Nakkiran, David Berthelot, Jiatao Gu, Huangjie Zheng, Tianrong Chen, Miguel Angel Bautista, Navdeep Jaitly, and Josh Susskind. Normalizing flows are capable generative models. In ICML, 2025.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023. 17
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features as perceptual metric. In CVPR, 2018. 3, 20
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. In NeurlPS, 2020. 21
Yue Zhao, Yuanjun Xiong, and Philipp Krähenbühl. Image and video tokenization with binary spherical quantization. In ICLR, 2025. 17
Anlin Zheng, Xin Wen, Xuanyang Zhang, Chuofan Ma, Tiancai Wang, Gang Yu, Xiangyu Zhang, and Xiaojuan Qi. Vision foundation models as effective visual tokenizers for autoregressive image generation, 2025. 17
Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation. NeurIPS, 2022.
Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar, Fast training of diffusion models with masked transformers. TMLR. 8
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint 2..7222 2021. 17
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang. Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wengi Shao, Junjun Hc, Yingtong Xiong, Wenwen Qu, Peng Sun. Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng. Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. In- ternv13: Exploring advanced training and test-time recipes for open-source multimodal models, 2025. 17
Rui Zhu, Yingwei Pan, Yehao Li, Ting Yao, Zhenglong Sun, Tao Mei, and Chang Wen Chen. SD- DiT: Unleashing the power of self-supervised discrimination in diffusion transformer. In CVPR, 2024a. 17
Yongxin Zhu, Bocheng Li, Hang Zhang, Xin Li, Linli Xu, and Lidong Bing. Stabilize the latent space for image autoregressive modeling: A unified perspective. In NewrIPS, 2024b. 17
16

---

Preprint
A EXTENDED RELATED WORK
Recent studies have investigated leveraging semantic representations for reconstruction, particularly in MILMs where diffusion decoders conditioned on semantic tokens (Sun et al., 2024; Chen et al., 2025b; Pan et al., 2025; Tong et al., 2025). While this improves visual quality, the reliance on large pretrained diffusion decoders makes reconstruc- tions less faithful to the input, limiting their effectiveness as true autoencoders. Very recently, UniLIP (Tang et al., 2025) employs a one-step convolutional decoder on top of Intern ViT (Zhu al., 2025), achieving reconstruction quality surpassing SD- VD- However, UniLIP relies on additional large-scale fine-tuning of pretrained ViT', arguing that frozen pretrained representation encoders lacks sufficient visual detail. In contrast, we show this is not the case: frozen representation en- coders achieves comparable reconstruction performance while enabling much faster convergence in diffusion training.
Representation encoder as autoencoder:
Another line of related work also try to utilize representation encoders directly as tokenizers. VFM- Tok (Zheng et al., 2025) and DiGIT (Zhu et al., 2024b) applies vector-quantization directly to pre- trained representation encoders like Dino or SigLIP. These approaches transform representation en- coders into an effective tokenizer for AR models, but still suffer from the information capacity bottleneck brought by quantization.
Compressed image tokenizers. Autoencoders have long been used compress images into low-dimensional representations for reconstruction (Hinton & Salakhutdinov, 2006; Vincent et al., 2008). VAEs (Kingma & Welling, 2014) extend this paradigm by mapping inputs to Gaussian distributions. while VQ-VAEs (Oord et al., 2017; Razavi et al., 2019) introduce discrete latent codes. VQGAN (Esser et al., 2021) adds adversarial objectives, and ViT- ViT-VQGAN (Esser et al., 2021; Cao et al., 2023) modernizes the architecture with Vision Transformers (ViTs) (Dosovitskiy et al., 2021). Other advances include multi-stage quantization (Lee al., 2022; Zheng et al., 2022), lookup-free schemes (Mentzer et al., 2024; Zhao et al., 2025), token-efficient designs such as TiTok and DCAE (Yu et al., 2024b; Chen et al., 2025c), and structure-preeserving approaches like EQ- VAE (Kouzelis et al., 2025a). (Hansen-Estruch et al., 2025) further explores the scaling behavior of VAEs. LARP, CRT, REPA-E (Wang et al., 2025a; Ramanujan et al., 2025; Yu et al., 2025) tried to improve VAE with generative priors via back-propagation. In contrast, we dispense with aggressive compression and instead adopt a pretrained representation encoders as encoder. This avoids the en- coder collapsing into shallow features optimized only for reconstruction loss, while providing strong pretrained representations that serve as a robust latent space.
Generative models. Modern image generation is dominated by two paradigms: autoregressive (AR) models and diffusion models. AR models (Ramesh et al., 2022; Yu et al., 2022; Parmar et al., 2018; Chen et al., 2020a) generate images sequentially, token by token, and benefit from powerful language-model architectures but often suffer from slow sampling. Diffusion models (Zhu et al., 2024a; Ho et al., 2020; Nichol & Dhariwal, 2021; Rombach et al., 2022; Ma et al., 2024; Peebles & Xie, 2023) instead learn to iteratively denoise noisy signals, offering superior sample quality and scalability, though at the cost of many sampling steps. In this work, we build on diffusion models but adapt them to high nigh-dimensional latent spaces provided by pretrained representation encoders. We find representation encoders provide faster convergence and improved scaling behavior.
Robust decoders for generation. Recent works suggest that incorporating masking or latent de- noising losses can improve tokenizer training. I-DeTok (Yang et al., 2025) shows that combining both losses yields strong VAES for second-stage MAR (Li et al., 2024b; Fan et al., 2025b) generation, while RobusTok (Qiu et al., 2025) demonstrates that training with perturbed tokens makes decoders more robust. Notably, Yang et al. (2025) report that denoising loss only increase performance when encoder and decoder are jointly trained, whereas we observe substantial gains in generation quality with frozen encoders.
Visual representation learning. Visual representations primarily fall into two broad families: self-supervised encoders that learn invariances from augmented views or masked prediction (He et al., 2019; Chen et al., 2020b; Grill et al., 2020; Zhou et al., 2021; He et al., 2021), and language- supervised encoders trained from image-text pairs (Radford et al., 2021: Zhai et al., 2023; Chen et al., 2024; Tschannen al., 2025). Recent work (Fan et al., 2025a; Tschannen et al., 2025; Wang
17

---

Preprint
et al., 2025d) scales both lines billion-scale corpora, yielding robust, semantically rich features. In this work, we leverage these pretrained encoders directly: we freeze the encoder, and train a lightweight decoder to form Representation Autoencoder (RAE). We show that such RAEs are effective for both reconstruction and generation.
B PROOFS
B. PROOF OF LOWER BOUND FOR TRAINING LOSS
Theorem 1, Assuming x ~ p(x) E IR", ER","~ N(0,I) 0, Letx; = (1 (1- t)x + te, consider the function family
Go = (g(xx.t) Bf(xxx,t A R f: 00 [0.1]xx=->0
(1)
where d < refers stack of standard DiT blocks whose width is smaller than the token dimension from the representation encoder, and A, denote the input and output linear projections, respectively. Then for any Gd.
Lig. 0) = Bx-p)))) [lg(( - -x)1l3]d N
Σ i=d+1
A
(2)
where X are the eigenvalues of the covariance matrix of the random variable W Notably, when 2 n, Ga contains the unique minimizer to Lig, 0).
Proof. By Albergo et al. (2023), the distribution Pt of x; satisfies po = p(x), P1 = LN and 0.p+V V (wp) where is the optimal velocity predictor defined as u(x,, t) E'e xx Also, by Theorem 2.7 in Albergo et al. (2023), there exists f E (CC((C) (R")(((")))) ))"; [0, 1)) that uniquely minimizes the L(f, 0) and perfectly approximates
By our training setting, it's reasonable to assume that x ~ p(x) and E ~ No are independent. x Py(y) satisfies (y = Jg- NO I.) +
Then the distribution of the objective y x)p(x)dxx. Clearly, Py has full support R" and is strictly positive, indicating y has non- zero probability anywhere in Ra. Similarly, for X2 = (1 t)x + te, given any t, Px, (w) = No, t (I,)(w x) (1-))(("'' (, )dx also has full support on R and is strictly positive, indicating x, has a non zero probability anywhere in R" well.
Recall that for any function f X' + y Im(f) = [f(x): : E X}. Then for linear transformation f(x Mx with M Rddxn, Im(f) (Mx :x:xx E RR]; we denote this as Im (M). Now, for any E Ga, lm(g) (Ax x R") C [By y E Rd] Im (B). Since rank B) S d, dim lm(g) S rank(B) < < n, therefore Im(g) C Im(B)
C R"
Now, given Ga and the deterministic pair (x E (R" R", 00, 1J). by Projection Theorem,
(lo(x,, 1) y112 1, 1
where uy E Im(g) is the unique minimizer and ug y is orthogonal to Im(g). Since || 1|2 2 we can take expectation on both sides
inf -x-pl.))-- [Ilo(xx,))) y113 gEga
>
inf gEga
B[[lu, - -
inf ue S inf S:dim S.d
E [[lu- - y|
E [| |P>y|
(4)
'aailyy of functions f: 10 RD RD that is continuous in t for all (x, t) E 00 1] R2. and f(.t) is
continuously differentiable function from RR" RR".
18

---

Preprint
where Ps denote the projection matrix from IR" onto S. Without loss of generality, assume
E[x] = 0², then Eq can be expanded as
inf -xxx,xx))--W001111llxxxxx y1l 2 gESia
Ely
sup S;dim SSd
Σ [((s))]]
==1
Tr(CCov(y)
sup S;dim SSd
Tr(Cov (Psy)
Σ
A
Σ A Σ
A
(5)
i=++1
where Eq 5 is obtained via Ky-Fan Maximum Principle.
When d n, sups [I| Psy
?] [] leading trivial lower bound in Eq 5, and Ga
(CC [R") [0,
B.2 PROOF LOWER BOUND FOR INFERENCE LOSS
Theorem 2. Consider the same setup as Theorem 1. Let xı be the initial random variables in the sampling process, and
X = =ODE(g X1, 1-
X ODEES ,1, 0)
where ODE(S,x, x, refers to any ODE solver that integrates f from time i using x as the initial condition. We further assume for any (x,y,t, E (R", R", [0, 1]), there exists constant L >> 0 such that
- s L|[ - yll
then
|x] Xxol
(6)
f=d+1
Proof. We first define forward ODE that integrates from 0 Xg x-1, and
dxy = g(x t)dt
dxf = fr (x t)dt
Then
|x x = (x (x² g(x .tll
> f (xT, g(xx :) -11 (1f -x1 (x(x,)))
2 |||| L||xī x7
(7)
where A denotes the approximation to f for Ga. Applying Gronwall's Lemma, have
1 || e/=111x - x f Lh e
dds
(1 -
|| - Xull = |
x 2
Σ A
(8)
L i=d+1
where by Theorem 1, |A|| 2 Σ 12
Ei=4+1 A,.
Nonnceetered x with F[x] will additionally introduce || |s//|22 to the lower bound; we ignore this term since most data processing pipelines will center the data.
19

---

Preprint
C RAE IMPLEMENTATION
C.I. ENCODER NORMALIZATION
For any given frozen representation encoders, we discard any [CLS] [REG] token produced by the encoder, and keep the all of patch tokens. We then apply a layer normalization to each token independently, to ensure each token has zero mean and unit variance across channels. We note that all representation encoders we use adopt the standard ViT architecture (Dosovitskiy et al., 2021), which have already applied layer normalization after the last transformer block. Therefore, we only need to cancel the affine parameters of the layer normalization in representation encoders. This does not affect the representation quality of representation encoders, as it a linear transformation.
Practical Notes. Specifically, we use DINOv2 with Registers (Darcet et al., 2025). Since DINOv2 only provides variants with Pe 14, we interpolate the input images to 224 x 224 but set Pa = 16, ensuring the model still produces 256 tokens while reconstructing 256 x 256 images.
C.2 DECODER TRAINING DETAILS
Datasets. We primarily use ImageNet-IK: for training all decoders. Most experiments are conducted at resolution of 256 x 256. For 512-resolution synthesis without decoder upsampling, train decoders directly on 512 x 512 images.
Decoder Architecture. The decoder takes the token embeddings produced by the frozen encoder takes the token embedding reconstructs them back into the pixel space using the same patch size as the encoder. As a result, it can generate images with the same spatial spatial resolution as the encoder's s inputs input. Following He et al. (2021), we prepend a learnable [CL.) token decoder's input sequence and discard it after decoding.
Discriminator Architecture. We include the majority of our decoder training details in Table 12. We follow most of the design choices in StyleGAN-T (Sauer al., 2023), except for using frozen Dino-S/8 (Caron et al., 2021) instead of Dino-S/16 as the discriminator. We found using Dino-S/8 stabilizes training and avoid the decoder to generate adversarial patches. We also remove the virtual batch norm in Sauer et al. (2023) and use the standard batch norm instead. All input is interpolated to 224 224 resolution before feeding into the discriminator.
Table 12: Training configuration for decoder and discriminator.
Component
Decoder
Discriminator
optimizer
Adam
Adam
max learning rate
2 x 10 4
2 x 10 4
min learning rate
x 10-5 5
2 x 10-5
learning rate schedule
cosine decay (0.5, 0.9)
cosine decay (0.5, 0.9) 0.0
optimizer betas
weight decay
0.0
batch size
512
512
warmup
epoch ViT-(B. XL)
I epoch
loss
4 + LPIPS + GAN
adv.
Model
Dino (frozen)
LPIPS start epoch disc, start epoch
0
adv. loss start epoch
8
Training epochs
16
10
Losses. For training the decoder, we use a mixture of L1, LPIPS (Zhang et al., 2018) and adversarial loss (Goodfellow al., 2014):
== E(x), D(z)
Crec(x) WL LP (uAA))
20

---

Preprint
Figure 8: Reconstruction examples. From left to right: input image, RAE (DINOv2-B), RAE (SigLIP2-B), RAE (MAE-B), SD-VAE. Zoom in for details.
where E, are the encoder and decoder of RAE. We set wiz and wg 0.75. We use the same losses in StyleGAN-T (Sauer et al., 2023) for discriminator, and GAN loss as in Esser et al. (2021). We also adopt the adaptive weight À for GAN loss proposed in Esser al. (2021) balance the scales of reconstruction and adversarial losses. À is defined as:
VeCreel V GAN x|
Augmentations. For data augmentation, we first resize the input image to 384 384 and then ran- domly crop to 256 x 256. We also apply differentiable augmentations with default hyperparameters in Zhao et al. (2020) before discriminator.
C.3 VISUALIZATIONS
We present visualizations of reconstructions from different RAEs. As shown in Figure 8, all RAEs achieve satisfactory reconstruction fidelity.
D DIFFUSION MODEL IMPLEMENTATION
Datasets. We primarily ImageNet-IK IK (Russakovsky al., 2015) for diffusion training. Most experiments are conducted at resolution of 256 x 256. For 512-resolution synthesis without decoder upsampling. we train diffusion models directly on 512 512 images.
Models. By default, we use LightningDiT (Yao al., 2025) as the backbone of our diffu- sion model. We use continuous time for- mulation of flow matching and restrict the timestep input to real values in [0, 1]. Fol- lowing prior work (Song et al., 2021), we re- place the timestep embedding with Gaussian Fourier embedding layer. We also add Abso- lute Positional Embeddings (APE) the in- put tokens in addition to ROPE, though we do observe significant performance difference with or without APE.
Model Contig
Dim
Num Hem-Heedss
Depth
S
384
6
12
B
768
12
12
L
1024
16
24
XL
1152
16
28
XXL
1280
16
32
H
1536
16
32
G
2048
16
40
T
2688
21
40
Table 13: Model configurations for different sizes.
For DH generally follow the same architecture DiT, and does reapply APE for the DDT head input. We use linear layer map the DH encoder output to the DH decoder
21

---

Preprint
dimension when the dimension of DiT and DDT head mismatches. We provide detailed model configuration in Table 13.
Compute. For all models based on RAE, and pixel inputs, we use patch sizes of 2 and 16, respectively. Across all 256 x 256 experiments, the diffusion model processes token sequence of length 256. Consequently, the computational cost of the DiT backbone remains identical across RAE, VAE, and pixel settings, with differences arising only from the patchification step. Since patchification accounts for less than I 1% of the total GFLOPs, training DiT on different autoencoders introduces only negligible compute overhead.
a patch size of 1. For baseline experiments on VAE
Optimization. For DIT, strictly follow the optimization strategy in Lightning (Yao al., 2025), using AdamW with a constant learning rate of 2.0 10 a batch size of 1024 and an EMA weight of 0. 0.9999. We do not observe instability or abnormal training dynamics with this recipe on DIT. For DiT we find using the recipe in (Yao al., 2025) leads to loss spikes later epochs and slow EMA model convergence at early epochs. We instead use a linear decay from 2.0 x 10-4 2.0 x 10- with a constant warmup of 40 epochs. To encourage the convergence of EMA model, we change the EMA weight from 0.9999 to 0.9995. Additionally, we use gradient clipping of 1.0 for DiT Other optimization hyperparameters are the same as DiT. All models are trained for 80 epochs unless otherwise specified. We only report EMA model performance.
Sampling. We use standard ODE sampling with Euler sampler and 50 steps by default. We find the performance generally converges above 50 steps. We use the same sampling hyperparameters for both DiT and Di
Computation. We use PyTorch/XLA on TPU for all training and inference on RAE. For evaluation, we use a single v6e-8 to generate 50K samples. For the 800 epoch Di -XL (DINOv2-B result, conduct the generation TensorFlow GPU support. We use an internal JAX codebase for training baseline models on VAEs.
machine with 4 A 1100s and evaluate FID on CPU due lack of
E SAMPLING DETAILS FOR FID EVALUATION
For the FID--5K evaluation on RAE, followed the protocol used in (Tian al., 2024; Li al., 2024b; Ren et al., 2025; Wang et al., 2025c), sampling 50 images from each class for a total of 50,000 images. The reference statistics were taken from the ADM pre-computed statisties (Dhariwal & Nichol, 2021) over the full ImageNet dataset. Another commonly adopted sampling strategy is uniformly sample class labels 50K times and generate images accordingly (Peebles & Xie, 2023; Ma et al., 2024; Yu et al., 2025). It is worth noting that this strategy is not equivalent to the per-class sampling scheme-althouge although random sampling asymptotically converges the per-class version, the resulting FID scores differ slightly in practice.
To ensure fair comparison, we re-evaluate previous state-of-the-ee methods that did not use class- balanced sampling with class-balanced sampling. We also evaluate our method with random sam- pling. As shown in Table 14, all methods' FID improves consistently with class-balanced sampling.
We also note that the original ImageNet training set is inherently unbalanced, with class sizes rang- ing from approximately 732 to 1,300 samples (Russakovsky al., 2015). Therefore, the com- nampling-namely, mon assumption behind both balanced and random sampling namely, that the label distribution is uniform-is not entirely accurate. However, since 895 classes contain exactly 1,300 samples, the dataset exhibits high degree of near-equivalence among most classes. This partial balance may partly explain why balanced sampling consistently yields better results, as it more closely approx- imates the true label distribution of the training set. As the FID values approach the lower ranges, these subtle details begin to have a greater impact. We therefore want to raise awareness within the community. Finally, while the FID metric remains useful indicator of generative quality, its absolute value becomes less meaningful as generation fidelity continues to improve.
F THEORY EXPERIMENT SETUP
In this section we list the setup of experiments in Section for overfitting images.
22

---

Preprint
Generation @ 256 w/o guidance Generation @ 256 w/ guidance
Method
Epochs
Random
Balanced
Random
Balanced
gFTD
IST
gFID
IS
gFID
IST
gFID
IS
Autoregressive
VAR (Tian al., 2024)
350
1.92
323.1
1.73
350.2
MAR (Li et al., 2024b)
800
2.35
227.8
1.55
303.7
xAR-H (Ren al., 2025)
800
1.24
301.6
Latent Diffusion with VAE
SiT (Ma al,, 2024)
1400
8.61
131.7
8.54
132.0
2.06
270.3
1.95
259.5
REPA (Yu el al., 2025)
800
5.90
157.8
5.78
158.3
1.42
305.7
1.29
306.3
DDT (Wang et al., 2025c)
6.27
154.7
1.40
303.6
1.26
310.6
REPA-E (Leng et al., 2025)
800
1.83
217.3
1.70
217.3
1.26
314.9
1.15
304.0
Latent Diffusion with RAE (Ours)
DI (DINOv2-B)
800
1.60
242.7
1.51
242.9
1.28
262.9
1.13
262.6
Performance of different methods using different sampling strategies. The officially Table 14:
reported numbers are marked in gray.
Models. By default, we use a DiT with depth 12, width 768 and a attention head of 4. The depth varies in (384.512, 640, 768, 896) and width varies in {4, 12, 16, 243 in Figure 3. Other configurations are the same as Appendix D.
Targets. We use three images for overfitting experiments. and all numbers reported are the average on three independent run on each images. We resize all targets 256 256 and not use any data augmentation.
Optimizations & Sampling. For single target image. the batch size only influences the timestep. We therefore use a relatively small batch size of 32 and a constant learning rate 10-4, op- timized with AdamW (B (0.9, 0.95)). The model is trained for 1200 steps without EMA. For sampling, We use standard ODE sampling with Euler sampler and 25 steps by default.
G ADDITIONAL ABLATION STUDIES
G.1 GENERATION PERFORMANCE ACROSS ENCODERS
As shown in Table 15a, DINOv2-B achieves the best overall performance. MAE performs substan- tially worse in generation, despite yielding much lower rFID. This shows that a low rFID does not necessarily imply good image tokenizer. Therefore, use DINOv2-B the default encoder for our image generation experiments.
(a) gFlD and rFD of different encoders w/ and w/o moisy-mbust decoding.
(b) gFID and rtD of different DINOY2 sizes w/ and w/o noisy-robust decoding.
(c) Scaling for DINOv2-B.
Model
gFID
(FID
Model
gFID
rFI
DINOv2-B
4.1/4428
0.49/0.57
3.83/3.50
0.52/0.64
0.0
4.81
0.49
SigLIP2-B
6.69 /4.93
0.53 0.82
4.81 4.28
0.49/0.55
0.5
4.39
0.54
MAE B
0.16/0.22 16.14/8.38
L
6.77/6.09
0.52/0.59
0.8
4.28
0.57
1.0
4.20
0.60
Table 15: Ablations noise-augmented decoder training. Despite minor drop in rFID, the noise- augmented training strategy can greatly improve the gFID across different encoders and model sizes. Default setups marked in gray.
23

---

Preprint
Depth
Width
GFLLops
FID
2-768
2-1536
2-2048
2-2688
1152 (XL)
25.65
2.36
Dino-S
2.66
2.47
2.42
2.43
2048 (G)
53.14
2.31
Dino-B
2.49
2.24
2.16
2.22
2
2048 (G)
26.78
2.16
Dino-L.
N/A
2.95
2.73
2.64
Table 16: DDT head should be wide and shallow. A wide, shallow head yields lower FID than deeper (4-layer G) or narrower (6-layer XI.) ones.
Unguided gFID of different RAE and DDT head. Larger RAE benefits more from wider DDT head. d-w: DDT head with d layers and width Default setups are marked in gray.
Table 17:
G.2 DESIGN CHOICES FOR NOISE- AUGMENTED DECODING
We first analyze how noise-robust decoding affects reconstruction and generation. Table 15c shows that larger improves generative FID (gFID) consistently, but slightly worsens reconstruction FID (rFID). This supports our intuition: noise encourages the decoder to learn smoother mappings that generalize better to imperfect latents, improving generation quality, but reducing exact reconstruc- tion accuracy.
To test the robustness of this trade-off, we evaluate different encoders (Table 15a) with o 0.8. Across all encoders, noisy training improves gFID while mildly harming rFID. The effect is strongest for weaker encoders such as MAE-B, where gFID improves from 16.14 to 8.38. Finally, Table 15b shows that the benefit holds across encoder sizes, suggesting that robust decoder training broadly applicable.
Together, these results highlight general principle: decoders should not only reconstruct clean latents, but also handle their noisy neighborhoods. This simple change enables RAEs to serve as stronger backbones for diffusion models.
G.3 DESIGN CHOICES FOR THE DDT HEAD
We now investigate design variants of the DDT head to identify those that serve role more fectively. Two factors turn out to be crucial: (a) the head needs to be wide and shallow, and (b) its benefit depends on the size of the underlying RAE encoder.
Width and Depth. We first vary the architecture of the DDT head, sweeping both width and depth while keeping the total parameter count approximately fixed. As shown Table 16, a 2-layer, 2048- dim (G) head outperforms 6-layer, 1 1152-dim (XL) head by large margin, despite having similar GFlops. Moreover, a 4-layer 2048-dim head does not improve over the 2-layer version, even though it has double the GFlops. This suggests that wide and shallow head is more effective for denoising.
Dependence on Encoder Size. Next, we analyze how the effect of the DDT head scales with the size of the RAE encoder. We fix the DiT backbone DiT-XL and vary the DDT head width from 768 (B) to 1536 (H), 2048 (G), and 2688 (T). We train Di models on top of three RAEs: DINOv2-S, DINOv2-B, and DINOv2-L. As shown in Table 17, the optimal DDT head width in- creases as the encoder scales. When using DINOv2-S and DINOv2-B, the performance converges at a DDT head width of 2048 (G), while 2688 (T) head still brings performance gains on DINOv2-L. This suggests that the larger RAE encoders benefit more from wider DDT head.
By default,
2-layer, 2048-dim DDT head for all DH models.
H ADDITIONAL SCALING RESULTS
We examine how model scale affects training loss in Figure 9. Increasing the model's computational capacity leads DH to converge faster and reach lower final loss.
24

---

Preprint
DIToH-S DireK-g DITTH
Training Loss
DIT XL
0.40
0.35
My (WMM
200K
40CK
60ck
200K
1M
Training Iterationn
Figure 9: Training loss of DiT on DINOv2-B. We use EMA weight of 0.9 to smooth the loss.
I GUIDANCE
We primarily adopt AutoGuidance (Karras et al., 2025) our guidance method, as it is easier to tune than CFG with interval (Ho & Salimans, 2022; Kynkäänniemi et al., 2024) and consistently delivers better performance. CFG with interval is used only for the DiT-XL DINOv2-S with guidance result reported in Table 8.
AutoGuidance. We adopt AutoGuidance (Karras al., 2025) as primary guidance method. The idea is to use a weaker diffusion model to guide a stronger one, analogous to the principle of Classifier-Free Guidance (CFG) (Ho & Salimans, 2022). Empirically, models and earlier training checkpoints consistently yield stronger guidance effects. In practice, we use the smallest variant, DiT -S. as the guiding model, typically adopting an early checkpoint. Unless otherwise specified, AutoGuidance results obtained using guidance scale of 1.5 and 20-epoch checkpoint of DI. For the best result of 1.13 FID (DiTDI-Xt -XL DINOv2-B, 256 x 256), we sweep the guidance scale and use guidance scale of 1.42 with 14-epoch DiT checkpoint. Notably, training this base model requires only about 0.05% of the compute used to train the guided model (DiTDH-XL. -XI for 800 epochs).
observe that weaker base
Classifier-Free Guidance. We also evaluate CFG (Ho & Salimans, 2022) on RAE. Interestingly, CFG without interval does not improve FID; in fact, applying it from the first diffusion step increases FID. With Guidance Interval (Kynkäänniemi et al., 2024), CFG can achieve competitive FID after careful grid search over scale and interval. However, our final model (DiTDH-Xx -XL with DINOv2- B), the best CFG result remains inferior to AutoGuidance. Considering both performance and tuning default guidance method.
overhead, adopt AutoGuidance
J DESCRIPTIONS FOR FLOW-BASED MODELS
Diffusson Models (Ho al., 2020; Dhariwal & Nichol, 2021; Karras al., 2022) and more generally flow-based models (Albergo et al., 2023; Lipman et al., 2023; Liu et al., 2023) are a family of generative models that learn to reverse reference "noising" process. One of the most commonly used "noising" process is the linear interpolation between i.i.d Gaussian noise and clean data (Esser et al., 2024; Ma et al., 2024):
x, (1 f)x + te
where x ~ ( [o, 1]. and we denote x,'s 's distribution as (x(x) with Po = p(x) with pure noise, and simulates some differential
and P1 (0. Generation then starts at equation to progressively denoise the sample to a clean one. Specifically for flow-based models, the differential equations (an ordinary differential equation (ODE) or a stochastic differential equation (SDE)) are formulated through underlying velocity u(x,, t) and score function sx,,,t)
ODE
dx; v(x,, t)dt
1
SDE
dx; = v(xe, l)dl
2 (urs(xx, + /indw:
25

---

Preprint
where wy is any scalar-valued continuous function (Ma et al., 2024), and w, is the reverse-time Wiener process. The velocity vxx, t) is represented as conditional expectation
ex x t) [x, Ele x/x{]
and can be approximated with model vo by minimizing the following training objective
Evalicity (0) - f Ex,e [|00(xes 1) x)]?
The score function s(xe, 1) is also represented as conditional expectation
s(x,, t) TE xx]
Notably, is equivalent to up constant factor (Albergo et al., 2023), it's enough estimate only one of the two vectors.
K EVALUATION DETAILS
K.1 EVALUATION
We strictly follow the setup and use the same reference batches of ADM (Dhariwal Nichol, 2021) for evaluation, following their official implementation." We use TPUs for generating 50k samples and use one single NVIDIA A100 80GB GPU for evaluation.
In what follows, we explain the main concept of metrics that we used for the evaluation.
FID (Heusel et al., 2017) evaluates the distance between the feature distributions of real and generated images. relies the Inception-v3 network (Szegedy et al., 2016) and assumes both distributions follow multivariate Gaussians.
(Salimans et al., 2016) also uses Inception-v3, but evaluates logits directly. It mea- sures the KL divergence between the marginal label distribution and the conditional label distribution after softmax normalization.
Precision and recall (Kynkäänniemí et al., 2019) follow their standard definitions: preci- sion reflects the fraction of generated images that appear realistic, while recall reflects the portion the training data manifold covered by generated samples.
L UNCONDITIONAL GENERATION
We are also interested in how RAES perform in unconditional generation. To evaluate this, we train DH- -XI. on RAF latents without labels. Following RCG (Li al., 2024a), we set labels to null during training and use the same null label at generation time. While classifier-free guidance (CFG) does not apply in this setting, AutoGuidance remains applicable. We therefore train DH, XL for 200 epochs with AG (detailed in Appendix 1).
As shown in Table 18, our model achieves substan- tially better performance than DiT-XI. trained on VAE latents. Compared to RCG, method specif- ically designed for unconditional generation, our ap- proach attains competitive performance while being much simpler and more straightforward, without the need for two-stage generation.
Method
gFID
IS
DIT-XL+ VAE DI DINOv2-B (w/ AG)
30.68
32.73
123.12
RCG DIT-XI.
4.89
143.2
Table 18: Comparison of unconditional gener- ation on ImageNet 256 x 256.
"https://gi thtp://ttttttttttee cowpopaaaaopaaaaaaa ded-di
26

---

Preprint
SigLIP2-B
Figure 10: tSNE examples. We use 50 images per class from ImageNet validation set, and 100 classes for (SNE visualization.
M INTERPRETING THE DIFFUSABILITY RAE VIA TSNE
A key remaining question is how the choice of RAE encoder influences generation performance. While we demonstrated Appendix C.1 that the encoder choice dramatically impacts results, the underlying mechanism requires further explanation. We perform ISNE visualizations sentations from various RAE encoders, using SD-VAE as a baseline. To extract features for visual- ization, we use the [CLS] token for encoders that possess one (MAE, DINOv2), and apply global pooling over all patch tokens for those that do not (SigLIP2, SD- VAE).
the repre-
As illustrated in Figure 10, we observe a clear hierarchy in class separation: SD-VAE < MAE SigLIP2 < DINOv2. DINOv2 exhibits the most distinct clustering among classes, while MAE shows the least separation among the discriminative encoders, though it still surpasses SD-VAE. Intuitively, well-separated class clusters simplify the learning of the generative model; the class- conditioned velocity becomes more coherent within classes and distinct between them, making the diffusion process easier to fit.
N
VISUAL RESULTS
We show uncurated 512 x 512 samples sampled from our most performant model: DiTPH on DINOv2-B with autoguidance scale 1.5.
27

---

Preprint
iinLO DIGH
Figure 11: Uncurated 512 x 512 DIT DH -XL samples.
Figure 12: Uncurated 512 x 512 DH -XL samples.
AutoGudance Scale = 1.5
AutoGudance Scale = 1.5 Class label "husky" (250)
Class label "golden retriever" (207)
28

---

Preprint
Figure 13: Uncurated 512 x 512 DH DH -XL samples.
Figure 14: Uncurated 512 x 512 DH -XL samples.
AutoGudance Scale = 1.5 Class label "cliff" (972)
AutoGudance Scale = 1.5 Class label "macaw" (88)
29

---

Preprint
BIYYINH
GUST
Figure 15: Uncurated 512 x 512 DIT DH -XL samples.
Figure 16: Uncurated 512 x 512 DH -XL samples.
AutoGudance Scale = 1.5
AutoGudance Scale = 1.5 Class label "balloon" (417)
Class label "arctic fox" (279)
30


Reviews:
[{"review_id": "xRBFt5NefU", "reviewer": "Reviewer_tJTk", "summary": "This paper revisits the training paradigm of image tokenizers for image generation models. It addresses the limitations of the standard SD-VAE–style tokenizer, whose training primarily depends on reconstruction losses and therefore often yields latent spaces with limited semantic structure. The authors propose an unconventional approach: replacing the learnable encoder with a frozen, pretrained vision encoder and training only the decoder for image reconstruction. To mitigate the resulting challenges—such as a substantially higher latent dimensionality and a more fragile decoder—the paper introduces several architectural and training refinements to the DiT models. The resulting tokenizer demonstrates improved image generation quality and faster convergence.", "questions": "- I understand that the authors aim to ensure fair comparisons by controlling the model size in terms of parameter count. However, having the same number of parameters does not necessarily imply comparable efficiency across all aspects. Could the authors also report the wall-clock training and inference speeds, as well as the FLOPs, of the proposed model relative to the baselines?\n\n- I understand that the rebuttal period is limited, and I'm not requesting significant experiments here. However, for future submissions, including moderate-scale text-to-image experiments would be valuable to more convincingly demonstrate the practical applicability of the proposed method.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper targets a very important problem of rich semantics in image tokenization. The proposed tokenizer design is simple yet very effective. \n\n- The practical and theoretical justifications for the architectural design are very convincing. \n\n- The empirical results are strong on ImageNet-level evaluations.", "weaknesses": "**Presentation**\n\nMy major concern with this paper is that it seems to have been submitted in a huge rush. A throughout proofreading and reformatting is very necessary. Examples: \n\n- I failed to find Figure 3 and Figure 4 in the main manuscript. \n\n- Line 175: the two sentences beginning with “Because RAE…” are repetitive.\n\n- Line 345: missing space before 'To'.\n\n- Figure 8 appears earlier than Figure 7 in the manuscript. \n\n\n**Technical**\n\n- In Line 013 of the abstract. I personally believe it is not appropriate to classify traditional VAEs as U-Nets. U-Nets typically include skip connections between the encoder and decoder, which are not applicable in image tokenization models.\n\n- In Line 086 of Introduction. Calling the latent space of RAE 'discrete' can be very misleading. Readers might confuse it with typical latent space such as VQ. If the authors wish to retain this term, a clear explanation is required early in the paper; otherwise, readers will remain unclear about its meaning until Section 4.3.\n\n- In line 306, the statement “noise smooths the latent distribution and mitigates OOD issues for the decoder, but also removes fine details” is unconvincing. Since the additional noise is introduced only during training, and the decoder receives clean latents during inference, the argument that noise removes fine details does not hold.\n\n- In Table 9, compared to direct scaling tokens, the upsample method actually increases the gFID by 42%, which can hardly be considered as 'comparable' as stated by the authors. This raises concerns about the scalability of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QnifmKvyWl", "reviewer": "Reviewer_diB6", "summary": "This paper proposes a new replacement to VAE autoencoder (used for downstream Diffusion Transformer models) with Representation Autoencoder (RAE). The authors argue that the traditional UNET based VAEs are computationally heavy and provide a weak latent space that leads to lower performance. Their RAE takes a pretrained representation encoder (like DINOv2) and lightweight ViT-based decoder to solve this issue. This new Autoencoder is not directly applicable to the DiT architectures 1) The image reconstruction is limited, hence a wide DiT is used 2) noise schedule shift is employed to account for increased dimensions 3) The latents are augmented with noise to inject stochasticity. To scale however, a shallow DDT head is attached to the DiT backbone which achieves FID scores of 1.18 on ImageNet-256", "questions": "1) The fundamental question is what property beyond reconstruction makes a representation diffusable?  and why does DINOv2 succeed where MAE fails?\n\n2) What is the qualitative trade-off of noise-augmented decoding, if the FID improves but is the reconstruction compromised?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1)  The proposed shallow DiT + RAE method achieves an FID of 1.18 on ImageNet 256x256 and 1.13 on 512x512, outperforming all prior diffusion models.\n\n2) The method converges faster than VAE-based baselines  for example 10x speedup over VA-VAE while maintaining the FID scores.\n\n3) The RAE also employs an efficient decoder. This part also contributes to make the model faster.\n\n4) The paper provides a theoretical proof and empirical evidence, that the DiT width should be at least equal to the dimensionality of the tokens.", "weaknesses": "1) The paper does not necessarily prove that the better reconstruction quality equals to better generation quality. This still raises questions on the diffusibility of the the latent space proposed in the paper.\n\n2) The model is suitable for the high computation models, for the highly compressed models in VAE which can have very high compression ratios, this model might not be comparable in the computation cost do to the requirement of dimensionality and DiT width constraints.\n\n3) Again Shallow DiT only benefits high dimension latents , for example it can perform worse that DiT-XL trained on low dim SD-VAE.\n\n4) The new training scheme is unstable as compared to  the standard DiT which is stable with a constant learning rate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QiHYMsawxU", "reviewer": "Reviewer_qR4u", "summary": "This paper discusses how to replace the VAE encoder in LDMs with pretrained representation encoders (e.g., DINOv2). The authors conduct a thorough analysis around this topic, and the paper is well-written, clear, and provides meaningful insights.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Introducing semantic tokenizers into generative diffusion models is a valuable and timely exploration. This idea may also be extended to unified MLLMs to alleviate the representational conflict between generation and understanding tasks caused by different semantic and pixel-level granularity requirements.\n\n2. The paper is highly readable and the proposed method is elegant. The analyses on model width, noise schedule, and the proposed DDT head provide clear insights and further improve the effectiveness of semantic tokenizers for generation tasks.\n\n3. Extensive experiments and analyses demonstrate that the proposed method achieves SOTA performance.", "weaknesses": "1. The results in Table 1 require further clarification.  RAEs with frozen encoders achieve consistently better reconstruction quality (rFID) than SD-VAE (lines 156–159). However, concurrent works [1, 2] reported that semantic tokenizers tend to decrease reconstruction fidelity.  Additional explanations for why rFID improves, along with more experiments, could help further clarify this point.\n\n2. DINOv2 was trained on hundreds of millions of images [3], which may include ImageNet-like, object-centric samples. This raises potential concerns about evaluation fairness. It would be helpful if the authors could clarify how to avoid the data leakage risk.\n\n3. The notably low gFID on ImageNet may raise concerns about potential overfitting. Since incorporating semantic latents could simplify the latent structure [4], a (large) pretrained tokenizer might more easily memorize simplified semantic distributions. Further OOD evaluations would help confirm the generality of the improvement.\n\n4. It would be interesting to explore whether a smaller-width DiT, combined with a simple learnable linear module, could achieve comparable performance of the d>n setting. For example, consider the model $s_{\\theta}(x_t) = Wx_t + Ug(V^Tx_t)$ where $V \\in \\mathbb{R}^{n \\times r}$ and $r \\ll n$. According to Theorem 4.2 in [5], the oracle score can be expressed as a linear term with a low-rank correction. This could potentially help alleviate the need for a large model width.\n\n5. It would be interesting to discuss whether such semantic tokenizers could be incorporated into unified MLLMs (e.g., for joint generation, understanding, and editing) to mitigate representational conflicts between different tasks [6].\n\n[1] Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Generation.\n\n[2] Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models.\n\n[3] DINOv2: Learning Robust Visual Features without Supervision.\n\n[4] Masked Autoencoders Are Effective Tokenizers for Diffusion Models.\n\n[5] Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?\n\n[6] Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oahaj8phnF", "reviewer": "Reviewer_UBC1", "summary": "The paper introduced Represenation Autoencoders, a new paradigm for choosing the latent encoder and decoder while training diffusion transformers. Rather than the conventionally used autoencoder SD-VAE, the authors claim that utilizing a preexisting foundation model as the encoder is more suited for training diffusion models since they contain semantic representation as well as portray excellent reconstruction ability. With the newly proposed autoencoder and additional design choices, specifically, (a higher dimensinonal latent space for the transfomer, utilizing a DDT head at the end of the diffusion transformer, a revised noise schedule to suit for Dinov2 latent space) the authors achieve state-of-the-art results in 256x256 and 512x512 generation in ImageNet 1M dataset. With the proposed approach additionally, the authors obtain a representation that seems to be cfg robust.", "questions": "1. What is the compute in GFLOPS due to the extra 163 M parameters introduced in RAE from Table 6 in RAE? Does this exceed the SDVAE compute cost?\n2. Figure 8a and b are a bit confusing; the convergence scores at 10^11 training iterations is different for Dit-DH-XL, Could the authors please clarify this?\n3.  Figure 2 portrays that the encoder of RAE has a compute cost of 22GFlops. However, the authors have not clarified which Dinov2 version this refers to. Could the authors please clarify this? \n4. Could the authors provide the performance for Table 7 DiT-DH-XL at 20 and 80 epochs in the presence of cfg?\n5. Can a smaller encoder-decoder be trained for SDVAE so that it is tuned for ImageNet 1M. Would such an encoder present better reconstruction performance with smaller number of parameters?\n6. By the design choices, would RAE need a big diffusion model and a large dino v2 encoder to train diffusion models on smaller datasets like FFHQ? \n7. In Table 9, Upsample is spelled wrong.\n8. While training  diffusion transformers for smaller datasets like FFHQ wouldn't the limitation of large latent space hold and wouln't the model be severely overparametrized? Additionally, if the method is not generalizable towards text to image generation and infeasible towards smaller datasets, wouldn't this limit the scope of the work?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. The introduced methods, along with the design choices, significantly reduce the number of convergence epochs of training diffusion transformers in the ImageNet 1M dataset.\n2. The proposed method seems to be robust with repect to cfg and achieves similar performance with and without cfg. This may enable faster generation with less compute overhead. I would advise the authors to consider this angle while presenting this work as a model that is robust to different cfg values as well as portray good performance without cfg would be useful to the community.\n3. The authors have tried extensive experiments to make the DiT model work with Dino-v2  based representation encoder on ImageNet dataset. \n4. Finally the paper is well written and each design choice is clearly explained.", "weaknesses": "1. The authors have done a good job presenting the rFID numbers and extensive experiments for Dinov2, however, Dinov2 is known to discard information relevant for perfect reconstruction during the encoding process. (Ex:- In Fig 9 , on the bus, all the text-based information is lost, moreover, the headlight bulbs fused into one, and the lines in the bus got joined or bent ) Given that such information cannot be captured by the encoder, how can a generation process be guided by such features? Moreover, looking at Fig 13, 14, 15, there are multiple cases like this where the generated images lose semantic details.\n2. How easy is it for RAE to generalize for text-to-image generation? Dinov2 is known to not preserve colours, textual information and straight lines. Would this impact the performance of RAE and its generalizability towards text-to-image generation?\n3. Dinov2 features are not discriminative towards instances of the same object class, for example:  if two people are present in the scene, Dinov2 would give similar features for the hands of the two people. Assuming the text prompt is \"raise the left hand for the person on the left\" would this cause a problem towards the model being able  to identify this with text since all four hands in this case would be given similar features? Some examples can be seen in Figure 11/13 where multiple parrots/ dogs fuse to one.\n4. The SDVAE was originally trained for LAION dataset, and the size of the encoder and decoder was made to obtain the best performance in LAION. How does RAE autoencoder perform in LAION? Is a bigger encoder-decoder needed for RAE to work with the LAOIN dataset? As an actionable item, I would like the author's clarification of the performance RAE encoder-decoder on LAION, specifically on the PSNR, SSIM and rFID \n5.  There is a performance oscillation for  DiT-DH-XL with different versions of Dinov2 encoders as portrayed in Table 6, and they don't scale linearly with parameter sizes. Why is this? Would this mean that the models may not scale with larger foundation model encoders in the future?\n6. In Ln 448-450, the authors claim that \"the diffusion models trained for 256x256 could be generalized to 512x512\" is this claim valid for other datasets than ImageNet where more number of semantic objects are involved in the same scene?\n7. I personally applaud the authors efforts in trying to make the model work with Dino-v2 latent space, however a concern I have is , is such kind of extensive encoder-dependent tuning necessary if we are to swap in a different encoder? Wouldn't this reduce the utility of the approach?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)