Diffusion Transformers with Representation Autoencoders

## Abstract

Abstract Latent generative modeling, where a pretrained autoencoder maps pixels into a latent space for the diffusion process, has become the standard strategy for Diffusion Transformers (DiT); however, the autoencoder component has barely evolved. Most DiTs continue to rely on the original VAE encoder, which introduces several limitations: outdated backbones that compromise architectural simplicity, low-dimensional latent spaces that restrict information capacity, and weak representations that result from purely reconstruction-based training and ultimately limit generative quality. In this work, we explore replacing the VAE with pretrained representation encoders (e.g., DINO, SigLIP, MAE) paired with trained decoders, forming what we term Representation Autoencoders (RAEs). These models provide both high-quality reconstructions and semantically rich latent spaces, while allowing for a scalable transformer-based architecture. Since these latent spaces are typically high-dimensional, a key challenge is enabling diffusion transformers to operate effectively within them. We analyze the sources of this difficulty, propose theoretically motivated solutions, and validate them empirically. Our approach achieves faster convergence without auxiliary representation alignment losses.
Using a DiT variant equipped with a lightweight, wide DDT head, we achieve strong image generation results on ImageNet: 1.51 FID at 256 × 256 256\times 256 (no guidance) and 1.13 at both 256 × 256 256\times 256 and 512 × 512 512\times 512 (with guidance). RAE offers clear advantages and should be the new default for diffusion transformer training. Project page: rae-dit.github.io

## 1 Introduction

The evolution of generative modeling has been driven by a continual redefinition of *where* and *how* models learn to represent data. Early pixel-space models sought to directly capture image statistics, but the emergence of latent diffusion (Vahdat et al., 2021; Rombach et al., 2022) reframed generation as a process operating within a learned, compact representation space. By diffusing in this space rather than in raw pixels, models such as Latent Diffusion Models (LDM) (Rombach et al., 2022) and Diffusion Transformers (DiT) (Peebles & Xie, 2023; Ma et al., 2024) achieve higher visual fidelity and efficiency, powering the most capable image and video generators of today.

Despite progress in diffusion backbones, the autoencoder defining the latent space remains largely unchanged. The widely used SD-VAE (Rombach et al., 2022) still relies on heavy channel-wise compression and a reconstruction-only objective, producing low-capacity latents that capture local appearance but lack global semantic structure crucial for generalization and generative performance of diffusion models (Song et al., 2025). In addition, SD-VAE, built on a legacy convolutional design, remains computationally inefficient (see Fig. [2](https://arxiv.org/html/2510.11690v1#S1.F2)).

Meanwhile, visual representation learning has undergone a rapid transformation. Self-supervised and multimodal encoders such as DINO (Oquab et al., 2023), MAE (He et al., 2021), JEPA (Assran et al., 2023) and CLIP / SigLIP (Radford et al., 2021; Tschannen et al., 2025) learn semantically structured visual features that generalize across tasks and scales and provide a natural basis for visual understanding. However, latent diffusion remains largely isolated from this progress, continuing to diffuse in reconstruction-trained VAE spaces rather than semantically meaningful representational ones. Recent work attempts to improve latent quality *indirectly* through REPA-style (Yu et al., 2025; Yao et al., 2025; Leng et al., 2025) alignment with external encoders, but these methods introduce extra training stages, auxiliary losses, and tuning complexity.

This separation stems from long-standing assumptions about the *incompatibility* between semantic and generative objectives. It is widely believed that encoders trained to capture semantics are *unsuited* for faithful reconstruction, since they *“focus on high-level information and can only reconstruct an image with high-level semantic similarities.*” (Yu et al., 2024b) In addition, diffusion models are believed to perform poorly in high-dimensional latent spaces (Skorokhodov et al., 2025; Yao et al., 2025; Esser et al., 2024; Liu et al., 2024), leading practitioners to favor low-dimensional VAE latents over the typically much higher-dimensional representations of semantic encoders. In this work, we show that *both assumptions might be wrong*. We demonstrate that *frozen* representation encoders, even those explicitly optimized for semantics over reconstruction, can be repurposed into powerful autoencoders for generation, yielding reconstructions superior to SD-VAE without architectural complexity or auxiliary losses. Furthermore, we find that diffusion transformer training can be stable and efficient in these higher-dimensional latent spaces. With the right architectural adjustments, higher-dimensional representations are not a liability but an *advantage*, offering richer structure, faster convergence, and better generation quality. Notably, higher-dimensional latents introduce effectively *no extra compute or memory costs* since the token count is fixed (determined by the patch size) and the channels are projected to the DiT hidden dimension in the first layer.

We formalize this insight through *Representation Autoencoders* (RAEs), a new class of autoencoders that replace the VAE with a pretrained representation encoder (e.g., DINO) paired with a trained decoder. RAEs produce latent spaces that are semantically rich, structurally coherent, and *diffusion-friendly*, linking semantic and generative modeling through a shared latent representation.

While feasible in principle, adapting diffusion transformers to these high-dimensional semantic latents requires careful design. Original DiTs, designed for compact SD-VAEs, struggle with the increased dimensionality due to:
(1) *Transformer design:* DiTs cannot fit even a single image unless their width exceeds the token dimension, implying width must scale with latent dimensionality; (2) *Noise scheduling:* resolution-based schedule shifts (Chen, 2023; Hoogeboom et al., 2023; Esser et al., 2024), derived from pixel- and VAE-based inputs, neglect token dimensionality, motivating a dimensionality-dependent shift; (3) *Decoder robustness:* unlike VAEs trained on continuous latent distributions (Kingma & Welling, 2014), RAE decoders learn from discretely supported latents but must reconstruct samples from a diffusion model that follow a continuous distribution, which we address by noise-augmented decoder training. Finally, we introduce a new DiT variant, $\text{DiT}^{\text{DH}}$, inspired by DDT (Wang et al., 2025c) but motivated by a different design perspective. It augments the standard DiT architecture with a lightweight, *shallow yet wide* head, enabling the diffusion model to scale in width without incurring quadratic computational costs. Empirically, this design further enhances diffusion transformer training in high-dimensional RAE spaces.

Empirically, RAEs demonstrate strong visual generation performance. On ImageNet, our RAE-based $\text{DiT}^{\text{DH}}$ achieves FIDs of 1.51 at 256×256 without any guidance, and 1.13 at both 256×256 and 512×512 with AutoGuidance (Karras et al., 2025), showing the effectiveness of RAEs as an alternative to conventional VAEs in diffusion transformer training. More broadly, these results reframes autoencoding from a compression mechanism into a representation foundation, one that enables diffusion transformers to train more efficiently and generate more effectively.

Figure: Figure 2: Comparison of SD-VAE and RAE (DINOv2-B). The VAE relies on convolutional backbones with aggressive down- and up-sampling, while the RAE uses a ViT architecture without compression. SD-VAE is also more computationally expensive, requiring about 6× and 3× more GFLOPs than RAE for the encoder and decoder, respectively. GFlops are evaluated on one $256\times 256$ image.
Refer to caption: x2.png

## 2 Related works

Here, we discuss previous work on the line of representation learning and reconstruction/generation. We present a more detailed related work discussion in [Appendix A](https://arxiv.org/html/2510.11690v1#A1).

Representation for Reconstruction.
Recent work explores enhancing VAEs with semantic representations: VA-VAE (Yao et al., 2025) aligns VAE latents with a pretrained representation encoder, while MAETok (Chen et al., 2025a), DC-AE 1.5 (Chen et al., 2025d), and l-DEtok (Yang et al., 2025) incorporate MAE- or DAE (Vincent et al., 2008)-inspired objectives into VAE training. Such alignment greatly improves reconstruction and generation performance of VAEs, yet its reliance on heavily compressed, low-dimensional latents still limits both reconstruction fidelity and representation quality. In contrast, we reconstruct directly from representation encoders features without compression. We show that, with a simple ViT decoder on top of frozen representation encoders features, it achieves reconstruction quality comparable to or better than SD-VAE (Rombach et al., 2022), while preserving substantially stronger representations.

Representation for Generation.
Recent work also explores using semantic representations to improve generative modeling.
REPA (Yu et al., 2025) accelerates DiT convergence by aligning its middle block with representation encoders features. DDT (Wang et al., 2025c) further improves convergence by decoupling DiT into an encoder–decoder and applying REPA loss to the encoder output. REG (Wu et al., 2025) introduces a learnable token into the DiT sequence and explicitly aligns it with a representation encoders representation. ReDi (Kouzelis et al., 2025b) generates both VAE latents and PCA components of DINOv2 features within a diffusion model. In contrast, we train diffusion models directly on representation encoders and achieve faster convergence. We also recommend  (Dieleman, 2025) for a broader perspective on this topic.

## 3 High Fidelity Reconstruction from Frozen Encoders

In this section, we challenge the common assumption that pretrained representation encoders, such as DINOv2 (Oquab et al., 2023) and SigLIP2 (Tschannen et al., 2025), are unsuitable for the reconstruction task because they *“emphasize high-level semantics while downplaying low-level details”* (Tang et al., 2025; Yu et al., 2024b). We show that, with a properly trained decoder, frozen representation encoders can in fact serve as strong encoders for the diffusion latent space. Our Representation Autoencoders (RAE) pair frozen, pretrained representation encoders with a ViT-based decoder, yielding reconstructions on par with or even better than SD-VAE. More importantly, RAEs alleviate the fundamental limitations of VAEs (Kingma & Welling, 2014), whose heavily compressed latent space (e.g. SD-VAE maps $256^{2}$ images to $32^{2}\times 4$ (Esser et al., 2021; Rombach et al., 2022) latents) restricts reconstruction fidelity and more importantly, representation quality.

Our training recipe for the RAE decoder is as follows.
Given an input ${\mathbf{x}}\in\mathbb{R}^{3\times H\times W}$ and the frozen representation encoder $E$ with patch size $p_{e}$ and hidden size $d$, we obtain $N=HW/p_{e}^{2}$ tokens with channel $d$. A ViT decoder $D$ with patch size $p_{d}$ maps them back to pixels with shape $3\times H\frac{p_{d}}{p_{e}}\times W\frac{p_{d}}{p_{e}}$; By default we use $p_{d}=p_{e}$, so the reconstruction matches the input resolution. For all experiments on $256\times 256$ images, the encoder produces 256 tokens, matching the token count of most prior DiT-based models trained with SD-VAE latents (Peebles & Xie, 2023; Yu et al., 2025; Ma et al., 2024). The decoder $D$ is trained with a combination of L1, LPIPS (Zhang et al., 2018), and adversarial losses (Goodfellow et al., 2014), following common practice in VAEs:

$$ $\displaystyle z=E(x),\hat{x}$ $\displaystyle=D(z)$ $\displaystyle\mathcal{L}_{rec}(x)=\omega_{L}\operatorname{LPIPS}(\hat{x},x)+$ $\displaystyle\operatorname{L1}(\hat{x},x)+\omega_{G}\lambda\operatorname{GAN}(\hat{x},x),$ $$

We provide implementation details about the decoder architecture, hyperparameters, coefficients, and GAN training details in Appendix [C](https://arxiv.org/html/2510.11690v1#A3).

We select three representative encoders from different pretraining paradigms: DINOv2-B (Oquab et al., 2023) ($p_{e}$=14, $d$=768), a self-supervised self-distillation model; SigLIP2-B (Tschannen et al., 2025) ($p_{e}$=16, $d$=768), a language-supervised model; and MAE-B (He et al., 2021) ($p_{e}$=16, $d$=768), a masked autoencoder. For DINOv2, we also study different model sizes S,B,L ($d$=384,768,1024). Unless otherwise specified, we use an ViT-XL decoder for all RAEs. We use FID score (Heusel et al., 2017) computed on the reconstructed ImageNet (Russakovsky et al., 2015) validation set as our main metric for reconstruction quality, denoted as rFID.

**(a) Encoder choice. All encoders outperform SD-VAE.**
| Model | rFID |
| --- | --- |
| DINOv2-B | 0.49 |
| SigLIP2-B | 0.53 |
| MAE-B | 0.16 |
| SD-VAE | 0.62 |

#### Reconstruction, scaling, and representation.

As shown in Table [1a](https://arxiv.org/html/2510.11690v1#S3.T1.sf1), RAEs with frozen encoders achieve consistently better reconstruction quality (rFID) than SD-VAE. For instance, RAE with MAE-B/16 reaches an rFID of 0.16, clearly outperforming SD-VAE and challenging the assumption that representation encoders cannot recover pixel-level detail.

We next study the scaling behavior of both encoders and decoders. As shown in [Table 1c](https://arxiv.org/html/2510.11690v1#S3.T1.sf3), reconstruction quality remains stable across DINOv2-S, B, and L, indicating that even small representation encoders models preserve sufficient low-level detail for decoding. On the decoder side ([Table 1b](https://arxiv.org/html/2510.11690v1#S3.T1.sf2)), increasing capacity consistently improves rFID: from 0.58 with ViT-B to 0.49 with ViT-XL. Importantly, ViT-B already outperforms SD-VAE while being $14\times$ more efficient in GFLOPs, and ViT-XL further improves quality at only one-third of SD-VAE’s cost.

We also evaluate representation quality via linear probing on ImageNet-1K in [Table 1d](https://arxiv.org/html/2510.11690v1#S3.T1.sf4). Because RAEs use frozen pretrained encoders, they directly inherit the representation of the underlying representation encoders. Since RAEs use frozen pretrained encoders, they retain the strong representations of the underlying representation encoders. In contrast, SD-VAE achieves only $\sim$8% accuracy.

## 4 Taming Diffusion Transformers for RAE

With RAE demonstrating good reconstruction quality, we now proceed to investigate the *diffusability* (Skorokhodov et al., 2025) of its latent space; that is, how easily its latent distribution can be modeled by a diffusion model, and how good the generation performance can be.

Before turning to generation, we fix the encoder to study generation capabilities. [Table 1a](https://arxiv.org/html/2510.11690v1#S3.T1.sf1) shows that MAE, SigLIP2, and DINOv2 all achieve lower reconstruction rFID than SD-VAE, with MAE the best among them. However, reconstruction alone does not determine generation quality. Empirically, DINOv2 produces the strongest generation results, so unless otherwise noted, we adopt DINOv2 as our default encoder and defer the full comparison to [Section G.1](https://arxiv.org/html/2510.11690v1#A7.SS1).

Following standard practice, we adopt the flow matching objective (Lipman et al., 2023; Liu et al., 2023) with linear interpolation ${\mathbf{x}}_{t}=(1-t){\mathbf{x}}+t\bm{\varepsilon}$,
where ${\mathbf{x}}\sim p({\mathbf{x}})$ and $\bm{\varepsilon}\sim{\mathcal{N}}(0,\mathbf{I})$, and train the model to predict the velocity $v({\mathbf{x}}_{t},t)$ (see [Appendix J](https://arxiv.org/html/2510.11690v1#A10)). We use LightningDiT (Yao et al., 2025), a variant of DiT (Peebles & Xie, 2023), as our model backbone.
We adopt a patch size of 1, which results in a sequence length of 256 for all RAEs on $256\times 256$ images, matching the token length used by VAE-based DiTs (Peebles & Xie, 2023; Yu et al., 2025; Yao et al., 2025).
Since the computational cost of DiT depends primarily on the sequence length, using DiT on RAE therefore effectively incurs no additional overhead compared to its VAE-based counterparts.
We evaluate our models using FID computed on 50K samples generated with 50 steps with the Euler sampler (denoted as gFID), and all quantitative results are trained for 80 training epochs on ImageNet at $256\times 256$ unless otherwise specified. More training details are included in [Appendix D](https://arxiv.org/html/2510.11690v1#A4).

**Table 2: Standard DiT struggles to model RAE’s latent distribution.**
|  | RAE | SD-VAE |
| --- | --- | --- |
| DiT-S | 215.76 | 51.74 |
| DiT-XL | 23.08 | 7.13 |

DiT does not work out of the box.
To our surprise, the standard diffusion recipe fails with RAE (see [Table 2](https://arxiv.org/html/2510.11690v1#S4.T2.fig1)). Training directly on RAE latents causes a small backbone such as DiT-S to completely fail, while a larger backbone like DiT-XL significantly underperforms it’s counterpart with the SD-VAE latents.

To investigate this observation, we raise several hypotheses detailed below, which we will discuss in the following sections:

#### 

### 4.1 Scaling DiT Width to Match Token Dimensionality

Figure: Figure 3: Overfitting to a single sample. Left: increasing model width lead to lower loss and better sample quality; Right: changing model depth has marginal effect on overfitting results.
Refer to caption: x3.png

To better understand the training dynamics of diffusion transformers with RAE latents, we first construct a simplified experiment. Rather than training on the entire ImageNet, we randomly select a single image, encode it by RAE, and test whether the diffusion model can *reconstruct* it.

[Table 2](https://arxiv.org/html/2510.11690v1#S4.T2.fig1) shows that although RAE underperforms SD-VAE, DiT performance improves with increased capacity. To dissect this effect, we vary model width while fixing depth. Using a DiT-S, we increase the hidden dimension from $384$ to $784$. As shown in [Figure 3](https://arxiv.org/html/2510.11690v1#S4.F3), sample quality is poor when the model width $d<$ token dimension $n=768$, but improves sharply and reproduces the input almost perfectly once $d\geq n$. Training losses exhibit the same trend, converging only when $d\geq n$.

One might suspect that this improvement still arises from the larger model capacity. To disentangle this effect, we fix the width at $d=384$ and vary the depth of the SiT-S model. As shown in [Figure 3](https://arxiv.org/html/2510.11690v1#S4.F3), even when doubling the depth from $12$ to $24$, the generated images remain artifact-heavy, and the training losses shown in [Figure 3](https://arxiv.org/html/2510.11690v1#S4.F3) fail to converge to similar level of $d=768$.

Together, the results indicate that for generation in RAE’s latent space to succeed, the diffusion model’s width must match or exceed the RAE’s token dimension. At first glance, this appears to contradict the common belief that data manifolds have low intrinsic dimensionality (Pope et al., 2021), allowing generative models such as GANs to operate effectively within that manifold without scaling to the full data dimension (Sauer et al., 2022). We argue that this contradiction arises from the formulation of diffusion models ([Appendix J](https://arxiv.org/html/2510.11690v1#A10)): injecting Gaussian noise directly to the data (e.g., in the construction of $x_{t}$) throughout training effectively extends the data distribution’s support to the entire space, thereby “diffusing” the data manifold into a full-rank one, requiring model capacity that scales with the full data dimensionality.

In the following, we provide a theoretical justification for this conjecture:

theoremTrainBound
Assuming ${\mathbf{x}}\sim p({\mathbf{x}})\in{\mathbb{R}}^{n},\bm{\varepsilon}\sim{\mathcal{N}}(0,\mathbf{I}_{n})$, $t\in[0,1]$. Let ${\mathbf{x}}_{t}=(1-t){\mathbf{x}}+t\bm{\varepsilon}$, consider the function family

$$ $\displaystyle{\mathcal{G}}_{d}=\{g({\mathbf{x}}_{t},t)={\bm{B}}f({\bm{A}}{\mathbf{x}}_{t},t):{\bm{A}}\in{\mathbb{R}}^{d\times n},{\bm{B}}\in{\mathbb{R}}^{n\times d},f:[0,1]\times{\mathbb{R}}^{d}\to{\mathbb{R}}^{d}\}$ (1) $$

where $d<n$, $f$ refers to a stack of standard DiT blocks whose width is smaller than the token dimension from the representation encoder, and ${\bm{A}},{\bm{B}}$ denote the input and output linear projections, respectively. Then for any $g\in{\mathcal{G}}_{d}$,

$$ $\displaystyle\mathcal{L}(g,\theta)$ $\displaystyle=\int_{0}^{1}\mathbb{E}_{{\mathbf{x}}\sim p({\mathbf{x}}),\bm{\varepsilon}\sim{\mathcal{N}}(0,\mathbf{I}_{n})}\big[\|g({\mathbf{x}}_{t},t)-(\bm{\varepsilon}-{\mathbf{x}})\|^{2}\big]\mathrm{d}t\geq\sum_{i=d+1}^{n}\lambda_{i}$ (2) $$

where $\lambda_{i}$ are the eigenvalues of the covariance matrix of the random variable $W=\bm{\varepsilon}-{\mathbf{x}}$.

Notably, when $d\geq n$, ${\mathcal{G}}_{d}$ contains the unique minimizer to ${\mathcal{L}}(g,\theta)$.

###### Proof.

See [Section B.1](https://arxiv.org/html/2510.11690v1#A2.SS1).
∎

In our toy setting where $p({\mathbf{x}})=\delta({\mathbf{x}}-{\mathbf{x}}_{0})$, we have $W\sim{\mathcal{N}}(-{\mathbf{x}},\mathbf{I}_{n})$ and $\lambda_{i}=1$ for all $i$. Thus by Theorem [3](https://arxiv.org/html/2510.11690v1#S4.F3), the lower bound of the average loss becomes $\tilde{{\mathcal{L}}}(g,\theta)\geq\frac{1}{n}\sum_{i=d+1}^{n}1=\frac{n-d}{n}$. As shown in [Figure 3](https://arxiv.org/html/2510.11690v1#S4.F3), this theoretical bound is consistent with our empirical results.

**Table 3: Overfitting losses. Compared between different combinations of model width and token dimension.**
|  | DiT-S | DiT-B | DiT-L |
| --- | --- | --- | --- |
| DINOv2-S | $3.6\mathrm{e}{-2}$ ✓ | $1.0\mathrm{e}{-3}$ ✓ | $9.7\mathrm{e}{-4}$ ✓ |
| DINOv2-B | $5.2\mathrm{e}{-1}$ ✗ | $2.4\mathrm{e}{-2}$ ✓ | $1.3\mathrm{e}{-3}$ ✓ |
| DINOv2-L | $6.5\mathrm{e}{-1}$ ✗ | $2.7\mathrm{e}{-1}$ ✗ | $2.2\mathrm{e}{-2}$ ✓ |

We further extend our investigation to a more practical setting by examining three models of varying width, {DiT-S, DiT-B, DiT-L}. Each model is overfit on a single image encoded by {DINOv2-S, DINOv2-B, DINOv2-L}, respectively, corresponding to different token dimensions. As shown in [Table 3](https://arxiv.org/html/2510.11690v1#S4.T3.fig1), convergence occurs only when the model width is at least as large as the token dimension (e.g., DiT-B with DINOv2-B), while the loss fails to converge otherwise (e.g., DiT-S with DINOv2-B).

###### Proof.

#### 

### 4.2 Dimension-Dependent Noise Schedule Shift

Many prior works (Teng et al., 2023; Chen, 2023; Hoogeboom et al., 2023; Esser et al., 2024) have observed that, for inputs ${\mathbf{z}}\in{\mathbb{R}}^{C\times H\times W}$, increasing the spatial resolution ($H\times W$) reduces information corruption at the same noise level, impairing diffusion training. These findings, however, are based mainly on pixel- or VAE-encoded inputs with few channels (e.g., $C\leq 16$). In practice, the Gaussian noise is applied to both spatial and channel dimensions; as the number of channels increases, the effective “resolution” per token also grows, reducing information corruption further. We therefore argue that proposed resolution-dependent strategies in these prior works should be generalized to the effective data dimension, defined as the number of tokens times their dimensionality.

**Table 4: Impact of schedule shift.**
|  | gFID |
| --- | --- |
| w/o shift | $23.075\,045\,3$ |
| w/ shift | 4.81 |

We adopt the shifting strategy of Esser et al. (2024): for a schedule $t_{n}\in[0,1]$ and input dimensions $n,m$, the shifted timestep is defined as $t_{m}=\frac{\alpha t_{n}}{1+(\alpha-1)t_{n}}$ where $\alpha=\sqrt{m/n}$ is a dimension-dependent scaling factor. We follow (Esser et al., 2024) in using $n=4096$
as the base dimension and set $m$ to the effective data dimension of RAE. As shown in [Table 4](https://arxiv.org/html/2510.11690v1#S4.T4), this yields significant performance gains, showing its importance for training diffusion models in the high-dimensional RAE latent space.

#### 

### 4.3 Noise-Augmented Decoding

Unlike VAEs, where latent tokens are encoded as a continuous distribution ${\mathcal{N}}(\mu,\sigma^{2}\mathbf{I})$ (Kingma & Welling, 2014), the RAE decoder $D$ is trained to reconstruct images from the discrete distribution $p({\mathbf{z}})=\sum_{i}\delta({\mathbf{x}}-{\mathbf{z}}_{i})$, where $\{{\mathbf{z}}_{i}\}$ denotes the training set processed by the RAE encoder $E$. At inference time, however, the diffusion model may generate latents that are noisy or deviate slightly from the training distribution due to imperfect training and sampling Abuduweili et al. (2024). This could introduce a notable out-of-distribution challenge for $D$, which degrades sampling quality.

To mitigate this issue, inspired by prior works on Normalizing Flows (Dinh et al., 2017; Ho et al., 2019; Zhai et al., 2025), we augment the RAE decoder training with an additive noise ${\mathbf{n}}\sim{\mathcal{N}}(0,\sigma^{2}\mathbf{I})$. Concretely, rather than decoding directly from the clean latent distribution $p({\mathbf{z}})$, we train $D$ on a smoothed distribution $p_{{\mathbf{n}}}({\mathbf{z}})=\int p({\mathbf{z}}-{\mathbf{n}}){\mathcal{N}}(0,\sigma^{2}\mathbf{I})({\mathbf{n}})\mathrm{d}{\mathbf{n}}$ to enhance the decoder’s generalization to the denser output space of diffusion models. We further introduce stochasticity into $\sigma$ by sampling it from $|{\mathcal{N}}(0,\tau^{2})|$, which helps regularize training and improve robustness.

**Table 5: Impact of $p_{\mathbf{n}}({\mathbf{z}})$.**
|  | gFID | rFID |
| --- | --- | --- |
| ${\mathbf{z}}\sim p({\mathbf{z}})$ | 4.81 | 0.49 |
| ${\mathbf{z}}\sim p_{{\mathbf{n}}}({\mathbf{z}})$ | 4.28 | 0.57 |

We analyze how $p_{\mathbf{n}}({\mathbf{z}})$ affects reconstruction and generation. As shown in [Table 5](https://arxiv.org/html/2510.11690v1#S4.T5), it improves gFID but slightly worsens rFID. This trade-off is expected: adding noise smooths the latent distribution and, therefore, helps reduce OOD issues for the decoder, but also removes fine details, lowering reconstruction quality. We conduct more ablation experiments on $\tau$ and different encoders in [Section G.2](https://arxiv.org/html/2510.11690v1#A7.SS2).

Figure: Figure 4: DiT w/ RAE reaches much faster convergence and better FID than SiT or REPA.
Refer to caption: x5.png

We combine all of the above techniques to train a DiT-XL model on RAE latents. Our improved diffusion recipe achieves a gFID of 4.28 ([Figure 4](https://arxiv.org/html/2510.11690v1#S4.F4)) after only 80 epochs and 2.39 after 720 epochs in RAE’s latent space. With same model size, this not only surpasses prior diffusion baselines (e.g., SiT-XL (Ma et al., 2024)) trained on VAE latents (achieving a $47\times$ training speedup), but also outperforms the convergence speed of recent methods based on representation alignment (e.g., REPA-XL (Yu et al., 2025)), achieving a $16\times$ training speedup. In the following sections, we investigate ways to make RAE generation more efficient and effective, pushing it toward state-of-the-art performance.

#### 

## 5 Improving the Model Scalability with Wide Diffusion Head

As discussed in [Section 4](https://arxiv.org/html/2510.11690v1#S4), within the standard DiT framework, handling higher-dimensional RAE latents requires scaling up the width of the entire backbone, which quickly becomes computationally expensive. To overcome this limitation, we draw inspiration from DDT (Wang et al., 2025c) and introduce the DDT head—a shallow yet wide transformer module dedicated to denoising. By attaching this head to a standard DiT, we effectively increase model width without incurring quadratic growth in FLOPs. We refer to this augmented architecture as $\text{DiT}^{\text{DH}}$ throughout the remainder of the paper.
We also conduct experiment of the design choice of DDT head in [Section G.3](https://arxiv.org/html/2510.11690v1#A7.SS3)

Figure: Figure 5: The Wide DDT Head.
Refer to caption: x6.png

Wide DDT head. Formally, a $\text{DiT}^{\text{DH}}$ model consists of a base DiT $M$ and an additional wide, shallow transformer head $H$.
Given a noisy input $x_{t}$, timestep $t$, and an optional class label $y$, the combined model predicts the velocity $v_{t}$ as

$$ $\displaystyle z_{t}$ $\displaystyle=M(x_{t}\mid t,y),$ $\displaystyle v_{t}$ $\displaystyle=H(x_{t}\mid z_{t},t),$ $$

Figure: (a) $\text{DiT}^{\text{DH}}$ scales much better than DiT with RAE latents.
Refer to caption: x7.png

$\text{DiT}^{\text{DH}}$ converges faster than DiT. We train a series of $\text{DiT}^{\text{DH}}$ models with varying backbone sizes ($\text{DiT}^{\text{DH}}$-S, B, L, and XL) on RAE latents. We use a 2-layer, 2048-dim DDT head for all $\text{DiT}^{\text{DH}}$ models. Performance is compared against the standard DiT-XL baseline. As shown in [Figure 6a](https://arxiv.org/html/2510.11690v1#S5.F6.sf1), $\text{DiT}^{\text{DH}}$ is substantially more FLOP-efficient than DiT. For example, $\text{DiT}^{\text{DH}}$-B requires only $\sim$40% of the training FLOPs yet outperforms DiT-XL by a large margin; when scaled to $\text{DiT}^{\text{DH}}$-XL under a comparable training budget, $\text{DiT}^{\text{DH}}$ achieves an FID of 2.16—nearly half that of DiT-XL.

**Table 6: $\text{DiT}^{\text{DH}}$ outperforms DiT across RAE encoder sizes.**
| Model | DINOv2 |  |  |
| --- | --- | --- | --- |
| S | B | L |  |
| DiT-XL | $3.502\,643\,111$ | $4.276\,128\,739$ | $6.086\,614\,197$ |
| $\text{DiT}^{\text{DH}}$-XL | $2.420\,268\,923$ | $2.158\,400\,731$ | $2.726\,684\,653$ |

$\text{DiT}^{\text{DH}}$ maintains its advantage across RAE scales.
We compare $\text{DiT}^{\text{DH}}$-XL and DiT-XL on three RAE encoders—DINOv2-S, DINOv2-B, and DINOv2-L. As shown in [Table 6](https://arxiv.org/html/2510.11690v1#S5.T6), $\text{DiT}^{\text{DH}}$ consistently outperforms DiT, and the advantage grows with encoder size.
For example, with DINOv2-L, $\text{DiT}^{\text{DH}}$ improves FID from 6.09 to 2.73.
We attribute this robustness to the DDT head. Larger encoders produce higher-dimensional latents, which amplify the width bottleneck of DiT. $\text{DiT}^{\text{DH}}$ addresses this by satisfying the width requirement discussed in [Section 4](https://arxiv.org/html/2510.11690v1#S4) while keeping features compact. It also filters out noisy information that becomes more prevalent in high-dimensional RAE latents.

### 5.1 State-of-the-Art Diffusion Transformers

**Table 7: Class-conditional performance on ImageNet 512$\times$512 with guidance. $\text{DiT}^{\text{DH}}$ with 400-epoch training achieves an strong FID score of 1.13.**
| Method | Generation@512 |  |  |  |
| --- | --- | --- | --- | --- |
| gFID$\downarrow$ | IS$\uparrow$ | Prec.$\uparrow$ | Rec.$\uparrow$ |  |
| BigGAN-deep (Brock et al., 2019) | 8.43 | 177.9 | 0.88 | 0.29 |
| StyleGAN-XL (Sauer et al., 2022) | 2.41 | 267.8 | 0.77 | 0.52 |
| VAR (Tian et al., 2024) | 2.63 | 303.2 | - | - |
| MAGVIT-v2 (Yu et al., 2024a) | 1.91 | 324.3 | - | - |
| XAR (Ren et al., 2025) | 1.70 | 281.5 | - | - |
| ADM | 3.85 | 221.7 | 0.84 | 0.53 |
| SiD2 | 1.50 | - | - | - |
| DiT | 3.04 | 240.8 | 0.84 | 0.54 |
| SiT | 2.62 | 252.2 | 0.84 | 0.57 |
| DiffiT (Hatamizadeh et al., 2024) | 2.67 | 252.1 | 0.83 | 0.55 |
| REPA | 2.08 | 274.6 | 0.83 | 0.58 |
| DDT | 1.28 | 305.1 | 0.80 | 0.63 |
| EDM2 (Karras et al., 2024) | 1.25 | - | - | - |
| $\text{DiT}^{\text{DH}}$-XL (DINOv2-B) | 1.13 | 259.6 | 0.80 | 0.63 |

Convergence. We compare the convergence behavior of $\text{DiT}^{\text{DH}}$-XL with previous state-of-the-art diffusion models (Peebles & Xie, 2023; Ma et al., 2024; Yu et al., 2025; Gao et al., 2023; Yao et al., 2025) in terms of FID without guidance. In [Figure 6b](https://arxiv.org/html/2510.11690v1#S5.F6.sf2), we show the convergence curve of $\text{DiT}^{\text{DH}}$-XL with training epochs/GFLOPs, while baseline models are plotted at their reported final performance. $\text{DiT}^{\text{DH}}$-XL already surpasses REPA-XL, MDTv2-XL, and SiT-XL around $5\times 10^{10}$ GFLOPs, and by $5\times 10^{11}$ GFLOPs it achieves the best FID overall, requiring over 40$\times$ less compute.

Table: Table 8: Class-conditional performance on ImageNet 256$\times$256. RAE reaches an FID of 1.51 without guidance, outperforming all prior methods by a large margin. It also achieves an FID of 1.13 with AutoGuidance (Karras et al., 2025). We identified an inconsistency in the FID evaluation protocol in prior literature and re-ran the sampling process for several baselines. This resulted in higher baseline numbers than those originally reported. Further details are discussed in [Figure 7](https://arxiv.org/html/2510.11690v1#S5.F7).

Figure: Figure 7: Qualitative samples from our model trained at $512\times 512$ resolution with AutoGuidance. The RAE-based DiT demonstrates strong diversity, fine-grained detail, and high visual quality.
Refer to caption: assets/superimage-class-89.png

Scaling.
We compare $\text{DiT}^{\text{DH}}$ with recent methods of different model at different scales. As shown in [Figure 6c](https://arxiv.org/html/2510.11690v1#S5.F6.sf3), increasing the size of $\text{DiT}^{\text{DH}}$ consistently improves the FID scores. The smallest model, $\text{DiT}^{\text{DH}}$-S, reaches a competitive FID of 6.07, already outperforming the much larger REPA-XL. When scaling from $\text{DiT}^{\text{DH}}$-S to $\text{DiT}^{\text{DH}}$-B, the FID improves significantly from 6.07 to 3.38, surpassing all prior works of similar or even larger scale. The performance continues to improve with $\text{DiT}^{\text{DH}}$-XL, setting a new state-of-the-art result of 2.16 at 80 training epochs.

Performance. Finally, we provide a quantitative comparison between $\text{DiT}^{\text{DH}}$-XL, our most performant model, with recent state-of-the-art diffusion models on ImageNet $256\times 256$ and $512\times 512$ in [Table 8](https://arxiv.org/html/2510.11690v1#S5.T8) and [Table 7](https://arxiv.org/html/2510.11690v1#S5.T7). Our method outperforms all prior diffusion models by a large margin, setting new state-of-the-art FID scores of 1.51 without guidance and 1.13 with guidance at $256\times 256$. On $512\times 512$, with 400-epoch training, $\text{DiT}^{\text{DH}}$-XL further achieves an FID of 1.13 with guidance, surpassing the previous best performance achieved by EDM-2 (1.25).

Following the quantitative results, we present qualitative samples from our best models in [Figure 7](https://arxiv.org/html/2510.11690v1#S5.F7). These visualizations exhibit both high semantic diversity and fine-grained details comparable to ground-truth ImageNet samples, consistent with the achieved state-of-the-art FID. We provide additional visualization samples in [Appendix M](https://arxiv.org/html/2510.11690v1#A13) and unconditional generation results in [Appendix L](https://arxiv.org/html/2510.11690v1#A12).

Remarks on FID evaluation. To construct the 50,000 samples used for conditional FID evaluation, we note that previous works such as DDT (Wang et al., 2025c), VAR (Tian et al., 2024), MAR (Li et al., 2024b), and xAR (Ren et al., 2025) typically evaluate using exactly 50 images per class, while others employ uniform random sampling across the 1,000 class labels. Although uniform random sampling asymptotically approaches the balanced sampling case, we surprisingly observed that class-balanced sampling consistently achieves around 0.1 lower FID scores. To ensure fair comparison, we re-evaluate several recent methods with accessible checkpoints, such as SiT (Peebles & Xie, 2023), REPA (Yu et al., 2025), REPA-E (Leng et al., 2025) using class-balanced sampling and update their reported scores accordingly. A more detailed comparison is provided in [Appendix E](https://arxiv.org/html/2510.11690v1#A5).

## 6 Discussions

### 6.1 How can RAE extend to high-resolution synthesis efficiently?

A central challenge in generating high-resolution images is that resolution scales with the number of tokens: doubling image size in each dimension requires roughly four times as many tokens. To address this, we let the decoder handle resolution scaling by allowing its patch size patch size $p_{d}$ to differ from the encoder patch size $p_{e}$. When $p_{d}=p_{e}$, the output matches the input resolution; setting $p_{d}=2p_{e}$ produces a $2\times$ upsampled image, reconstructing a $512\times 512$ image from the same tokens used at $256\times 256$.

**Table 9: Comparison on ImageNet 512 $\times$ 512. Decoder upsampling achieves competitive FID compared to direct 512-resolution training. Both models are trained for 400 epochs.**
| Method | #Tokens | gFID $\downarrow$ | rFID$\downarrow$ |
| --- | --- | --- | --- |
| Direct | 1024 | 1.13 | 0.53 |
| Upsample | 256 | 1.61 | 0.97 |

Since the decoder is decoupled from both the encoder and the diffusion process, we can reuse diffusion models trained at $256\times 256$ resolution, simply swapping in an upsampling decoder to produce $512\times 512$ outputs without retraining. As shown in [Table 9](https://arxiv.org/html/2510.11690v1#S6.T9), this approach slightly increases rFID but achieves competitive gFID, while being 4$\times$ more efficient than quadrupling the number of tokens.

### 6.2 Does DiT DH \text{DiT}^{\text{DH}} work without RAE?

In this work, we propose and study RAE and $\text{DiT}^{\text{DH}}$. In [Section 4](https://arxiv.org/html/2510.11690v1#S4), we showed that RAE with DiT already brings substantial benefits, even in the absence of $\text{DiT}^{\text{DH}}$. Here, we turn the question around: can $\text{DiT}^{\text{DH}}$ still provide improvements, without the latent space of RAE?

**Table 10: Performance on VAE. $\text{DiT}^{\text{DH}}$ yields worse FID than DiT, despite using extra compute for the wide DDT head.**
|  | VAE | DINOv2-B |
| --- | --- | --- |
| DiT-XL | 7.13 | 4.28 |
| $\text{DiT}^{\text{DH}}$-XL | 11.70 | 2.16 |

To investigate, we train both DiT-XL and $\text{DiT}^{\text{DH}}$-XL on SD-VAE latents with a patch size of $2$, alongside DINOv2-B for comparison, for 80 epochs, and report unguided FID. As shown in [Table 10](https://arxiv.org/html/2510.11690v1#S6.T10), $\text{DiT}^{\text{DH}}$-XL performs even worse than DiT-XL on SD-VAE, despite the additional computation introduced by the diffusion head. This indicates that the DDT head provides little benefit in low-dimensional latent spaces, and its primary strength arises in high-dimensional diffusion tasks introduced by RAE.

### 6.3 The role of structured representation in high-dimensional diffusion?

$\text{DiT}^{\text{DH}}$ achieves strong performance when paired with the high-dimensional latent space of RAE. This raises a key question: is the structured representation of RAE essential, or would $\text{DiT}^{\text{DH}}$ work equally well on unstructured high-dimensional inputs such as *raw pixels*?

**Table 11: Comparison on pixel diffusion. Pixel Diffusion has much worse FID than diffusion on DINOv2-B.**
|  | Pixel | DINOv2-B |
| --- | --- | --- |
| DiT-XL | 51.09 | 4.28 |
| $\text{DiT}^{\text{DH}}$-XL | 30.56 | 2.16 |

To evaluate this, we train DiT-XL and $\text{DiT}^{\text{DH}}$-XL directly on raw pixels. For $256\times 256$ images with a patch size of 16, the resulting DiT input token dimensionality is $16\times 16\times 3=768$, matching that of the DINOv2-B latents. We report unguided FID after 80 epochs. As shown in [Table 11](https://arxiv.org/html/2510.11690v1#S6.T11), $\text{DiT}^{\text{DH}}$ outperforms DiT on pixels, but both models perform far worse than their counterparts trained on RAE latents. These results demonstrate that high dimensionality alone is not sufficient: the structured representation provided by RAE is crucial for achieving strong performance gains.

## 7 Conclusion

In this work, we challenge the belief that pretrained representation encoders are too high-dimensional and too semantic for reconstruction or generation. We show that a frozen representation encoder, paired with a lightweight trained decoder, forms an effective Representation Autoencoder (RAE). On this latent space, we train Diffusion Transformers in a stable and efficient way with three added components: (1) match DiT width to the encoder token dimensionality; (2) apply a dimension-dependent shift to the noise schedule; and (3) add decoder noise augmentation so the decoder handles diffusion outputs. We also introduce DiT^DH, a shallow-but-wide diffusion transformer head that increases width without quadratic compute. Empirically, RAEs enable strong visual generation: on ImageNet, our RAE-based $\text{DiT}^{\text{DH}}$-XL achieves 1.51 FID at $256\times 256$ (without guidance) and 1.13 at both $256\times 256$ and $512\times 512$ (with guidance). We believe RAE latents serve as a strong candidate for training diffusion transformers in the future.

## 8 Acknowlegments

The authors would like to thank Sihyun Yu, Jaskirat Singh, Shusheng Yang, Xichen Pan, Chenyu Li, Bingda Tang, Fred Lu, David Fan, Amir Bar, Shangyuan Tong for insightful discussions and feedback on the manuscript. This work was mainly supported by the Google TPU Research Cloud (TRC)
program and the Open
Path AI Foundation. ST is supported by Meta AI Mentorship Program. SX also acknowledges support from the MSIT IITP grant (RS-2024-00457882) and the NSF award
IIS-2443404.

## References

- Abuduweili et al. (2024)
Abulikemu Abuduweili, Chenyang Yuan, Changliu Liu, and Frank Permenter.
Enhancing sample generation of diffusion models using noise level correction.
*TMLR*, 2024.
- Albergo et al. (2023)
Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden.
Stochastic interpolants: A unifying framework for flows and diffusions.
*arXiv preprint arXiv:2303.08797*, 2023.
- Assran et al. (2023)
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas.
Self-supervised learning from images with a joint-embedding predictive architecture.
In *CVPR*, 2023.
- Brock et al. (2019)
Andrew Brock, Jeff Donahue, and Karen Simonyan.
Large scale gan training for high fidelity natural image synthesis.
In *ICLR*, 2019.
- Cao et al. (2023)
Shiyue Cao, Yueqin Yin, Lianghua Huang, Yu Liu, Xin Zhao, Deli Zhao, and Kaigi Huang.
Efficient-vqgan: Towards high-resolution image generation with efficient vision transformers.
In *ICCV*, 2023.
- Caron et al. (2021)
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.
Emerging properties in self-supervised vision transformers.
In *ICCV*, 2021.
- Chen et al. (2025a)
Hao Chen, Yujin Han, Fangyi Chen, Xiang Li, Yidong Wang, Jindong Wang, Ze Wang, Zicheng Liu, Difan Zou, and Bhiksha Raj.
Masked autoencoders are effective tokenizers for diffusion models.
In *ICML*, 2025a.
- Chen et al. (2025b)
Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, and Ran Xu.
Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset, 2025b.
- Chen et al. (2025c)
Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang, Haotian Tang, Muyang Li, Yao Lu, and Song Han.
Deep compression autoencoder for efficient high-resolution diffusion models.
In *ICLR*, 2025c.
- Chen et al. (2025d)
Junyu Chen, Dongyun Zou, Wenkun He, Junsong Chen, Enze Xie, Song Han, and Han Cai.
Dc-ae 1.5: Accelerating diffusion model convergence with structured latent space, 2025d.
- Chen et al. (2020a)
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever.
Generative pretraining from pixels.
In *ICML*, 2020a.
- Chen et al. (2025e)
Shoufa Chen, Chongjian Ge, Shilong Zhang, Peize Sun, and Ping Luo.
Pixelflow: Pixel-space generative models with flow.
*arXiv preprint arXiv:2504.07963*, 2025e.
- Chen (2023)
Ting Chen.
On the importance of noise scheduling for diffusion models.
*arXiv preprint arXiv:2301.10972*, 2023.
- Chen et al. (2020b)
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.
A simple framework for contrastive learning of visual representations.
In *ICML*, 2020b.
- Chen et al. (2024)
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al.
Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.
In *CVPR*, 2024.
- Darcet et al. (2025)
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski.
Vision transformers need registers.
In *ICLR*, 2025.
- De Fauw et al. (2019)
Jeffrey De Fauw, Sander Dieleman, and Karen Simonyan.
Hierarchical autoregressive image models with auxiliary decoders.
*arXiv preprint arXiv:1903.04933*, 2019.
- Dhariwal & Nichol (2021)
Prafulla Dhariwal and Alexander Nichol.
Diffusion models beat GANs on image synthesis.
In *NeurIPS*, 2021.
- Dieleman (2025)
Sander Dieleman.
Generative modelling in latent space, 2025.
URL [https://sander.ai/2025/04/15/latents.html](https://sander.ai/2025/04/15/latents.html).
- Dinh et al. (2017)
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.
Density estimation using real nvp.
In *ICLR*, 2017.
- Dosovitskiy et al. (2021)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.
An image is worth 16x16 words: Transformers for image recognition at scale.
In *ICLR*, 2021.
- Esser et al. (2021)
Patrick Esser, Robin Rombach, and Björn Ommer.
Taming transformers for high-resolution image synthesis.
In *CVPR*, 2021.
- Esser et al. (2024)
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach.
Scaling rectified flow transformers for high-resolution image synthesis.
In *ICML*, 2024.
- Fan et al. (2025a)
David Fan, Shengbang Tong, Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rabbat, Nicolas Ballas, Yann LeCun, Amir Bar, et al.
Scaling language-free visual representation learning.
In *ICCV*, 2025a.
- Fan et al. (2025b)
Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian.
Fluid: Scaling autoregressive text-to-image generative models with continuous tokens.
In *ICLR*, 2025b.
- Gao et al. (2023)
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan.
Mdtv2: Masked diffusion transformer is a strong image synthesizer.
*arXiv preprint arXiv:2303.14389*, 2023.
- Goodfellow et al. (2014)
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
Generative adversarial nets.
In *NeurIPS*, 2014.
- Grill et al. (2020)
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.
Bootstrap your own latent-a new approach to self-supervised learning.
In *NeurIPS*, 2020.
- Hansen-Estruch et al. (2025)
Philippe Hansen-Estruch, David Yan, Ching-Yao Chung, Orr Zohar, Jialiang Wang, Tingbo Hou, Tao Xu, Sriram Vishwanath, Peter Vajda, and Xinlei Chen.
Learnings from scaling visual tokenizers for reconstruction and generation.
In *ICML*, 2025.
- Hatamizadeh et al. (2024)
Ali Hatamizadeh, Jiaming Song, Guilin Liu, Jan Kautz, and Arash Vahdat.
DiffiT: Diffusion vision transformers for image generation.
In *ECCV*, 2024.
- He et al. (2019)
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.
Momentum contrast for unsupervised visual representation learning. arxiv e-prints, art.
In *CVPR*, 2019.
- He et al. (2021)
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.
Masked autoencoders are scalable vision learners.
In *CVPR*, 2021.
- Heusel et al. (2017)
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
GANs trained by a two time-scale update rule converge to a local nash equilibrium.
In *NeurIPS*, 2017.
- Hinton & Salakhutdinov (2006)
Geoffrey E Hinton and Ruslan R Salakhutdinov.
Reducing the dimensionality of data with neural networks.
*science*, 313(5786), 2006.
- Ho & Salimans (2022)
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance, 2022.
- Ho et al. (2019)
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel.
Flow++: Improving flow-based generative models with variational dequantization and architecture design.
In *ICML*, 2019.
- Ho et al. (2020)
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
In *NeurIPS*, 2020.
- Hoogeboom et al. (2023)
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans.
Simple diffusion: End-to-end diffusion for high resolution images.
In *ICML*, 2023.
- Hoogeboom et al. (2025)
Emiel Hoogeboom, Thomas Mensink, Jonathan Heek, Kay Lamerigts, Ruiqi Gao, and Tim Salimans.
Simpler diffusion (sid2): 1.5 fid on imagenet512 with pixel-space diffusion.
In *CVPR*, 2025.
- Jabri et al. (2023)
Allan Jabri, David Fleet, and Ting Chen.
Scalable adaptive computation for iterative generation.
In *ICML*, 2023.
- Karras et al. (2022)
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.
Elucidating the design space of diffusion-based generative models.
In *NeurIPS*, 2022.
- Karras et al. (2024)
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine.
Analyzing and improving the training dynamics of diffusion models.
In *CVPR*, 2024.
- Karras et al. (2025)
Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine.
Guiding a diffusion model with a bad version of itself.
In *NeurIPS*, 2025.
- Kingma & Welling (2014)
Diederik P Kingma and Max Welling.
Auto-encoding variational bayes.
In *ICLR*, 2014.
- Kouzelis et al. (2025a)
Theodoros Kouzelis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Komodakis.
Eq-vae: Equivariance regularized latent space for improved generative image modeling.
In *ICML*, 2025a.
- Kouzelis et al. (2025b)
Theodoros Kouzelis, Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Komodakis.
Boosting generative image modeling via joint image-feature synthesis.
In *NeurIPS*, 2025b.
- Kynkäänniemi et al. (2019)
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila.
Improved precision and recall metric for assessing generative models.
In *NeurIPS*, 2019.
- Kynkäänniemi et al. (2024)
Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen.
Applying guidance in a limited interval improves sample and distribution quality in diffusion models.
In *NeurIPS*, 2024.
- Lee et al. (2022)
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han.
Autoregressive image generation using residual quantization.
In *CVPR*, 2022.
- Leng et al. (2025)
Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng.
Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers.
In *ICCV*, 2025.
- Li et al. (2024a)
Tianhong Li, Dina Katabi, and Kaiming He.
Return of unconditional generation: A self-supervised representation generation method.
In *NeurIPS*, 2024a.
- Li et al. (2024b)
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He.
Autoregressive image generation without vector quantization.
In *NeurIPS*, 2024b.
- Lipman et al. (2023)
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le.
Flow matching for generative modeling.
In *ICLR*, 2023.
- Liu et al. (2024)
Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Chase Lambert, Joao Souza, Suhail Doshi, and Daiqing Li.
Playground v3: Improving text-to-image alignment with deep-fusion large language models, 2024.
- Liu et al. (2023)
Xingchao Liu, Chengyue Gong, and Qiang Liu.
Flow straight and fast: Learning to generate and transfer data with rectified flow.
In *ICLR*, 2023.
- Ma et al. (2024)
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie.
Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers.
In *ECCV*, 2024.
- Mentzer et al. (2024)
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen.
Finite scalar quantization: VQ-VAE made simple.
In *ICLR*, 2024.
- Nichol & Dhariwal (2021)
Alex Nichol and Prafulla Dhariwal.
Improved denoising diffusion probabilistic models.
In *ICML*, 2021.
- Oord et al. (2017)
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.
Neural discrete representation learning.
In *NeurIPS*, 2017.
- Oquab et al. (2023)
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.
Dinov2: Learning robust visual features without supervision.
*TMLR*, 2023.
- Pan et al. (2025)
Xichen Pan, Satya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, Ji Hou, and Saining Xie.
Transfer between modalities with metaqueries, 2025.
- Parmar et al. (2018)
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran.
Image transformer.
In *ICML*, 2018.
- Peebles & Xie (2023)
William Peebles and Saining Xie.
Scalable diffusion models with transformers.
In *ICCV*, 2023.
- Pope et al. (2021)
Phillip Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein.
The intrinsic dimension of images and its impact on learning.
In *ICLR*, 2021.
- Qiu et al. (2025)
Kai Qiu, Xiang Li, Hao Chen, Jason Kuen, Xiaohao Xu, Jiuxiang Gu, Yinyi Luo, Bhiksha Raj, Zhe Lin, and Marios Savvides.
Image tokenizer needs post-training, 2025.
URL [https://arxiv.org/abs/2509.12474](https://arxiv.org/abs/2509.12474).
- Radford et al. (2021)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In *ICML*, 2021.
- Ramanujan et al. (2025)
Vivek Ramanujan, Kushal Tirumala, Armen Aghajanyan, Luke Zettlemoyer, and Ali Farhadi.
When worse is better: Navigating the compression-generation tradeoff in visual tokenization.
In *NeurIPS*, 2025.
- Ramesh et al. (2022)
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.
Hierarchical text-conditional image generation with clip latents.
*arXiv preprint arXiv:2204.06125*, 1(2):3, 2022.
- Razavi et al. (2019)
Ali Razavi, Aaron van den Oord, and Oriol Vinyals.
Generating diverse high-fidelity images with vq-vae-2.
In *NeurIPS*, 2019.
- Ren et al. (2025)
Sucheng Ren, Qihang Yu, Ju He, Xiaohui Shen, Alan Yuille, and Liang-Chieh Chen.
Beyond next-token: Next-x prediction for autoregressive visual generation.
*arXiv preprint arXiv:2502.20388*, 2025.
- Rombach et al. (2022)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.
High-resolution image synthesis with latent diffusion models.
In *CVPR*, 2022.
- Russakovsky et al. (2015)
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.
Imagenet large scale visual recognition challenge.
*International journal of computer vision*, 115(3):211–252, 2015.
- Salimans et al. (2016)
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.
Improved techniques for training GANs.
In *NeurIPS*, 2016.
- Sauer et al. (2022)
Axel Sauer, Katja Schwarz, and Andreas Geiger.
Stylegan-xl: Scaling stylegan to large diverse datasets.
In *SIGGRAPH*, 2022.
- Sauer et al. (2023)
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila.
Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis.
In *ICML*, 2023.
- Skorokhodov et al. (2025)
Ivan Skorokhodov, Sharath Girish, Benran Hu, Willi Menapace, Yanyu Li, Rameen Abdal, Sergey Tulyakov, and Aliaksandr Siarohin.
Improving the diffusability of autoencoders.
In *ICML*, 2025.
- Song et al. (2025)
Kiwhan Song, Jaeyeon Kim, Sitan Chen, Yilun Du, Sham Kakade, and Vincent Sitzmann.
Selective underfitting in diffusion models.
*arXiv preprint arXiv:2510.01378*, 2025.
- Song et al. (2021)
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.
Score-based generative modeling through stochastic differential equations.
In *ICLR*, 2021.
- Sun et al. (2024)
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang.
Generative multimodal models are in-context learners.
In *CVPR*, 2024.
- Szegedy et al. (2016)
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.
Rethinking the Inception architecture for computer vision.
In *CVPR*, 2016.
- Tang et al. (2025)
Hao Tang, Chenwei Xie, Xiaoyi Bao, Tingyu Weng, Pandeng Li, Yun Zheng, and Liwei Wang.
Unilip: Adapting clip for unified multimodal understanding, generation and editing, 2025.
- Teng et al. (2023)
Jiayan Teng, Wendi Zheng, Ming Ding, Wenyi Hong, Jianqiao Wangni, Zhuoyi Yang, and Jie Tang.
Relay diffusion: Unifying diffusion process across resolutions for image synthesis.
In *ICLR*, 2023.
- Tian et al. (2024)
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang.
Visual autoregressive modeling: Scalable image generation via next-scale prediction.
In *NeurIPS*, 2024.
- Tong et al. (2025)
Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu.
Metamorph: Multimodal understanding and generation via instruction tuning.
In *ICCV*, 2025.
- Tschannen et al. (2025)
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai.
Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features, 2025.
- Vahdat et al. (2021)
Arash Vahdat, Karsten Kreis, and Jan Kautz.
Score-based generative modeling in latent space.
In *NeurIPS*, 2021.
- Vincent et al. (2008)
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol.
Extracting and composing robust features with denoising autoencoders.
In *ICML*, 2008.
- Wang et al. (2025a)
Hanyu Wang, Saksham Suri, Yixuan Ren, Hao Chen, and Abhinav Shrivastava.
Larp: Tokenizing videos with a learned autoregressive generative prior.
In *ICLR*, 2025a.
- Wang et al. (2025b)
Shuai Wang, Ziteng Gao, Chenhui Zhu, Weilin Huang, and Limin Wang.
Pixnerd: Pixel neural field diffusion.
*arXiv preprint arXiv:2507.23268*, 2025b.
- Wang et al. (2025c)
Shuai Wang, Zhi Tian, Weilin Huang, and Limin Wang.
Ddt: Decoupled diffusion transformer, 2025c.
- Wang et al. (2025d)
Xiao Wang, Ibrahim Alabdulmohsin, Daniel Salz, Zhe Li, Keran Rong, and Xiaohua Zhai.
Scaling pre-training to one hundred billion data for vision language models.
*arXiv preprint arXiv:2502.07617*, 2025d.
- Wu et al. (2025)
Ge Wu, Shen Zhang, Ruijing Shi, Shanghua Gao, Zhenyuan Chen, Lei Wang, Zhaowei Chen, Hongcheng Gao, Yao Tang, Jian Yang, Ming-Ming Cheng, and Xiang Li.
Representation entanglement for generation:training diffusion transformers is much easier than you think, 2025.
- Yang et al. (2025)
Jiawei Yang, Tianhong Li, Lijie Fan, Yonglong Tian, and Yue Wang.
Latent denoising makes good visual tokenizers, 2025.
- Yao et al. (2025)
Jingfeng Yao, Bin Yang, and Xinggang Wang.
Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models.
In *CVPR*, 2025.
- Yu et al. (2022)
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.
Scaling autoregressive models for content-rich text-to-image generation.
*TMLR*, 2022.
- Yu et al. (2024a)
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al.
Language model beats diffusion–tokenizer is key to visual generation.
In *ICLR*, 2024a.
- Yu et al. (2024b)
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen.
An image is worth 32 tokens for reconstruction and generation.
In *NeurIPS*, 2024b.
- Yu et al. (2025)
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie.
Representation alignment for generation: Training diffusion transformers is easier than you think.
In *ICLR*, 2025.
- Zhai et al. (2025)
Shuangfei Zhai, Ruixiang Zhang, Preetum Nakkiran, David Berthelot, Jiatao Gu, Huangjie Zheng, Tianrong Chen, Miguel Angel Bautista, Navdeep Jaitly, and Josh Susskind.
Normalizing flows are capable generative models.
In *ICML*, 2025.
- Zhai et al. (2023)
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.
Sigmoid loss for language image pre-training.
In *ICCV*, 2023.
- Zhang et al. (2018)
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang.
The unreasonable effectiveness of deep features as a perceptual metric.
In *CVPR*, 2018.
- Zhao et al. (2020)
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han.
Differentiable augmentation for data-efficient gan training.
In *NeurIPS*, 2020.
- Zhao et al. (2025)
Yue Zhao, Yuanjun Xiong, and Philipp Krähenbühl.
Image and video tokenization with binary spherical quantization.
In *ICLR*, 2025.
- Zheng et al. (2025)
Anlin Zheng, Xin Wen, Xuanyang Zhang, Chuofan Ma, Tiancai Wang, Gang Yu, Xiangyu Zhang, and Xiaojuan Qi.
Vision foundation models as effective visual tokenizers for autoregressive image generation, 2025.
- Zheng et al. (2022)
Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung.
Movq: Modulating quantized vectors for high-fidelity image generation.
In *NeurIPS*, 2022.
- (106)
Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar.
Fast training of diffusion models with masked transformers.
*TMLR*.
- Zhou et al. (2021)
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong.
ibot: Image bert pre-training with online tokenizer.
*arXiv preprint arXiv:2111.07832*, 2021.
- Zhu et al. (2025)
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang.
Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models, 2025.
- Zhu et al. (2024)
Yongxin Zhu, Bocheng Li, Hang Zhang, Xin Li, Linli Xu, and Lidong Bing.
Stabilize the latent space for image autoregressive modeling: A unified perspective.
In *NeurIPS*, 2024.

## Appendix A Extended Related Work

#### Representation encoder as autoencoder.

Recent studies have investigated leveraging semantic representations for reconstruction, particularly in MLLMs where diffusion decoders are conditioned on semantic tokens (Sun et al., 2024; Chen et al., 2025b; Pan et al., 2025; Tong et al., 2025). While this improves visual quality, the reliance on large pretrained diffusion decoders makes reconstructions less faithful to the input, limiting their effectiveness as true autoencoders. Very recently, UniLIP (Tang et al., 2025) employs a one-step convolutional decoder on top of InternViT (Zhu et al., 2025), achieving reconstruction quality surpassing SD-VAE. However, UniLIP relies on additional large-scale fine-tuning of pretrained ViTs, arguing that frozen pretrained representation encoders lacks sufficient visual detail. In contrast, we show this is not the case: frozen representation encoders achieves comparable reconstruction performance while enabling much faster convergence in diffusion training.

Another line of related work also try to utilize representation encoders directly as tokenizers. VFMTok (Zheng et al., 2025) and DiGIT (Zhu et al., 2024) applies vector-quantization directly to pretrained representation encoders like Dino or SigLIP. These approaches transform representation encoders into an effective tokenizer for AR models, but still suffer from the information capacity bottleneck brought by quantization.

#### Compressed image tokenizers.

Autoencoders have long been used to compress images into low-dimensional representations for reconstruction (Hinton & Salakhutdinov, 2006; Vincent et al., 2008). VAEs (Kingma & Welling, 2014) extend this paradigm by mapping inputs to Gaussian distributions, while VQ-VAEs (Oord et al., 2017; Razavi et al., 2019) introduce discrete latent codes. VQGAN (Esser et al., 2021) adds adversarial objectives, and ViT-VQGAN (Esser et al., 2021; Cao et al., 2023) modernizes the architecture with Vision Transformers (ViTs) (Dosovitskiy et al., 2021). Other advances include multi-stage quantization (Lee et al., 2022; Zheng et al., 2022), lookup-free schemes (Mentzer et al., 2024; Zhao et al., 2025), token-efficient designs such as TiTok and DCAE (Yu et al., 2024b; Chen et al., 2025c), and structure-preserving approaches like EQ-VAE (Kouzelis et al., 2025a). (Hansen-Estruch et al., 2025) further explores the scaling behavior of VAEs. LARP, CRT, REPA-E (Wang et al., 2025a; Ramanujan et al., 2025; Yu et al., 2025) tried to improve VAE with generative priors via back-propagation.
In contrast, we dispense with aggressive compression and instead adopt a pretrained representation encoders as encoder. This avoids the encoder collapsing into shallow features optimized only for reconstruction loss, while providing strong pretrained representations that serve as a robust latent space.

#### Generative models.

Modern image generation is dominated by two paradigms: autoregressive (AR) models and diffusion models. AR models (De Fauw et al., 2019; Ramesh et al., 2022; Yu et al., 2022; Parmar et al., 2018; Chen et al., 2020a) generate images sequentially, token by token, and benefit from powerful language-model architectures but often suffer from slow sampling. Diffusion models (Ho et al., 2020; Nichol & Dhariwal, 2021; Rombach et al., 2022; Ma et al., 2024; Peebles & Xie, 2023) instead learn to iteratively denoise noisy signals, offering superior sample quality and scalability, though at the cost of many sampling steps. In this work, we build on diffusion models but adapt them to high-dimensional latent spaces provided by pretrained representation encoders. We find representation encoders provide faster convergence and improved scaling behavior.

#### Robust decoders for generation.

Recent works suggest that incorporating masking or latent denoising losses can improve tokenizer training. l-DeTok (Yang et al., 2025) shows that combining both losses yields strong VAEs for second-stage MAR (Li et al., 2024b; Fan et al., 2025b) generation, while RobusTok (Qiu et al., 2025) demonstrates that training with perturbed tokens makes decoders more robust. Notably, Yang et al. (2025) report that denoising loss only increase performance when encoder and decoder are jointly trained, whereas we observe substantial gains in generation quality with frozen encoders.

#### Visual representation learning.

Visual representations primarily fall into two broad families: self-supervised encoders that learn invariances from augmented views or masked prediction (He et al., 2019; Chen et al., 2020b; Grill et al., 2020; Zhou et al., 2021; He et al., 2021), and language-supervised encoders trained from image–text pairs (Radford et al., 2021; Zhai et al., 2023; Chen et al., 2024; Tschannen et al., 2025). Recent work (Fan et al., 2025a; Tschannen et al., 2025; Wang et al., 2025d) scales both lines on billion-scale corpora, yielding robust, semantically rich features. In this work, we leverage these pretrained encoders directly: we freeze the encoder, and train a lightweight decoder to form a Representation Autoencoder (RAE). We show that such RAEs are effective for both reconstruction and generation.

## Appendix B Proofs

### B.1 Proof of Lower Bound for Training Loss

*

###### Proof.

By Albergo et al. (2023), the distribution $\rho_{t}$ of ${\mathbf{x}}_{t}$ satisfies $\rho_{0}=p({\mathbf{x}})$, $\rho_{1}={\mathcal{N}}(0,\mathbf{I}_{n})$, and $\partial_{t}\rho+\nabla\cdot(v\rho)=0$
where $v$ is the optimal velocity predictor defined as $v({\mathbf{x}}_{t},t)=\mathbb{E}[\bm{\varepsilon}-{\mathbf{x}}|{\mathbf{x}}_{t}]$. Also, by Theorem 2.7 in Albergo et al. (2023), there exists $f^{*}\in C^{0}((C^{1}({\mathbb{R}}^{n}))^{n};[0,1])$(^1^11family of functions $f:[0,1]\times{\mathbb{R}}^{n}\to{\mathbb{R}}^{n}$ that is continuous in $t$ for all $({\mathbf{x}},t)\in[0,1]\times{\mathbb{R}}^{n}$, and $f(\cdot,t)$ is a continuously differentiable function from ${\mathbb{R}}^{n}$ to ${\mathbb{R}}^{n}$.) that uniquely minimizes the ${\mathcal{L}}(f,\theta)$ and perfectly approximates $v$.

By our training setting, it’s reasonable to assume that ${\mathbf{x}}\sim p({\mathbf{x}})$ and $\bm{\varepsilon}\sim{\mathcal{N}}(0,\mathbf{I}_{n})$ are independent. Then the distribution of the objective ${\mathbf{y}}=\bm{\varepsilon}-{\mathbf{x}}\sim p_{{\mathbf{y}}}({\mathbf{y}})$ satisfies $p_{{\mathbf{y}}}({\mathbf{y}})=\int_{{\mathbb{R}}^{n}}{\mathcal{N}}(0,{\mathbf{I}}_{n})({\mathbf{y}}+{\mathbf{x}})p({\mathbf{x}})\mathrm{d}{\mathbf{x}}$.
Clearly, $p_{{\mathbf{y}}}$ has full support on ${\mathbb{R}}^{n}$ and is strictly positive, indicating ${\mathbf{y}}$ has a non-zero probability anywhere in ${\mathbb{R}}^{n}$. Similarly, for ${\mathbf{x}}_{t}=(1-t){\mathbf{x}}+t\bm{\varepsilon}$, given any $t$, $p_{{\mathbf{x}}_{t}}({\mathbf{w}})=\int_{{\mathbb{R}}^{n}}{\mathcal{N}}(0,t^{2}{\mathbf{I}}_{n})({\mathbf{w}}-{\mathbf{x}})\frac{1}{(1-t)}p(\frac{{\mathbf{x}}}{1-t})\mathrm{d}{\mathbf{x}}$
also has full support on ${\mathbb{R}}^{n}$ and is strictly positive, indicating ${\mathbf{x}}_{t}$ has a non-zero probability anywhere in ${\mathbb{R}}^{n}$ as well.

Recall that for any function $f:{\mathcal{X}}\to{\mathcal{Y}}$, $\text{Im}(f)=\{f(x):x\in{\mathcal{X}}\}$. Then for linear transformation $f({\mathbf{x}})={\bm{M}}{\mathbf{x}}$ with ${\bm{M}}\in{\mathbb{R}}^{d\times n}$, $\text{Im}(f)=\{{\bm{M}}{\mathbf{x}}:{\mathbf{x}}\in{\mathbb{R}}^{n}\}$; we denote this as $\text{Im}({\bm{M}})$. Now, for any $g\in{\mathcal{G}}_{d}$, $\text{Im}(g)=\{{\bm{B}}f({\bm{A}}{\mathbf{x}}):{\mathbf{x}}\in{\mathbb{R}}^{n}\}\subseteq\{{\bm{B}}{\mathbf{y}}:{\mathbf{y}}\in{\mathbb{R}}^{d}\}=\text{Im}({\bm{B}})$. Since $\text{rank}({\bm{B}})\leq d$, $\dim\text{Im}(g)\leq\text{rank}({\bm{B}})\leq d<n$, therefore $\text{Im}(g)\subseteq\text{Im}({\bm{B}})\subset{\mathbb{R}}^{n}$.

Now, given $g\in{\mathcal{G}}_{d}$ and the deterministic pair $({\mathbf{x}}_{t},{\mathbf{y}},t)\in({\mathbb{R}}^{n},{\mathbb{R}}^{n},[0,1])$, by Projection Theorem,

$$ $\displaystyle\|g({\mathbf{x}}_{t},t)-{\mathbf{y}}\|^{2}\geq\|{\mathbf{u}}_{g}-{\mathbf{y}}\|^{2}$ (3) $$

where ${\mathbf{u}}_{g}\in\text{Im}(g)$ is the unique minimizer and ${\mathbf{u}}_{g}-{\mathbf{y}}$ is orthogonal to $\text{Im}(g)$.
Since $\|\cdot\|^{2}\geq 0$, we can take expectation on both sides

$$ $\displaystyle\inf_{g\in{\mathcal{G}}_{d}}\mathbb{E}_{{\mathbf{x}}\sim p({\mathbf{x}}),\bm{\varepsilon}\sim{\mathcal{N}}(0,\mathbf{I}_{n})}\big[\|g({\mathbf{x}}_{t},t)-{\mathbf{y}}\|^{2}\big]$ $\displaystyle\geq\inf_{g\in{\mathcal{G}}_{d}}\mathbb{E}\big[\|{\mathbf{u}}_{g}-{\mathbf{y}}\|^{2}\big]$ $\displaystyle\geq\inf_{{\mathbf{u}}\in S;\dim S\leq d}\mathbb{E}\big[\|{\mathbf{u}}-{\mathbf{y}}\|^{2}\big]$ $\displaystyle\geq\inf_{S;\dim S\leq d}\mathbb{E}\big[\|{\mathbf{y}}\|^{2}-\|{\bm{P}}_{S}{\mathbf{y}}\|^{2}\big]$ (4) $$

where ${\bm{P}}_{S}$ denote the projection matrix from ${\mathbb{R}}^{n}$ onto $S$. Without loss of generality, we assume $\mathbb{E}[{\mathbf{x}}]=0$(^2^22Non-centered ${\mathbf{x}}$ with $\mathbb{E}[{\mathbf{x}}]=\mu$ will additionally introduce $\|\mu\|^{2}-\|{\bm{P}}_{S}\mu\|^{2}$ to the lower bound; we ignore this term since most data processing pipelines will center the data.), then Eq [4](https://arxiv.org/html/2510.11690v1#A2.E4) can be expanded as

$$ $\displaystyle\inf_{g\in{\mathcal{G}}_{d}}\mathbb{E}_{{\mathbf{x}}\sim p({\mathbf{x}}),\bm{\varepsilon}\sim{\mathcal{N}}(0,\mathbf{I}_{n})}\big[\|g({\mathbf{x}}_{t},t)-{\mathbf{y}}\|^{2}\big]$ $\displaystyle\geq\sum_{i=1}^{n}\mathbb{E}[{\mathbf{y}}_{i}^{2}]-\sup_{S;\dim S\leq d}\sum_{i=1}^{n}\mathbb{E}[({\bm{P}}_{S}{\mathbf{y}})_{i}^{2}]$ $\displaystyle=\operatorname{Tr}(\mathrm{Cov}({\mathbf{y}}))-\sup_{S;\dim S\leq d}\operatorname{Tr}(\mathrm{Cov}({\bm{P}}_{S}{\mathbf{y}}))$ $\displaystyle\geq\sum_{i=1}^{n}\lambda_{i}-\sum_{i=1}^{d}\lambda_{i}=\sum_{i=d+1}^{n}\lambda_{i}$ (5) $$

where Eq [5](https://arxiv.org/html/2510.11690v1#A2.E5) is obtained via Ky-Fan Maximum Principle.

When $d\geq n$, $\sup_{S}\mathbb{E}[\|{\bm{P}}_{S}{\mathbf{y}}\|^{2}]=\mathbb{E}[\|{\mathbf{y}}\|^{2}]$, leading to a trivial lower bound in Eq [5](https://arxiv.org/html/2510.11690v1#A2.E5), and ${\mathcal{G}}_{d}=C^{0}((C^{1}({\mathbb{R}}^{n}))^{n};[0,1])$.
∎

###### Proof.

### B.2 Proof of Lower Bound for Inference Loss

###### Theorem 1 .

Consider the same setup as Theorem [3](https://arxiv.org/html/2510.11690v1#S4.F3). Let ${\mathbf{x}}_{1}$ be the initial random variables in the sampling process, and

$$ $\displaystyle{\mathbf{x}}_{0}$ $\displaystyle=\text{ODE}(g,{\mathbf{x}}_{1},1\to 0)$ $\displaystyle{\mathbf{x}}_{0}^{*}$ $\displaystyle=\text{ODE}(f^{*},{\mathbf{x}}_{1},1\to 0)$ $$

where $\text{ODE}(f,{\mathbf{x}},t\to s)$ refers to any ODE solver that integrates $f$ from time $t$ to $s$ using ${\mathbf{x}}$ as the initial condition. We further assume for any $({\mathbf{x}},{\mathbf{y}},t)\in({\mathbb{R}}^{n},{\mathbb{R}}^{n},[0,1])$, there exists constant $L>0$ such that

$$ $\displaystyle\|f^{*}({\mathbf{x}},t)-f^{*}({\mathbf{y}},t)\|$ $\displaystyle\leq L\|{\mathbf{x}}-{\mathbf{y}}\|$ $$

then

$$ $\displaystyle\|{\mathbf{x}}_{0}^{*}-{\mathbf{x}}_{0}\|\geq\frac{1-e^{-L}}{L}\sum_{i=d+1}^{n}\lambda_{i}$ (6) $$

###### Proof.

We first define a forward $ODE$ that integrates from $0\to 1$ ${\mathbf{x}}_{\overleftarrow{t}}:={\mathbf{x}}_{-t}$, and

$$ $\displaystyle\mathrm{d}{\mathbf{x}}_{\overleftarrow{t}}$ $\displaystyle=g({\mathbf{x}}_{\overleftarrow{t}},t)\mathrm{d}t$ $\displaystyle\mathrm{d}{\mathbf{x}}_{\overleftarrow{t}}^{*}$ $\displaystyle=f^{*}({\mathbf{x}}_{\overleftarrow{t}}^{*},t)\mathrm{d}t$ $$

Then

$$ $\displaystyle\frac{\mathrm{d}}{\mathrm{d}t}\|{\mathbf{x}}_{\overleftarrow{t}}^{*}-{\mathbf{x}}_{\overleftarrow{t}}\|$ $\displaystyle=\|f^{*}({\mathbf{x}}_{\overleftarrow{t}}^{*},t)-g({\mathbf{x}}_{\overleftarrow{t}},t)\|$ $\displaystyle\geq\|f^{*}({\mathbf{x}}_{\overleftarrow{t}},t)-g({\mathbf{x}}_{\overleftarrow{t}},t)\|-\|f^{*}({\mathbf{x}}_{\overleftarrow{t}}^{*},t)-f^{*}({\mathbf{x}}_{\overleftarrow{t}},t)\|$ $\displaystyle\geq\|\Delta\|-L\|{\mathbf{x}}_{\overleftarrow{t}}^{*}-{\mathbf{x}}_{\overleftarrow{t}}\|$ (7) $$

where $\Delta$ denotes the approximation error to $f^{*}$ for $g\in{\mathcal{G}}_{d}$. Applying Gronwall’s Lemma, we have

$$ $\displaystyle e^{L\overleftarrow{t}}\|{\mathbf{x}}_{\overleftarrow{t}}^{*}-{\mathbf{x}}_{\overleftarrow{t}}\|\bigg|^{1}_{0}$ $\displaystyle\geq\int_{0}^{t}e^{Ls}\|\Delta\|\mathrm{d}s$ $\displaystyle\implies\|{\mathbf{x}}_{0}^{*}-{\mathbf{x}}_{0}\|=\|{\mathbf{x}}_{\overleftarrow{1}}^{*}-{\mathbf{x}}_{\overleftarrow{1}}\|$ $\displaystyle\geq\frac{(1-e^{-L})}{L}\sum_{i=d+1}^{n}\lambda_{i}$ (8) $$

where by Theorem [3](https://arxiv.org/html/2510.11690v1#S4.F3), $\|\Delta\|\geq\sum_{i=d+1}^{n}\lambda_{i}$.
∎

###### Theorem 1 .

###### Proof.

## Appendix C RAE Implementation

### C.1 Encoder Normalization

For any given frozen representation encoders, we discard any [CLS] or [REG] token produced by the encoder, and keep the all of patch tokens. We then apply a layer normalization to each token independently, to ensure each token has zero mean and unit variance across channels. We note that all representation encoders we use adopt the standard ViT architecture (Dosovitskiy et al., 2021), which have already applied layer normalization after the last transformer block. Therefore, we only need to cancel the affine parameters of the layer normalization in representation encoders. This does not affect the representation quality of representation encoders, as it is a linear transformation.

Practical Notes. Specifically, we use DINOv2 with Registers (Darcet et al., 2025). Since DINOv2 only provides variants with $p_{e}=14$, we interpolate the input images to $224\times 224$ but set $p_{d}=16$, ensuring the model still produces $256$ tokens while reconstructing $256\times 256$ images.

### C.2 Decoder Training Details

Datasets. We primarily use ImageNet-1K for training all decoders. Most experiments are conducted at a resolution of $256\times 256$. For $512$-resolution synthesis without decoder upsampling, we train decoders directly on $512\times 512$ images.

Decoder Architecture.
The decoder takes the token embeddings produced by the frozen encoder takes the token embedding reconstructs them back into the pixel space using the same patch size as the encoder. As a result, it can generate images with the same spatial spatial resolution as the encoder’s inputs input. Following He et al. (2021), we prepend a learnable [CLS] token decoder’s input sequence and discard it after decoding.

Discriminator Architecture.
We include the majority of our decoder training details in Table [12](https://arxiv.org/html/2510.11690v1#A3.T12). We follow most of the design choices in StyleGAN-T (Sauer et al., 2023), except for using a frozen Dino-S/8 (Caron et al., 2021) instead of Dino-S/16 as the discriminator. We found using Dino-S/8 stabilizes training and avoid the decoder to generate adversarial patches. We also remove the virtual batch norm in Sauer et al. (2023) and use the standard batch norm instead. All input is interpolated to $224\times 224$ resolution before feeding into the discriminator.

**Table 12: Training configuration for decoder and discriminator.**
| Component | Decoder | Discriminator |
| --- | --- | --- |
| optimizer | Adam | Adam |
| max learning rate | $2\times 10^{-4}$ | $2\times 10^{-4}$ |
| min learning rate | $2\times 10^{-5}$ | $2\times 10^{-5}$ |
| learning rate schedule | cosine decay | cosine decay |
| optimizer betas | (0.5, 0.9) | (0.5, 0.9) |
| weight decay | 0.0 | 0.0 |
| batch size | 512 | 512 |
| warmup | 1 epoch | 1 epoch |
| loss | $\ell_{1}$ + LPIPS + GAN | adv. |
| Model | ViT-(B, L, XL) | Dino-S/8 (frozen) |
| LPIPS start epoch | 0 | – |
| disc. start epoch | – | 6 |
| adv. loss start epoch | 8 | – |
| Training epochs | 16 | 10 |

Losses.
For training the decoder, we use a mixture of L1, LPIPS (Zhang et al., 2018) and adversarial loss (Goodfellow et al., 2014):

$$ $\displaystyle z=E(x),\hat{x}$ $\displaystyle=D(z)$ $\displaystyle\mathcal{L}_{rec}(x)=\omega_{L}\operatorname{LPIPS}(\hat{x},x)+$ $\displaystyle\operatorname{L1}(\hat{x},x)+\omega_{G}\lambda\operatorname{GAN}(\hat{x},x),$ $$

where $E,D$ are the encoder and decoder of RAE.
We set $\omega_{L}=1.$ and $\omega_{G}=0.75$. We use the same losses as in StyleGAN-T (Sauer et al., 2023) for discriminator, and a GAN loss as in Esser et al. (2021). We also adopt the adaptive weight $\lambda$ for GAN loss proposed in Esser et al. (2021) to balance the scales of reconstruction and adversarial losses. $\lambda$ is defined as:

$$ $\lambda=\frac{\|\nabla_{\hat{x}}\mathcal{L}_{rec}\|}{\|\nabla_{\hat{x}}\operatorname{GAN}(\hat{x},x)\|+\epsilon},$ $$

Augmentations. For data augmentation, we first resize the input image to $384\times 384$ and then randomly crop to $256\times 256$. We also apply differentiable augmentations with default hyperparameters in Zhao et al. (2020) before discriminator.

### C.3 Visualizations

We present visualizations of reconstructions from different RAEs. As shown in [Figure 8](https://arxiv.org/html/2510.11690v1#A3.F8.fig1), all RAEs achieve satisfactory reconstruction fidelity.

Figure: Figure 8: Reconstruction examples. From left to right: input image, RAE (DINOv2-B), RAE (SigLIP2-B), RAE (MAE-B), SD-VAE. Zoom in for details.
Refer to caption: x10.png

## Appendix D Diffusion Model Implementation

Datasets. We primarily use ImageNet-1K (Russakovsky et al., 2015) for diffusion training. Most experiments are conducted at a resolution of $256\times 256$. For $512$-resolution synthesis without decoder upsampling, we train diffusion models directly on $512\times 512$ images.

**Table 13: Model configurations for different sizes.**
| Model Config | Dim | Num-Heads | Depth |
| --- | --- | --- | --- |
| S | 384 | 6 | 12 |
| B | 768 | 12 | 12 |
| L | 1024 | 16 | 24 |
| XL | 1152 | 16 | 28 |
| XXL | 1280 | 16 | 32 |
| H | 1536 | 16 | 32 |
| G | 2048 | 16 | 40 |
| T | 2688 | 21 | 40 |

Models.
By default, we use LightningDiT (Yao et al., 2025) as the backbone of our diffusion model.
We use a continuous time formulation of flow matching and restrict the timestep input to real values in $[0,1]$.
Following prior work (Song et al., 2021), we replace the timestep embedding with a Gaussian Fourier embedding layer.
We also add Absolute Positional Embeddings (APE) to the input tokens in addition to RoPE, though we do not observe significant performance difference with or without APE.

For $\text{DiT}^{\text{DH}}$, we generally follow the same architecture as DiT, and does not reapply APE for the DDT head input. We use a linear layer to map the $\text{DiT}^{\text{DH}}$ encoder output to the $\text{DiT}^{\text{DH}}$ decoder dimension when the dimension of DiT and DDT head mismatches. We provide a detailed model configuration in [Table 13](https://arxiv.org/html/2510.11690v1#A4.T13).

Compute.
For all models based on RAE, we use a patch size of $1$. For baseline experiments on VAE and pixel inputs, we use patch sizes of $2$ and $16$, respectively.
Across all $256\times 256$ experiments, the diffusion model processes a token sequence of length $256$. Consequently, the computational cost of the DiT backbone remains identical across RAE, VAE, and pixel settings, with differences arising only from the patchification step.
Since patchification accounts for less than 1% of the total GFLOPs, training DiT on different autoencoders introduces only a negligible compute overhead.

Optimization.
For DiT, we strictly follow the optimization strategy in LightningDiT (Yao et al., 2025), using AdamW with a constant learning rate of $2.0\times 10^{-4}$, a batch size of 1024 and an EMA weight of 0.9999. We do not observe instability or abnormal training dynamics with this recipe on DiT. For $\text{DiT}^{\text{DH}}$, we find using the recipe in (Yao et al., 2025) leads to loss spikes at later epochs and slow EMA model convergence at early epochs. We instead use a linear decay from $2.0\times 10^{-4}$ to $2.0\times 10^{-5}$ with a constant warmup of 40 epochs. To encourage the convergence of EMA model, we change the EMA weight from 0.9999 to 0.9995. Additionally, we use gradient clipping of $1.0$ for $\text{DiT}^{\text{DH}}$. Other optimization hyperparameters are the same as DiT. All models are trained for 80 epochs unless otherwise specified. We only report EMA model performance.

Sampling.
We use standard ODE sampling with Euler sampler and 50 steps by default. We find the performance generally converges above 50 steps. We use the same sampling hyperparameters for both DiT and $\text{DiT}^{\text{DH}}$.

Computation.
We use PyTorch/XLA on TPU for all training and inference on RAE. For evaluation, we use a single v6e-8 to generate 50K samples.
For the 800 epoch $\text{DiT}^{\text{DH}}$-XL (DINOv2-B) result, we conduct the generation on a machine with 4 A100s and evaluate FID on CPU due to a lack of TensorFlow GPU support.
We use an internal JAX codebase for training baseline models on VAEs.

## Appendix E Sampling details for FID Evaluation

For the FID-50K evaluation on RAE, we followed the protocol used in  (Tian et al., 2024; Li et al., 2024b; Ren et al., 2025; Wang et al., 2025c), sampling 50 images from each class for a total of 50,000 images. The reference statistics were taken from the ADM pre-computed statistics (Dhariwal & Nichol, 2021) over the full ImageNet dataset.
Another commonly adopted sampling strategy is to uniformly sample class labels 50K times and generate images accordingly (Peebles & Xie, 2023; Ma et al., 2024; Yu et al., 2025). It is worth noting that this strategy is not equivalent to the per-class sampling scheme–although random sampling asymptotically converges to the per-class version, the resulting FID scores differ slightly in practice.

To ensure a fair comparison, we re-evaluate previous state-of-the-art methods that did not use class-balanced sampling with class-balanced sampling. We also evaluate our method with random sampling.
As shown in [Table 14](https://arxiv.org/html/2510.11690v1#A5.T14), all methods’ FID improves consistently with class-balanced sampling.

We also note that the original ImageNet training set is inherently unbalanced, with class sizes ranging from approximately 732 to 1,300 samples (Russakovsky et al., 2015). Therefore, the common assumption behind both balanced and random sampling—namely, that the label distribution is uniform—is not entirely accurate. However, since 895 classes contain exactly 1,300 samples, the dataset exhibits a high degree of near-equivalence among most classes. This partial balance may partly explain why balanced sampling consistently yields better results, as it more closely approximates the true label distribution of the training set.
As the FID values approach the lower ranges, these subtle details begin to have a greater impact. We therefore want to raise awareness within the community. Finally, while the FID metric remains a useful indicator of generative quality, its absolute value becomes less meaningful as generation fidelity continues to improve.

Table: Table 14: Performance of different methods using different sampling strategies. The officially reported numbers are marked in gray.

## Appendix F Theory Experiment Setup

In this section we list the setup of experiments in [Section 4](https://arxiv.org/html/2510.11690v1#S4) for overfitting images.

#### Models.

By default, we use a DiT with depth $12$, width $768$ and a attention head of $4$. The depth varies in $\{384.512,640,768,896\}$ and width varies in $\{4,12,16,24\}$ in [Figure 3](https://arxiv.org/html/2510.11690v1#S4.F3). Other configurations are the same as [Appendix D](https://arxiv.org/html/2510.11690v1#A4).

#### Targets.

We use three images for overfitting experiments, and all numbers reported are the average on three independent run on each images. We resize all targets to $256\times 256$ and not use any data augmentation .

#### Optimizations & Sampling.

For a single target image, the batch size only influences the timestep. We therefore use a relatively small batch size of 32 and a constant learning rate of $2\times 10^{-4}$, optimized with AdamW ($\beta=(0.9,0.95)$). The model is trained for 1200 steps without EMA.
For sampling, We use standard ODE sampling with Euler sampler and 25 steps by default.

## Appendix G Additional Ablation Studies

**(a) gFID and rFID of different encoders w/ and w/o noisy-robust decoding.**
| Model | gFID | rFID |
| --- | --- | --- |
| DINOv2-B | 4.81 / $4.276\,128\,739$ | 0.49 / 0.57 |
| SigLIP2-B | 6.69 / 4.93 | 0.53 / 0.82 |
| MAE-B | 16.14 / 8.38 | 0.16 / 0.28 |

### G.1 Generation Performance across Encoders

As shown in [Table 15a](https://arxiv.org/html/2510.11690v1#A7.T15.sf1), DINOv2-B achieves the best overall performance. MAE performs substantially worse in generation, despite yielding much lower rFID. This shows that a low rFID does not necessarily imply a good image tokenizer. Therefore, we use DINOv2-B as the default encoder for our image generation experiments.

### G.2 Design choices for noise-augmented decoding

We first analyze how noise-robust decoding affects reconstruction and generation. Table [15c](https://arxiv.org/html/2510.11690v1#A7.T15.sf3) shows that larger $\tau$ improves generative FID (gFID) consistently, but slightly worsens reconstruction FID (rFID). This supports our intuition: noise encourages the decoder to learn smoother mappings that generalize better to imperfect latents, improving generation quality, but reducing exact reconstruction accuracy.

To test the robustness of this trade-off, we evaluate different encoders (Table [15a](https://arxiv.org/html/2510.11690v1#A7.T15.sf1)) with $\sigma=0.8$. Across all encoders, noisy training improves gFID while mildly harming rFID. The effect is strongest for weaker encoders such as MAE-B, where gFID improves from $16.14$ to $8.38$. Finally, Table [15b](https://arxiv.org/html/2510.11690v1#A7.T15.sf2) shows that the benefit holds across encoder sizes, suggesting that robust decoder training is broadly applicable.

Together, these results highlight a general principle: decoders should not only reconstruct clean latents, but also handle their noisy neighborhoods. This simple change enables RAEs to serve as stronger backbones for diffusion models.

### G.3 Design Choices for the DDT Head

**Table 16: DDT head should be wide and shallow. A wide, shallow head yields lower FID than deeper (4-layer G) or narrower (6-layer XL) ones .**
| Depth | Width | GFLops | FID $\downarrow$ |
| --- | --- | --- | --- |
| 6 | 1152 (XL) | 25.65 | $2.361\,707\,74$ |
| 4 | 2048 (G) | 53.14 | $2.305\,911\,967$ |
| 2 | 2048 (G) | 26.78 | $2.158\,400\,731$ |

We now investigate design variants of the DDT head to identify those that serve its role more effectively.
Two factors turn out to be crucial: (a) the head needs to be wide and shallow, and (b) its benefit depends on the size of the underlying RAE encoder.

#### Width and Depth.

We first vary the architecture of the DDT head, sweeping both width and depth while keeping the total parameter count approximately fixed.
As shown in [Table 17](https://arxiv.org/html/2510.11690v1#A7.T17), a 2-layer, 2048-dim (G) head outperforms a 6-layer, 1152-dim (XL) head by a large margin, despite having similar GFlops. Moreover, a 4-layer, 2048-dim head does not improve over the 2-layer version, even though it has double the GFlops.
This suggests that a wide and shallow head is more effective for denoising.

#### Dependence on Encoder Size.

Next, we analyze how the effect of the DDT head scales with the size of the RAE encoder.
We fix the DiT backbone as DiT-XL and vary the DDT head width from 768 (B) to 1536 (H), 2048 (G), and 2688 (T). We train $\text{DiT}^{\text{DH}}$ models on top of three RAEs: DINOv2-S, DINOv2-B, and DINOv2-L. As shown in [Table 17](https://arxiv.org/html/2510.11690v1#A7.T17), the optimal DDT head width increases as the encoder scales. When using DINOv2-S and DINOv2-B, the performance converges at a DDT head width of 2048 (G), while 2688 (T) head still brings performance gains on DINOv2-L. This suggests that the larger RAE encoders benefit more from a wider DDT head.

By default, we use a 2-layer, 2048-dim DDT head for all $\text{DiT}^{\text{DH}}$ models.

## Appendix H Additional Scaling Results

Figure: Figure 9: Training loss of $\text{DiT}^{\text{DH}}$ on DINOv2-B. We use an EMA weight of 0.9 to smooth the loss.
Refer to caption: x11.png

We examine how model scale affects training loss in [Figure 9](https://arxiv.org/html/2510.11690v1#A8.F9). Increasing the model’s computational capacity leads $\text{DiT}^{\text{DH}}$ to converge faster and reach a lower final loss.

## Appendix I Guidance

We primarily adopt AutoGuidance (Karras et al., 2025) as our guidance method, as it is easier to tune than CFG with interval (Ho & Salimans, 2022; Kynkäänniemi et al., 2024) and consistently delivers better performance. CFG with interval is used only for the DiT-XL + DINOv2-S with guidance result reported in [Table 8](https://arxiv.org/html/2510.11690v1#S5.T8).

AutoGuidance. We adopt AutoGuidance (Karras et al., 2025) as our primary guidance method. The idea is to use a weaker diffusion model to guide a stronger one, analogous to the principle of Classifier-Free Guidance (CFG) (Ho & Salimans, 2022). Empirically, we observe that weaker base models and earlier training checkpoints consistently yield stronger guidance effects. In practice, we use the smallest variant, $\text{DiT}^{\text{DH}}$-S, as the guiding model, typically adopting an early checkpoint. Unless otherwise specified, AutoGuidance results are obtained using a guidance scale of 1.5 and a 20-epoch checkpoint of $\text{DiT}^{\text{DH}}$-S. For the best result of 1.13 FID ($\text{DiT}^{\text{DH}}$-XL on DINOv2-B, $256\times 256$), we sweep the guidance scale and use a guidance scale of 1.42 with a 14-epoch $\text{DiT}^{\text{DH}}$-S checkpoint. Notably, training this base model requires only about 0.05% of the compute used to train the guided model ($\text{DiT}^{\text{DH}}$-XL for 800 epochs).

Classifier-Free Guidance. We also evaluate CFG (Ho & Salimans, 2022) on RAE. Interestingly, CFG without interval does not improve FID; in fact, applying it from the first diffusion step increases FID. With Guidance Interval (Kynkäänniemi et al., 2024), CFG can achieve competitive FID after careful grid search over scale and interval. However, on our final model ($\text{DiT}^{\text{DH}}$-XL with DINOv2-B), the best CFG result remains inferior to AutoGuidance. Considering both performance and tuning overhead, we adopt AutoGuidance as our default guidance method.

## Appendix J Descriptions for Flow-based Models

Diffusion Models (Ho et al., 2020; Dhariwal & Nichol, 2021; Karras et al., 2022) and more generally flow-based models (Albergo et al., 2023; Lipman et al., 2023; Liu et al., 2023) are a family of generative models that learn to reverse a reference “noising” process. One of the most commonly used “noising” process is the linear interpolation between i.i.d Gaussian noise and clean data (Esser et al., 2024; Ma et al., 2024):

$$ $\displaystyle{\mathbf{x}}_{t}=(1-t){\mathbf{x}}+t\bm{\varepsilon}$ $$

where ${\mathbf{x}}\sim p({\mathbf{x}}),\bm{\varepsilon}\sim{\mathcal{N}}(0,\mathbf{I}_{n}),t\in[0,1]$, and we denote ${\mathbf{x}}_{t}$’s distribution as $\rho_{t}({\mathbf{x}})$ with $\rho_{0}=p({\mathbf{x}})$ and $\rho_{1}={\mathcal{N}}(0,\mathbf{I})$. Generation then starts at $t=1$ with pure noise, and simulates some differential equation to progressively denoise the sample to a clean one. Specifically for flow-based models, the differential equations (an ordinary differential equation (ODE) or a stochastic differential equation (SDE)) are formulated through an underlying velocity $v({\mathbf{x}}_{t},t)$ and a score function $s({\mathbf{x}}_{t},t)$

$$ ODE $\displaystyle\mathrm{d}{\mathbf{x}}_{t}=v({\mathbf{x}}_{t},t)\mathrm{d}t$ SDE $\displaystyle\mathrm{d}{\mathbf{x}}_{t}=v({\mathbf{x}}_{t},t)\mathrm{d}t-\frac{1}{2}w_{t}s({\mathbf{x}}_{t},t)\mathrm{d}t+\sqrt{w_{t}}\mathrm{d}\bar{{\mathbf{w}}}_{t}$ $$

where $w_{t}$ is any scalar-valued continuous function (Ma et al., 2024), and $\bar{{\mathbf{w}}}_{t}$ is the reverse-time Wiener process. The velocity $v({\mathbf{x}}_{t},t)$ is represented as a conditional expectation

$$ $\displaystyle v({\mathbf{x}}_{t},t)=\mathbb{E}[\dot{{\mathbf{x}}}_{t}|{\mathbf{x}}_{t}]=\mathbb{E}[\bm{\varepsilon}-{\mathbf{x}}|{\mathbf{x}}_{t}]$ $$

and can be approximated with model $v_{\theta}$ by minimizing the following training objective

$$ $\displaystyle{\mathcal{L}}_{\text{velocity}}(\theta)=\int_{0}^{1}\mathbb{E}_{{\mathbf{x}},\bm{\varepsilon}}\bigg[\|v_{\theta}({\mathbf{x}}_{t},t)-(\bm{\varepsilon}-{\mathbf{x}})\|^{2}\bigg]\mathrm{d}t$ $$

The score function $s({\mathbf{x}}_{t},t)$ is also represented as a conditional expectation

$$ $\displaystyle s({\mathbf{x}}_{t},t)=-\frac{1}{t}\mathbb{E}[\bm{\varepsilon}|{\mathbf{x}}_{t}]$ $$

Notably, $s$ is equivalent to $v$ up to constant factor (Albergo et al., 2023), so it’s enough to estimate only one of the two vectors.

## Appendix K Evaluation Details

### K.1 Evaluation

We strictly follow the setup and use the same reference batches of ADM (Dhariwal & Nichol, 2021) for evaluation, following their official implementation.(^3^33[https://github.com/openai/guided-diffusion/tree/main/evaluations](https://github.com/openai/guided-diffusion/tree/main/evaluations)) We use TPUs for generating 50k samples and use one single NVIDIA A100 80GB GPU for evaluation.

In what follows, we explain the main concept of metrics that we used for the evaluation.

- •
FID (Heusel et al., 2017) evaluates the distance between the feature distributions of real and generated images. It relies on the Inception-v3 network (Szegedy et al., 2016) and assumes both distributions follow multivariate Gaussians.
- •
IS (Salimans et al., 2016) also uses Inception-v3, but evaluates logits directly. It measures the KL divergence between the marginal label distribution and the conditional label distribution after softmax normalization.
- •
Precision and recall (Kynkäänniemi et al., 2019) follow their standard definitions: precision reflects the fraction of generated images that appear realistic, while recall reflects the portion of the training data manifold covered by generated samples.

## Appendix L Unconditional Generation

We are also interested in how RAEs perform in unconditional generation. To evaluate this, we train $\text{DiT}^{\text{DH}}$-XL on RAE latents without labels. Following RCG (Li et al., 2024a), we set labels to null during training and use the same null label at generation time. While classifier-free guidance (CFG) does not apply in this setting, AutoGuidance remains applicable. We therefore train $\text{DiT}^{\text{DH}}$-XL for 200 epochs with AG (detailed in [Appendix I](https://arxiv.org/html/2510.11690v1#A9)).

**Table 18: Comparison of unconditional generation on ImageNet 256 $\times$ 256.**
| Method | gFID $\downarrow$ | IS $\uparrow$ |
| --- | --- | --- |
| DiT-XL + VAE | 30.68 | 32.73 |
| $\text{DiT}^{\text{DH}}$-XL + DINOv2-B (w/ AG) | 4.96 | 123.12 |
| RCG + DiT-XL | 4.89 | 143.2 |

As shown in [Table 18](https://arxiv.org/html/2510.11690v1#A12.T18), our model achieves substantially better performance than DiT-XL trained on VAE latents. Compared to RCG, a method specifically designed for unconditional generation, our approach attains competitive performance while being much simpler and more straightforward, without the need for two-stage generation.

## Appendix M Visual Results

We show uncurated $512\times 512$ samples sampled from our most performant model: $\text{DiT}^{\text{DH}}$-XL on DINOv2-B with autoguidance scale = 1.5.

Figure: Figure 10: Uncurated $512\times 512$ $\text{DiT}^{\text{DH}}$-XL samples. AutoGudance Scale = 1.5 Class label = ”golden retriever” (207)
Refer to caption: x12.png

Figure: Figure 12: Uncurated $512\times 512$ $\text{DiT}^{\text{DH}}$-XL samples. AutoGudance Scale = 1.5 Class label = ”cliff” (972)
Refer to caption: x14.png

Figure: Figure 14: Uncurated $512\times 512$ $\text{DiT}^{\text{DH}}$-XL samples. AutoGudance Scale = 1.5 Class label = ”arctic fox” (279)
Refer to caption: x16.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: - I understand that the authors aim to ensure fair comparisons by controlling the model size in terms of parameter count. However, having the same number of parameters does not necessarily imply comparable efficiency across all aspects. Could the authors also report the wall-clock training and inference speeds, as well as the FLOPs, of the proposed model relative to the baselines?

- I understand that the rebuttal period is limited, and I'm not requesting significant experiments here. However, for future submissions, including moderate-scale text-to-image experiments would be valuable to more convincingly demonstrate the practical applicability of the proposed method.
soundness: 3
strengths: - This paper targets a very important problem of rich semantics in image tokenization. The proposed tokenizer design is simple yet very effective. 

- The practical and theoretical justifications for the architectural design are very convincing. 

- The empirical results are strong on ImageNet-level evaluations.
summary: This paper revisits the training paradigm of image tokenizers for image generation models. It addresses the limitations of the standard SD-VAE–style tokenizer, whose training primarily depends on reconstruction losses and therefore often yields latent spaces with limited semantic structure. The authors propose an unconventional approach: replacing the learnable encoder with a frozen, pretrained vision encoder and training only the decoder for image reconstruction. To mitigate the resulting challenges—such as a substantially higher latent dimensionality and a more fragile decoder—the paper introduces several architectural and training refinements to the DiT models. The resulting tokenizer demonstrates improved image generation quality and faster convergence.
weaknesses: **Presentation**

My major concern with this paper is that it seems to have been submitted in a huge rush. A throughout proofreading and reformatting is very necessary. Examples: 

- I failed to find Figure 3 and Figure 4 in the main manuscript. 

- Line 175: the two sentences beginning with “Because RAE…” are repetitive.

- Line 345: missing space before 'To'.

- Figure 8 appears earlier than Figure 7 in the manuscript. 


**Technical**

- In Line 013 of the abstract. I personally believe it is not appropriate to classify traditional VAEs as U-Nets. U-Nets typically include skip connections between the encoder and decoder, which are not applicable in image tokenization models.

- In Line 086 of Introduction. Calling the latent space of RAE 'discrete' can be very misleading. Readers might confuse it with typical latent space such as VQ. If the authors wish to retain this term, a clear explanation is required early in the paper; otherwise, readers will remain unclear about its meaning until Section 4.3.

- In line 306, the statement “noise smooths the latent distribution and mitigates OOD issues for the decoder, but also removes fine details” is unconvincing. Since the additional noise is introduced only during training, and the decoder receives clean latents during inference, the argument that noise removes fine details does not hold.

- In Table 9, compared to direct scaling tokens, the upsample method actually increases the gFID by 42%, which can hardly be considered as 'comparable' as stated by the authors. This raises concerns about the scalability of the proposed method.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1) The fundamental question is what property beyond reconstruction makes a representation diffusable?  and why does DINOv2 succeed where MAE fails?

2) What is the qualitative trade-off of noise-augmented decoding, if the FID improves but is the reconstruction compromised?
soundness: 3
strengths: 1)  The proposed shallow DiT + RAE method achieves an FID of 1.18 on ImageNet 256x256 and 1.13 on 512x512, outperforming all prior diffusion models.

2) The method converges faster than VAE-based baselines  for example 10x speedup over VA-VAE while maintaining the FID scores.

3) The RAE also employs an efficient decoder. This part also contributes to make the model faster.

4) The paper provides a theoretical proof and empirical evidence, that the DiT width should be at least equal to the dimensionality of the tokens.
summary: This paper proposes a new replacement to VAE autoencoder (used for downstream Diffusion Transformer models) with Representation Autoencoder (RAE). The authors argue that the traditional UNET based VAEs are computationally heavy and provide a weak latent space that leads to lower performance. Their RAE takes a pretrained representation encoder (like DINOv2) and lightweight ViT-based decoder to solve this issue. This new Autoencoder is not directly applicable to the DiT architectures 1) The image reconstruction is limited, hence a wide DiT is used 2) noise schedule shift is employed to account for increased dimensions 3) The latents are augmented with noise to inject stochasticity. To scale however, a shallow DDT head is attached to the DiT backbone which achieves FID scores of 1.18 on ImageNet-256
weaknesses: 1) The paper does not necessarily prove that the better reconstruction quality equals to better generation quality. This still raises questions on the diffusibility of the the latent space proposed in the paper.

2) The model is suitable for the high computation models, for the highly compressed models in VAE which can have very high compression ratios, this model might not be comparable in the computation cost do to the requirement of dimensionality and DiT width constraints.

3) Again Shallow DiT only benefits high dimension latents , for example it can perform worse that DiT-XL trained on low dim SD-VAE.

4) The new training scheme is unstable as compared to  the standard DiT which is stable with a constant learning rate.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: See Weaknesses.
soundness: 3
strengths: 1. Introducing semantic tokenizers into generative diffusion models is a valuable and timely exploration. This idea may also be extended to unified MLLMs to alleviate the representational conflict between generation and understanding tasks caused by different semantic and pixel-level granularity requirements.

2. The paper is highly readable and the proposed method is elegant. The analyses on model width, noise schedule, and the proposed DDT head provide clear insights and further improve the effectiveness of semantic tokenizers for generation tasks.

3. Extensive experiments and analyses demonstrate that the proposed method achieves SOTA performance.
summary: This paper discusses how to replace the VAE encoder in LDMs with pretrained representation encoders (e.g., DINOv2). The authors conduct a thorough analysis around this topic, and the paper is well-written, clear, and provides meaningful insights.
weaknesses: 1. The results in Table 1 require further clarification.  RAEs with frozen encoders achieve consistently better reconstruction quality (rFID) than SD-VAE (lines 156–159). However, concurrent works [1, 2] reported that semantic tokenizers tend to decrease reconstruction fidelity.  Additional explanations for why rFID improves, along with more experiments, could help further clarify this point.

2. DINOv2 was trained on hundreds of millions of images [3], which may include ImageNet-like, object-centric samples. This raises potential concerns about evaluation fairness. It would be helpful if the authors could clarify how to avoid the data leakage risk.

3. The notably low gFID on ImageNet may raise concerns about potential overfitting. Since incorporating semantic latents could simplify the latent structure [4], a (large) pretrained tokenizer might more easily memorize simplified semantic distributions. Further OOD evaluations would help confirm the generality of the improvement.

4. It would be interesting to explore whether a smaller-width DiT, combined with a simple learnable linear module, could achieve comparable performance of the d>n setting. For example, consider the model $s_{\theta}(x_t) = Wx_t + Ug(V^Tx_t)$ where $V \in \mathbb{R}^{n \times r}$ and $r \ll n$. According to Theorem 4.2 in [5], the oracle score can be expressed as a linear term with a low-rank correction. This could potentially help alleviate the need for a large model width.

5. It would be interesting to discuss whether such semantic tokenizers could be incorporated into unified MLLMs (e.g., for joint generation, understanding, and editing) to mitigate representational conflicts between different tasks [6].

[1] Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Generation.

[2] Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models.

[3] DINOv2: Learning Robust Visual Features without Supervision.

[4] Masked Autoencoders Are Effective Tokenizers for Diffusion Models.

[5] Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?

[6] Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 4
questions: 1. What is the compute in GFLOPS due to the extra 163 M parameters introduced in RAE from Table 6 in RAE? Does this exceed the SDVAE compute cost?
2. Figure 8a and b are a bit confusing; the convergence scores at 10^11 training iterations is different for Dit-DH-XL, Could the authors please clarify this?
3.  Figure 2 portrays that the encoder of RAE has a compute cost of 22GFlops. However, the authors have not clarified which Dinov2 version this refers to. Could the authors please clarify this? 
4. Could the authors provide the performance for Table 7 DiT-DH-XL at 20 and 80 epochs in the presence of cfg?
5. Can a smaller encoder-decoder be trained for SDVAE so that it is tuned for ImageNet 1M. Would such an encoder present better reconstruction performance with smaller number of parameters?
6. By the design choices, would RAE need a big diffusion model and a large dino v2 encoder to train diffusion models on smaller datasets like FFHQ? 
7. In Table 9, Upsample is spelled wrong.
8. While training  diffusion transformers for smaller datasets like FFHQ wouldn't the limitation of large latent space hold and wouln't the model be severely overparametrized? Additionally, if the method is not generalizable towards text to image generation and infeasible towards smaller datasets, wouldn't this limit the scope of the work?
soundness: 2
strengths: 1. The introduced methods, along with the design choices, significantly reduce the number of convergence epochs of training diffusion transformers in the ImageNet 1M dataset.
2. The proposed method seems to be robust with repect to cfg and achieves similar performance with and without cfg. This may enable faster generation with less compute overhead. I would advise the authors to consider this angle while presenting this work as a model that is robust to different cfg values as well as portray good performance without cfg would be useful to the community.
3. The authors have tried extensive experiments to make the DiT model work with Dino-v2  based representation encoder on ImageNet dataset. 
4. Finally the paper is well written and each design choice is clearly explained.
summary: The paper introduced Represenation Autoencoders, a new paradigm for choosing the latent encoder and decoder while training diffusion transformers. Rather than the conventionally used autoencoder SD-VAE, the authors claim that utilizing a preexisting foundation model as the encoder is more suited for training diffusion models since they contain semantic representation as well as portray excellent reconstruction ability. With the newly proposed autoencoder and additional design choices, specifically, (a higher dimensinonal latent space for the transfomer, utilizing a DDT head at the end of the diffusion transformer, a revised noise schedule to suit for Dinov2 latent space) the authors achieve state-of-the-art results in 256x256 and 512x512 generation in ImageNet 1M dataset. With the proposed approach additionally, the authors obtain a representation that seems to be cfg robust.
weaknesses: 1. The authors have done a good job presenting the rFID numbers and extensive experiments for Dinov2, however, Dinov2 is known to discard information relevant for perfect reconstruction during the encoding process. (Ex:- In Fig 9 , on the bus, all the text-based information is lost, moreover, the headlight bulbs fused into one, and the lines in the bus got joined or bent ) Given that such information cannot be captured by the encoder, how can a generation process be guided by such features? Moreover, looking at Fig 13, 14, 15, there are multiple cases like this where the generated images lose semantic details.
2. How easy is it for RAE to generalize for text-to-image generation? Dinov2 is known to not preserve colours, textual information and straight lines. Would this impact the performance of RAE and its generalizability towards text-to-image generation?
3. Dinov2 features are not discriminative towards instances of the same object class, for example:  if two people are present in the scene, Dinov2 would give similar features for the hands of the two people. Assuming the text prompt is "raise the left hand for the person on the left" would this cause a problem towards the model being able  to identify this with text since all four hands in this case would be given similar features? Some examples can be seen in Figure 11/13 where multiple parrots/ dogs fuse to one.
4. The SDVAE was originally trained for LAION dataset, and the size of the encoder and decoder was made to obtain the best performance in LAION. How does RAE autoencoder perform in LAION? Is a bigger encoder-decoder needed for RAE to work with the LAOIN dataset? As an actionable item, I would like the author's clarification of the performance RAE encoder-decoder on LAION, specifically on the PSNR, SSIM and rFID 
5.  There is a performance oscillation for  DiT-DH-XL with different versions of Dinov2 encoders as portrayed in Table 6, and they don't scale linearly with parameter sizes. Why is this? Would this mean that the models may not scale with larger foundation model encoders in the future?
6. In Ln 448-450, the authors claim that "the diffusion models trained for 256x256 could be generalized to 512x512" is this claim valid for other datasets than ImageNet where more number of semantic objects are involved in the same scene?
7. I personally applaud the authors efforts in trying to make the model work with Dino-v2 latent space, however a concern I have is , is such kind of extensive encoder-dependent tuning necessary if we are to swap in a different encoder? Wouldn't this reduce the utility of the approach?


Decision: Accept (Poster)