Epsilon-VAE: Denoising as Visual Decoding
Authors: Long Zhao
Sanghyun Woo
Ziyu Wan
YANDONG LI
Han Zhang
Boqing Gong
Hartwig Adam
Xuhui Jia
Ting Liu
Venue: ICML 2025 poster (2025)
Keywords: Diffusion Model
VAE
Image Tokenizer
Rectified Flow
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: In generative modeling, tokenization simplifies complex data into compact, structured representations, creating a more efficient, learnable space. For high-dimensional visual data, it reduces redundancy and emphasizes key features for high-quality generation. Current visual tokenization methods rely on a traditional autoencoder framework, where the encoder compresses data into latent representations, and the decoder reconstructs the original input. In this work, we offer a new perspective by proposing denoising as decoding, shifting from single-step reconstruction to iterative refinement. Specifically, we replace the decoder with a diffusion process that iteratively refines noise to recover the original image, guided by the latents provided by the encoder. We evaluate our approach by assessing both reconstruction (rFID) and generation quality (FID), comparing it to state-of-the-art autoencoding approaches. By adopting iterative reconstruction through diffusion, our autoencoder, namely Epsilon-VAE, achieves high reconstruction quality, which in turn enhances downstream generation quality by 22% at the same compression rates or provides 2.3x inference speedup through increasing compression rates. We hope this work offers new insights into integrating iterative generation and autoencoding for improved compression and generation.

Epsilon-VAE: Denoising as Visual Decoding
Long Zhao
Sanghyun Woo
Ziyu Wan 12* Yandong Li Han Zhang' Boqing Gong' Hartwig Adam
Xuhui Jia
Ting Liu'
Abstract
autoregressive models, while continuous latents enable effi- cient learning in the denoising process of diffusion models. In either case, empirical results demonstrate that tokeniza- tion enhances generative performance. Here, we focus on continuous tokenization for latent diffusion models, which excel generating -igh-dimensional visual data.
In generative modeling, tokenization simplifies complex data into compact, structured represen- tations, creating more efficient, learnable space. For high-dimensional visual data, it reduces re- dundancy and emphasizes key features for high- quality generation. Current visual tokenization methods rely on traditional autoencoder frame- work, where the encoder compresses data into la- tent representations, and the decoder reconstructs the original input. In this work, we offer new perspective by proposing denoising as decoding, shifting from single-step reconstruction to itera- tive refinement. Specifically, we replace the de- coder with diffusion process that iteratively re- fines noise to recover the original image, guided by the latents provided by the encoder. We evalu- ate approach by assessing both reconstruction (rFID) and generation quality (FID), comparing it to state-of-the-art autoencoding approaches. By adopting iterative reconstruction through diffu- sion, our autoencoder, namely e-VAE, achieves high reconstruction quality, which in turn en- hances downstream generation quality by 22% the same compression rates or provides 2.3x in- ference speedup through increasing compression rates. We hope this work offers new insights into integrating iterative generation and autoencoding for improved compression and generation.
In this paper, revisit the conventional autoencoding pipeline, which typically consists of an encoder that com- presses the input into a latent representation and decoder that reconstructs the original data in single step. Instead of deterministic decoder, we introduce a diffusion pro- cess (Ho et al., 2020; Song et al., 2021), where the encoder still compresses the input into a latent representation, but reconstruction performed iteratively through denoising. This reframing turns the reconstruction phase into a progres- sive refinement process, where the diffusion model, guided by the latent representation, gradually restores the origi- nal data. While previous work (Preechakul et al., 2022) and concurrent work (Birodkar et al., 2024) have explored di diffusion mechanisms autoencoding, none have fully re- alized a practical diffusion--ased autoencoder. By carefully do-designnng architecture and objectives, firstly show that our approach outperforms state-of-the-art autoencoding paradigms in reconstruction fidelity, sampling efficiency, and resolution generalization.
To effectively implement our approach, several key design factors must be carefully considered. First, the architectural design must ensure that the diffusion decoder is effectively conditioned on the encoder latent representations. Second, the training objectives should leverage synergies with tra- ditional autoencoding losses, such as LPIPS (Zhang et al., 2018) 1) and GAN (Esser et al., 2021). Finally, diffusion- specific design choices play a crucial role, including: (I) model parameterization, which defines the prediction tar- get for the diffusion decoder; (2) noise scheduling, which shapes the optimization trajectory; and (3) the distribution of time steps during training and testing, which balances noise levels for effective learning and generation. Our study sys- tematically examines these components through controlled experiments, demonstrating their impact on achieving a high-performing diffusion- autoencoder. We show in the experiments that under the standard configuration (Rom- bach et al., 2022), method obtains a 40% improvement in
1. Introduction
Two dominant paradigms in modern visual generative mod- eling are autoregression (Radford et al., 2018) and diffu- sion (Ho et al., 2020). Tokenization is essential for both: discrete tokens allow step-by-step conditional generation in
'Work done as student researcher at Google. GGoole Deep- Mind *City University of Hong Kong. Conrepondencee Long Zhao <longzh@ google.com>, Xuhui Jia <xhjia@ google.com>>. Ting Liu <liuti@ google.com>..
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Epsilon- Denoising as Visual Decoding
terms of reconstruction quality, leading to 22% better image generation quality. More notably, we achieve x.3x higher inference throughput by increasing compression rates, while keeping competitive generation quality.
competing baseline (Esser et al., 2021), as it has become standard tokenizer training scheme widely adopted in state-of-the-art image and video generative models (Chang et al., 2022; Rombach et al., 2022; Yu et al., 2022; 2023; Kondratyuk et al., 2024; Esser et al., 2024).
In summary, our contributions are as follows: (1) introduc- ing a novel approach that fully leverages the capabilities of diffusion decoders for more practical diffusion- based au- toencoding, achieving strong rFID, high sampling efficiency (within to 3 steps), and robust resolution generalization; (2) presenting key design choices in both architecture and objectives to optimize performance; and (3) conducting ex- tensive controlled experiments that demonstrate our method achieves high-quality reconstruction and generation results, outperforming leading visual auto-encoding paradigms.
Diffusion. Given a data distribution pa and a noise distri- bution Pe. a diffusion process progressively corrupts clean data 20 Pe by adding noise ~ Pe and then reverses this corruption to recover the original data (Song & Ermon, 2019 Ho et al., 2020), represented as:
22 = 04 2o +
(2)
where t E [0, T] and e is drawn from a standard Gaus- sian distribution, Pe = 1)., The functions at and oy govern the trajectory between clean data and noise, af- fecting both training and sampling. The basic parameter- ization in Ho et al. (2020) defines or V1-of of with
2. Background
We start by briefly reviewing the basic concepts required to understand the proposed method. A more detailed summary of related work is deferred to Appx. B.
a, (Io' - -B.) $ sion coefficients B, are linearly interpolated values between Sn and - as 3, = Bo + 7-m (Br -0)), with start and end values are set empirically.
for discrete timesteps. The diffu-
Visual autoencoding. To achieve efficient and scalable high-resolution image synthesis, common generative mod- els, including autoregressive models (Razavi et al., 2019; Esser et al., 2021; Chang et al., 2022) and diffusion mod- els (Rombach et al., 2022), are typically trained in a low- resolution latent space by first downsampling the input im- age using a tokenizer. The tokenizer is generally imple- mented as a convolutional autoencoder consisting of an encoder, E, and decoder, G. Specifically,, the encoder, E, compresses
The forward and reverse diffusion processes are described by the following factorizations:
(( = II
g( (1 (
i=1
(3)
input image x: E TR. xxW33 into set of latent codes (i.e., tokens), E(x)=
and p( - p(xT) II ((
E /x//ff/fxx/ where f is the downsampling factor and n, is the latent channel dimensions. The decoder, G, then reconstructs the input from 2, such that (2))= =x
where the forward process g A ao)) transitions clean data To to noise aT while the reverse process p(æo:T) recovers clean data from noise. At denotes the time step interval or step size.
Training an autoencoder primarily involves several losses: reconstruction loss Cree. perceptual loss (LPIPS) LLPIPS, and adversarial loss Lav. The reconstruction loss minimizes pixel differences (i.e., typically measured by the E or & distance) between x and (z). The LPIPS loss (Zhang et al., 2018) enforces high-level structural similarities between inputs and reconstructions by minimizing differences in their intermediate features extracted from a pre-trained VGG network (Simonyan & Zisserman, 2015). The adversarial loss (Esser et al., 2021) introduces discriminator, D. which encourages more photorcalistic outputs by distinguishing between real images, D(æ)), and reconstructions, (z)( The final training objective is a weighted combination of these losses:
During training. the model learns the score function V log (ax) x - c, a. which represents gradient pointing to- ward the data distribution along the noise-to-data trajectory. In practice, the model se (22, is optimized by minimizing the score-matching objective:
Lxcrre = min (x [willo,se( t) + E||?],
(4)
where (t) defines the time-step sampling distribution and we a time-dependent weight. These elements together influence which time steps or noise levels are prioritized during training. Conceptually, the diffusion model learns the tangent of the trajectory at each point along the path. During sampling, it progressively recovers clean data from noise based on its predictions.
LVAE = Liee + ALPIPS LIPWS Auav
Ludvs
(1)
where the À values are weighting coefficients. In this paper, we consider the autoencoder optimized by Eq. 1 as our main
Rectified flow provides a specific parametrization of o and oy such that the trajectory between data and noise follows a
2

---

Epsilon- Denoising as Visual Decoding
"straight" path (Liu al., 2023; Albergo & Vanden-Eijnden, 2023). This trajectory is represented as:
Conditional denoising. Specifically, the input Pe is encoded by the encoder as E(x), and this encoding serves a condition to guide the subsequent denoising process. This reformulates the reverse process in Eq. 3 into a conditional form (Nichol & Dhariwal, 2021):
(1 t) zo + c,
(5)
where t E [0, 1]. In this formulation, the gradient along the trajectory, 6 - æo, is deterministic, often referred to as the velocity, The model ug t) is parameterized to predict velocity by minimizing:
p(æ0rr|z) = p(ær) II p( IP (8)
min (1, (a),) - (E ((ce-x)]' 1²] (6)
where the denoising process from the noise æт to the input Zo = æ, is additionally conditioned on 2 over time. Here, the decoder is no longer deterministic, as the process starts from random noise. For a more detailed discussion on this autoencoding formulation, we refer readers to Appx. A.
We note that this objective is equivalent to a score matching form (Eq. 4), with the weight Wi = (,- This equiv- alence highlights that alternative model parameterizations reduce to a standard denoising objective, where the primary difference lies in the time-dependent weighting functions and the corresponding optimization trajectory (Kingma & Gao, 2024).
A Architecture and conditioning. We adopt the standard U- Net architecture from Dhariwal & Nichol (2021) for our dif- fusion decoder G, while also exploring Transformer-based models (Peebles & Xie, 2023). For conditional denois- ing, we concatenate the conditioning signal with the input channel-wise, following the approach of diffusion-bart super-resolution models (Ho et al., 2022; Saharia al., 2022b). Specifically, low-rcsolution latents upsampled using nearest-neighbor interpolation to match the resolution of 2t, then concatenated along the channel dimension. In Appx. D, although we experimented with conditioning via AdaGN (Nichol Dhariwal, 2021), it did not yield signifi- cant improvement and introduced additional overhead, so we adopt channel concatenation.
During sampling, the model follows a simple probability flow ODE:
(7) Although a perfect straight path could theoretically be solved in a single step, the independent coupling between data and noise often results in curved trajectories, necessi- tating multiple steps to generate high-quality samples (Liu et al., 2023; Lee et al., 2024). In practice, we iteratively apply the standard Euler solver (Euler, 1845) to sample data from noise.
dæ; = (x), dl.
3.2. Objectives
3. Method
We adopt the standard autoencoding objective from Eq. to train e-VAE, with key modification: replacing the recon- struction loss free used for the standard decoder with the score-matching loss Lacore for training the diffusion decoder. Additionally, we introduce strategy to adjust the percep- tual L and adversarial Ladv losses to better align with the diffusion decoder training.
We introduce c- VAE, with an overview provided in Fig. 1. The core idea is to replace single-step, deterministic de- coding with an iterative, stochastic denoising process. By reframing autoencoding as a conditional denoising problem, we anticipate two key improvements: (1) more effective gen- eration of latent representations, allowing the downstream latent diffusion model to learn more efficiently, and (2) en- hanced decoding quality due to the iterative and stochastic nature of the diffusion process.
Velocity prediction. We adopt the rectified flow parame- terization, utilizing linear optimization trajectory between data and noise, combined with velocity-matching objective (Eq. 6). We inject the encoder output z into the objective by replacing ve (zt, t) with ((z,,t,.).
We systematically explore the design space of model ar- chitecture, objectives, and diffusion training configurations, including noise and time scheduling. While this work pri- marily focuses on generating continuous latents for latent diffusion models, the concept of iterative decoding could also be extended to discrete tokens, which we leave for future exploration.
Perceptual matching, The LPIPS loss (Zhang et al., 2018) minimizes the perceptual distance between the reconstruc- tions and real images using pre-trained models. typically VGG network (Esser et al., 2021; Yu et al., 2023; 2022). We apply this Ceature-matching objective to train c- VAE. However, unlike traditional autoencoders, e-VAE predicts velocity instead of directly reconstructing the image during training, making it infeasible to compute the LPIPS loss directly between the prediction and the target image. To address this, we leverage the simple reversing step from
3.1. Modeling
C- VAE retains the encoder & while enhancing the decoder G by incorporating a diffusion model, transforming the stan- dard decoding process into an iterative denoising task.
3

---

Epsilon VAE: Denoising as Visual Decoding
Training Pipeline
Resize & Concat
Input Image
Denoising Target
Estimated Input
Score Matching
Latents
Encoder
Diffusion Model
Perception & Trajectory Matching
Inference Pipeline
Resize Concat
Step
Step #2
Step #N
Input Image
Latents
Diffusion
Figure 1. An overview of e-V.. We frame visual decoding as an iterative denoising problem by replacing the autoencoder decoder with diffusson model, optimized using combination score, perception, and trajectory matching losses. During inference, images are reconstructed (or generated) from encoded (or sampled) latents through iterative denoising process. The number of sampling steps N can be flexibly adjusted within small NFF regimes (from to 3). We empirically confirm that e- eAV significantly outperforms standard VAE schema, even with just few steps.
Eq. to estimate zu from the prediction and æ, as follows:
where fake trajectories pe( 22) are contrasted with real trajectories g( To further stabilize training, we ap- ply the R2 gradient penalty to the discriminator param- eters (Mescheder et al., 2018). In Appx. D, explore alternative matching approaches, including the standard adversarial method of comparing individual reconstruc- tions æ with real images to. matching the trajectory steps æ + œt-At At (Xiao et al., 2022; Wang et al., 2024a), and our start-to-end trajectory matching æt + 2o, with the latter showing the best performance.
E/ 2z G( 2),
(9)
where th represents the reconstructed image estimated by the model at time We then compute the I.PIPS loss be- tween a and the target real image æ.
Denoising trajectory matching. The adversarial loss courages photorealistic outputs by comparing the reconstruc- tions to real images. We modify this to better align with a diffusion decoder. Specifically, our approach adapts the standard adversarial loss to enforce trajectory consistency rather than solely on realism. practice, we achieve this by minimizing the following divergence, Daav:
Final training objective combines Lucares LLPIPS, and Ladve with empirically adjusted weights (see Appx. C.2).
Note that applying LPIPS and adversarial losses the esti- mated one-step sample could lead to potential objective bias. However, we would like to emphasize that e- e-VAE differs significantly from traditional diffusion models in that its diffusion decoder is conditioned on encoded latents This conditioning provides strong prior about the input image to reconstruct, resulting in a more accurate estimated one- step sample than in typical diffusion scenarios. Therefore, we believe the potential for objective bias is considerably reduced e- VAE. Fine-tuning the diffusion decoder with frozen like Sargent avenue for further improvement, which we will explore in our future work.
min Eerps (Duiy
(10)
where Dady probability distance metric (Goodfellow al., 2014; Arjovsky al., 2017), and we adopt the basic non-saturating GAN (Goodfellow et al., 2014).
For adversarial training, we design time-dependent dis- criminator that takes time as input using AdaGN ap- proach (Dhariwal& & Nichol, 2021). To simulate the trajec- tory, we concatenate æy and x, along the channel dimension. The generator parameterized by O and the discriminator, parameterized by &, then optimized through minimax game as:
(2025) could be promising
3.3. Noise and time scheduling
min max Ludy Eeg [log D& (2o,, Fc. t)]
4
Noise scheduling. In diffusion models, noise scheduling involves progressively adding noise the data time
+ E (pe
[log (1 Do (2/, Lt, f))] (11)

---

Epsilon-VARE Denoising as Visual Decoding
by defining specific functions for o and of in Eq. 2. This process is erucial as it determines the signal-to-noise ra- tio, A = or which directly influences training dynamics. Noise scheduling can also be adjusted by scaling the inter- mediate states 2t with constant factor Y (0, 11, which shifts the signal-to-noise ratio downward. This makes train- ing more challenging over time while preserving the shape of the trajectory (Chen, 2023).
2023) for e-VAE. To evaluate model scaling, we test five decoder variants: base (B), medium (M), large (I.), extra- large (XL), and huge (H), by adjusting width and depth accordingly. Further model specifications are provided in Appx. C.I.
We experiment with the following two encoder configura- tions. c-VAE-lite: Hight-weight version with 6M param- eters, a downsampling rate of 16, and 8 latent channels; c- c-VAE-SD: standard version based on Stable Diffusion with 34M parameters, a downsampling rate of 8, and 4 latent channels. c- V-V---lite is intentionally designed as a more challenging setup and serves as the primary focus of analy- sis in the paper. For this configuration, we further explore downsampling rates of 4, 8, and 32, as well as latent di- mensions of 4, 16, and 32 channels. Both VAE and V-VAE are trained reconstruct 128 x 128 images under these controlled conditions. Additionally, we validate our method in the standard setup of c- c-VAE-SD, where we compare it against state-of-the-art VAEs.
In this work, we define a and or according to rectified flow formulation, while also scaling æy by 7, with the value chosen empirically. However, when y # 1, the variance of 2t changes, which can degrade performance (Karras al., 2022). To address this, we normalize the denoising input 2t by its variance after scaling, ensuring it preserves unit variance over time (Chen, 2023).
Time scheduling. Another important aspect in diffusion models is time scheduling for both training and sampling. controlled by xt) during training and At during sampling, as outlined in Eq. 3 and Eq. 4. A common choice for (t) is the uniform distribution (0 T). which applies equal weight to each time step during training. Similarly, uniform time steps At = F are typically used for sampling. How- ever, to improve model performance on more challenging time steps and focus on noisy regimes during sampling, the time scheduling strategy should be adjusted accordingly.
Evaluation. We evaluate the autoencoder on both recon- struction and generation quality using Fréchet Inception Dis- tance (FID) (Heusel al., 2017) the primary metric, and we also report PSNR and SSIM secondary metrics. For reconstruction quality (rFID), FID is computed at both train- ing and higher resolutions to assess generalization across resolutions. For generation quality (FID), we generate la- tents from the trained autoencoders and use them to train the DiT-XL/2 latent generative model (Peebles & Xie, 2023). This latent model remains fixed across all generation ex- periments, meaning improved autoencoder latents directly enhance generation quality.
In this work, we sample L from a logit-normal distribu- tion (Atchison& & Shen, 1980). which emphasizes interme- diate timesteps (Esser et al., 2024). During sampling, we apply a reversed logarithm mapping Plog defined as:
Piog(:; n) = log(m) log (f- (m - n) n)
(12)
lm) - log(n)
4.1. Reconstruction quality
where we set m 1 and 100, resulting in denser sampling steps early in the inference process.
Decoder architecture. We explore two major architectural designs: the UNet-based architecture from ADM (Dhariwal & Nichol, 2021) and the Transformer based DiT (Peebles & Xie, 2023). We compare various model sizes-ADM:((B M. L. XL, H} and DIT:(S, B, L, XL) with patch sizes of {4. 8). The results are summarized Fig. 2 (left). ADM consistently outperforms DiT across the board. While we observe rFID improvements in DIT when increasing the number of tokens by reducing patch size, this comes with significant computational overhead. The overall result aligns with the original design intentions: ADM for pixel-level generation and DiT for latent-level generation. For the following experiments, we use the ADM architecture for our diffusion decoder.
4. Experiments
We evaluate the effectiveness of e- cAV on image reconstruc- tion and generation tasks using the ImageNet (Deng al., 2009). The VAE formulation by Esser et al. (2021) serves as a strong baseline due to its widespread use in modern image generative models (Rombach et al., 2022; Peebles & Xie, 2023; Esser et al., 2024). We perform controlled exper- iments to compare reconstruction and generation quality by varying model scale, latent dimension, downsampling rates, and input resolution.
Model configurations. We use the encoder and discrimina- tor architectures from VQGAN (Esser et al., 2021) and keep consistent across all models. The decoder design follows BigGAN (Brock et al., 2019) for VAE and from ADM (Dhariwal & Nichol, 2021) for e- VAE. Additionally, we experiment with the DiT architecture (Peebles & Xie,
Compression rate. Compression can be achieved by ad- justing either the channel dimensions of the latents or the downsampling factor of the encoder. In Fig. 2 (middle and right), we compare VAE and e- VAE across these two aspects. The results show that e- VAE consistently outperforms VAE
5

---

Epsilon- VAE: Denoising as Visual Decoding
10
DIT/8
15
VAE E-VAE
VAE E-VAE
DIT/4
ADM
12
15
8
rFID
10
6
8
5
50
200
350
500
650
8
16
32
4
8
16
32
Decoder params (M)
Channel dimensions
Downsample factor
Figure 2.
Architecture and compression analysis. The VAE decoder uses either a UNet- base- ADM or Transformer- DiT (left). e-VAE and VAE are evaluated under different compression rates by varying latent channel dimensions (middle) or encoder downsampling factors (right). We follow the e- c-VAE-lite configuration in these experiments.
Comparisons to state-of-the-art VAEs. We provide im- age reconstruction results under the same configuration as VAEs in Stable Diffusion (SD-VAE): an encoder with 34M parameters and a chanmel dimension of 4 for 256 x 256 im- age reconstruction. We evaluate rFID, PSNR and SSIM on the full validation sets of ImageNet and COCO-2011 (Lin et al., 2014), with the results summarized in Tab. Our finds reveal that v-V outperforms state-of-the-ar VAEs when the decoder sizes are comparable, and its performance can be further improved by scaling up the decoder. This demonstrates the strong model scalability of our framework.
Table 1. ImageNet reconstruction results (rFID) at different resolutions using VAEs trained at 128 x 128. denotes training 128 128 followed by fine-tuning at a higher resolution.
Resolution (ImageNNet)
128
256
512
256
ADE (Rombach et al., 2022)
4.54
1.21
0.91
0.86
VAE (Sadat et al., 2024)
4.40
0.97
0.73
--VE-DD (B)
1.94
0.65
0.61
0.52
--VAE-SD (M)
1.58
0.55
0.53
0.47
--VAE-SD (L)
1.47
0.52
0.41
0.45
--VE--S (XL.
1.34
0.49
0.39
0.43
<-VAE-SD (H)
1.00
0.44
0.35
0.38
One-step e- VAE. Note that the denoising process of eAVE demonstrates promising results even with a single itera- tion. To show this, we provide a direct comparison between SD- VAE and our one-step c- VAE models in Tab. 3. This table presents image reconstruction quality on ImageNet 256 x 256 with the 8 x 8 downsampling factor. As shown, both e- VAE (B) and eAE (M) outperform SD-VAE across all metrics. These results confirm the effectiveness and efficiency of our one-step models compared to SD- SD-VAE. Consequently, this allows e-VAE to be adapted for scenarios with latency-sensitvv requirements, e-g., real-time visual- ization during image generation, by reducing the decoding step to a single pass.
in terms of rFID, particularly as the compression ratio in- creases. Specifically, as shown on the middle graph, e- e-V achieves lower rFIDs than VAE across all channel dimen- sions, with a notable gap at lower dimensions (4 and 8). On the right graph, VAV maintains lower rFIDs than VAE even as the downsampling factor increases, with the gap widening significantly at larger factors (16 and 32). Further- more, VAE delivers comparable or superior rFIDs even when the compression ratio is doubled, demonstrating its robustness and effectiveness in high-compression scenarios.
Resolution generalization. A notable feature of conven- tional autoencoders is their capacity generalize and recon- struct images at higher resolutions during inference (Rom- bach et al., 2022). To assess this, we conduct inference on images with resolutions of 256 x 256 and 512 x 512, using e-E and VAE models trained at 128 x 128. As shown in Tab. I, e. VAE effectiiely generalizes to higher res- olutions, consistently preserving its performance advantage over other VAEs. Furthermore, we find that fine-tuning mod- els at the target (higher) resolution leads to improvement at it, which is consistent with the observation made by Sadat et al. (2024). We hence utilize this multi-stage training strat- egy in the following experiments when the target resolution larger than 128 x 128.
4.2. Class- clasiconniiional image generation
We now evaluate the generative performance of e- VAE when combined with latent diffusion models (Rombach et al., 2022). We perform standard class-conditional image gener- ation tasks using the DiT-XL/2 model as our latent gener- ative model (Peebles & Xie, 2023). Further details on the training setup are provided in Appx. C.3. Tab, 4 presents the image generation results of e-VA and other competing VAEs resolutions of 256 x 256. The results show that e- VAE consistently outperforms other VAEs across differ- ent dowmsampling factors. In addition, we emphasize that c- VAE achieves favorable generation quality while using only 25% of the token length typically required by SD- SD-VAE.
6

---

Epsilon-VAE. Denoising as Visual Decoding
Table Comparisons with state-of-the-art image autoencoders. results are computed on ImageNet 50K validation set and COC0-2011 5K validation set. e- v-DD-D (M) achieves better reconstruction quality while having similar parameters (49M) in the decoder with other VAEs. Further improvements are obtained after we scale up to <-VA-SD (H) which has 355M decoder parameters.
Downsample factor
Discrete latent
Latent dim.
ImageNet PSNR
COCO
Method
rFID
SSIM
rFDD
PSNRT
SSIM
VQGAN (Esser al., 2021)
256
4.99
20.00
0.629
12.29
19.57
0.630
MaskGIT (Chang ct al., 2022)
256
2.28
LlamaGen (Sun et al., 2024)
2.19
20.79
0.675
8.11
20.42
0.678
16 16
SD-VAE (Rombach et al., 2022)
4
2.93
20.57
0.662
8.89
19.95
0.670
<-
-VAF-SD (M)
4
1.91
21.27
0.693
6.12
22.38
0.718
<- -VAE-SD (H)
4
1,35
22.60
0.711
4.18
24.26
0.830
VQGAN (Esser al., 2021)
1.19
23.38
0.762
5.89
23.08
0.771
ViT-VQGAN (Yu et al., 2022)
32
1.28
LlamaGin (Sun al., 2024)
0.59
24.45
0.813
4.19
24.20
0.822
SD-
VAF. SD-VAF
(Rombach al., 2022)
0.74
25.68
0.820
4.45
25.41
0.831
8 8
SDXL-VAL VAE (Podell et al., 2024)
4
0.68
26.04
0.834
4.07
25.76
0.845
VAE (Sadat et al., 2024)
4
0.87
26.02
0,740
<- VAE-SD (M)
4
0.47
27.65
0.841
3.98
25.88
0.850
e- V-AAE- (H)
4
0.38
29.49
0.851
3.65
26.01
0.856
Table Image reconstruction results of one-step e~VAE and SD-VAE ImageNet 256 x 256. A downsampling factor of include two variants of our model in the results: e- c-VAE-SD (B), which has similar inference speed to SD- VAE, and e- VAE-SD (M), which matches SD- VAE in the number of parameters.
Table 4. Benchmarking class-conditional image generation on ImageNet 256 x 256. We use the DiT-XT/2 architecture (Esser et al., 2024) latent diffusion models, and we not apply classifier- free guidance (Ho & Salimans, 2022).
x used for comparison.
Downsample factoor
Throughput (image/ks)
Method
FID.
Method
rFID.
PSNR
SSIM
SD-VAF (Rombach al., 2022)
3991
21.31
32 32
v-E--D (M)
3865
15.98
SD-VAE (Rombach et al., 2022)
0.74
25.68
0.820
VAE SD (H)
3870
14.26
<-V--D (B)
0.57
25.91
0.826
--VAE-SD (M)
0,51
26.45
0.830
SD-VAE (Rombach et al., 2022)
1220
14.59
16 16
<-VAE-S (M)
1192
10.68
VAE (H)
1180
9.72
Asym- VAE (Zha al., 2023)
Beem-VAE
502
10.85
2.3x
Omni- MAEi--VEEao (Wang et al., 2024b)
This token length reduction significantly accelerates latent diffusion model generation, leading to 2.3x higher infer- ence throughput while maintaining competitive generation quality. These results confirm that the performance gains from the reconstruction task successfully transfer to the gen- cration task, further validating the effectiveness of e- eAE..
8
SD-VAE (Rombach al., 2022)
522
11.63
VAF-SS (M)
491
9.39
<-VAE-SD (H)
477
8.85
self-attention scales quadratically with input resolution. In contrast, our convolution-basee UNet decoder offers more favorable linear scaling. As models grow, shifting complex- ity to the autoencoder helps reduce the burden on the latent model, leading to a more efficient overall system.
More importantly, e- VAE-SD achieves around 25% improve- ment in generation quality over SD-VAE at the 32 32 downsampling factor, alongside 3.2x inference speedup than SD-VAE at the 16 x 16 downsampling factor with comparable FID. We observed similar training speedups for latent diffusion models utilizing 6- c-VAE at this higher downsampling rate. These gains are more pronounced than those observed when increasing the downsampling factor from 8 x 16 16. These findings strongly suggest that the benefits of c- VAE and latent diffusion pipeline could be amplified with higher downsampling factors.
4.3. Ablation studies
We conduct a component-wise analysis to validate our key design choices, focusing on three critical aspects: architec- ture, objectives, and noise time scheduling. We evaluate the reconstruction quality (rFID) and sampling efficiency (NFE). The results are summarized in Tab. 5.
An additional advantage of scaling the autoencoder over the latent model lies in computational efficiency. Recent trends show latent diffusion models increasingly adopt Transformer architectures (Peebles & Xie, 2023), where
Baseline. Our evaluation begins with baseline model: an autoencoder with a diffusion decoder, trained solely using the score matching objective. This baseline follows the vanilla diffusion setup from Ho al. (2020), including their

---

Epsilon-VARE VAE: Denoising as Visual Decoding
Table Ablation study on key design choices for the e-VAE diffusion decoder. systematic evaluation the proposed architecture
(+), objectives (t), and noise & time scheduling (8). Each row progressively modifies or builds upon the baseline decoder, showing improvements in performance. The results are computed under the c-Ve configuration.
Ablation
NFE,
rFID
Baseline: DDPM-based diffusion decoder
1,000
(a) Diffusion - Rectified flow parameterization
100
24.11
(b) Uniform + Logit-normal time step sampling during training
50
23.44
(c) DDPM UNet + ADM UNet
50
22.04
(d) Perceptual matching db and 40
10
11.76
(e) Adversarial denoising trajectory matching on (af as)) (zea ai)
(1) Scale ae by y 0.6
(g) Uniform - Reversed logarithm time spacing during inference
6.24
40
7.2
E-VAE
Uniform
9.0
7.0
Logarithm
30
6.8
20
Diffusion
8.5
6.6
Flow
LPIPS
6.4
10
8.0
Adv.
6.2
10
0.2
0.4
0.6
5
NFE
Scale factor
NFE
Figure 3. Impact of our major diffusion decoder designs. Improved training objectives, particularly perceptual matching loss and adversarial denoising trajectory matching loss, significantly contribute better rFID scores and NFE (left). Effective noise scheduling by modulating the scaling factor further enhances rFID, with an optimum value of 0.6 in our experiments (middle). Lastly, adjusting time step spacing during inference ensures stable sampling in low NFF regimes (right). We report results under the e- V--ittt configuration.
UNet architecture, parameterization, and training configura- tions, while extending to conditional form as described in Eq. 8. Building on this baseline, we progressively introduce updates and evaluate the impact of our proposed method.
In (f), scaling reduces the signal-to-noise ratio (Chen, 2023), presenting challenges for more effective learning during training. Fig. 3 (middle) demonstrates that a scal- ing factor of 0.6 produces the best results. Finally, in (g), reversed logarithmic time step spacing during inference al- lows for denser evaluations noisier regions. Fig. 3 (right) demonstrates that this method provides more stable sam- pling in the lower NFE regime compared to the original uniform spacing.
Impact of proposals. In (a), transitioning from standard diffusion rectified flow (Liu et al., 2023) straightens the optimization path, resulting in significant gains in rFID and NFE. In (b), adopting a logit-normal time step distribution optimizes rectified flow training (Esser et al., 2024), further improving both rFID and NFE. In (c), updates to the UNet architecture (Nichol Dhariwal, 2021) contribute to en- hanced rFID scores. In (d), LPIPS loss applied to match reconstructions d with real images Fp. In (e), adversar- ial trajectory matching loss aligns (.), xx) with (æo, æx), the target transition in rectified flow. Both objectives im- prove model understanding of the underlying optimization trajectory, significantly enhancing rFID scores and NFE.
In Fig. 3 (right), reconstruction quality degrades when the number of denoising steps exceeds three. To enable large step sizes for the reverse process during inference, we in- troduce the denoising trajectory matching loss to implicitly model the conditional distribution p(zo| (z,), shifting the denoising distributions from traditional Gaussian to non- Gaussian multimodal forms (Xiao et al., 2022). However, the assumptions underlying this approach are most effective when the total number of denoising steps small. This reveals an optimal range of one to three inference steps. The degradation beyond this range also suggests that uni- form step spacing may longer be ideal. Accordingly, empirically explored alternative sampling strategies and found that reversed logarithmic schedule yields improved performance, as shown in the figure.
Up to this point, with the full implementation of Eq. 1, we can compare our proposal with the VAE (B) model, which achieves rFID score of 11.15. Our model, with score of 8.24, already surpasses this baseline. We further improve performance by optimizing noise and time scheduling within our framework, as described next.
8

---

Epsilon-VAil VAE: Denoising as Visual Decoding
AVE /C
E
2 3/4 Th CTH
GT
AVIE IC
A
SD-VAE
2 3/4
T CTH
I II IN
AVE OC
A!
22 3/4 M /CH
E-VAE
Figure Image reconstruction results under the SD-VAE configuration (Rombach et al., 2022) at the resolution 512 x 512. We find that vAE produces accurate visual details than SD-VAE in the highlighted regions with text human face. Best viewed when zoomed-in and color.
4.4. Visualization
objectives, and diffusion training, we demonstrate signifi- cant performance improvements. Our approach outperforms traditional autoencoders in both reconstruction and genera- tion quality, particularly in high-compression scenarios. We hope our concept of iterative generation during decoding inspires further advancements in visual autoencoding.
In addition to the quantitative results, Fig. 4 shows high- resolution image reconstruction samples produced by SD- VAE (Rombach al., 2022) and e-VAE at the resolution of 512 x 512. We observe that reconstructed images generated by e-VAE demonstrate a better visual quality than ones of SD-VAE. In particular, c-VAE maintains good visual quality for small text and human face. We provide more visual comparisons in Appx. E and throughout discussions on the major properties and advantages of e- e-VAE compared to traditional VAEs in Appx. A.
Acknowledgements
We would like to thank Xingyi Zhou, Weijun Wang, and Caroline Pantofaru for reviewing the paper and providing feedback. We thank Rui Qian, Xuan Yang, and Mingda Zhang for helpful discussion. We also thank the Google Kauldron team for technical assistance.
5. Conclusion
We present e- v-V,, effective visual tokenizer that in- troduces diffusion decoder into standard autoencoders, urning single-step decoding into a multi-step probabilis- tie process. By exploring key design choices in modeling,
Impact statement
Our work could lead improved autoencoding techniques which have the potential benefit generative modeling

---

Epsilon- VAE:: Denoising as Visual Decoding
various perspectives, including reducing training time and memory requirements, improving visual qualities, etc. Although our work does not uniquely raise any new ethical challenges, visual generative modeling is field with several ethical concerns worth acknowledging. For example, there are known issues around bias and fairness, either the representation of generated images (Menon et al., 2020) or the implicit encoding of stereotypes (Steed & Caliskan, 2021), as well as potential risks in privacy. To ensure that the benefits of this technology are harnessed responsibly, we encourage continued open discussions in the community around the development of these new technologies.
Chang, H., Zhang, H., Jiang, L.,, Liu, C., and Freeman,
W. T. MaskGIT: Masked generative image transformer. In CVPR, pp. 11315-11325, 2022.
Chen, J., Cai, H., Chen, J., Xie, E., Yang, S., Tang, H., Li, M., Lu, Y., and Han, S. Deep compression autoen- coder for efficient high-resolution diffusion models. arXiv preprint arXiv:2410. 10733, 2024.
Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, Generative pretraining from pixels. In ICML, pp- 1691-1703, 2020.
Chen, T. On the importance of noise scheduling for diffusion models. arXiv preprini urXiv:2301.10972. 2023.
References
Albergo, M. S. and Vanden-Eijnden, E. Building normaliz- ing flows with stochastic interpolants. In ICLR, 2023.
Chen, T., Li, I.., Saxena, S., Hinton, G., and Fleet, D. J. A generalist framework for panoptic segmentation of im- ages and videos. In ICCV, pp. 909-919, 2023.
Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein generative adversarial networks. In ICML, pp. 214-223, 2017.
Chen, Y., Girdhar, R., Wang, X., Rambhatla, S., and Misra, I. Diffusion autoencoders are scalable image tok- enizers, arXiv preprint arXiv:2501.18592 2025.
Atchison, and Shen, S. M. Logistic-normal distributions: Some properties and uses. Biometrika, 67(2):261-272, 1980.
Deng, J., Dong, W., Socher, R., Li, L.-J, Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In CVPR, pp. 248-254 2009.
Baldridge, J.. Bauer, J., Bhutani, M., Brichtova, N., Bun- ner, A., Chan, K., Chen, Y., Dieleman, S., Du, Y., Eaton-Rosen, Z. et al. Imagen 3. arXiv preprint arXiv:2408.07009 2024.
Dhariwal, P. and Nichol, A. Diffusion models beat GANs on image synthesis. In NeurIPS, 2021.
Ding, Z., Zhang, M., Wu, J., and Tu, Z. Patched denoising diffusion models for high-resolution image synthesis. In ICLR, 2024.
Birodkar, V. Barcik, G., Lyon, J., Ioffe, S., Minnen, D., and Dillon, J. V. Sample what you cant compress. arXiv preprint arXiv:2409.025:: 2024.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In CVPR, pp. 12873- 12883, 2021.
Blau, Y. and Michaeli, T. The perception-distortion tradeoff. In CVPR, pp. 6228-6237, 2018
Blau, Y. and Michaeli, T. Rethinking lossy compression:
The aate-distortion-pereeppie tradeoff. In ICML, pp. 675- 685, 2019.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F, Podell, D., Dockhorn, T., English, Z., Lacey, K. Good- win, A., Marek, Y., and Rombach, R. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024.
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang. Q. JAX: composable transformations of Python+NumPy programs, 2018. URL http://gis whw.bb.thhhu.coo com/jax-moe jax.
Euler, Institutionum calculi integralis, volume impen- sis Academiae imperialis scientiarum, 1845.
Brock, A., Donahue, J., and Simonyan, K. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In NeurIPS, 2014.
Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y.,
Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng, C., Wang, R., and Ramesh, A. Video generation models as world simulators. OpenAl Blog. 2024. URL https: /openaa..... www.coo//ooo..o..oo sora/.
Gupta, A., Yu, L., Sohn, K., Gu, X., Hahn, M., Fei-Fei, L., Essa, I., Jiang, L., and Lezama, J. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.066m 2023.
10

---

Epsilon-VAL VAE: Denoising as Visual Decoding
Heek, J., Levskaya, A., Oliver, A., Ritter, M., Rondepierre, B., Steiner, A., and van Zee, M. Flax: A neural network library and ecosystem for JAX, 2024. URL http:// github. cwm goog/e//llll....
Lee, S., Kim, B., and Ye, C. Minimizing trajectory eur- vature of ODE-based generative models. In ICMI., pp. 18957-18973, 2023.
Lee, S., Lin, Z., and Fanti, G. Improving the training of rectified flows. arXiv preprint arXiv:2405.20520, 2024.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. GANs Trained by Two Time-Scale Up- date Rule Converge to Local Nash Equilibrium. In NeurlPS, 2017.
Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregres- sive image generation without vector quantization. arXiv preprint arXiv:2406. 1838, 2024.
Hinton, G. E. and Salakhutdinov, R. R. Reducing the di- mensionality data with neural networks. Science, 313 (5786):504-577 2006.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft COCO: Common objects in context. In ECCV, pp. 740- 755, 2014.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12580 2022.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. In NeurIPS, 2020.
Liu, X., Gong, C., and Liu, Q Flow straight and fast: Learning to generate and transfer data with rectified flow. ICLR, 2023.
Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., and Salimans, T. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(4)):1-33, 2022.
Luo, S., Tan, Y., Huang, L., Li, J., and Zhao, H. La- tent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023.
Hoogeboom, E., Agustsson, E., Mentzer, F., Versari, L., Toderici, G., and Theis, L. High-fidelity image compres- sion with bosed generative models. arXiv preprint arXiv:2305, 18231, 2023a.
Ma, N., Goldstein, M., Albergo, M. S., Boffi, N. M., Vanden- Eijnden, E., and Xie, S. SiT: Exploring flow and diffusion- based generative models with scalable interpolant trans- formers. arXiv preprint arXiv:2401.087
2024.
Hoogeboom, E., Heek, J., and Salimans, T. simple diffusion: End-to-end diffusion for high resolution images. In ICML, pp. 13213-13232 2023b.
Menon, S., Damian, A., Hu, S., Ravi, N., and Rudin, C. PULSE: Self-supervised photo upsampling via latent space exploration of generative models. CVPR, pp- 2437-2445, 2020.
Karras, T., Laine, S., and Aila, T. A style-based genera- tor architecture for generative adversarial networks. In CVPR, pp. 1401-4410, 2019.
Mescheder, I.., Geiger, A., and Nowozin, S. Which training methods for GANs do actually converge? In ICML, pp. 3481-3490, 2018.
Karras, T., Aittala, M., Aila, T., and Laine. S. Elucidating the design space of diffusion- based generative models. In NeurlPS, 2022.
Nguyen, T. H. and Tran, A. SwiftBrush: One-step text-to- image diffusion model with variational score distillation. In CVPR, pp. 7807-7816, 2024.
Kingma, D. and Gao, R. Understanding diffusion objectives as the elbo with simple data augmentation. In NeurIPS, 2024.
Nichol, A. Q. and Dhariwal, Improved denoising diffusion probabilistic models. ICML, pp. 8162 8171, 2021.
Kingma, D. P. Auto-encoding variational bayes. arXiv preprint 13 2013.
Peebles, W. and Xie. Scalable diffusion models with transformers. In ICCV, pp. 4195 4205, 2023.
Kingma, D. P. and Ba. J. Adam: A method for stochastic optimization. In ICLR, 2015.
Kondratyuk, D., Yu, I.., Gu, X., Lezama, J., Huang, J., Hornung, R., Adam, H., Akbari, H., Alon, Y., Birodkar, al. VideoPoet: A large language model for zero-shot video generation. /CML, 2024.
Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. FiLM: Visual reasoning with a general conditioning layer. In AAAI, 2018.
Pernias, P., Rampas, D., Richter, M. L., Pal, C. J., and Aubreville, M. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. In ICLR, 2024.
Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metric for assess- ing generative models. In NewrIPS, 2019.

---

Epsilon-VARR VAE: Denoising as Visual Decoding
Podell, D., English, Z.,, Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. SDXL: Im- proving latent diffusion models for high-resolution image synthesis. In ICLR, 2024.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. In NeurlPS, 2019.
Song, Y., Sohl-Dickstein J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
Preechakul, K., Chatthee, N., Wizadwongsa, S., and Suwa- janakom, S. Diffusion autoencoders: Toward a meaning- ful and decodable representation. In CVPR, pp. 10619- 10629, 2022.
Steed, R. and Caliskan, A. Image representations learned with unsupervised pre-training contain human-like biases. ACM conference on fairness, accountability, and trans- 2021.
Radford, A., Narasimhan, K., Salimans, T., and Sutskever, 1. Improving language understanding by generative pre- training, OpenAl Blog, 2018.
parency, pp. 701-713, 2021.
Sun, P., Jiang, Y. Chen, S., Zhang, S., Peng, B., Luo, P., and Yuan, Z. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06530 2024.
Razavi, A., Van den Oord, A., and Vinyals, O. Generating diverse high-fidee high-fidelity images with VQ-VAE-2. In NewrlPS, 2019.
Van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., Graves, A., et al. Conditional image generation with pixelenn decoders. In NeurlPS, 2016.
Rombach, R., Blattmann, A.. Lorenz, D., Esser, and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Sadat, S., Buhmann, J., Bradley, D., Hilliges, O., and Weber, R. M. LAE: Lightweight and efficient variational autoencoders for latent diffusion models. arXiv preprint TrXiv:2405.14477 2024.
Van Den Oord, A., Vinyals, O., et al. Neural discrete repre- sentation learning. In NeurIPS, 2017.
Wang, F--Y,, Huang, Z., Bergman, A. W., Shen, D., Gao, P., Lingelbach, M., Sun, K., Bian, W., Song, G., Liu, Y.. et al. Phased consistency model. arXiv preprint arXiv:2405. 18407, 2024a.
Saharia, C., Chan, W., Chang, H., Lec, C., Ho, J., Salimans,
T., Fleet, D., and Norouzi, M. Palette: Image-to- to-image diffusion models. In ACM SIGGRAPH, pp. 1-10, 2022a.
Wang. J., Jiang, Y. Yuan, Z., Peng, B., Wu, Z., and Jiang, Y.GG. OmniTokenizer: A joint image-video tokenizer for visual generation. arXiv preprint arXiv:2406. 093999 2024b.
Saharia, C., Ho, J., Chan, W. Salimans, T., Fleet, D. J., and Norouzi, M. Image super-resolution via iterative refinement. IEEE TPAMI, 45(4):4713-4722) 2022b.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training GANs. In NeurIPS, 2016.
Wang. Z., Jiang, Y.. Zheng, H., Wang, P., He. P., Wang, Z. A., Chen, W., and Zhou, M. Patch diffusion: Faster and more data-efficient training of diffusson models. In NeurIPS, 2024c.
Sargent, K., Hsu, K., Johnson, J., Fei-Fei, L., and Wu, Flow to the mode: Mode-seeking diffusion autoencoders for state-of-the-art image tokenization. arXiv preprint 1xXiv:2503.11056 2025.
Wu, Y. and He, K. Group normalization. In ECCV, pp. 3-19, 2018
Xiao, Z., Kreis, K., and Vahdat, A. Tackling the generative learning trilemma with denoising diffusion GANs. In ICLR, 2022.
Sauer, A., Lorenz, D., Blattmann, A., and Rombach, R. Adversarial diffusion distillation. In ECCV, pp. 87-103, 2024.
Shannon, C. E. al. Coding theorems for discrete source with a fidelity criterion. IRE Nat. Cony. Rec, 4(142-16) 163): 1959.
Yang, R. and Mandt, S. Lossy image compression with conditional diffusion models. In NeurlPS, 2024.
Yu, J., Li, X., Koh, J. Y., Zhang, H., Pang. R., Qin, J., Ku, A., Xu, Y., Baldridge, J., and Wu, Y. Vector-quantized image modeling with improved VQGAN. In ICLR, 2022.
Shi, J., Wu, C., Liang. J., Liu, X., and Duan, N. DiVAE: Photorealistic images synthesis with denoising diffusion decoder. arXiv preprint arXiv:2206.00388 2022.
Yu, L., Cheng, Y., Sohn, K., Lezama, J., Zhang, H., Chang, H., Hauptmann, A. G., Yang, M.-H., Hao, Y., Essa, I., et al. MAGVIT: Masked generative video transformer. In CVPR, pp. 10459-10469, 2023.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for scage image recognition. In ICLR, 2015.
12

---

Epsilon- VAE: Denoising as Visual Decoding
Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng. Y., Gupta, A., Gu, X., Haupt- mann, A. G., et al. Language model beats diffusion- tokenizer is key to visual generation. In ICLR, 2024a.
Yu, Q. Weber, M., Deng, X., Shen, X., Cremers, D., and Chen, L.-C. An image is worth 32 tokens for reconstruc- tion and generation. arXiv preprint arXiv:2406.07550 2024b.
Zhang. R., Isola, P., Efros, A. A., Shechtman, E., and Wang,
O. The unreasonable effectiveness of deep features perceptual metric. In CVPR. pp. 586-595, 2018.
Zhao, Y., Xiong, Y., and Krähenbühl, Image and video tokenization with binary spherical quantization. arXiv preprint Xiv:2406.0054444 2024.
Zhu, Z., Feng, X., Chen, D., Bao, J., Wang. L.,, Chen, Y., Yuan, L., and Hua, G. Designing a better asym- metric VQGAN for StableDiffusion. arXiv preprint
13

---

Epsilon- Denoising as Visual Decoding
A. Discussion
challenge becomes more pronounced as models, datasets, or resolutions scale up. A promising future direction is patch- based diffusion (Ding et al., 2024; Wang et al., 2024c), which partitions the input into smaller, independently pro- cessed patches. This approach has the potential to reduce memory usage and enable faster parallel decoding.
Distribution-aware compression. Traditional image compression methods optimize the rate-distortion trade- off (Shannon et al., 1959), prioritizing compactness over input fidelity. Building on this, we also aim to capture the broader input distribution during compression, generat- ing compact representations suitable for latent generative models. This approach introduces an additional dimension to the trade-off, perception or distribution fidelity (Blau & Michaeli, 2018), which aligns more closely with the rate- distortion-perceptic framework (Blau & Michaeli, 2019).
B. Related work
Image tokenization. Image tokenization is crucial for effec- tive generative modeling, transforming images into compact, structured representations, A common approach employs an autoencoder framework (Hinton & Salakhutdinov, 2006), where the encoder compresses images into low-dimensional latent representations, and the decoder reconstructs the origi- nal input. These latent representations can be either discrete commonly used in autoregressive models (Van den Oord et al., 2016; Van Den Oord et al., 2017; Chen et al., 2020; Chang et al., 2022; Yu et al., 2023; Kondratyuk et al., 2024), or continuous, as found in diffusion models (Ho et al., 2020; Dhariwal & Nichol, 2021; Rombach et al., 2022: Peebles & Xie, 2023; Gupta et al., 2023; Brooks et al., 2024). The foundational form of visual autoencoding today originates from Van Den Oord et al. (2017). While advancements have been made in modeling (Yu et al., 2022; 2024b; Chen et al., 2024), objectives (Zhang et al., 2018; Karras et al., 2019; Esser et al., 2021), and quantization methods (Yu et al., 2024a; Zhao et al., 2024), the core prroding-andddecond scheme remains largely the same.
Iterative and stochastic decoding. A key question within the vate-distortion te-ddiitortion-perppttion perception trade-off is whether the it- erative, stochastic nature of diffusion decoding offers ad- vantages over traditional single-step, deterministic meth- ods (Kingma, 2013). The strengths of diffusion (Ho et al., 2020) lie in its iterative process, which progressively refines the latent space for more accurate reconstructions, while stochasticity allows for capturing complex variations within the distribution. Although iterative methods may appear less efficient, our formulation is optimized to achieve opti- mal results in just three steps and also supports single-step decoding, ensuring decoding efficiency remains practical (see Fig. 3 (left)). While stochasticity might suggest the risk of "hallucination" in reconstructions, the outputs remain faithful to the underlying distribution by design, producing perceptually plausible results. This advantage is particularly evident under extreme compression scenarios (see Fig. 5), with the degree of stochasticity adapting based on compres- sion levels (see Fig. 6).
In this work, we propose new perspective by replacing the traditional decoder with diffusion process. Specifically, our new formulation retains the encoder but introduces a conditional diffusion decoder. Within this framework, we systematically study various design choices, resulting in a significantly enhanced autoencoding setup.
Multi-step vs. single-step decoding. While replacing single-step decoding with an iterative process may seem counter-intuitive due to increased computational cost, the diffusion-based decoder addresses this concern in three key ways. First, it offers scalable inference, where even single- step variant already outperforms a plain VAE decoder, and additional steps further enhance quality (see Tab. 3). Second, provides controllable trade-off's between computation and visual fidelity, allowing the number of steps to be adjusted at inference time based on application needs. Third, as shown in Tab. 4, it enables training under higher compression ra- tios, which helps offset the added cost of iterative decoding by reducing the size of latent representations.
Additionally, we refer to the recent work MAR (Li et al., 2024), which leverages diffusion to model per-token dis- tribution in autoregressive frameworks. In contrast, our approach models the overall input distribution in autoen- coders using diffusion. This difference leads to distinct applications of diffusion during generation. For instance, MAR generates samples autoregressivelly, decoding each token iteratively using diffusion, token by token. In our method, we first sample all tokens from the downstream generative model and then decode them iteratively using diffusion as whole.
Scalability. As discussed in Sec. 4.1, our diffusion- based decoding method maintains the resolution generalizabil- ity typically found in standard autoencoders. This feature is highly practical: the autoencoder is trained on lower- resolution images, while the subsequent latent generative model is trained on latents derived from righer-resolution inputs. However, we acknowledge that memory overhead and throughput become concerns with our UNet-based dif- fusion decoder, especially for high-resolution inputs. This
Image compression. Our work shares similarities with re- cent image compression approaches that leverage diffusion models. For example, Hoogeboom et al. (2023a); Birodkar et al. (2024) use diffusion to refine autoencoder residuals, enhancing high-frequency details. Yang & Mandt (2024) employs diffusion decoder conditioned on quantized dis-
14

---

Epsilon- VAE: Denoising as Visual Decoding
Increasing downsample factor (VAE)
Increasing downsample factor (E-VAE)
Ground Truth
8x
16x
32x
8x
16x
32x
Figure Reconstruction results with varying downsampling ratios. e- e-VAE maintains both high fidelity and perceptual quality, even under extreme downsampling conditions, whereas VAE fails to preserve semantic integrity. Best viewed when zoomed-in in and in color.
crete codes and omits the GAN loss. However, these meth- ods primarily focus on the traditional rate-distortion tradeoff, balancing rate (compactness) and distortion (input fidelity) (Shannon et al., 1959), with the goal of storing and transmit- ting data efficiently without significant loss of information.
Diffusion decoder. Several studies (Preechakul et al., 2022: Shi et al., 2022; Pernias et al., 2024; Nguyen & Tran, 2024; Sauer al., 2024; Luo et al., 2023) have explored diffu- sion decoders conditioned on compressed latents of the input, which are relevant to our work. We outline the key differences between these works and e-VAE: First, prior works have not fully leveraged the synergy between dif- fusion decoders and standard VAE training objectives. In this work, we enhance ofte--of-the-rn VAE objectives by replacing the reconstruction loss with score matching loss and adapting LPIPS and GAN losses to ensure compatibility with diffusion decoders. These changes yield significant improvements autoencoding performance, as evidenced by lower rFID scores and faster inference. Second, we are the first to investigate various parameterizations (e.g., g., ep- silon and velocity) and demonstrate that modern velocity parameterization, coupled with optimized train and test-time
In this work, we emphasize perception (distribution fidelity) alongside the rate-distortion tradcoff, ensuring that recon- structions more closely align with the overall data distri- bution (Heusel al., 2017; Zhang et al., 2018; Blau & Michaeli, 2019), thereby enhancing the decoded results from the sampled latents of downstream generative models. We achieve this by directly integrating the diffusion process into the decoder, unlike Hoogeboom et al. (2023a); Birod- kar et al. (2024). Moreover, unlike Yang Mandt (2024), we do not impose strict rate-distortion regularization in the latent space and allow the GAN loss to synergize with our approach.
15

---

Pasilon-VAE, Denoising as Visual Decoding
Ground Truth
oownsample factor Bx
Downsample factor 16x
Downsample factor
Figure VAE reconstruction results with varying random seeds and downsampling ratios. At lower compression levels, the reconstruction behaves more deterministically, whereas higher compression introduces stochasticity, enabling more flexible reconstruction of plausible inputs. Best viewed when zoomed-in and in color.
noise scheduling, provides substantial benefits. These en- hancements improve both reconstruction performance and sampling efficiency. Third, previous diffusion-based de- coders (Preechakul et al., 2022: Shi et al., 2022; Pernias al., 2024), which often rely on ad-hoe techniques like distillation or consistency regularization to speed up infer- ence (Nguyen& & Tran, 2024; Sauer et al., 2024; Luo al.,, 2023), our approach achieves fast decoding (1 to 3 steps) without such techniques. This is made possible by integrat- ing our proposed objectives and parameterizations. Last but not least, e- VAE exhibits strong resolution generalization capabilities, a key property of standard VAEs. In contrast, models like DiffusionAA (Preechakul al., 2022) and Di- VAE (Shi et al., 2022) either lack this ability are inher- ently limited. For example, DiVAE's bottleneck add/concat design restricts its capacity to generalize across resolutions.
presents a diffusion-based tokenizer which learns compact visual representations for image generation. Its main in- sight is that a single diffusion learning objective is capable of training scalable image tokenizers. More than that, our method demonstrates that traditional autoencoding losses such LPIPS and GAN losses are complimentary to the dif- fusion target, leading better reconstruction quality. This design substantially differ our work from DiTo.
While following different motivation, Lee et al. (2023) essentially also proposes a VAE with a denoising decoder but uses the encoding as the "initial noise" instead of as conditioning for a standard diffusion model starting from standard Gaussian distribution. This idea could be poten- tially used for speeding up the proposed approach, which we will explore in the future.
Image generation. Recent advances in image generation span a wide range of approaches, including VAEs (Kingma, 2013), GANs (Goodfellow al., 2014), autoregressive mod- els (Chen et al., 2020) and diffusion models (Song et al., 2021; Ho et al., 2020). Among these, diffusion models have emerged as the leading approach for generating high- dimensional data such images (Saharia et al., 2022a; Baldridge et al., 2024; Esser et al., 2024) and videos (Brooks et al., 2024; Gupta et al., 2023), where the gradual refine- ment of global structure is crucial. The current focus in diffusion based generative models lies in advancing archi-
SWYCC (Birodkar et al., 2024) also explores joint learn- ing of continuous encoders and decoders using a diffusion model. However, SWYCC differs fundamentally from our approach: it replaces the GAN loss with diffusion-based loss, while we focus on identifying optimal synergies be- tween traditional autoencoding losses (including GAN loss) and diffusion-based decoding. Our goal is to identify an optimal strategy for combining these elements, rather than simply substituting one for another.
Another closely related work, DiTo (Chen al., 2025), also
16

---

Epsilon-VARE Denoising as Visual Decoding
Table
for decoder variants.
Table 7. Image reconstruction results on ImageNet 128 128.
Model
Channel dim.
Depth multipliers
of blocks
Configuration
NFE
rFID
Base (B)
64
[,, 4]
Baseline (c) in Tab. S:
Medium (M)
96
[,,
Inject conditioning by channel-wise concatenation
50
22.04
Large (L)
128
[,, 4
Inject conditioning by AdaGN
50
22.01
large- (XL)
128
2,,
4
Baseline (e) in Tab,
Huge (H)
256
2,,4,
Matching distribution of di and ahe
N/A
Matching trajectory of at aPg
8.24
Matching trajectory of an -+ at-ar
10.53
tectures (Rombach et al., 2022; Peebles & Xie, 2023; Hooge- boom et al., 2023b), parameterizations (Karras et al., 2022; Kingma & Gao, 2024; Ma et al., 2024; Esser et al., 2024), or better training dynamics (Nichol & Dhariwal, 2021; Chen, 2023; Chen et al., 2023). However, tokenization, an essen- tial component in modern diffusion models, often receives less attention.
steps, followed by a constant rate 0.0001 for a total of one million steps. The batch size is 256, with data augmenta- tions including random cropping and horizontal flipping. An exponential moving average of model weights is maintained with decay rate of 0.999. All models are implemented in JAX/Flax (Bradbury et al., 2018; Heek et al., 2024) and trained on TPU-v5lite pods.
In this work, we focus on providing compact continuous latents without applying quantization during autoencoder training (Rombach et al., 2022), as they have been shown to be effective in statc-of-the-art latent diffusion models (Rom- bach et al., 2022; Saharia et al., 2022a; Peebles Xie, 2023; Esser et al., 2024; Baldridge et al., 2024). We com- pare autoencoding performance against the baseline approach (Esser et al., 2021) using the DiT framework (Pee- bles & Xie, 2023) as the downstream generative model.
C.3. Latent diffusion models
We follow the setting in Peebles & Xie (2023) to train the latent diffusion models for unconditional image generation on the ImageNet dataset. The DiT-XL/2 architecture is used for all experiments. The diffusion hyperparameters from ADM (Dhariwal & Nichol, 2021) are kept. To be specific, use fmax 1000 linear variance schedule ranging from 0.0001 to 0.02, and results are generated using 250 DDPM sampling steps. For simplicity and training stability, we remove the variational lower bound loss term during training, which leads to a slight drop in generation qualities.
C. Experiment setups
In this section, we provide additional details on our experi- ment configurations for reproducibility.
All models are trained with Adam (Kingma & Ba, 2015) with weight decay. We use constant learning rate of 0.0001 and a batch size of 256. Horizontal flipping and random cropping are used for data augmentation. We maintain an exponential moving average of DiT weights over training with a decay of 0.9999. We use identical training hyperparameters across all experiments and train models for one million steps in total. No classifier-fret guidance (Ho & Salimans, 2022) is employed in all the experiments. Inference throughputs are computed Tesla H100 GPU.
C.1. Model specifications
Tab. summarizes the primary architecture details for each decoder variant. The channel dimension the number of channels of the first U-Net layer, while the depth multipliers are the multipliers for subsequent resolutions. The number of residual blocks denotes the number of residual stacks contained in each resolution.
C.2. Implementation details
During the training of discriminators, Esser et al. (2021) introduced an adaptive weighting strategy for Audv. However, we notice that this adaptive weighting does not introduce any benefit which consistent with the observation made by Sadat et al. (2024). Thus, we set Aadv 0.5 in the experiments for more stable model training across different configurations.
D. Additional experimental results
We note that all experiments conducted this section are under the e- e-VAE-lite configuration.
Conditioning. In addition injecting conditioning via channel-wise concatenation, we explore providing condi- tioning to the diffusion model by adaptive group normal- ization (AdaGN) (Nichol & Dhariwal, 2021; Dhariwal & Nichol, 2021). To achieve this, we resize the conditioning (i.e., encoded latents) via bilinear sampling to the desired
The autoencoder loss follows Eq. 1, with weights set to ALPIS = 0.5 and Audv = 0.5. We use the Adam opti- mizer (Kingma & Ba, 2015) with 31 = and 32 = 0.999, applying linear learning rate warmup over the first 5,000
17

---

Epsilon- VAE: Denoising as Visual Decoding
Table Model scaling and resolution generalization analysis. Five model variants are trained and evaluated. AnD represents the absolute differences (or relative ratio) in rFID between the corresponding model size variants of VAE and c-VAE. denotes resolution generalization experiments. To fairly evaluate the impact of e- VAE under controlled model parameters, we highlight three groups of model variants with comparable parameters, using different colors.
ImageNet 128 x 128
ImageNet 256 x 256
ImageNet 512 x 512
Model
G params (M)
rFID)
A
rFID
A
rFID
AID
VAE (B)
10.14
11.15
5.74
3.69
VAE (M)
22.79
9.26
4.63
2.69
VAE (L)
40.48
8.49
4.78
2.78
VAB (XL)
65.27
7.58
4.42
2.41
VAE (H)
161.81
7.12
4.29
2.37
C- VAE (B)
20.63
6.24
4.91 (44.0%)
3,90
1.84 (32.0%)
2.06
1.63 (44.2%)
c- VAE (M)
49.33
5,42
3.84 (41.5%)
2.79
1.84 (39.7%)
2,02
0.67 (24.9%)
VAE (L)
88.98
4.71
3.78 (44.5%)
2.60
2.03 (43.8%)
1.92
0.86 (30,9%)
VAE (XL)
140.63
4.18
3.40 (44.9%)
2.38
2.04 (46.2%)
1.82
0.59 (24.5%)
VAE (H)
355.62
4.04
3.08 (43.3%)
2.31
1.98 (46.2%)
1.78
0.59 (24.9%)
Table Unconditional image generation quality. The DLT-XL/2 is trained on latents provided by the trained autocncoders, VAE and e- c-VAE, with varying model sizes using ImageNet. We evaluate the generation quality at resolutions of 128 x 128 and 256 x 256 using four standard metries. Additionally, we report rFID determine the improvement trend observed in reconstruction task extends to the generation task. We highlight three groups of model variants with comparable parameters.
ImageNet 128 x 128
ImageNet 256 x 256
Model
rFID
FID]
18
Prec.t
Rec.t
rFID
FID]
18
Prec.
Rec.
VAE (B)
11.15
36.8
17.9
0.48
0.53
5.74
46.6
23.4
0,45
0.56
VAE (M)
9.26
34.6
18.2
0.49
0.55
4.63
44.7
23.8
0.47
0.58
VAE (L)
8.49
33.9
18.4
0,50
0.56
4.78
44.3
24.7
0.47
0.59
VAE (XL)
7.58
31.7
19.3
0.51
0,57
4.42
43.1
24.9
0.47
0.59
VAE (H)
7.12
30.9
19.8
0.52
0.57
4.29
41.6
25.9
0.48
0.59
e- VAE (B)
6.24
29.5
20.7
0.53
0.59
3.90
39.5
25.2
0.46
0.61
C- VAE (M)
5.42
27.6
21.2
0.55
0.59
2,79
35.4
26.2
0.51
0.62
~VAA (1.)
4.71
27.3
22.1
0.55
0.59
2.60
34.8
26.5
0.51
0.63
VAE (XL)
4.18
25.3
22.7
0.55
0.59
2.38
34.0
27.4
0.53
0.63
VAE (H)
4.04
24.9
23.0
0.56
0.60
2.31
33.2
27.5
0.54
0.64
resolution of each stage in the U-Net model, and incorpo- rates it into each residual block after group normalization operation (Wu & He, 2018). This is similar adaptive instance norm (Karras al., 2019) and FiLM (Perez al., 2018). We report the results in Tab. 7 (top), where we find that channel-wist concatenation and AdaGN obtain simi- lar reconstruction quality in terms of rFID. Because of the additional computational cost required by AdaGN, we thus apply channel -wiee concatenation in our model by default.
step At from (0,t). Then, we construct the real trajectory by computing Zi-ae via Eq. and concatenating it with æ;, while the fake trajectory is obtained in a similar way but using Eq. 9 instead. Tab. 7 (bottom) shows the comparison. We observe that matching trajectory æ; - 2o yields better performance than matching trajectory + It-At, con- firming the effectiveness of the proposed objective which is designed for the rectified flow formulation.
Comparisons with plain diffusion ADM. Under the same training setup of Tab. 5, directly trained plain diffu- sion model (ADM) for comparison, which resulted in rFID score of 38. 28.26. Its conditional form is already provided as bascline in Tab. 5, achieving 28 28.22. This demonstrates that our conditional form p( x4, 2) offers a better approx- imation of the true posterior q q( 2y) compared to the standard form p( lax). ). By further combining LPIPS and GAN loss, we achieve rFID of 8.24, outperforming its VAE counterpart, which achieves 11.15. With better training configurations, our final rFID improves to 6.24. This pro-
Trajectory matching. The proposed denoising trajec- tory matching objective matches the start-to-end trajec- tory + Zo by default. One alternative choice is to directly matching the distribution of 2 and zo without coupling on Σt. However, we find this formulation leads to unstable training and could not produce reasonable re- sults. Here, we present the results when matching the trajec- tory of Φ; - Tt-Ar, which is commonly used in previous work (Xiao al., 2022; Wang al., 2024a). Specifically, for each timestep / during training, we randomly sample
18

---

Epsilon- VAE: Denoising as Visual Decoding
GT
SD-VAE
E-VAE
GT
SD-VAE
E-VAE
A FURNACE ST.
TURRACE ST.
FURNACE ST.
CRAPO S7
CRAPO ST.
CRAPO ST
Figure? 7. Image reconstruction results under the SD-VA VA-VAE configuration (Rombach et al., 2022) at the resolution of 256 x 256. VAB produces significantly better visual details than SD-VAE when reconstructing local regions with complex textures or structures, such as human faces and small texts. Best viewed when zoomed-in in and in color.
19

---

Epsilon-VAR. VAE: Denoising as Visual Decoding
Table Benchmarking class-conditional image generation ImageNet 256 x 256. We use the DiT-XL/2 architecture (Esser al., 2024) for latent diffusion models and apply classifier free guidance (Ho & Salimans, 2022).
Downsample factor
Method
FID
SD-VAE (Rombuch al., 2022)
3,51
8 x 8
v-V--D (M)
2.83
VAE-SD (H)
2.69
gression, from plain diffusion ADM to e-VAE, underscores the significance of our proposals and their impact.
Model scaling. We investigate the impact of model scaling by comparing VAE and VAE across five model variants, all trained and evaluated at a resolution of 128 x 128, as summarized in Tab. 8. The results demonstrate that c- c-VAE consistently achieves significantly better rFID scores than VAF, with an average relative improvement of over 40%, and even the smallest e-VAE model outperforms VAE at largest scale. While the U-Net-based decoder of c-VAE has about twice as many parameters as standard decoder of VAE, grouping models by similar sizes, highlighted in different colors, shows that performance gains are not simply due to increased model parameters.
Tab. 9 presents the unconditional image generation results of VAE and e-VAE at resolutions of 128 128 and 256 x 256. In addition to FID, report Inception Score (IS) (Salimans et al., 2016) and Precision/Recall (Kynkäänniemi et al., 2019) as secondary metrics. The results show that c-V consistently outperforms VAE across all model scales. No- tably, e- e-VAE (B) surpasses VAE (H), consistent with our car- lier findings in Sec. 4.1. These results further demonstrate the effectiveness of e~VAE from the generation perspective.
Results with classifier-free guidance. We provide addi- tional results with classifier-free guidance (Ho & Salimans, 2022) under the 8 x 8 downsample factor in Tab. 10. We find that c- vAE (M) performs relatively 20% better than SD- VAE and further improvements are obtained after we scale up our model to cAVAE (H). These results are consistent with the results without classifier-free guidance in Tab, 4, confirming the effectiveness of our model.
E. Additional visual results
We provide additional visual comparisons between V-E and SD-VAE at the resolution of 256 256 (Fig. 7). Our observations indicate that e-VAE delivers significantly better visual quality than SD-VAE, particularly when reconstruct- ing local regions with complex textures structures, such as human faces and small text.
20


Reviews:
[{"review_id": "DpQ9Hibh2s", "reviewer": "Reviewer_NjT6", "summary": "This paper proposes to use a diffusion decoder in an autoencoder training for image generation. The autoencoder is trained with a diffusion loss, together with a LPIPS loss and a GAN loss defined on the one-step generation. The authors show that the proposed method outperforms prior state-of-the-art autoencoders for both reconstruction and image generation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive feedback. Below we provide a point-by-point response to your questions. Please let us know if you have any further questions.\n\n**Q1: The LPIPS and GAN loss are applied on the estimated one-step sample, which may not be accurate and may cause objective bias in theory. Although this can potentially be addressed with finetuning a diffusion decoder with frozen $z$.**\n\nWe acknowledge the reviewer's point regarding potential objective bias due to applying the LPIPS and GAN loss on the estimated one-step sample. However, we would like to emphasize that our Epsilon-VAE differs significantly from traditional diffusion models in that its diffusion decoder is conditioned on encoded latents. This conditioning provides a strong prior about the input image to reconstruct, resulting in a more accurate estimated one-step sample than in typical diffusion scenarios. Therefore, we believe the potential for objective bias is considerably reduced. We agree that finetuning the diffusion decoder with frozen $z$ is a promising avenue for further improvement and will explore this in future work.\n\n**Q2: Besides rFID, how much improvement it gets from the additional LPIPS and GAN losses for the actual visual quality in images?**\n\nWe find that the LPIPS loss enhances textural fidelity and structural coherence in generated images, while the GAN loss contributes to sharper high-frequency details. These observations align with their established roles in traditional VAEs. Detailed visual comparisons demonstrating these improvements will be included in the revised version.\n\n**Q3: The zoom-in boxes in Fig. 4 seem to be not very clear even for the GT.**\n\nThanks for pointing out this. This is because the compressed images were included in the original paper to ensure a reasonable file size. We will provide the original, uncompressed images in the revised version.", "claims_and_evidence": "The key claim of this paper is that a diffusion-loss based autoencoder (with LPIPS and GAN losses) can outperform the prior autoencoders while being efficient efficient. The authors evaluate rFID, PSNR, SSIM for reconstruction, and generation FID, the results show that the proposed method outperforms prior works. Visualization is also consistent with the claim.", "methods_and_evaluation_criteria": "Diffusion models are scalable probabilistic models. Using diffusion loss is thus a promising way to better model the probabilistic decoding and for learning image tokens. Evaluation metrics include rFID, PSNR, SSIM, and generation FID, which are standard in this field. Ideally human evaluation can be added, while it may not be necessary given the large amount of visual samples.", "theoretical_claims": "Theoretical proofs are not the focus of this work. The formulations are correct.", "experimental_designs_or_analyses": "The experiments are solid. Authors evaluate rFID, PSNR, SSIM, gFID and show visual samples for comparisons with prior methods for both reconstruction and generation. The number of parameters is also listed in supplementary material (Table 7) and shows a fair comparison. In ablation study, Table 4 shows the effect of different design choices which help understanding the importance of different components for the method.", "relation_to_broader_scientific_literature": "This work is one of the first works that show using the diffusion loss for the decoder can help training the autoencoders for reconstruction and generation on latent. The proposed method also serves as a novel autoencoder that achieves state-of-the-art quality while being efficient.", "essential_references_not_discussed": "I did not find key missing related works.", "other_strengths_and_weaknesses": "See prior sections for strengths. Using diffusion loss for autoencoder training is an important and promising direction to explore, and this work is one of the first in this direction that shows solid improvement on common benchmark ImageNet which are general images. Autoencoders are of key importance for generative models, this work proposes a novel autoencoder with state-of-the-art performance, therefore it is an important contribution to the field.\n\nMinor weakness: The LPIPS and GAN loss are applied on the estimated one-step sample, which may not be accurate and may cause objective bias in theory. Although this can potentially be addressed with finetuning a diffusion decoder with frozen z.", "other_comments_or_suggestions": "The zoom-in boxes in Fig. 4 seems to be not very clear even for the GT.", "questions_for_authors": "Besides rFID, how much improvement it gets from the additional LPIPS and GAN losses for the actual visual quliaty in images?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n5VqzGjp2H", "reviewer": "Reviewer_RVFr", "summary": "The author propose a new autoencoder paradigm, in which decoder is in the form of a diffusion model. The advantage of this is that it has better reconstruction quality compared with standard VAE. The proposed architecture is straight-forward, by directly upsampling encoded latents and then run diffusion at the input resolution to get the original signal backs. They introduce adversarial loss to encourage small reconstruction error from the diffusion decoder. In their main experiment, they use existing architecture, such as VQGAN encoder and discriminator for this purpose. They demonstrate epsilon-VAE achieve better reconstruction than other latent space VAE.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Below we provide a point-by-point response to your questions. Please let us know if you have any further questions.\n\n**Q1: Clarification on high-level insights.**\n\nThank you for raising this concern. Rather than opposing latent diffusion, we offer a complementary perspective by demonstrating that the diffusion process can also be applied to the outer structure of the latent diffusion model – the autoencoder part. Our goal is to revisit the standard autoencoder framework and integrate diffusion into the decoding step to enhance the entire model’s generative capacity. We empirically show that a latent diffusion DiT integrated with Epsilon-VAE significantly outperforms its SD-VAE counterpart, with the performance gap being significant in high compression regime.\n\n**Q2: Diffusion-based iterative decoding.**\n\nWhile replacing single-step decoding with an iterative process may seem counter-intuitive due to increased computational cost, the diffusion-based decoder addresses this concern in three key ways. First, it offers scalable inference, where even a single-step variant already outperforms a plain VAE decoder, and additional steps further enhance quality (please refer to Figure 3, left and right, and Table 7 in the original paper, where a single-step Epsilon-VAE-lite achieves 7.18 rFID and a VAE obtains 11.15 rFID in the same setting).  Second, it provides controllable trade-offs between computation and visual fidelity, allowing the number of steps to be adjusted at inference time based on application needs. Third, as shown in Section 4.2 of the main paper and our response to Q3 below, it enables training under higher compression ratios, which helps offset the added cost of iterative decoding by reducing the size of latent representations.\n\nAn additional advantage of scaling the autoencoder over the latent model lies in computational efficiency. Recent trends show latent diffusion models increasingly adopt Transformer architectures (e.g., DiT), where self-attention scales quadratically with input resolution. In contrast, our convolution-based UNet decoder offers more favorable linear scaling. *As models grow, shifting complexity to the autoencoder helps reduce the burden on the latent model, leading to a more efficient overall system.*\n\n**Q3: A benchmark on computation at various decoding resolutions.**\n\nIn response to the reviewer's valuable point regarding computational cost at various decoding resolutions, we conducted experiments with Epsilon-VAE using a 16 x 16 downsampling factor. Our analysis reveals that: (1) the dominant factor in the overall memory footprint during the generation process is the LDM; (2) increasing the decoding resolution leads to a more substantial increase in the memory requirements of the LDM compared to the Epsilon-VAE decoder itself; (3) Epsilon-VAE outperforms SD-VAE under the same 8 x 8 downsampling factor at the 256 x 256 resolution, but with slightly worse throughput and memory efficiency; and (4) importantly, by employing a 16 x 16 downsampling factor instead of 8 x 8, Epsilon-VAE demonstrates a significant 2.3x inference speedup and a 3.3x reduction in total memory cost compared to SD-VAE, while maintaining comparable image generation performance (highlighted in bold). This demonstrates the efficiency gains achieved with our proposed approach in mitigating the potential for undesirable decoding memory scaling.\n\n| Method | Resolution | Downsample factor | Throughput | LDM memory (GB) | Decoder memory (GB) | FID w/o CFG |\n| -------- | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |\n| **SD-VAE** | **256 x 256** | **8 x 8** | **522** | **78.8** | **1.7** | **11.63** |\n| Epsilon-VAE (M) | 256 x 256 | 8 x 8 | 491 | 78.8 | 2.8 | 9.39 |\n| Epsilon-VAE (M) | 128 x 128 | 16 x 16 | 3910 | 5.9  | 1.6 | 11.14 |\n| **Epsilon-VAE (M)** | **256 x 256** | **16 x 16** | **1192** | **20.7** | **3.7** | **10.68** |\n| Epsilon-VAE (M) | 512 x 512 | 16 x 16 | 240 | 82.6 | 6.9 | 9.20 |", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Method-wise, I am not convinced about the epsilon-VAE idea. At a high-level, they are proposing a direction opposite to latent diffusion. Instead of doing diffusion at latent space (which was first proposed in Stable Diffusion for budge reduction), it does that instead in the original input space. It is not suprising that they can achieve better reconstruction performance, (similar argument is already proven by Imagen and latter work) but the computation is much higher. Besides, it is counter-intuitive to replace a single step decoding with a iterative process, which is more computationally heavy. \n\nEvaluation criteria makes sense.", "theoretical_claims": "no theoretical claims.", "experimental_designs_or_analyses": "The experiment designs make sense. However, I would argue a benchmark on computation at various decoding resolution is necessary. The memory could be quadratically increased with the input size, which is not desirable.", "relation_to_broader_scientific_literature": "Many generative models (e.g., image, video, and other LMM) use latent diffusion style to train. The proposed framework could speed up the training by combining the autoencoding and generation step together.", "essential_references_not_discussed": "They have good coverage.", "other_strengths_and_weaknesses": "Strength:\n1. Overall the authors did a great job for experiment design which has a thorough coverage of their design choices and potential use cases, e.g., image-conditioned diffusion training. \n\n2. The proposed framework is generic, which works with existing off-the-shelf encoder network and diffusion architecutres. \n\nWeakness:\n1. As is stated above, my major concern is on its high-level insights. They are proposing an interesting direction opposite to latent diffusion. Instead of doing diffusion at latent space (which was first proposed in Stable Diffusion for budge reduction), it does that instead in the original input space. It is not suprising that they can achieve better reconstruction performance, (similar argument is already proven by Imagen and latter work) but the computation is much higher. Experiment evidences wise, it does not justify why such a design choice is worthy to pursuit in my opinion. Besides, it is counter-intuitive to replace a single step decoding with a iterative process, which is more computationally heavy.", "other_comments_or_suggestions": "none", "questions_for_authors": "none.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "guaAHpiXou", "reviewer": "Reviewer_FhpZ", "summary": "In this work, the authors propose using denoising diffusion model as the decoder in autoencoder for image reconstruction and generation. $\\epsilon$-VAE develops denoising decoder conditioned on the learnable latents. The work includes solid experiments in validating the design choices for image reconstruction. Further DiT-based latent diffusion models on $\\epsilon$-VAE latents achieves comparable performance as popular SD-VAE.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. Below we provide a point-by-point response to all of your questions. Please let us know if you have any further questions.\n\n**Q1: The reported performance in Table 3 is worse than numbers reported in the original DiT paper.**\n\nWe did the following modifications to the original DiT training recipe as mentioned in Section C of the original paper, leading to a slight drop in performance: (1) we reimplement the pipeline with JAX for training models on TPUs; (2) for simplicity and training stability, we remove the variational lower bound loss term; (3) we reduce the number of training steps to 1M to conserve compute.\n\n**Q2: It would greatly help validate the contribution of the work by showing how ϵ-VAE works with CFG.**\n\nWe provided results with CFG under the 8 x 8 downsample factor in the table below, where we find that Epsilon-VAE (M) performs relatively 20% better than SD-VAE and further improvements are obtained after we scale up our model to Epsilon-VAE (H). These results are consistent with the results without CFG, confirming the effectiveness of our model. We will provide more detailed results of other models and under different downsample factors in the revised version.\n\n| Method | FID w/ CFG |\n| -------- | :-------: |\n| SD-VAE | 3.51 |\n| Epsilon-VAE (M) | 2.83 |\n| Epsilon-VAE (H) | 2.69 |\n\n**Q3: In latent diffusion, is the DiT setting optimized for Epsilon-VAE or is it also benefitting standard DiT with SD-VAE?**\n\nOur training protocol for latent diffusion was consistently applied across all evaluated VAE models. The three minor modifications to the original DiT training recipe, as outlined in our response to Q1, were implemented solely to conserve computational resources. Our current setting does not favor either our Epsilon-VAE or the standard SD-VAE, but it ensures  a fair basis for comparison.\n\n**Q4: Can authors provide more details of how inference is conducted on higher resolutions with models trained at 128 x 128 images?**\n\nInference on higher resolutions works out-of-the-box. Since our model is a fully convolutional UNet, it can directly process images with resolutions exceeding the 128 x 128 training size without any modifications to the input.", "claims_and_evidence": "The authors claim superior performance on image reconstruction which is well supported by the experimental results. But I find the generation performance of $\\epsilon$-VAE latents not fully examined. In particular, in Table 3, the authors shows better FID than DiT with SD-VAE under the diffusion setting reported in this work. However, the reported performance is worse than numbers reported in the original DiT paper. Also, the authors only report results without CFG. Whereas CFG is broadly applied to diffusion models and should be compared to fully validate the performance of proposed $\\epsilon$-VAE.", "methods_and_evaluation_criteria": "Yes, evaluation and benchmark make sense.", "theoretical_claims": "Theoretical claims in the work are valid.", "experimental_designs_or_analyses": "Experimental designs and analysis are valid.", "relation_to_broader_scientific_literature": "The paper is very related to the previous latent diffusion models which relies on autoencoder to acquire latent space. The work follows the pipeline and aims at improving the autoencoder design.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. My main question is the performance of $\\epsilon$-VAE on image generation as mentioned in \"Claims And Evidence\". It would greatly help validate the contribution of the work by showing how $\\epsilon$-VAE works with CFG. \n2. In latent diffusion, DiT with $\\epsilon$-VAE shows better performance than SD-VAE under the setting of this work. Is the setting optimized for $\\epsilon$-VAE or is it also benefitting standard DiT with SD-VAE?\n3. Can authors provide more details of how inference is conducted on higher resolutions with models trained at 128 x 128 images?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c6BjOdxUj9", "reviewer": "Reviewer_3ACU", "summary": "- The paper proposes an alternative autoencoder that itself uses a (conditional) diffusion / rectified flow model as decoder replacing the standard VAE for latent diffusion models (LDMs).\n- For that, the paper experimentally explores the design space in terms of decoder / denoiser architecture, the implementation of the conditioning, the training objective as a combination of different loss functions, the sampling of noise levels during training, and the time schedule for sampling at test time. Within this exploration, the authors propose \n  - a specific combination of loss terms including a novel adversarial trajectory matching loss replacing the patch-wise GAN loss in the usual VAE training for LDMs,\n  - a reversed logarithmic mapping for the timesteps during sampling at test time to have denser steps early in the inference process.\n- The experimental evaluation shows advantages in reconstruction over other VAE baselines, which also translate to advantages in generation FID when combinated with a LDM. The approach enables a 4x stronger spatial compression with comparable FID than the well established StableDiffusion VAE + LDM pipeline, which results in a 2.3x increase of throughput (image/sec).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. We will rephrase our claims and reduce the redundancy as suggested, and include suggested related work in the revision. Below we provide a point-by-point response to all of your questions. Please let us know if you have any further questions.\n\n**Q1: Because of the denoising procedure with 3 network forward evaluations used in the paper, the decoding is more expensive than with a standard VAE, which the paper misses to openly discuss. This limits its application in cases with real-time visualization, e.g., during the generation process of images.**\n\nThank you for pointing out this limitation, and we will include a detailed discussion of it in the revised version. However, we would like to note that our denoising process demonstrates promising results even with a single iteration (please refer to Figure 3, left and right, in the original paper). Consequently, this allows our model to be adapted for scenarios with latency-sensitive requirements, such as real-time visualization during image generation as mentioned, by reducing the decoding step to a single pass.\n\n**Q2: How would the pipeline of eps-VAE and LDM benefit from even higher downsampling factors? Could the efficiency of the diffusion model be improved (both in terms of training and sampling) without hurting the generation quality significantly?**\n\nTo address the questions regarding higher downsampling factors, we present additional results for a 32 x 32 downsampling factor in the table below, comparing them to our 16 x 16 results. Notably, Epsilon-VAE achieves a 25% improvement in generation quality over SD-VAE at the 32 x 32 factor, alongside a 3.2x inference speedup than SD-VAE at the 16 x 16 factor with comparable FID (highlighted in bold). We observed similar training speedups for latent diffusion models utilizing Epsilon-VAE at this higher downsampling rate. These gains are more pronounced than those observed when increasing the downsampling factor from 8 x 8 to 16 x 16 (where we observe 22% quality improvement and 2.3x inference speedup, respectively, as shown in the original paper). These findings strongly suggest that the benefits of the Epsilon-VAE and LDM pipeline are amplified with higher downsampling factors. We will include detailed results in the revised paper.\n\n| Method | Downsample factor | Throughput | rFID | FID w/o CFG |\n| -------- | :-------: | :-------: | :-------: | :-------: |\n| **SD-VAE** | **16 x 16** | **1220** | **2.93** | **14.59** |\n| SD-VAE | 32 x 32 | 3991 | 9.33 | 21.31 |\n| Epsilon-VAE (M) | 16 x 16 | 1192 | 1.91 | 10.68 |\n| **Epsilon-VAE (M)** | **32 x 32** | **3865** | **3.80** | **15.98** |\n\n**Q3: Why does the reconstruction performance degrade when increasing the number of steps starting from 3?**\n\nTo enable large step sizes for the reverse process during inference, we introduced the denoising trajectory matching loss to implicitly model the conditional distribution $p(x_0|x_t)$, shifting the denoising distributions from traditional Gaussian to non-Gaussian multimodal forms (please refer to [1*] for a detailed discussion on this). However, the assumptions underpinning this approach are most effective when the total number of denoising steps remains small. Consequently, there appears to be an optimal range or \"sweet spot\" for the number of total inference steps. We will elaborate on this phenomenon and provide further clarification in the revised version.\n\n[1*] Tackling the Generative Learning Trilemma with Denoising Diffusion GANs, ICLR 2022.\n\n**Q4: The qualitative comparisons in the paper and the appendix are not very convincing showing only very minor differences in high-frequency details.**\n\nThis is because in the original paper, we show compressed high-resolution images with Epsilon-VAEs using low downsampling factors. We will include additional uncompressed visual results under more changeling settings (e.g., 128 x 128 images with 16 or 32 downsampling factors) in the revised version to highlight our advantages.\n\n**Q5: The proposed reversed logarithm mapping for the timesteps during sampling lacks intuition and would benefit from a visualization.**\n\nAs detailed in our response to Q3, the denoising trajectory matching loss results in denoising distributions that deviate from the standard Gaussian. This deviation suggests that the conventional uniform sampling approach may no longer be optimal. Hence, we empirically investigated alternative sampling strategies and found the reversed logarithm mapping to yield the best performance. We will clarify the intuition and provide additional visualizations as suggested in the revision.", "claims_and_evidence": "Most claims are supported by convincing evidence except for:\n- The authors claim that the approach \"enhances downstream generation quality by 22% and provides 2.3x inference speedup\" (lines 32 ff., left column), which can be misunderstood as the experimental evaluation only shows that either the first or the second can be achieved but not both at the same time, by changing the downsampling factor of the autoencoder. It should be made clear that it achieves 22% better generation quality for the same downsampling factor or 2.3x inference speedup with comparable FID by increasing the downsampling factor.\n- The authors \"anticipate two key improvements\" with one being \"more effective generation of latent representations, allowing the downstream latent diffusion model to learn more efficiently\" (lines 142 ff., left column). The paper only provides evidence for improved generation quality, but it remains unclear whether this is because of a higher level of detail achieved by the denoising-based VAE decoder or whether the different latent space also enables more efficient training of the LDM, e.g., in terms of faster convergence.", "methods_and_evaluation_criteria": "Both methods and evaluation criteria make sense for the problem at hand:\n- The paper extensively explores and ablates the design choices for their approach with all of them contributing to the final performance.\n- The evaluation uses ImageNet in different resolutions and COCO as well as (r)FID, PSNR, and SSIM, which are established benchmark datasets and metrics for image reconstruction and generation.", "theoretical_claims": "There are no theoretical claims that require proofs.", "experimental_designs_or_analyses": "All experimental designs and analyses seem to be valid.", "relation_to_broader_scientific_literature": "The paper is using a lot of ideas from recent improvements of denoising generative models like the rectified flow schedule with velocity parameterization, the VAE encoder architecture from StableDiffusion / Latent Diffusion Models, a similar combination of loss functions for training with reconstruction, LPIPS, and adversarial loss terms (but adjusted for denoising), denoiser architectures from ADM and DiT, a logit-normal distribution for timestep sampling during training from StableDiffusion 3, but it explores the best configuration for a different setting being: Lightweight generative image compression instead of training the second-stage LDM, e.g, for text-to-image generation.\nFor this particular task, certain design questions have to be answered differently showed by the advantage of the UNet-based ADM architecture over a Transformer-based DiT, for example.\nThere is a relevant related work [6] that proposes a similar two(/ three with VAE)-stage approach with two diffusion models but at a higher level of (semantic) compression, still using the standard VAE for the first image compression.\n\nThere has not been that much prior work focusing on improving the VAE / tokenizer part of latent diffusion or autoregressive approaches, but a lot of concurrent work [1, 2, 3, 4]. Prior works mostly used existing pre-trained VAEs from StableDiffusion or trained a similar architecture with different hyperparameters and downsampling factors like MAR [5] for example.\n\n- [1] Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models. ICLR 2025\n- [2] Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models. arxiv 2025\n- [3] Exploring Representation-Aligned Latent Space for Better Generation. arxiv 2025\n- [4] FlexTok: Resampling Images into 1D Token Sequences of Flexible Length. arxiv 2025\n- [5] Autoregressive Image Generation without Vector Quantization. NeurIPS 2024\n- [6] Würstchen: An efficient architecture for large-scale text-to-image diffusion models. ICLR 2024", "essential_references_not_discussed": "I am not aware of any essential missing references. Many related works are discussed in the appendix, but not in the main paper. It might make sense to briefly describe the most relevant ones in the main paper, e.g., [1].\nA later version of the paper could discuss the concurrent work mentioned in the above review section.\n\n[1] Würstchen: An efficient architecture for large-scale text-to-image diffusion models. ICLR 2024", "other_strengths_and_weaknesses": "Strengths:\n- The paper is well-written and easy to understand.\n- The experimental results are convincing.\n  - The proposed approach achieves improvements in reconstruction, which are shown to translate to improvements in generation when combined with a LDM.\n  - It enables a larger downsample factor than the standard VAE while providing comparable (even slightly better) generation quality.\n  - The extensive ablation study (table 4) validates the effectiveness of all design choices.\n- The paper introduces additional technical contributions like the adversarial denoising trajectory matching and the reversed logarithm time spacing during inference.\n\nWeaknesses:\n- Because of the denoising procedure with 3 network forward evaluations used in the paper, the decoding is more expensive than with a standard VAE, which the paper misses to openly discuss. This limits its application in cases with real-time visualization, e.g., during the generation process of images.\n- The approach seems to be limited in terms of downsampling factors and leveraging the full potential of the denoising generative paradigm:\n  - The main paper misses to explore even larger compression factors in combination with training a LDM, which could be interesting to see whether even more efficient training of the expensive diffusion model is possible.\n  - For both uniform or logarithmic spacing of timesteps for sampling, the reconstruction FID degrades for more than 3 denoising steps which is counterintuitive (see figure 3).\n- The paper is partially a bit repetitive:\n  - Velocity prediction (lines 145 ff., right column) and eq. (9) already discussed before together with rectified flows (lines 102 ff., right column) and eq. (6).\n  - Noise scheduling (lines 199 ff., right column) is related work that also has been partially addressed already in the related work section.\n  - Model configurations (lines 238 ff., right column) and decoder architecture (lines 312 ff., left column) overlap in content.\n- The qualitative comparisons in the paper and the appendix are not very convincing showing only very minor differences in high-frequency details.\n- The proposed reversed logarithm mapping for the timesteps during sampling lacks intuition and would benefit from a visualization.", "other_comments_or_suggestions": "No other comments or suggestions", "questions_for_authors": "1. How would the pipeline of eps-VAE and LDM benefit from even higher downsampling factors? \n  - Could the efficiency of the diffusion model be improved (both in terms of training and sampling) without hurting the generation quality significantly?\n2. Why does the reconstruction performance degrade when increasing the number of steps starting from 3?\n\nConvincing responses to these questions could alleviate my concerns regarding the limitations of the proposed approach.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a new autoencoder paradigm where the decoder is a diffusion model. The paper shows that this setup achieves better reconstruction quality than standard VAEs. The adversarial loss introduced reconstructs a small error from the diffusion encoder. The paper is deemed interesting for proposing a diffusion-based decoder in a traditional VAE setup. The experiments are solid and extensive.

The main concern that remains seems to stem from the paper exploring the diffusion process in the decoding step, which is more expensive than other options. Due to the nature of research, I don't consider the slower and more expensive method a problem since it explores new avenues. The other concerns raised by the reviewers were addressed in the rebuttal. While there were some minor concerns regarding the lack of human evaluation and potential biases in the loss measures, the majority of the reviewers agreed that the paper is interesting.

Given the positive evaluation, I recommend the paper for acceptance.
Decision: Accept (poster)