Improving the Diffusability of Autoencoders
Authors: Ivan Skorokhodov
Sharath Girish
Benran Hu
Willi Menapace
Yanyu Li
Rameen Abdal
Sergey Tulyakov
Aliaksandr Siarohin
Venue: ICML 2025 poster (2025)
Keywords: autoencoders
latent diffusion
image generation
video generation
DCT
diffusability
fourier transform
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We explore the spectral properties 📊 of modern autoencoders 🤖 used for image 🖼️ / video 📹 latent diffusion 🌬️ training 🏋️ and find 🔍 that a simple downsampling ⬇️ regularization 📏 can substantially boost 🚀 their downstream 🌊 LDM performance 📈💥.

Abstract: Latent diffusion models have emerged as the leading approach for generating high-quality images and videos, utilizing compressed latent representations to reduce the computational burden of the diffusion process. While recent advancements have primarily focused on scaling diffusion backbones and improving autoencoder reconstruction quality, the interaction between these components has received comparatively less attention. In this work, we perform a spectral analysis of modern autoencoders and identify inordinate high-frequency components in their latent spaces, which are especially pronounced in the autoencoders with a large bottleneck channel size. We hypothesize that this high-frequency component interferes with the coarse-to-fine nature of the diffusion synthesis process and hinders the generation quality. To mitigate the issue, we propose scale equivariance: a simple regularization strategy that aligns latent and RGB spaces across frequencies by enforcing scale equivariance in the decoder. It requires minimal code changes and only up to $20$K autoencoder fine-tuning steps, yet significantly improves generation quality, reducing FID by 19% for image generation on ImageNet-1K 256x256 and FVD by at least 44% for video generation on Kinetics-700 17x256x256. The source code is available at https://github.com/snap-research/diffusability.

Improving the Diffusability of Autoencoders
Ivan Skorokhodov
Sharath Girish Sergey Tulyakov
Benran Hu"
Willi Menapace' Aliaksandr Siarohin
Yanyu Li'
Rameen Abdal
Abstract
35
DIT-XL/2 + FluxAE (vanilla)
1200
Latent diffusion models have emerged as the lead- ing approach for generating high-quality images and videos, utilizing compressed latent repre- sentations to reduce the computational burden of the diffusion process. While recent advance- ments have primarily focused scaling diffu- sion backbones and improving autoencoder recon- struction quality, the interaction between these components has received comparatively less atten- tion. In this work, we perform a spectral analysis of modern autoencoders and identify inordinate high-frequency components in their latent spaces, which are especially pronounced in the autoen- coders with large bottleneck channel size. We hypothesize that this high-frequency component interferes with the coarse-to- tosse-tt-finss fine nature of the diffusion synthesis process and hinders the gener- ation quality. To mitigate the issue, we propose seale equivariance: a simple regularization strat- egy that aligns latent and RGB spaces across fre- quencies by enforeing scale equivariance in the de- coder. It requires minimal code changes and only up to 20K autoencoder fine-tuning steps, yet sig- nificantly improves generation quality, reducing FID by 19% for image generation on ImageNet- IK 2562 and FVD by at least 44% for video gen- eration on Kinetics-700 17 2562. The source code is available at htttps://githee com/ shap-reeearcch/lfffuaal
DIT-XL/2 FluxAE SE
DIT-XL/2 Cog VidooX AB (vanilla)
1000
DIT-XL/2 VogVidexX AE + SE
25
20
200
10
50000 100000 150000 200800 250000 300000 350000 4050000
Iteration
Figure I. Convergence speed of DIT-XL/2 on top of vanilla FluxAE vs FluxAE fine-tuned for 10K steps with scale equiv- ariance (SE) regularization on 1 and on top of Cog VideoX-AR AE vs CogVideoX-AE with SE on Kinetics- 17 256 Our regularization improves the performance of image and video LDMs by refinng the frequency profile of their autoen- coders' latent spaces.
of diffusion models impractical. Latent diffusion models (LDMs) (Vahdat et al., 2021; Rombach et al., 2022) have become the main approach in mitigating this issue, demon- strating remarkable success in generating high-resolution images (Black Forest Labs, 2023; Betker et al., 2023; Esser et al., 2024) and videos (Brooks et al., 2024; Yang et al., 2024; Kong et al., 2024). A typical LDM consists of two main components: an autoencoder and a diffusion backbone. Most recent breakthroughs have been driven by scaling up diffusion backbones (Peebles Xie, 2022), while autoen- coders (AEs) have received comparatively less attention.
Recently, the research community has begun focusing more improving the autoencoders, recognizing their crucial impact on overall performance, but most effort has been con- centrated on enhancing reconstruction quality (Black Forest Labs, 2023; Hong et al., 2022; Agarwal et al., 2025; HaCo- hen et al., 2024; Chen et al., 2025) and achieving higher compression ratios (Agarwal et al., 2025; HaCohen et al., 2024; Chen et al., 2025) to accelerate the diffusion process. However, we argue that critical yet under-explored aspect, which we refer to as diffusability! also plays a key role in
1. Introduction
In recent years, diffusion models (DMs) have emerged as the dominant generative modeling paradigm in computer vision. However, the high dimensionality of visual data poses a significant challenge, making the direct application
Snap Inc. CCamegie Mellon University- Correspondence to: Ivan Skorokhodov <iskorokhodov @ gmail.com²>, Aliaksandr Siarohin filiaksandr.siarohin gmail.com>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Diffusabiilty describes how easily distribution can be mod- eled by diffuusio: high diffusability indicates that the distribution is easy to fit, while low diffusability implies a higher difficulty.

---

Improving the Diffusability of Autocncoders
determining the utility of autoencoders. Indeed, all three quality, compression efficiency, and diffusability-aarr essential for the practical effectiveness of LDMs. Specifically, inaccurate reconstruction sets an upper bound on generation fidelity, low compression efficiency leads to slow and costly generation, and poor diffusability necessitates the use of heavier, more expensive, and sophis- ticated diffusion backbones, further limiting LDM quality.
hen et al., 2024), consistently demonstrating improved LDM performance ImageNet-1K (Deng al., 2009) 256², re- ducing FID by 19% for DiT-XL, and Kinetics-700 (Carreira et al., 2019) 17 x 256 reducing FVD by at least 44%.
2. Related Work
Diffusion models. Diffusion models (Vahdat et al., 2021; Rombach et al., 2022; Song et al., 2021; Ho et al., 2020; Nichol & Dhariwal, 2021; Karras et al., 2022) have emerged as the dominant framework for generative modeling, sur- passing traditional approaches like GANs (Goodfellow et al., 2014; Karras et al., 2019) and VAES (Kingma & Welling, 2013). Diffusion models express generation as denoising process producing the generated content by progressively denoising an initial noise sample. Owing their efficiency and scalability, foundational generative models (Saharia et al., 2022; Ho et al., 2022; Yang et al., 2024; Podell et al., 2023; Blattmann et al., 2023; Polyak et al., 2024) have made significant strides in synthesizing visually stunning and se- mantically aligned images and videos.
Diffusion models possess a unique property of being coarse- to-fine in nature (Dieleman, 2024; Ning et al., 2024): in the denoising process, they synthesize low-frequency signal components first and add them later. It is a beneficial trait since it allows to defer error accumulation to higher frequency parts of the spectrum, which aligns well with how humans perceive quality: we sensitive to the image structure and composition, but oblivious of its fine-grained textural details. However, when applying the diffusion process in the latent spaces of pre- trained autoencoders, the correspondence between latent low-frequue wow-freueeccy components and their RGB counterparts may be lost, hindering the spectral autoregression property.
ones on top of
Initially applied to low-resolution visual content in the pixel space (Vahdat al., 2021; Ho et al., 2020; Nichol & Dhariwal, 2021; Karras et al., 2022), they have soon been extended to higher resolutions. In Latent Diffusion Models (LDMs) (Vahdat et al., 2021; Rombach et al., 2022) high-resolution visual content is modeled in the com- pact latent space produced by a variational autoencoder (VAE) (Kingma & Welling, 2013) within a two-stage frame- work. Latent Flow Models (LFMs) (Dao et al., 2023; Liu et al., 2024), follow the same approach but leverage Recti- fied Flows (RFs) to enable faster and more stable sampling,
In this work, we identify correlation between the spectral properties of the latent space and its diffusability. We an- alyze the spectral characteristics of latent representations across several widely used image and video autoencoders. Our investigation reveals a prominent high-frequency com- ponent in these latent spaces, deviating significantly from the spectral distribution of RGB signals. This component be- comes even more pronounced as the channel size increases, which the recent autoencoders use to improve reconstruction. We hypothesize that the flat spectral distribution induced by the strong fighffrequency component harms the spectral au- toregression property. Moreover, we demonstrate that these high-frequency components substantially influence the final RGB result, and their inaccurate modeling can introduce noticeable visual artifacts. Finally, we show that standard KL regularization is insufficient to address spectrum defects and, in some cases, may even amplify the issue.
Recent work attributes the success of diffusion models to a form of implicit spectral autoregression (Rissanen et al., 2023; Ning et al., 2024) implied by the progressive removal of noise during sampling, resulting in the generation of vi- sual content in a coarse-to-fine manner. Such result holds in the pixel-space of natural images, based on its pattern of decreasing spectral power (Ruderman, 1997). We show that popular autoencoders (Black Forest Labs, 2023; Agar- wal et al., 2025; HaCohen et al., 2024; Yang et al., 2024) have a less pronounced pattern of decreasing spectral power, inhibiting implicit spectral autoregressive generation. Build- ing on this observation, this work proposes a regularization scheme that re-establishes this property, consistently show- ing improved LDM performance and avoiding the need for explicit coarse-to- fine generation.
To mitigate these spurious tigh-frequency components in latent representations, we propose a simple and effective regularization strategy. Our approach involves aligning the latent space and the RGB space at different frequencies. This is achieved by enforcing scale equivariance in the de- coder ensuring that downsampled latents correspond to downsampled RGB representations. Our method requires minimal modifications and few additional autoencoder fine- tuning steps, yet significantly enhances diffusability across various architectures, ultimately improving the quality of generated samples. We validate our approach on both im- age and video autoencoders, including FluxAE (Black For- est Labs, 2023), Cosmos Tokenzze (Agarwal et al., 2025), Cog VideoX-AE AE (Hong et al., 2022), and LTX-AE (HaCo-
Image and video autoencoders. Due to the success of LDMs, a lot of effort has been devoted to the development of better AEs. Image LDMs (Rombach et al., 2022) and early video diffusion models (Blattmann et al., 2023; Guo et al., 2023) employ spatial AE with a compression ra-
2

---

Improving the Diffusability of Autocncoders
tio of 1x8x8. The rapid advancement of video diffusion models poses the demand for 3D AES that jointly compress spatial and temporal dimensions to further improve effi- ciency (Hong et al., 2022; Zhou et al., 2024; Kong et al.,
1.0
RGB
FluxAE 24
FluXAE d. 4
FluxAE d 32
FluxAE d.
-FlxXA 48
Normalized Amplitude
FluxAE d. = 16
FluxAE d. = 64
2024). Among them, Open-Sora (Zheng et al., 2024) in- herits the 1x8x8 spatial AE and trains a decoupled 4x temporal AE on top of its latent space, while others tend to build a hierarchical spatio-temporal AE with 3D causal convolutions (Xing al., 2024; Wu al., 2024; Chen et al., 2024a; Zhao et al., 2024; Sadat et al., 2024; Mansen-Esttuchh et al., 2025). To accelerate the model and improve the reconstruction, Open-Sora-Plae (Lin et al., 2024) and Cos- mos mosTokenizerr (Agarwal et al., 2025) propose to employ the wavelet transform of the input. Another popular trend to further increase the compression ratio to reduce the number of tokens in the latent space (Xie al., 2024; Tian et al., 2024; HaCohen et al., 2024), thus enabling more ef- ficient denoising process. In addition to the continuous AEs explored in this work, multiple discrete AEs (Wang et al., 2024; Tang et al., 2024; Agarwal et al., 2025) are proposed to aid autoregressive tasks. Esteves et al. (2024) leverages a wavelet transform to produce latents corresponding to different frequency components.
0.6
0.4
0.2
0
10
20
30
40
50
60
Zigzag Frrequency Index
Figure 2. Latent frequency profiles of FluxAE autoencoders of varying bottleneck sizes, and also RGB (of the same 32 spatial dimension). One can notice two things: 1) the latent space of an autoencoder exhibits different power profile from RGB; and 2) high frequency amplitudes increase with the latent channel size.
high-frequency components become more pronounced. Ad- ditionally, we demonstrate that the widely adopted KL regu- larization only increases the strength of these components. Finally, Section 3.3 presents a straightforward method to improve the diffusability of a latent space of an autoencoder by enhancing its spectral properties.
AEs for compression. Many works train neural-basee AEs for image compression (Ballé et al., 2016; 2018; Minnen et al., 2018; Cheng et 2020), typically with 16x 16 down- sampled latents which are discrete and entropy constrained. Video compression AEs involve autoregressive AEs (Li Sheng et al., 2022) with explicit frame- wise formulations that utilize motion vectors implicit modeling (Mentzer al., 2022). These approaches tar- get high-quality reconstruction with low bitrates and adopt complex designs for learnable entropy models. They typi- cally employ a larger number of latent bottleneck channels (96-192), which is not generally suited for the generation task, thus we do not consider them in this work.
3.1. Background: Blockwise 2D DCT
et al., 2021; 2023;
The discrete cosine transform (DCT) (Ahmed et al., 2006) over a 2D signal is transformation converting the signal's representation between the spatial and frequency domains. DCT, in particular, represents the original input signal as coefficients for a set of horizontal and vertical cosine basis oscillating with different frequencies. More formally, given a 2D signal block A E RBx whose values Pay denote the pixel intensity at position (z. the two-dimensional type-I1 DCT yields a frequency-domain block D E R BBxB where Dwo captures the coefficient for the corresponding horizontal and vertical cosine bases:
Concurrent works. Independently from us, AF- LDM (Zhou et al., 2025) enforces shift equivariance in both the autoencoder and I.M,, and EQ-VAE (Kouzelis et al., 2025) proposes scale/shift equivariance regularization for autoencoders, but with a different motivation of improving the models' equivariance to spatial transformations.
B-1B-1
Duv = =(()a(v))
Σ
Σ
Pay fzz uf,,,,
-= w=
u = 0,
where alu) = 122BB
3. Improving Diffusability
+ 0,
We begin this section by discussing the spectral decompo- sition of 2D signals and providing some background on discrete cosine transform in Section 3. In Section 3.2, we analyze the spectral properties of latent spaces across dif- ferent autoencoders and compare them to those of the RGB space. Our main insight is that the frequency profile of the latent space includes large-magnitude high-frequency com- ponents. We also show that as the channel size increases, the
f(z, u) = co8
(2x- 1) 1uar
2B
In practice, split the input 2D signal into non-overlapping blocks of size B x B and treat each channel independently.
By analyzing RGB images and latents in the DCT frequency domain, we produce a frequency profile that relates to the energy of the signal at every frequency. A zigzag frequency index is used map each DCT block D E R B into a
3

---

Improving the Diffusability of Autocncoders
1.0
FluxAE; KL B =
FluxAE; KL. f 10
10a
RGB
FluxAE+ FT-SE 0.1
FluxAE; KL B 10 FluxAE, KL. $ 10
Flux AE (vanilla)
FluxAE FT-SE a 0.25
Normalized Amplicude
0.8
FluxAE; KI. 8 10
FluxAE; KI. 8 10-2
Normalized Amplitude
FluxAB+ + FT-SE a 0.01
FluxAE FI-SE 0.05
FluxAE FT-SE
FluxAB + FT-SE a 0.5
FluxAE; KI. 10-5
FluxAE, KL B 10
0.6-
10-1
0.4
0.2
10-
30
50
60
10
30 Zigzng Frequency Index
Zigrag Frequency Index
Figure 3. Spectrums for FluxAE autoencoders trained (from scratch) with different KL regularization strengths. KL regular- ization is double-edged sword: it pushes the latents distribution closer to standard Gaussian (the distribution the reverse diffusion process starts with), that LDM has less work do (Vahdat et al., 2021), but it also introduces figh-frequency components into the latents due the random noise addition (see Figure 13), which LDM is forced to model as well. Figure 13 shows the influence of noise addition on the frequency profile.
Figure 4. DCT Spectrum of the FluxAE latents with and without scale equivariance (SE) regularization. Fine-tuning AEs with SE brings the spectrum closer to the RGB domain, the higher the regularization strength.
to the RGB profile. (ii) As the number of channels in the autoencoder's bottleneck increases, high-frequency ponents become more pronounced. We hypothesize that larger bottleneck allows the autoencoder to capture finer, high-frequency details. Initially, limited capacity prioritizes smoother, www-frequency information. But as the capac- ity expands, the model encodes additional high- frehfrreuencyy content, distributing it unevenly and in the unstructured manner across channels. This finding is of particular inter- est the number of channels is positively correlated with autoencoder's reconstruction quality.
one-dimensional sequence following the standard zigzag ordering as in JPEG (Wallace, 1991), which indexes the DCT coefficients from lowest frequency Do.o to highest one DB-1,B-1- Formally, let zigzag( (u, v) {O...., B² 1} denote the ranks of the coefficients Duo in the ascending frequency order. Given block, we compute its DCT and produce the normalized amplitudes for each frequency com-
A common way to regularize the latent space for latent diffu- sion models (LDMs) is to employ a variational autoencoder (VAE) (Kingma & Welling, 2013) framework with KL divergence term, encouraging the latent distribution to align with the standard Gaussian prior. Since the reverse diffusion process also starts with the same standard normal distribu- tion, such KL regularization is argued to simplify the job for the diffusion model (Vahdat et al., 2021), since it now has less "work" to do. However, as we show in Figure which compares FluxAE with varying levels KL regu- larization, higher KL regularization introduces more high frequencies due to the underlying noise addition process, creating harmful side-effect, reducing diffusability.
ponent (u, 2) as:
Duv Do.u
Aue
We define the frequency profile as the sequence of normal- ized amplitudes in the standard zigzag order.
When analyzing the frequency profiles of videos (or latent codes with an additional time dimension), we still rely on per-frame 2D DCT since the temporal and spatial domains possess different spectral properties.
3.2. Spectral Analysis of the Latent Space
Previous work (Ning et al., 2024; Dieleman, 2024; Rissanen et al., 2023) interprets diffusion models as autoregressive ones, but in the spectral domain: when noise level is high, low frequencies are generated, then, as the level of noise lowers during sampling, progressively higher frequencies are generated. This is an attractive property since allows the model leverage the cleaner lower frequencies conditioning signal for the current prediction. However, the strength of this autoregressive pattern is directly related to the shape of the frequency profile for the signal to generate. Since the white noise that applied as part of the diffusion process has a flat frequency profile, it follows that the flatter
We begin our analysis by observing the frequency profile of the latent space in the Flux (Black Forest Labs, 2023) family of autoencoders to establish a relationship with dif- fusability, For the purpose of this study, we train a family of FluxAE models with various channel sizes for 100k steps (where performance saturates in this setting) and, for each of them, compute the averaged frequency profile over 256 samples, all channels, and all DCT blocks. Figure 2 presents the frequency profiles of both Flux autoencoders and RGB space, from which we observe: (i) The Flux profile exhibits significantly larger high- freeurrey components compared

---

Improving the Diffusability of Autocncoders
easily learn alter its latent frequency profile to encode the inputs in the low-frequency region of the spectrum without sacrificing the reconstruction quality much (see Figure 5), While this regularization, which we name Chopping High Frequencies (CHF), improves the spectrum, we develop a much simpler procedure to achieve the same effect without the need to perform the error-prone DCT transform (the details of CHF are described in Appendix C). The simplest way to achieve high-frequency truncation is through direct downsampling, which we discuss next.
RGB
FluxAB
FluxAE + CHF
Downsampling involves resizing both the input æ and the latent representation 2 by a fixed scale, yielding è and E, re- spectively. This process effectively removes a portion of the high-frequency components from both the RGB and latent signals. In practice, we use x2 bilinear downsampling for all the experiments. Regularization is then enforced by ensuring that 7 and the decoder's ss reconstruction of the downsampled latent Dec(ž) remain consistent through an additional reconstruction loss. The autoencoder is trained using the following objective:
0%
25%
50%
75%
Figure 5. RGB and autoencoder reconstructions with progressively erased DCT gehhffrequenyy components. RGB faces minimal degradation (top), as a higher percentage of the latent DCT spec- trum is removed, but the Flux AE reconstructions (middle) quickly degrade when the high-ffequency components from the latents are heing removed. autoencoder to rely more on the low frequency region of the latents and leads to better compression and resilience to high- geeefrryy error accumulation in diffusion models.
nigh-frequeney cutoff regularization forces the
((x) = d(x,, Dec(=)) - (ec()) + 3LKL-
(2)
Here, d(:;) :) represents a distance measure for reconstruction which we instantiate as mean squared error loss and percep- tual losses (Zhang et al., 2018) following prior work (Rom- bach et al., 2022), a is the regularization strength (we use a = 0.25 for the main experiments). The term LXL is VAE's (Kingma & Welling, 2013) KL regularization, if applicable (we do not use it when we train with our regu- larization). This regularization effectively enforces scale equivariance in the decoder, which is the basis for its name.
the frequency profile of the signal, the lower the cleanliness of low frequencies that can act as conditioning for the model. For a flat frequency profile, no autoregressive generation is possible as all frequencies would be crased at the same speed by white noise. We also hypothesize that higher frequencies components are harder to model than lower frequency components for the following reasons and thus should be avoided: (i) they have higher dimensionality"; (ii) they are generated only in the final steps of sampling, thus must emerge more rapidly; (iii) they are more susceptible to error accumulation over time (Li & van der Schaar, 2024).
In Figure 4, illustrate the effect of scale equivariance on the spectrum of FluxAE. Our proposed regularization effec- tively reduces the high-frequene components of the signal, bringing it closer to the spectral characteristics of the RGB space. This successfully achieves objective (i) for effec- tive regularization. Meanwhile, Figure 8 demonstrates that scale equivariance preserves more content compared the baseline, as more and more fregurryy components suppressed. thereby fulfilling objective (ii). Finally, Figure 6 visualizes intermediate steps in the diffusion trajectory. The regularized model exhibits a noticeably smoother and more structured progression, following a healthier coarse-to-fine -ine generation process.
Motivated by this analysis, we propose scale equivariance regularization for the autoencoder's latent space.
3.3. Scale Equivariance Regularization
Effective regularization should achieve two key objectives: (i) to suppress high-frequency components in the latent space and (ii) to prevent the decoder from amplifying these components, as their impact on the final result is what ulti- mately matters. This can be accomplished by aligning the spectral properties of the latent and RGB spaces at differ- ent frequencies. A way to achieve this consists in explicitly chopping off a portion of the high frequencies in both spaces and training the decoder to reconstruct the truncated RGB signal from the truncated latent representation. Our prelim- inary experiments demonstrated that an autoencoder can
4. Experiments
Data. We trained all the autoencoders on wn-the-wild data which do not overlap with ImageNet-IK (Deng et al., 2009) or Kinetics-700 (Carreira et al., 2019) to make sure that there is no data leak in the autoencoders, and that they remain pereral- For this, we used our internal image and video datasets of the 256 resolution, which
F.g.. going from 256² to 512² resolution adds little structural content, but increases the dimensionality 3 times.
5

---

Improving the Diffusability of Autocncoders
Training details. All the LDM models are trained for 400k steps with 10k warmup steps following the rectified flow dif- fusion parametrization (Albergo & Vanden-Eijnden, 2022; Dao et al., 2023; Esser et al., 2024). Following Esser et al. (2024), we use a logit-normal training noise distribution. We use either 2 x 2 or 1 x 1 patchification in DiT (Peebles & Xie, 2022) to match the compute between DiT' trained on top autoencoders with different compression ratios. Our video DiT is direct adaption of the image one where we additionally unroll the temporal axis, following the recent works on video diffusion models (Yang et al., 2024). We do not use patchification for the temporal axis in video DiTs. In contrast to prior work (e.g., Rombach et al. (2022)), we average the KL. loss across the latent channels and resolu- tions: this has no theoretical impact, but allows compare autoencoders with different bottleneck sizes.
Figure 6. Denoising trajectories (steps 1, 16, 32, 128 and 256 out of 256) for DiT-XL trained with FluxAE (top) and FluXAE+SE. DiT-XL with vanilla FluxAE exhibits prominent high-frequent artifacts early on in the trajectory.
Table/ 1. Quantitative performance on ImageNet (Deng et al., 2009)
without guidance. The original DiT scores are provided for refer-
ence from Table 4 of (Peebles& & Xie, 2022).
Stage II
Stage
FID
FDD 4
FluxAE (vanilla)
25.41
536.2
Inference details. We run DiT inference with 256 steps without classifier-free guidance (Ho & Salimans, 2022) for quantitative evaluations since different models are too sensi- tive to it and should be tuned separately (Karras et al., 2024; Kynkäänniemi et al., 2024).
DIT-B/2
+
30.51
575.4
FluxAR+ FT-SE (ours)
18.06
450.6
FluxAE (vanilla)
12.42
306.72 73
DIT-L/2
+ FT
14.48
333.54
FluxAE + FT-SE (ours)
9.61
236.43
FluxAE (vanilla)
12.21
282.8
4.1. Improving Existing Autoencoders
+ FT
10.62
262.2
DIT-XL/2
FluxAE+ + FT-SE (ours)
9.85
235.8
We apply our training pipeline on top of 3 different autoen- coders. For each autoencoder, we trained it while freez- ing the last output layers avoid breaking their adversar- ial fine-tuning, which should have no impact on the latent space (Chen et al., 2025). We emphasize that none of the ex- plored modern autoencoders publicly released their training pipelines. For the pretrained snapshots of autoencoders, we used the original snapshots available in the di ffuuerss library (von Platen et al., 2022).
+IM steps
3.27
85.86
CMS-AE, (vanilla)
11.69
360.83
DiT-B/I
CMS-AE; + FT
13.59
375.19
CMS-AE, + FT-SB (ours)
11.85
354.22
D/T-B/2 (orig)
43.47
DiT-L/2 (orig)
23.33
SD- SD-VAE-ft-MEE
DiT-XLJ2 (orig)
19.47
+ 7M steps (orig)
12.03
similar in distribution of concepts and aesthetics to the publicly available in-the-wild datasets like COYO (Byeon et al., 2022) and Panda- 70M (Chen et al., 2024b). To control for the impact of the data (and also the training recipe), we trained a separate autoencoder baseline for each setup without using our proposed regularization. In several cases, just fine-tuning on such in-the-wild data already yields better diffusion performance (e.g., see DiT-XL results in Table 1).
Improving image autoencoders. For the image autoen- coder, we used FluxAE (Black Forest Labs, 2023) with 8 x 8 compression ratio and 16 latent channels (since it is the most popular modern autoencoder in the commu- nity) and CMS-AE, (Agarwal et al., 2025) with 16 x 16 compression ratio and 16 channels as a high-compression autoencoder. For all the experiments (unless stated other- wise), we fine-tuned it for just 10,000 training steps with batch size of 32 (320,000 total seen images) using 2x and 4x downsampling ratios, chosen randomly during forward pass of the regularization loss. DiT training on top of the unchanged Flux Autoencoder is labeled as "vanilla", Autoencoders fine-tuned for 10,000 steps with our proposed SE regularization is denoted via the "+ FT-SE" suffix. To control for the fine-tuning data and training pipeline, we fine- tuned each autoencoder without adding our regularization as an additional loss (denoted via "+ FT"".
Evaluation. We evaluate image DiT models via FID and FDD (Frechet Distance computed on top of DI- NOv2 (Oquab et al., 2023) features), where the latter was shown to be a more reliable metric (Stein et al., 2024; Karras et al., 2024). We evaluate video DiT models with FVD (Unterthiner et al., 2018), FID, and FDD, except for ablations where we rely on 5,000 samples. For image models, we use 50,000 samples without any optimization for class balancing. To evaluate autoencoders, we used PSNR, SSIM, LPIPS and FID metrics computed on 512 samples from ImageNet and Kinetics-700 for image and video autoencoders, respectively.
For the LDM benchmark, we utilized ImageNet (Deng et al., 2009) at 256 x 256 resolution. We used the DiT (Peebles & Xie, 2022) model as the backbone since it is the most
6

---

Improving the Diffusability of Autocncoders
Figure 7. Uncurated samples on ImageNet 256 256 from DiT-XL trained on top of FluxAE (top) DIT-XL with "FluXÁE FT-SE" (bottom). 256 steps with the guidance scale of 3.0. More visualizations are in Appendix D.
Table Results on Kinetics-708 (Carreira et al., 2019) for DiT
Table 3. Ablating KL. regularization weight 8 for FluxAE fine- tuning in terms of the reconstruction quality and downstream DiT- S/2 and DIT-L/2 (Peebles & Xie, 2022) training. Increasing the KL in general improves the LDM's performance for smaller models, but at the expense of worsened AE reconstruction (and reduced training stability). which can limit scalability (also observed by (Esser et al., 2024)) and results in worse performance of DiT- L/2. Our SE regularization lcads to improved LDM performance
trained on top of various autoencoders. See Section 4.1 for details.
Stage
Stage I
FVDjoK
FDD
FIDrox
CV-AE (vanilla)
650.40
650.97
28.85
DIT-B/2
CV-AE + FT
447.26
593.02
19.45
CV-AE FT-SE (ours)
252.26
466.15 15
12.19
CV-AE (vanilla)
268. 26
407.23
12.02
DIT-XL/2
CV-AE FT
270.66
402.91
12.78
CV-AE FT-SE (ours)
135.15
245.27
8.59
without hurting the reconstruction and scales well to larger models.
LTX-AE (vanilla)
854.47
814.49
50.99
Method
DIT-S/2 FDDsx
DIT-L/2 FDDsk
AE PSNRs12
DIT-B/I
LTX-AE FT
876.61
823.71
50.18
LTX-AE FT-SE (ours)
389.56
642.80
22.88
FlxXAE (vanilla)
992.05
415.87
30.20
KL =
968.26
472.08
29.97
3 = 10
1018.6
425.35
30.29
3 = 10-6
1095.2
612.12
19.66
popular modern latent diffusion backbone. Compared to the original paper, incorporated several recent advance- ments into the DiT architecture to improve the baseline performance, as described in Appendix B. Qualitative sam- ples from DiT-XL/2 are provided in Figure 7. The results are shown in Table 1. One can observe that our proposed regularization greatly improves the diffusabilitt of the down- stream LDM model, allowing to achieve 19% lower FID compared the vanilla Flux AE and 8% lower FID com- pared to the Flux AE, fine-tuned in our training pipeline without the SE regularization.
KI. -10-
940.13
403.99
29.21
8 = -10
974.67
404.61
30.22
-10-3
982.91
425.24
29.51
KL 10-2
1946.5
1737.47
10.82
KL 10-1
929.58
472.74
23.72
FT-SE (ours)
924.28
369.15
30.37
Similar to the image autoencoder experiments, we train a DiT model Kinetics-700 (Carreira et al., 2019) three variants: 1) a "vanilla" autoencoder snapshot; 2) the vanilla autoencoder fine-tuned for 20,000 steps in our training pipeline (denoted as "FFT"); and 3) the autoencoder snapshot, fine-tuned with our downsampling regularization (denoted as "FT-SE"), For LTX-AE, we used a reduced patchification resolution of 1 x to compensate for its extreme compres- sion ratio. The results are presented in Table 2: DIT model top SE-regularized autoencoders has drastically better performance: 44% and 54% lower FVD for CV-AE and LTX-AE, respectively. Our training pipeline allowed to achieve better DiT-B training for CV-AE (650.4 vs 447.3 FVDIok), but led to worse scores for LTX-AE (854.4 vs 876.6), which we found less stable to train. Adding our reg- ularization strategy greatly improves the LDM performance in each case. One can also observe that the boost for video autoencoders is larger than in the image domain (at least - -44% reduced FVD1oK for DiT-B for video generation vs - 7% reduced FID for DiT-XL for image generation). We attribute this to two factors. First, improvements in Frechet Distances (Heusel et al., 2017) do not scale linearly (i.e.,
The improvement for CMS-AE, is reduced with the main reason being that our training pipeline hurts its performance (we explored over 10 different hyperparameters setups to tune the vanilla model): after fine-tuning it for 10,000 steps with only reconstruction losses (it does not use KL regu- larization by default), the downstream FID performance increases by 14% from 11.69 to 13.59.
Improving video autoencoders. For video autoencoders, we used Cog Vi (Hong et al., 2022) (CV-AB) with 4 x 8 x 8 compression and 16 latent channels and LTX- AE (HaCohen et al., 2024) with 8 32 32 compression and 32 latent channels. The latter serves as a strong high- compression autoencoder baseline. All the video AEs are fine-tuned for 20,000 training steps on the joint image and video dataset with the batch size of 32. Image batches are treated as single-frama videos which is possible due to the causal structure of the video autoencoders (Yu et al., 2023).
7

---

Improving the Diffusability of Autocncoders
30.0
FlaxAE (vanilla)
0.9
PlatA (vanilla)
0.5
FlaxAE (vanillo)
27.5
FlaxAF FT
FlaxAE + FTSSE
0.8
FlaxAE FT-SE
F +
0.4
200
FlxxAF
FlaxAE + FT-SE
PSNR
0.7
FDDz2
25.0
0.3
150
0.6
22.5
0.2
PRRAE (vanilla)
0.5
PlxxAE FT
100
20.0
0.1
FlaxAE PT-SE
0.0
0.2
0.4
0.6
0.0
0.2
01.4
0.6
0.0
0.2
0.4
0.6
0.0
01.2
01.4
0.6
DCT Cut Ratio
DCT Cut Ratio
DCT Cut Ratio
DCT Cut Ratio
Figure 8. Effect of DCT spectrum cutting, We plot reconstruction metrics on ImageNet for the baseline FluxAE and its finetuned version, with and without scale equivariance. As more high-freq frequency DCT coefficients are removed from the latents, the AE with regularization consistently achieves the best reconstruction quality.
four reconstruction metries - PSNR, SSIM, LPIPS (Zhang et al., 2018), and FID, on 50,000 samples from ImageNet and Kinetics for image and video autoencoders, respectively. Reconstruction quality remains similar across the models.
30.75
FluzAE FT-SE
0.065
FluxAl: (vanilla)
20.50
PSNR
30.25
0.060
30.00
FlaxAE FT-SE
0.055
29.75
Can LDM performance be improved by tweaking the KL weight instead? In Table 3, we show that increasing the KL strength can indeed improve the LDM performance for DiT-S/2, but it inevitably hurts reconstruction, as shown by the PSNRs, which bottlenecks the scalability of larger LDM models. In contrast, our proposed scale equivariance allows to achieve good LDM performance without hurting the reconstruction quality of the autoencoder. For these ablations, we trained the DīT-S/2 variants for 200K training steps and DiT-L/2 variants for 400K steps, and ran inference for 80 steps without classifier-free guidance. One can see that for small compute budget, increasing the KL strength is beneficial: the best LDM score is obtained with the high- est KL 3 0.1. But it severely affected the reconstruction quality of the autoencoder, which limited its scaling: the cor- responding DÍT-L/2 LDM variant ranked among the worst. At the same time, our developed regularization performs well for all DiT variants and does limit scaling.
Flus AF (vanille)
29.50
0.050
10 Regulatization Weight
T0
10
Regulatization Weight
Figure 9. Reconstruction quality with varying SE regularization strength for line-tuning. We find the value of 0.25 be optimal in maintaining reconstruction quality compared to the Base while also improving generation quality shown in Tables and 2.
Table Influence of spectrum regularization on the reconstruction quality for image and vidco autoencoders, Image autoencoders are evaluated on 50,000 images from ImageNet-1 ones are on 50,000 videos from Kinetics-700.
IK K, while the video
Method
PSNR
SSIM f
LPIPS
FID
FVDxx
FluxAE
30.243
0.883
0.054
0.183
+TT-SE (ours)
30.474
0.888
0.055
0.550
CMS-AE,
23.230
0.638
0.181
1.077
+FT-SE (ours)
24.558
0.677
0.166
1,570
CogVideoX-AE
34.961
0.947
0.073
2,992
4.614
+FT-SE (ours)
35.399
0.948
0.067
2,986
3.328
LTX-AE
30,897
0,886
0.152
5.928
36.783
Can the quality improvement come from implicit time shifting? SE smoothes the latent space. which can result in "implicit time shifting" (Gao et al., 2024): it eliminates gigh-rreueency components from the latents, allowing the model to spend more compute on low-frequency ones, es- pecially at inference time (see B). We trained a family of DÍT-B/2 models for various location and scale coeffi- cients for logit-normal noise distribution (Esser et al., 2024). Figure 10 presents the FDDsk results with 128 inference steps (and without CFG) for time shifting coefficients of [0.1, 0.25,, 0.5, 0.75, 2, 3, 4, 5, 6, 8, 10). Our regularized FluxAF-FT+SE achieves the best results across all the se- tups, confirming that our improved quality is not due to improved diffusability, rather than implicit time shifting, Note that our original noise schedule was tuned for the vanilla setup, which is why the default location of 0 and scale perform the best for non-regularized AEs.
+FT-SE (ours)
30,386
0.885
0.137
5.303
34.148
their smaller values are progressively harder to improve). Next, causal video autoencoders have less regular latent structure: the first frame in a video is encoded into the same representation size as the subsequent chunks, which leaves more room to enhance the diffusability of the latent space.
We additionally trained DiT-XL/2 model for the CV-AE family to explore the scalability of our regularization. For this large-scale setup, our SE regularization improved the FVDiox score by almost twice.
4.2. Ablations
Does scale equivariance hurt reconstruction? scale equiv- ariance in VAES improves downstream generation quality in terms of FID (Tables 1 and 2). We now examine its impact on AE reconstruction quality. Table 4 presents results across
Effect of SE regularization strength. In Figure 9, we
8

---

Improving the Diffusability of Autocncoders
crepancies between latent and RGB spectral properties and demonstrated that they lead worse LDM synthesis quality. Building on this insight, we developed a regularization strat- egy that aligns the latent and RGB spaces across different frequencies. Our approach maintains reconstruction fidelity and improves diffusion training by suppressing spurious high- high-frequency details in the latent code. Potential future di- rections include exploring more advanced frequency- -based regularizations, adaptive compression methods and scale equivariance regularization in the temporal axis for video autoencoders to further optimize the trade-off between re- construction quality, compression rate, and diffusability.
750
Location: -0.50
Scale: 1.0
Location: -0.25
Scale: 1.0
Location: 0.00
Scale: 1.0 (vanilla)
700
Location: 0.00
Scale: 1.0 (FT.SE)
Location:
Scale:
FDDox
650
Location:
Scale: 1.5
Location: 0.25
Seale: 1.0
600-
Location: 0.50
Scale: 1.0
Location: 1.00
Scale: 1.0
550
500
0
10
Time Shifting Coefficient
Impact Statement
Figure 10. Inference time time shifting and train-time noise sched- ule ablation for DiT-B/2 models trained on top of vanilla FluxAE, and also on top of our regularized autoencoder. Our regularized BluxAE-FT+SS achieves the best results across all the setups, confirming that our improved quality is not due to improved diffus- ability, rather than implicit time shifting. The optimal time shifting coefficient is marked with a star.
This work focuses on improving the representations in au- toencoders that serve as backbone for latent diffusion training, ultimately enhancing generative performance. Our improvements can facilitate beneficial applications such as boosting creativity, supporting educational content creation, and reducing computational overhead in generative work- flows. Beyond these considerations, we do not identify addi- tional ethical or societal implications beyond those already known accompany large-scale generative modeling.
show the effect of varying the loss weight on the FluxAE performance on ImageNet. Increasing the SE regularization strength naturally worsens the total reconstruction quality since the decoder trained to generalize its performance across both low frequency and high frequency latents (via downsampling) while having the same capacity. We choose the value of 0.25 to maintain reconstruction performance compared to the base AE model while improving the gener- ation quality as shown in Tables and 2.
References
Agarwal, N., Ali, A., Bala, M., Balaji, Y., Barker, E.,, Cai, T., Chattopadhyay, P., Chen, Y., Cui, Y., Ding, Y., et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.0355 2025.
Effect of DCT spectrum cutting. To examine the impact of downsampling regularization, we evaluate reconstruc- tion quality by progressively removing high-frequency DCT components from the latents. Figure 8 presents reconstruc- tion metrics on 512 samples from the ImageNet validation set for the baseline FluxAE and fine-tuning, with and with- out regularization. As the cut ratio increases on the x-axis, indicating the removal of more high fighffreuuenc components, the AE with regularized latents consistently achieves the best reconstruction quality across all metrics. This under- scores the regularization role in aligning the spectral proper- ties of the latent and RGB spaces.
Ahmed, N., Natarajan, T., and Rao, K. R. Discrete cosine transform. IEEE transactions on Computers, 100(1):90- 93, 2006.
Albergo. M. S. and Vanden-Eüjnden, E. Building normal- izing flows with stochastic interpolants. arXiv preprint arXiv:2209. 15571, 2022.
Ballé, J., Laparra, V., and Simoncelli, E. P. End-to- end optimized image compression. arXiv preprint 1rXiv:1611.0111 2016.
Ballé, J., Minnen, D., Singh, S., Hwang, S. J., and Johnston.
N. Variational image compression with scale hyperprior. arXiv preprint arXiv:1802.0111
5. Conclusion
2018.
We have shown that modern Latent Diffusion Models (LDMs) rely just as critically on their autoencoders as on the more frequently investigated diffusion architectures. While prior work has largely focused improving reconstruc- tion quality and compression rates for the autoencoders, our study focuses on diffusability, revealing how latent spec- tra with excessive high- fieeueryy components can hamper the downstream diffusion process. Through a systematic analysis of several autoencoders, we uncovered stark dis-
Betker, J., Goh, G., Jing, L., Brookst, T., Wang, J., Li, L., Ouyang, L.,, Zhuang, J., Lee, J., Guot, Y., Manassra, W., Dhariwal, P., Chu, C., Jiaot, Y., and Ramesh, A. Improving image generation with better captions. atttps:/ccle http://dd....p......// com/ haanrapper//////// dall-e-3. poff 2023. Accessed: 2023-11-14.
Black Forest Labs. Flux. https: / com/ black-forest laatttllas/ffe flux, 2023.
9

---

Improving the Diffusability of Autocncoders
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z.., Voleti, V., Letts, A., al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311. 15117,, 2023.
Dieleman, S. Diffusion is spectral autoregression, 2024. URL https:://anndes att..aii//22/// /
mhectral-autoreggeee autoregression. html.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, et al. Scaling rectified flow transformers for high-rcsolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng, C., Wang, R., and Ramesh, A. Video generation models as world simulators. 2024. URL. https://openai: www.rreaarrcc/////oo///.o..a.cae wideo-generation-moon
Esteves, C., Suhail, M., and Makadia, A. Spectral image worlswoolddsss swarrdddsimulatete www.mmmtttooo.mommm rookeiizr... arXiv preprint arXiv:2412.09007, 2024.
Byeon, M., Park, В., Kim, H., Lee, S., Baek, W., and Kim, S. Coyo-700m: Image-text pair dataset. petps://githine com/kakaobrain/ doyo-dataset. 2022.
Gao, P., Zhuo, L.,, Liu, D., Du, R., Luo, X., Qiu, I.., Zhang, Y., Lin, C., Huang, R., Geng, S., et al. Lumina-t22 12min Transforming text into any modality, resolution, and du- ration via flow-based large diffusion transformers. arXiv preprint arXiv:2405.0594. 2024.
Carreira, J., Noland, E., Hillier, C., and Zisserman, A. A short note on the kinetics 700 human action dataset. arXiv preprint XXKV:1907.0096 2019.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S.. Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), 2014.
Chen, J., Cai, H., Chen, J., Xie, E., Yang, S., Tang, H., Li, M., Lu, Y., and Han, S. Deep compression autoen- coder for efficient high- highuroeslution diffusion models. arXiv preprint ar 11v:24400.10733, 2024a.
Guo, Y., Yang. C., Rao, A., Wang, Y., Qiao, Y., Lin, D., and Dai, B. Animatediff: Animate your personalized text-to- text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.0477 2023.
Chen, J., Cai, H., Chen, J., Xie, E., Yang, S., Tang, H., Li, M., Lu, Y., and Han, S. Deep compression autoen- coder for efficient high-resolution diffusion models. In The Thirteenth International Conference Learning Representationss, 2025, URL https: http:p:://ppeeeee //tpp//ppppee...ee///epp...e..er net
Gupta, A., Yu, L., Sohn, K., Gu, X., Hahn, M., Li, F.- F., Essa, I., Jiang, L., and Lezama, J. Photorealistic video generation with diffusion models. In European Conference on Computer Vision, pp. 393-411. Springer, 2024.
Chen, T. and Li, L. Fit: Far-reaching interleaved transform- ers. arXiv preprini arXiv:2305. 12689, 2023.
HaCohen, Y., Chiprut, N., Brazowski, B., Shalem, D., Moshe, D.,, Richardson, E., Levin, E., Shiran, G., Zabari, N., Gordon, O., et al. Ltx-video: Realtime video latent diffusion. arXiv preprint arXiv:2501.00103 2024.
Chen, T.SS., Siarohin, A., Menapace, W., Deyneka, E., Chao, H.-w., Jeon, B. E., Fang. Y., Lee, H.-Y., Ren, J., Yang, M.- H. and Tulyakov, S. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024b.
Hansen Cansen-Estruch. P., Yan, D., Chung, C.-Y., Zohar, O., Wang. J., Hou, T., Xu, T., Vishwanath, S., Vajda, P., and Chen, X. Learnings from scaling visual tokenizers for reconstruc- tion and generation. arXiv preprint arXiv:2501.09755 2025.
Cheng, Z., Sun, H., Takeuchi, M., and Katto, J. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7939-7948, 2020.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Dao. Q. Phung, H., Nguyen, B., and Tran, A. Flow match- ing in latent space. arXiv preprint arXiv:2307.08690 2023.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.1 125.2 2022.
Deng. J., Dong, W., Socher, L.- Li, and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 2009.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
10

---

Improving the Diffusability of Autocncoders
Ho, J., Chan, W., Saharia, C., Whang, Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:22100023003, 2022.
Li, J., Li, B., and Lu, Y. Neural video compression with diverse contexts. In Proceedings of the IEEF/CVE Con- ference Computer Vision and Pattern Recognition, pp. 2023.
Li, Y. and van der Schaar, M. On error propagation of diffusion models. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: / opwnrevieww.o. wwt.:/iwwww.eeeeee.......e..ee www.roommmmtdtttiiiioooee
Hong, W., Ding, M., Zheng, W. Liu, X., and Tang, J. Cogvideo: Large-scale pretraining for text-to-video gener- gener- ation via transformers. arXiv preprint arXiv:2205.1588 2022.
Lin, B., Ge, Y., Cheng, X., Li, Z., Zhu, B., Wang, S., He, X., Ye, Y., Yuan, S., Chen, L.., et al. Open-sora plan: Open-source large video generation model. arXiv preprint arXiv:2412.00113, 2024.
Jabri, A., Fleet, D. J., and Chen, Scalable adaptive computation for iterative generation. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, JMLR.org. 2023.
Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 03003, 2022.
Karras, T., Laine, S., and Aila, T. A style-based genera- architecture for generative adversarial networks. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Liu, X., Zhang, X., Ma, J., Peng, J., and Liu, Q. Instaflow: One step is enough for high-quality diffusion- text- to-image generation, 2024. URL https: /arxiv.. 230.0/6300.000
Karras, T., Aittala, M., Aila, T., and Laine. S. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35: 16565-26577, 2022.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv: 17X1v:1711.05101, 2017.
Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T., and Laine, S. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24174-24184, 2024.
Menapace, W., Siarohin, A., Skorokhodov, I., Deyneka, E., Chen, T.S., Kag, A., Fang. Y., Stoliar, A., Ricci, E., Ren, J., et al. Snap video: Scaled spatiotemporal transformers for text-to-video synthesis. arXiv preprint ANNv:2402.14797,, 2024.
Kingma, and Gao, R. Understanding the diffusion objective weighted integral of elbos. arXiv preprint arXiv:2303.000 00048 2023.
Mentzer, F., Toderici, G., Minnen, D., Hwang, S.-J., Caelles, S., Lucie, M., and Agustsson, E. Vet: A video com- pression transformer. arXiv preprint arXiv:2206.07307, 2022.
Kingma, D. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
Mescheder, I.., Geiger, A., and Nowozin, S. Which training methods for gans actually converge? International conference on machine learning. 2018.
Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong, J., Li, X., Wu, B., Zhang, J., et al. Hunyuan- video: A systematic framework for large vidco generative models. arXiv preprint arXiv:24/2.03600 2024.
Minnen, D., Ballé, J., and Toderici, G. D. Joint autoregres- sive and hierarchical priors for learned image compres- sion. Advances in neural information processing systems, 31, 2018.
Kouzelis, T., Ioannis, K., Spyros, G., and Nikos, K. Eq- vae: Equivariance regularized latent space for improved generative image modeling. In arxiv, 2025.
Nichol, A. Q. and Dhariwal, Improved denois- ing diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171, 2021. www.httss://prceeeeeeee 1139/nichol2la.ho
Kynkäääniemm, T., Aittala, M., Karras, T., Laine, S., Aila, T., and Lehtinen, J. Applying guidance in a limited interval improves sample and distribution quality in diffusion models. arXiv preprint arXiv:2404.07724 2024.
mlr .tt..reess//////mmmmmmmmmm..m.
Ning, M., Li, M., Su, J., Jia, H., Liu, L., Beneš, M., Salah, A. A., and Ertugrul, I. O. Detdiff: Intriguing properties image generative modeling in the det space. arXiv preprint arXiv:24/2.1500 2024.
Li, Li, B., and Lu, Y. Deep contextual video compression. Advances in Neural Information Processing Systems, 34: 18114 14112 2021.

---

Improving the Diffusability of Autocncoders
Oquab, M., Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Howes, R., Huang, P.Y., Xu, H., Sharma, V., Li, S.-W., Galuba, W., Rabbat, M., As- sran, M., Ballas, N., Synnaeve, G., Misra, I,, Jegou, H.,
Song. Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative model- ing through stochastic differential equations, 2021. URL https: /ttp:///axiivvv/ii http::/aaxxv.or/// wtwp:xii.orr/bb////1110 13456.
Mairal, J., Labatut, P., Joulin, A., and Bojanowski. Dinov2: Learning robust visual features without supervi- sion, 2023.
Stein, G., Cresswell, J., Hosseinzadeh, R., Sui, Y., Ross, B., Villecroze, V., Liu, Z., Caterini, A. L., Taylor, E., and Loaiza-Ganem, G. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 2024.
Tang, A., He, T., Guo, J., Cheng, X., Song, L., and Bian, J. Vidtok: A versatile and open-source video tokenizer. arXiv preprint arXiv:24122 13061, 2024.
Polyak, A., Zohar, A., Brown, A., Tjandra, A., Sinha, A., Lee, A., Vyas, A., Shi, B., Ma, C.-Y., Chuang, C.-Y., et al. Movie gen: A cast of media foundation models. arXiv preprint urXiv:2410.13720 2024.
Tian, R., Dai, Q., Bao, J., Qiu, K., Yang, Y., Luo, C., Wu, Z., and Jiang, Y.-G. Reducio! generating 1024x 1024 vidco within 16 seconds using extremely compressed motion latents. arXiv preprint 0rXiv:2411.13553 2024.
Rissanen, S., Heinonen, M., and Solin, A. Generative mod- elling with inverse heat dissipation. In The Eleventh International Conference on Learning Representations, 2023. URL https:t//////..... httpps///perrreveeee... forum? id=4PJUBT9F201..
Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. Towards accurate generative models of video: A new metric & challenges. arXiv preprint
2018.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffiusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Vahdat, A., Kreis, K., and Kautz, J. Score-based gener- ative modeling latent space. 2021. arXiv preprint urXiv:2106.00 05931, 2021.
von Platen, P., Patil, S., Lozhkov, A., Cuenca, P., Lam- bert, N., Rasul, K., Davaadorj. M., Nair, D., Paul, S., Berman, W., Xu, Y., Liu, S., and Wolf, T. Diffusers: State-of-the-art diffusion models. petps://giihhee com con/hugginggface /iiicee/iiii difffsers.r. 2022.
Ruderman, L. Origins of scaling in natural images. Vision Research, 37(23):3385-333 3398, 1997. ISSN 0042-6989. doi: http::/ddiinn//000000066s4444929
URL celeeneeaarrccll//a/////222/////
Wallace, G. K. The jpeg still picture compression standard. Communications of the ACM, 34(4):30 44, 1991.
Sadat, S., Buhmann, J., Bradley, D., Hilliges, O., and Weber, R. M. Litevae: Lightweight and efficient variational autoencoders for latent diffusion models. arXiv preprint arXiv:2405. 14477, 2024.
Wan, T., Wang, A., Ai, B., Wen, B., Mao, C., Xie, C.-W., Chen, D., Yu, F., Zhao, H., Yang, J., et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.203114 2025.
Saharia, C., Chan, W. Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif- fusion models with deep language understanding. Ad- vances in neural information processing systems, 35: 36479-36494, 2022.
Wang. J., Jiang, Y., Yuan, Z., Peng. B., Wu, Z., and Jiang, Y.-G. Omnitokenizer: A joint image-video tokenizer for visual generation. arXiv preprint arXiv:2406.0994 049999999 2024.
Sheng, X., Li, Li, Li,, D., and Lu, Y. Temporal context mining for learned video compression. IEEE Transactions Multimedia, 25:7311-7322, 2022.
Wu, P., Zhu, Liu, Y., Zhao, L., Zhai, W., Cao, Y., and Zha, Z.-J. Improved video vae for latent video diffusion model. arXiv preprint M.Xi::4411.06444 2024.
12

---

Improving the Diffusability of Autocncoders
Xie, E., Chen, J., Chen, J., Cai, H., Tang, H., Lin, Y., Zhang, Z., Li, M., Zhu, L.,, Lu, Y., et al. Sana: Efficient high- resolution image synthesis with linear diffusion trans- formers, arXiv preprint arXiv:2410. 10629, 2024.
Xing, Y., Fei, Y., He, Y., Chen, J., Xie, J., Chi, X., and Chen, Q- Large motion video autoencoding with cross-modal video vae. arXiv preprint urXiv:2412.17888
2024.
Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J. Yang, Y., Hong, W. Zhang, X., Feng, G., al. Cogvideox: Text-to-video diffusion models with an pert transformer. arXiv preprint arXir:2408.06007, 2024.
Yu, L., Lezama, J., Gundavarapu, B., Versari, L., Sohn, K., Minnen, D., Cheng, Y., Gupta, A., Gu, X., Haupt- mann, A. G., al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint 0rXiv:23100057333 2023.
Zhang. R., Isola. P., Efros, A. A., Shechtman, and Wang, O. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
Zhao, S., Zhang, Y., Cun, X., Yang, S., Niu, M., Li, X., Hu, W., and Shan, Y. Cy-vae: compatible video vae for latent generative video models. arXiv preprint arXiv:2405.20279 2024.
Zhao, Y. Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M., Wright, L., Shojanazeri, H., Ott, M., Shleifer, S., al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304. / /277, 2023.
Zheng, Z., Peng. X., Yang. T., Shen, C., Li, S., Liu, H., Zhou, Y., Li, T., and You, Y. Open-sora: Democratizing efficient video production for all, 2024. URL https: //github. com hpca tech Paae//Oppnn-rrree Sora.
Zhou, Y., Wang, Q., Cai, Y., and Yang, H. Allegro: Open the black box of commercial-level video generation model. arXiv preprint arXiv:2410.154
2024.
Zhou, Y., Xiao, Z., Yang, S., and Pan, X. Alias-free latent diffusion models: Improving fractional shift equivariance of diffusion latent space. In CVPR, 2025.
13

---

Improving the Diffusability of Autocncoders
A. Limitations.
identify the following limitations of our work and the proposed regularization:
1. While we did our best to verify that our framework works in the most general setup possible, testing 4 different autoencoders across 2 different domains (image and videos), our study would be more complete when verified across other diffusion parametrivations (Karras et al., 2022; Ho et al., 2020; Kingma & Gao, 2023) or architectures (Karras et al., 2024).
2. We observed that our regularization still affects the reconstruction slightly: for example, Table 4 shows that FluxAE FID increased from 0.183 to 0.55 (though for some AEs, like Cog it improves). We are convinced that this FID increase could be mitigated by training with adversarial losses, which we omitted in this work for simplicity.
3. There is might improve the results (see Table 9), adding small KI. regularization (which we disabled in the end for our regularization for simplicity).
mild sensitivity hyperparameters: for example, found that varying the SHF regularization weight
4. None of the explored autoencoders released their training pipelines, and it non-trivial to fine-tune them even without any extra regularization. For example, observed that any fine-tuning of DC-AE (Chen et al., 2025) was leading to divergent reconstructions in our training pipeline (we explored dozens of different hyperparameter setups).
We leave the exploration of these limitations for future work.
14

---

Improving the Diffusability of Autocncoders
B. Implementation Details
DiT model details. To strengthen the baseline DiT performance, we integrated into the latest advancements from the diffusion model literature. Namely, we used self conditioning (Jabri et al., 2023) and RoPE (Su et al.. 2024) positional embeddings, Besides, we switched to the rectified flow diffusion parametrization (Albergo & Vanden- Vanden-Eijnden, 2022; Liu et al., 2022; Vahdat et al., 2021), which was recently shown to have better scalability with a fewer amount of inference steps (Esser et al., 2024).
Following (Jabri et al., 2023; Chen & Li, 2023; Gupta et al., 2024; Menapace et al., 2024), we employ self-conditioning for our DiT training and inference. During training with a 90% probability, we run an auxiliary forward pass with the DiT model, take its activations from the last block (i.e., right before the "unpatchify" projection), project them with a linear layer and add as residuals to the input tokens after patchification of the main training forward pass. For that auxiliary forward pass, following RIN (Jabri et al., 2023), we use the same noise level and "no-grad" context (i.e., we do not backpropagate through the auxiliary forward pass).
DiT training details. All the DIT models are trained for 400,000 steps with 10,000 warmup steps of the learning rate from O to 0.0003 and then its gradual decay towards 0,00001. We used weight decay of 0.01 and Adam (Loshchilov, 2017) optimizer with beta coefficients of 0.9 and 0.99. We used posterior sampling from the encoder distribution for VAE-based autoencoders. In contrast to the original work, we found it helpful to do learning rate decay to 0.00001 using the cosine learning rate schedule. We used the same model sizes for DīT-S (small), DiT-B (base), DiT-L (large) and DÝT-XL (extra large), as the original work (Peebles & Xie, 2022):
DiT-S: hidden dimensionality of 384, 12 transformer blocks, and 6 attention heads in the multi-head attention.
DiT-B: hidden dimensionality of 768, 12 transformer blocks, and 12 attention heads in the multi-head attention.
DiT-L: hidden dimensionality of 1024, 24 transformer blocks, and 16 attention heads in the multi attention.
DiT-XL: hidden dimensionality of 1152, 28 transformer blocks, and 16 attention heads in the multi-head attention.
We used gradient clipping with the norm of 16 for all the DiT models. Our models were trained in the FSDP (Zhao et al., 2023) framework with the full sharding strategy on single node of 8x NVidia A100 80GB GPUs or 8x NVidia H100 80GB GPUs (depending on their availability in our computational cluster).
For CV-AE, since it is considerably slower than other autoencoders, we trained LDMs on pre-extracted latents. For this, we pre-extracted them on random 17-frames clips. In essence, this reduces the total dataset size, but since we do the same procedure for the entire Cog VideoX-AE family, the models are comparable between each other.
Autoencoders training details. Since none of the autoencoders had their training pipelines released, we had to develop the training recipes for each of the autoencoder baselines individually which would not be detrimental to neither their reconstruction capability nor downstream diffusion performance. To do this, we ablated multiple hyperparameters (the most important ones being lcarning rate and KL regularization strength) to arrive to a proper setup. We chose the KL weight in such a way that the KL penalty maintains approximately the same magnitude as the pre-trained checkpoint.
Each autoencoder is trained with Adam (Loshchilov, 2017) optimizer, with betas of 0.9 and 0.99, and weight decay of 0.01. The learning rate was grid-searched individually for each autoencoder and is provided in Table 5. In all the cases, used mixed precision training with BFloat16.
During training, we maintained an exponential moving average of the weights (Karras et al., 2024), initialized from the same parameters as the starting model, and having a half life of 5,000 steps.
We emphasize that, when applying our regularization strategy on top of an autoencoder baseilne, we do not alter other hyperparameters (like learning rate), except for KL regularization which we disable for SE-regularized models (even though we found it helpful in some of our explorations).
For each autoencoder, we freeze the last output layers of the decoder. The motivation is the following: they were fine-tuned with the adversarial loss, which we want to exclude from the equation without hurting the ability of an autoencoder to model textural details which FID would be sensitive to (Rombach et al., 2022) and which do not influence the latent space properties. Namely, we freeze the last normalization and output convolution layers. In each case, the amount of frozen parameters constitute negligible amount of total parameters.
15

---

Improving the Diffusability of Autocncoders
Other hyperparameters for autoencoders training are provided in Table 5.
Table 5. Hyperparameters for the autoencoders explored in the current work. We had to tweak the hyperparameters for various autocncoders to prevent the divergence of the baseline training.
Hyperparameter
FluxAE
CMS-AE,
CV-AE
LTX-AE
Domain
image
image
video
video
Compression rate
8 x
16 x 16
4 x 8 x 8
8 x 32 x 32
Latent channels
16
16
16
32
Number fune-tuning steps
10,000
10,000
20,000
20,000
Image batch size
32
32
64
64
Video batch size
0
0
32
32
Default KL 3 weight
0.001
0.0
0.001
0.0001
Learning rate
0.00001
0.0001
0.0003
0.00005
Number of parameters
83.8M
44M
211.5M
419M
Training resolution
256 x 256
256 x 256
17 x 256 x 256
17 x 256 256
MSE loss weight
5
LPIPS loss weight
1.0
1.0
1.0
Gradient clipping norm
50
50
50
Num upsampling blocks frozen Is output convolution frozen?
1 Yes
3
0
0
Yes
Yes
Yes
Time shifting for rectified flow. In Section 4.2, we discussed the influence of time shifting the DiT model. Here, we provided the details of its implementation. We adapt the time shifting schedule from Lumina-T2X (Gao et al., 2024) for rectified flow parametrization by rescaling the time steps at inference time using the following formula:
1 t s s + (1-
t' =
where t is the time step, going from 1 (full noise) to 0 (clean image), and is the shifting coefficient. One can see that for 8 = 1, = 4, which is the default unshifted behaviour.
16

---

Improving the Diffusability of Autocncoders
C. Additional Experiments
In Section 3, we outlined the base scale equivariance strategy to regularize the spectrum of an autoencoder which has strong advantage of being very easy to implement by a practitioner. However, could be beneficial to possess more advanced tools for finer-grained control over the latent space spectral properties. This section outlines them and provides the corresponding ablation.
C.1. Explicitly Chopping off High-Frequency Components
Rather than applying downsampling to produce latents and RGB targets for regualrization, possible to replace some ratio of high-frequency components with zeros. To do so, DCT is applied the latents and RGB targets where chosen set of frequency components are masked out. The modified components are then translated back to the spatial domain by inverse DCT to form the training latents and reconstruction targets.
(x)) = d(x,, Dec(=)) + d(D= M), Dec() * M)) + Lreg,
where D and D represent DCT and its inverse, respectively. M B B binary mask indicating which frequencies to zero out defined as follows:
1, if zigzag(u, (u, v) < -N,NN,
M(u,v) v)
(5)
0, otherwise.
N controls the frequency cutoff. We provide the ablation for this strategy in Table 6.
Table 6. Ablations for explicit high-frequency chop off for DIT-S/2 trained for 200,000 iterations top of Flux AE with such regularization. While it can achieve better results some of baselines than naive downsampling, we opt out the latter strategy to its simplicity. For the non-zigzag order ablation, cut across each and independently
Stage II
Stage I
FDDsK
DIT-S/2
FluxAE chop off 90% (non-7igzag order)
912.4
DIT-S/2
FluxAE chop off 70% (non-zigzag order)
915.6
DIT-S/2
FluxAE chop off 30% (non-zigzag order)
929.7
DiT-S/2
FluxAE chop off 10% (non-zigzag order)
916.5
DIT-S/2
FluxAE + chop off 90% (zigzag order)
935.5
DIT-S/2
FluxAE + chop off 70% (zigzag order)
932.8
DIT-S/2
FluxAF chop off 30% (zigzag order)
962.9
DiT-S/2
FluxAE chop off 10% (zigzag order)
930.1
DIT-S/2
FluxAE (vanilla)
992.0
DIT-S/2
FluxAE with optimal (out of 8) KL 3
929.6
In Figure 5, we provided the visualizations for a FluxAE resiliense with and without such chopping high-frequency regularization for 50% HF dropout rate. In Figure 11, we provide an equivalent visualization for SE-fine-tune tuned FluxAE: while it less resilient to frequency dropout than CHF, but still noticeably better than the vanilla model.
C.2. Soft Penalty for High-Frequency Components
Instead of directly removing some of the components, which might become too strict regularization signal, one can consider penalizing the amplitudes of high-frequency components in soft manner. Concretely, given B B block, construct the following weight penalty matrix:
W.. = (u + BPP
(6)
Next, the soft regularization loss itself computed as:
Lsoftreg
Dav Wue-
(7)
M,30
17

---

Improving the Diffusability of Autocncoders
FluxAE
FluxAE + SE
25% Figure 11. RGB and FluxAE reconstruction with/without scale equivariance regularization for diffcrent percentages of chopped-off high frequency components.
0%
50%
75%
During training, when enabled, we add it to the main loss with the weigh 7. We found it beneficial in some of our experiments when it is added with a small coefficient (e.g., 0.01). While it is possible to achieve higher results with more fine-grained regularization, we opt to use the simpler version since we believe it would be easier to employ by the community.
To ablate its importance, we trained DiT-B/2 model on top of FluxAE models, fine-tuned with a different strength y. The results presented in Table 7.
Table 7. Ablating the regularization strength 7 of the soft frequency regularization Lusing-
Stage
Stage
FID 5k
FDD 5k
FluxAE FT-SE 0.001
26.43
497.14
FluxAE FT-SE 0.025
25.46
477.61
FluxAE + FT-SE 0.01
26.72
487.06
DIT-B/2
FluxAE FT-SE 7 -0.05
24.28
458.11
FluxAE FT-SE 7 =0.1
25.84
461.97
C.3. ImageNet 512" experiments
We trained our DiT-L/2 for class-conditional 5122 ImageNet-IK generation for 400K steps for FluxAE (Black Forest Labs, 2023), the results are presented in Table 8.
C.4. Ablating regularization strength a
To ablate the importance of the regularization strength a, we train FluxAE for 10,000 steps with a varying strength. The results are presented in Table 9.
18

---

Improving the Diffusability of Autocncoders
Figure 12. Illustration of the zigzag indexing order of DCT.
Table Class-conditional generation results on ImageNet- IK 512? without guidance. The original DiT paper reports the results after 3M training steps, while we use 400K steps for our models.
Stage II
Stage
FID
FDD
FluxAE (vanilla)
13.13
249.4
DIT-L/2
FluxAE FT
13.69
267.7
FluxAE FT-SE (ours)
11.63
203.5
DiT-XL/2 (orig) 3M steps
SD-VAE-ft-SE
12.03
Table 9. Ablating the regularization strength of proposed scale equivariance regularization.
Stage II
Stage I
FID
FDD
FluxAE FT-SE a 0.01
33.99
641.95
FluxAE FT-SE a 0.05
33.86
645.94
FluxAE FT-SE 0.1
28.62
586.91
DIT-B/2
FluxAE FT-SE 0.25
26.84
558.36
FluxAE + FT-SE 0.5
29.63
569.92
FluxAE + FT-SE a
33.22
612.45
C.5. From scratch training
To conduct whether our regularization provides benefits for from scratch training and when the same data is being used for both AE and LDM, we conducted additional experiments for FluxAE and Cog Vidcoo-AAAA. We trained FluxAE from scratch on ImageNet 2562 with/without our regularization and then trained DIT-B/2 top of the latent spaces of these two autoencoders. Then, we did the same for AA for Kinetics-700 17 x 2562, The results of AE reconstruction performance (after 300K training steps for FluxAE and 60K training steps for Cog VideoX-AB) and LDM generation performance after 400K training steps are presented in Table 10. One can observe that our regularization improves the performance in both cases.
C.6. Compute cost analysis
One might argue that the improved performance of our regularization comes from the extra computation which the autoencoder is using during training. this section, we show that not the case by providing an analysis of its
19

---

Improving the Diffusability of Autocncoders
Table 10. From scratch training results for FluxAE (Black Forest Labs, 2023) and CogVideoX-AE (Yang et al., 2024) See Appendix C.5 for details.
Method
PSNR t
LPIPS +
gFDDsk +
gFlDsk
gFVDsk J
FluxAE
29.39
0.059
502.44
25.43
+FT-SE (ours)
28.73
0.065
472.26
22.32
CogVideoX-AB
37.08
0.049
614.95
19.87
336.33
+FT-SE (ours)
36.61
0.053
515.68
17.40
237.28
computational cost.
We measured the FLOPs of FluxAE (for the batch size of and resolution of 2562) using the popular fvcore library. The entire encoder-decoder pass has 447 GFLOPS, and is split between the encoder/decoder as 136 vs 311 GFLOPS. Our regularization reuses the encoder pass and only runs the decoder with x2 or x4 reduced resolution (the scale sampled randomly during training). This results in 77.6 or 19.4 extra GFLOPs of the decoder, which is almost exactly 1/4 or 1/16 of the decoder compute or +17% or 4.5% the total forward pass. Since we sample x2 or x4 downsampling factor equally randomly, this results in 10.75% of total FLOPs overhead for our regularization.
To strengthen our point
experiment where the baseline FluxAE was fine-tuned strictly for 2 times longer (for 20K instead of 10K iterations). The resulted DiT-B/2 model achieved FIDsk and FDDsk of only 33.99 and 642.7 versus the scores of 25.87 and 551. .27 respectively for our regularized FluxAE+FT-SE model, fine-tuned for only 10K steps.
further,
20

---

Improving the Diffusability of Autocncoders
D. Additional visualizations
D.I. Spectra visualizations
10P
RGB
noise o 0.25
RGB noise 0.01
RGB noise 0.5
RGB noise 0.05
RGB noise σ
Normalized Amplitude
RGB noise 0.1
10-1
10
20
30
40
50
60
Zigzag Frequency Index
Figure 13. Spectrum of RGB under different noising levels: noise inflates the high-frequency component (when normalized). This results into an undesirable side-effect of KL regularization.
WanAE KL ß 0
WanAE KL 10
WanAE d,
WanAF de 32
WanAE KL B = 10-7
WanAE KI. B 10-3
WanAE d. 8
WanAE de 48
Normalizeel Amplitude
WanAF 10-6
WanAF 10-2
Normalizzd Amplitude
0.8
WanAE =
WanAE
WanAE d. 64
WanAE KI. 10-5
WanAE KL 10
0.2
30 Zigzag Frequency Index
Zigzag Frequency Index
(a) WanAF, KI. regularization.
(b) WanAF, channel dimensions.
LTX-AE LEX-AE 10
1.0
LTX- dE =
LTX AE d; 128
LTX-AE 10-7
LTX-AE 10
LTX-AE LTX-AE d
B 10-5
LIX-AE B 10
LTX-AE LTX-AE d.
Nornalized Amplitde
Normalzzed Amplitde
LTX-AEd; 64
LTX-AE
LTX-AE 10
0.6
0.4
0.2
10
20
30
10
50
60
Zigzag Frequency Index
Zigzag Frequency Index
(c) LTX-AF, KL regularization.
(d) LTX-AF, channel dimensions.
Figure Latent spectra of WanAF (Wan al., 2025) and I.X-AEE (HaCohen al., 2024) trained from scratch Kinetics-7000 17 x 256 for 100K steps with varying KI. regularization strength 3 or channel size.
21

---

Improving the Diffusability of Autocncoders
D.2. Samples visualizations
ZFT
Figure 15. Uncurated samples from DiT-XL/2 for FluxAE (top), FluxAE + FT (middle) and FluxAE + SE (bottom) on class-conditional ImageNet 256 x 256 for random classes. During inference, used 256 steps with the guidance scale of 3.0.
22

---

Improving the Diffusability of Autocncoders
SVB
BOR
Figure 16. Uncurated samples from DiT-XL/2 trained for IM steps top FluxAE SE (boltom) on class-conditional ImageNet 256 256 for random classes. During inference, we used 256 steps with the guidance scale of 3.0.
Figure 17. Uncurated samples from DiT-XL/2 trained for IM steps on top FluxAE SE (bottom) on class-conditional ImageNet 256 x 256 for class 88. During inference, used 256 steps with the guidance scale of 3.0.
Figure 18. Uncurated samples from DiT-XI/2 trained for IM steps on top FluxAE SE (bottom) on class-conditional ImageNet 256 x 256
for class 130. During inference, we used 256 steps with the guidance scale of 3.0.
23

---

Improving the Diffusability of Autocncoders
Figure 19. Uncurated samples from DiT-XL/2 trained for 1M steps on top FluxAE + SE (bottom) on class candiionn ImageNet 256 x 256
for class 279, During inference, we used 256 steps with the guidance scale of 3.0.
Figure 20. Uncurated samples from DiT-XL/2 trained for IM steps on top FluxAE + SE (bottom) on class-conditional ImageNet 256 x 256
for class 555. During inference, used 256 steps with the guidance scale of 3.0.
24

---

Improving the Diffusability of Autocncoders
FERTNIRRE WITEES
F
yecapatioa
B BROBAI
SE KIROEAT
I
KONG INAVORRAEE
Figure Uncurated samples from DIT-XL/2 trained for 400K steps on top FluxAE SE (bottom) on class-conditional ImageNet 512 x 512 for random classes. During inference, we used 256 steps with the guidance scale of 3.0.
25

---

Improving the Diffusability of Autocncoders
1
Figure Uncurated samples from DIT-B/1 for CMS-AE, (top), CMS-AE, FT (middle) CMS-AE, (bottom) on class-
conditional ImageNet 256 256. During inference, used 256 steps with the guidance scale of 1.5.
26

---

Improving the Diffusability of Autocncoders
Figure 23. Uncurated samples from DiT-XL/2 for CogVideoX-AE (top). CogVideoX-AE + FT (middle) and Cog VogVeoeoX-AE SE (bottom) on class-conditional Kinetics 17 256. During inference, we used 256 steps with the guidance scale 3.0.
27

---

Improving the Diffusability of Autocncoders
Figure 24. Uncurated samples from DiT-B/2 for Cog (top), Cog VideoX-AE FT (middle) and Cog CogVideoX-AE. + SE (bottom)
on class-conditional Kinetics
256. During inference, used 256 steps with the guidance scale
28

---

Improving the Diffusability of Autocncoders
x
Figure 25. Uncurated samples from DiT-B/1 for LTX-AE (top), LTX-AE FT (middle) and LTX-AE + SE (bottom) on clas-ccodittonal
Kinetics
256. During inference, 256 steps with the guidance scale
29

---

Improving the Diffusability of Autocncoders
E. Failed experiments
Over course the project, we explored several other ideas to regularize the latent space of autoencoders. While they did pan out that well, we still discuss them in this section to spur potential future exploration.
MiniL MiniLLM--reguzaation training. LSGM (Vahdat et al., 2021) showed that the correct objective of LDM training optimizes both autoencoder and LDM at the same time:
C(x,0,0, v) = (oo xx)
| Pu (x + (z)
B (xn 1- log Py
E (xolx))) [log qs (zo|x)] + E (x |- log pa (2o)]
(8)
reconsiruction term
negative encoder entropy
cross eatropy
where [(x, 0, , 16) is the ELBO objective, 4s Pv are encoder/decoder, and is the LDM (or any other latent generative model). This motivated us to explore training small LDM model together with the autoencoder expecting that it would make its latent space more diffusable. However, even with extensive hyperparameter search, was not outperforming the vanilla baseline.
Lipszhitz regularization. Dao et (2023) derived the following upper bound on the Wasserstein distance W2 (Po. pa) between the ground truth (x() and recovered (x(x distributions, when the distribution is learned in the latent space of an autoencoder with encoder f and decoder gr:
S.' [ ||vo(Zr, t) - 6z (Zr, 2) dzdt,
W (po. Po) S I|A (x) + L3 el+2l
/ee/
where er and are the ground-truth and recovered velocity estimators in the rectified flow framework (Liu al., 2022), Ly- is the Lipschitz constant of the decoder and É is the Lipschitz constant of the learned velocity estimator 0g This upper bound inspired to minimize the Lipschitz constant of the decoder. To do this, we used Ri Re-regularization (Mescheder et al., 2018) from the GAN literature (Goodfellow et al., 2014). This was yielding promising initial results and working almost on par with the scale equivariance regularization, but second-order differentiation for Rí regularization was entailing much slower training speed and engineering struggles (it has poor compatibility with FSDP). This is why we proceeded with scale equivariance which was performing slightly better and much simpler conceptually.
Temporal scale equivariance. We briefly explored temporal scale-equivariance regularization, but surprisingly, it was leading to improved results. We hypothesize that the temporal domain is of different nature compared to the spatial one since the temporal freeufrry components are more noticeable by human eye than the low-frequency ones.
30


Reviews:
[{"review_id": "DNhu3IF9zI", "reviewer": "Reviewer_MRKn", "summary": "The authors analyze the latent space of autoencoders widely used for latent diffusion models and identify that the spectrum of autoencoders typically deviate from that of natural images. In particular, latent spaces have stronger high frequency components compared to RGB images. These high frequency components will be challenging for the diffusion model to learn and could impede performance. The authors propose a simple regularization to align the spectrum of the latent space with that of RGB images. The authors train the autoencoder to reconstruct a downsampled version of the RBG image from a downsampled latent code. This removes the high frequency components from both the RGB image and latent and enforces scale equivariance. The authors present results across image and video generation and demonstrate that this autoencoder regularization improves the downstream performance of diffusion models. Extensive comparisons with KL regularization are presented as well as additional spectral regularization methods in the appendix.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thorough feedback. In what follows, we carefully respond to each of the points raised. All comments and suggestions will be fully reflected in the revised manuscript.\n\n> From-scratch training\n\nOur main motivation of fine-tuning instead of training from scratch is three-fold:\nBe able to compare against strong established benchmarks;\nEliminate the possibility that we might have some “bug” in the training pipeline, which our regularization is rectifying (please, note that *none* of the explored SotA AEs release their training code).\nCarry more value to the community of improving popular autoencoders.\n\nThat being said, we launched multiple experiments for from-scratch training for FluxAE and CogVideoAE. Moreover, we opted for using public datasets to have fully self-sufficient experiments. Namely, we trained FluxAE from-scratch on ImageNet for 200K steps and CogVideoAE from-scratch on Kinetics-700 for 60K steps (it’s a heavyweight AE and slower to train). Due to the limited rebuttal period time, we only have the DiT-B/2 results till 300K steps on ImageNet, and till 250K steps on Kinetics. For both datasets, the regularized AEs lead to better LDM convergence. DiT-B/2 for non-regularized vs regularized AEs perform as follows:\n- For ImageNet:\n    - DinoFID@5k: 569.93 vs 561.4 \n    - DFID@5k: 27.94 vs 28.79\n- For Kinetics:\n    - DDinoFID@5k: 652.5 vs 561.21\n    - DFID@5k: 21.54 vs 19.66\n    - DFVD@5k: 265.94 vs 379.88\n\nIn all cases, our regularization improves the diffusability. We’ll include the full training results (400K) in the revised version of the paper.\n\n> ​​Training on public data\n\nWe thank the reviewer for pointing this out, we include the results on public data together with the from-scratch training experiments in the previous message.\n\n> Some training details are under-explained (e.g. self-cond)\n\nOur self-conditioning mechanism follows prior work without any modifications (i.e., RIN, FIT, WALT). Namely, during training with a 90% probability, we run an auxiliary forward pass with the DiT model, take its activations from the last block (i.e., right before the “unpatchify” projection), project them with a linear layer and add as residuals to the input tokens after “patchification” of the main training forward pass. For that auxiliary forward pass, following RIN, we use the same noise level $\\sigma$ and “no-grad” context (i.e., we do not backpropagate through the auxiliary forward pass)\n\nWe added the self-conditioning discussion into the “Implementation details” appendix. We would be grateful to the reviewer if they point out any further missing implementation details and we would happily include them into the submission.\n\n> The DiT training details section in appendix A ends on a trailing sentence.\n\nWe thank the reviewer for pointing out that writing mistake. The sentence was intended to convey: “In essence, this reduces the total dataset size, but since we do the same procedure for the entire CogVideoX-AE family, the models are comparable between each other.” We fixed the error.\n\n> Explanation of non-monotonic KL influence on reconstruction quality (Table 3).\n\nYes, FluxAE was indeed unstable when fine-tuned with KL regularization for some of the KL β weights: it was stable for β of 0, 1e-7, 1e-4, and 1e-3, but not for 1e-6, 1e-5, 1e-2, and 1e-1. For from-scratch training (which we were doing for Figure 3), it was stable for the entire range (from 0.0 to 0.1), but after some threshold of β <= 1e-4, there was almost no difference in PSNR or FID. Our intuition is that high-capacity autoencoders (like FluxAE) can accommodate high KL penalty for their latents (for from-scratch training, we started noticing degradation only for β >= 1e-3), and their reconstruction quality is governed by other factors, making KL interference less predictable (up to a certain factor).\n\n\n\nShould the reviewer have further comments, we would be glad to incorporate them fully into our manuscript.", "claims_and_evidence": "The claims are presented by clear and convincing evidence. The spectral analysis, in particular, is insightful and clearly motivates their proposed approach. The authors present comprehensive analysis of the effect of their regularization on both the autoencoder and the downstream generative models. The comparison against KL regularization, the current standard, is comprehensive.", "methods_and_evaluation_criteria": "The authors are concerned with improving the suitability of autoencoders for downstream generative modeling. The authors evaluate primarily on ImageNet 256 and Kinetics 700 which are suitable benchmarks for latent diffusion modeling. Their evaluation of their autoencoders (reconstruction metrics, spectral analysis) and generative models (FID) is convincing.", "theoretical_claims": "The authors do not present any proofs.", "experimental_designs_or_analyses": "The experimental design is sound and the experimental comparisons are fair. One limitation is that the authors always start from an existing high-quality autoencoder and then fine-tune it with their additional regularization. While much more computationally feasible, this raises the question of whether their regularizer conveys the same benefit when training an autoencoder from scratch. I do not think that this is a big limitation as their method can always be introduced towards the end of training to improve align the latent spectrum. However, it does limit the ramifications of their findings somewhat. When training a new autoencoder, it is not entirely clear what the optimal procedure is.", "relation_to_broader_scientific_literature": "While diffusion models, and latent diffusion models in particular, have exploded in popularity, I think there has been comparatively less focus on what makes a \"good\" autoencoder for latent diffusion. People often use publicly available autoencoders for which the training decisions may not be entirely transparent. I think that this area is currently under-explored, and this paper is a welcome remedy to that.", "essential_references_not_discussed": "The discussion of related work is comprehensive.", "other_strengths_and_weaknesses": "Strenghts:\n1. This work focuses on an under-explored problem: What makes a \"good\" autoencoder for latent diffusion? I think that studying this problem is challenging in part because evaluation requires both training an autoencoder and a downstream diffusion model in its latent space. I welcome work in this area.\n2. The proposed regularization is well-motivated. I find the \"spectral autoregression\" interpretation of diffusion models to be intuitive and it's nice to see this intuition motivating a technique that seems to work well in practice. The spectral analysis throughout is insightful.\n3. The proposed regularization is simple to implement, increasing the likelihood of adoption. I appreciated the discussion of more complex alternatives in the appendix.\n4. The comparison with KL regularization is comprehensive.\n\nWeaknesses:\n1. The work focuses only on fine-tuning pre-trained autoencoders. The regularization could behave differently when learning a model from scratch with additional losses (e.g. adversarial). This limits the takeaways from their work somewhat. \n2. The autoencoders are fine-tuned on private internal datasets which harms reproducibility. The authors do verify (with their fine-tuning only ablation) that the dataset shift doesn't contribute to the performance boost.\n3. Some of the implementation details are bit under-explained. The authors mention that they incorporate self-conditioning in the DiT, but do not provide precise implementation details.", "other_comments_or_suggestions": "The DiT training details section in appendix A ends on a trailing sentence.", "questions_for_authors": "1. How exactly is the self-conditioning implemented? Is it the latent self-conditioning from the RIN paper, or just self-conditioning on the current data prediction?\n2. The behavior of the KL regularization seems somewhat non-monotonic (Table 3). I would expect increased regularization to monotonically harm reconstruction, but this does not appear to strictly be the case. For instance, 10e−6 achieves worse reconstruction than 10e-3. Any insight into why this is the case? Did you observe training instabilities?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SBcL4zhZLx", "reviewer": "Reviewer_tijt", "summary": "This paper explores the latent spaces of autoencoders within latent diffusion models (LDMs), specifically examining spectral discrepancies between latent and RGB spaces. The authors introduce the concept of diffusability, which quantifies how effectively a distribution can be modeled by a diffusion process. They hypothesize that high-frequency components in the latent space degrade diffusability, thereby reducing both the efficiency and generation quality of LDMs. To mitigate this issue, they propose a scale equivariance regularization strategy, which enforces spectral alignment between the latent and RGB spaces by removing high-frequency components. Empirical evaluations demonstrate that this approach improves image generation performance by 19% and video generation by 44% compared to existing LDMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments, which have greatly improved our work. Below, we address each concern.\n\n> Ambiguous caption in Figure 2  \n\nFigure 2 shows spectra of latent codes from real images encoded with from-scratch trained FluxAE with varying bottleneck sizes. We clarified the caption accordingly.\n\n> Exploring influence of bottleneck dimensionality on high-frequency components with more architectures  \n\nUpon closer inspection, we discovered a broadcasting issue in our spectrum computation pipeline, which led to distorted plots in the original submission. Updated FluxAE results are [[here]](https://anonymous.4open.science/r/ae-diff-icml-rbtl-815F/figures/fluxae-ch.png). We further added analyses for two additional architectures: [[WanAE]](https://anonymous.4open.science/r/ae-diff-icml-rbtl-815F/figures/wanae-ch.png) (which has the same bottleneck dimensionalities as CogVideoX-AE, but considerably faster to train) and [[LTX-AE]](https://anonymous.4open.science/r/ae-diff-icml-rbtl-815F/figures/ltxae-ch.png), trained with increasing bottleneck channel sizes. As one can see from these figures, autoencoders with higher channel sizes tend to possess high-frequencies of larger relative magnitude.\n\n> Fig. 3 does not show clear KL-high-frequency trend  \n\nThe FluxAE KL plot previously had the same broadcasting issue; corrected results are [[here]](https://anonymous.4open.science/r/ae-diff-icml-rbtl-815F/figures/fluxae-kl.png). We included similar analyses for [[WanAE]](https://anonymous.4open.science/r/ae-diff-icml-rbtl-815F/figures/wanae-kl.png) and [[LTX-AE]](https://anonymous.4open.science/r/ae-diff-icml-rbtl-815F/figures/ltxae-kl.png). These demonstrate a clearer relationship between KL and high-frequency energy.\n\nThe influence of KL on the high-frequency spectrum can be attributed to its role in injecting random noise during the encoding stage—an effect present both during training and inference. As KL regularization increases, so does the level of injected noise. Since random Gaussian noise has a uniform power spectrum, this flattens the frequency distribution, disproportionately inflating the high-frequency tail. We illustrate this in [[this plot]](https://anonymous.4open.science/r/ae-diff-icml-rbtl-815F/figures/rgb-noise.png), where progressively increasing Gaussian noise (added to normalized RGB signals in [[–1, 1]]) results in a visible elevation of the high-frequency content. A similar effect was also independently observed in concurrent work on [[SwD distillation (Figure 1)]](https://arxiv.org/pdf/2503.16397).\n\nThis reveals a nuanced trade-off: KL regularization aligns latents to the standard normal prior, facilitating downstream diffusion (as noted by LSGM), yet increases high-frequency energy, potentially hindering diffusability. We believe this trade-off warrants further attention.\n\n> Eq. 2 missing SE regularization loss weight  \n\nWe thank the reviewer for pointing this out and corrected the equation.\n\n> Table 3 unclear relationship between KL strength and high-frequency components  \n\nThere may be a misunderstanding: Table 3 illustrates KL's negative effect on diffusability, whereas its impact on high-frequency energy appears in Figure 3. To strengthen this analysis, we added results from DiT-L [[here]](https://anonymous.4open.science/r/ae-diff-icml-rbtl-815F/figures/new-kl-table.png).\n\nThis table shows increased KL generally boosts small-model LDM performance, but at the expense of poorer reconstruction and stability, limiting scalability (consistent with SD3 findings). In contrast, our regularization improves LDM performance without harming reconstruction, scaling well to larger models.\n\n> Clarify “diffusability”: does it include efficiency and quality, and are there metrics?  \n\nWe use \"diffusability\" strictly for generation quality, independent from efficiency. All else equal, increased diffusability should enhance LDM generation quality. Unfortunately, we found no reliable quantitative metrics correlating consistently with diffusability, including diffusion loss magnitude or decoder Lipshitz constants (as theoretically connected via Eq. 28 in [[LFM]](https://arxiv.org/pdf/2307.08698)). We believe it is still important to introduce such a term to the community  to bring attention to this property, since it remains largely ignored these days.\n\n> Clarify how spectral analysis suggested high-frequency components impact diffusability  \n\nWe initially sought high-compression autoencoders but quickly found that increasing bottleneck size significantly reduces diffusability, keeping other factors constant. Then, we concurrently worked on cascaded latent diffusion pipelines, which required autoencoders to support downsampling, motivating us to explore their spectral characteristics. This revealed how increased bottleneck dimensionality relates to diminished diffusability, informing our central hypothesis.\n\nWe would greatly welcome any further comments the reviewer might have.", "claims_and_evidence": "The authors’ hypothesis is grounded in the spectral analysis of latent and RGB spaces, as presented in  Figures 2 and 3. However, certain aspects of their analysis remain unclear.\n\nFigure 2: The caption lacks clarity regarding whether the spectra of various channels in FluxAE correspond to reconstructions or latent representations. The distinction between “comparison between the reconstructions” and “the latent space of an autoencoder” is not explicitly addressed, leaving ambiguity in the interpretation of the results. Additionally, given the limited scope of the analysis (only applied on FluxAE), this claim appears insufficiently substantiated. The absence of empirical verification across multiple architectures weakens the argument that this trend is a fundamental characteristic of diffusion models in general.\n\nFigure 3: The authors claim in Section 3.2 that “higher KL regularization introduces more high frequencies”. However, the figure does not provide clear evidence supporting this statement. No explicit trend is observed between the scale of KL regularization and the power of high-frequency components, contradicting the claim made in the text. This lack of alignment between theoretical justification and empirical results raises concerns about the robustness of the proposed hypothesis.\n\nGiven these limitations in the spectral analysis, the motivation for the proposed regularization technique appears to be based on a weak foundation. The authors do not provide sufficiently clear or convincing evidence to establish a strong causal link between their observations and the claimed effects on diffusion model performance. A more rigorous analysis, including evaluations across multiple architectures and additional spectral studies, would be necessary to substantiate their claims.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense. However, a more comprehensive evaluation across multiple architectures for image generation would be necessary to substantiate the claim that their method effectively improves diffusability in LDMs more broadly.\n\nAs the strength of the regularization is controlled, mentioned to be 0.25 in the paper, the scale factor for scale equivariance term should be included.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experimental design seems to be well-constructed and appropriate for the research objectives. The authors effectively evaluate their method on both image and video generation tasks, which demonstrates the broader applicability of their proposed regularization technique. Furthermore, the ablation study provides valuable insights by comparing the autoencoder's reconstruction quality with and without the regularization term, highlighting the impact of their approach on the model’s performance.\nHowever, there is still an issue with the unclear trend in Table 3, where the relationship between the KL regularization scale factor and high-frequency components is not clearly demonstrated.", "relation_to_broader_scientific_literature": "The authors propose a regularization method that truncates these high frequencies, drawing from earlier work on spectral regularization techniques. By aligning the spectral properties of latent and RGB spaces, their approach improves generation quality, offering a novel perspective on latent space manipulation in generative models.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. As the goal of this paper is to improve diffusability, could you clarify whether the term diffusability encapsulates both the efficiency and generation quality of diffusion models? Specifically, does an increase in diffusability directly imply an improvement in generation quality, as suggested in your results? If not, are there any proposed methods or metrics to quantitatively assess diffusability in the context of latent diffusion models?\n\n2. As I mentioned above, in your analysis, the hypothesis regarding the impact of high-frequency components on diffusability is derived from spectral properties. Could you provide a more detailed explanation of how the spectral analysis led to this hypothesis? Specifically, what were the key observations or trends that informed this conclusion?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X1UIm2138Z", "reviewer": "Reviewer_JqrQ", "summary": "The paper observes higher frequency component in VAE's latent space than those in normal RGB images and these high frequency components have greater magnitude with larger channels and stronger KL regularization. Therefore, it proposes a novel regularization technique -- scale equivariance (SE) -- to improve the diffusibility of VAE. Specifically, SE suppresses the high-frequency component in latent space by introducing an additional loss for the ground-truth downsampled image $\\tilde{x}$ and reconstructed corresponding downsampled latent vector $\\text{Dec}(\\tilde{z})$. Extensive experiments have been conducted on different VAE to demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable remarks. Below, we address each raised concern.\n\n> Why high-frequency components have higher dimensionality?\n\nWe agree this could be clearer. By \"low-frequency components,\" we mean DCT coefficients required to reconstruct a feature map downsampled by factor k per spatial dimension. High-frequency components are remaining coefficients needed to fully reconstruct the original map. Since DCT coefficients scale quadratically with k, we have:\n\nNumber of high-frequency components = (k² − 1) × Number of low-frequency components\n\nFor k=2 (our experiments), high-frequency components thus have three times higher dimensionality. We'll clearly state this in the final version.\n\n> It is unclear why higher frequencies are only generated in the final steps.\n\nThis claim originates from prior literature (Rissanen et al., \"Generative Modelling With Inverse Heat Dissipation\"; also \"Diffusion is spectral autoregression\" and DCTDiff). Intuitively, high-frequency components are too easily erased by noise in the early denoising steps, which is why the model can only pick them up for smaller noise levels, which relate to the final denoising steps. We will extend the discussion of this topic in the earliest revision.\n\n> It is unclear why higher frequencies are more prone to error accumulation.\n\nThis observation is also an already explored phenomenon in the broader diffusion literature. A solid reference we can point to is Li et al. “On Error Propagation of Diffusion Models” (ICLR 2024) with Figure 2 of their paper serving as a clear illustration of such behaviour. Intuitively, the denoising process of a diffusion model can be represented as an ODE trajectory. And ODEs are known to accumulate errors very quickly even for simple processes (e.g., even for the simplest y’ = y equation, forward Euler solver would accumulate the error proportional to h² with each step of size h, and this error compounds exponentially as steps progress). Then, since diffusion models generate high frequencies later in the trajectory (per our previous response), the error accumulation is amplified specifically for them.\n\n> Could SE interact with other AE properties, rather than spectral properties alone?\n\nIn Table 6 of the appendix, we provide the ablation for direct *high frequency chop-off*: it only has influence on spectral properties (and this influence is the same as for SE), eliminating the interaction of other factors. It noticeably improves the diffusability, but we opt for the equivalent SE regularization since it’s much simpler and less error-prone to implement and should be easier to adopt by the community.\n\n> Computational cost and potentially unfair AE training budget comparison.\n\nWe measured the FLOPs of FluxAE (for the batch size of 1 and resolution of 256²) using [[fvcore]](https://github.com/facebookresearch/fvcore). The entire encoder-decoder pass has 447 GFLOPS, and is split between the encoder/decoder as 136 vs 311 GFLOPS. Our regularization reuses the encoder pass and only runs the decoder with x2 or x4 reduced resolution (the scale sampled randomly during training). This results in 77.6 or 19.4 extra GFLOPs of the decoder, which is almost exactly 1/4 or 1/16 of the decoder compute or +17% or 4.5% of the total forward pass. Since we sample x2 or x4 downsampling factor equally randomly, this results in ~10.75% of total FLOPs overhead for our regularization.\n\nTo strengthen our point even further, we ran an experiment where the baseline FluxAE was fine-tuned strictly for 2 times longer (for 20K instead of 10K iterations). The resulted DiT-B/2 model achieved FID@5k and DinoFID@5k of only 33.99 and 642.7 vs the corresponding metrics of 25.87 and 551.27 of our FluxAE+FT-SE, fine-tuned for only 10K steps.\n\n> Balancing reconstruction and SE regularization.\n\nWe appreciate the raised concern and we provide the ablation over the SE strength in the table [[here]](https://anonymous.4open.science/r/ae-diff-icml-rbtl-815F/figures/se-reg-ablation.png). We remain cautious about reducing the influence of the main reconstruction term not to lose in terms of the reconstruction quality.\n\n> Training AE and LDM on the same dataset.\n\nWe conducted from-scratch FluxAE and CogVideoAE training on ImageNet/Kinetics datasets, followed by DiT-B/2 training, as detailed in our response to Reviewer MRKn. In both cases, SE regularization consistently improved results.\n\n> Unclear if suppressing high frequencies motivates improved performance.\n\nWe emphasize that merely suppressing high frequencies is insufficient; it is also crucial to prevent the decoder from arbitrarily amplifying them (we provide a detailed discussion on this in Sec. 3.3). This ensures alignment between the diffusion process’s strength—its ability to generate low-frequency components—and human perception, which is more tolerant to errors in low frequencies. Our experiments with direct frequency chop-off (described above) articulates this more explicitly.", "claims_and_evidence": "The effectiveness of the proposed method is well supported in the experiment section. Nevertheless, I have the following questions:\n1. For hypothesis (i) in line 210 (right column), how do the results in Figure 4 imply that high-frequency components are higher dimensional? Adding a new loss does not necessarily increase the maximum dimensionality VAE can model.\n2. For hypothesis (ii), how does Figure 5 show higher frequencies are generated only in the final steps of sampling? I thought Figure 5 only considers the reconstructed images with VAE and there is no any sampling in diffusion.\n3. For hypothesis (iii), how does Figure 6 demonstrate higher frequencies are more susceptible to error accumulation over time? Is it possible to provide more quantitive evidences for this claim (rather than sampled noisy images during the diffusion denoising process)?\n4. Empirically, it is observed using scale equivariance can improve the performance of diffusion model on different metrics. But I do not find any quantitive results that illustrate generation performance or the diffusibility of VAE is negatively correlated with the presence of higher frequencies components? It is possible that scale equivariance might interact with other factors in latent space of VAE and inadvertently improve the generation performance of the ultimate generative model.", "methods_and_evaluation_criteria": "The paper adopts standard metrics for VAE and diffusion models. But I have some important concerns regarding the computation budget of baseline and scale equivariance. \n1. Regarding equation (2), if I understand correctly, the additional loss term regarding the downsampling introduces additional gpu memory cost when doing forward and backward. Can you provide the flop of each VAE training / fine-tuning iteration? I am wondering whether it is fair to compare with baseline without scale equivariance given they might use different compute budget with the same number of iterations. \n2. Also, is it possible to adapt dynamic downsampled latent vectors and images during the training rather than introducing a new loss term for it? In other words, we do not use the first term in Equation 2, and vary the downsampled ratio for each batch images and vectors (of course, we still need to do one forward to get latent z with original x). Can we still achieve better performance than the original baseline? \n3. Can we introduce a coefficient to balance the weight between original reconstruction objective and downsampled one? What is the optimal coefficient (empirically)?", "theoretical_claims": "This paper does not introduce any new theory or proof.", "experimental_designs_or_analyses": "The paper has conducted extensive evaluations using various VAE models, including FluxAE, CogVideoX-AE, and LTX-AE and across different datasets, including ImageNet-1K and Kinetics-700. In all cases, the dataset (in-the-wild data) used to train VAE is different from the one training diffusion model. What if they are the same? Do scale equivariance still help filter out higher frequencies and improve generation performance?", "relation_to_broader_scientific_literature": "The proposed method is both simple and novel. I am not aware of any prior work using downsampled image to reduce high frequency components for VAE.", "essential_references_not_discussed": "I do not find any important yet missing cited works.", "other_strengths_and_weaknesses": "See above for each section.", "other_comments_or_suggestions": "I do not have major comments and suggestions other than questions listed above.", "questions_for_authors": "I have listed all my questions in each part. While the proposed method does improve the performance, my major concern is (1) whether the motivation of the method (suppressing high-frequency components) leads to the improving performance (2) whether it is fair to compare with original baseline given the same iteration with possibly additional computation budget from a new loss term (3) whether it is possible to simplify the method using reconstruction loss with dynamic downsampled ratio or further improve the generation performance with weights balancing two reconstruction losses. \n\nIf the authors can resolve some of my concerns (mostly about 1, 2). I am happy to increase my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1IY72pE5gT", "reviewer": "Reviewer_3gRN", "summary": "This paper finds that the pre-trained VAE for visual generation exhits larger high-frequency components than the original RGB images, in the lens of spectral analysis using 2D DCT.\n\nTo improve the latent diffusion generative modeling, the authors proposed to align the spectral property of image latents with that of RGB images by enhancing the VAE reconstruction ability of the low-frequency signals.\n\nResults on image generation and video generation show that the proposed Scale Equivariance regularization improves the generation quality.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer’s valuable feedback and constructive recommendations. Below, we systematically respond to each issue highlighted. We will ensure comprehensive incorporation of all suggestions into our manuscript.\n\n> It is not clear why high-frequency components have higher dimensionality.\n\nReviewer JqrQ has raised the same concern, and due to the limited space, we politely refer to our argumentation in that other response.\n\n> It is not clear why high-frequency components are more susceptible to error accumulation.\n\nSimilarly to the previous question, we kindly refer the reviewer to our response to Reviewer JqrQ on the same matter.\n\n> Can SE be helpful just because it improves the reconstruction of low-frequency components?\n\nNot quite. PSNR/SSIM scores are extremely sensitive to low-frequency reconstructions, and all the autoencoders (vanilla, fine-tuned and fine-tuned+SE) perform on par in terms of these metrics.\n\n> From-scratch training.\n\nDue to the space limit, we are again forced to refer the reviewer to our response on the same question to Reviewer MRKn. We apologize for this inconvenience.\n\n> The improved performance can be due to a different reason: high-frequency components of images have a lower entropy and should be easier to model.\n\nIn this [[plot]](https://anonymous.4open.science/r/ae-diff-icml-rbtl-815F/figures/entropy.png), we visualize the entropy of the latents for regularized/non-regularized FluxAE at various frequencies, and also RGB. Entropy is computed by obtaining a histogram of 100 bins for each frequency and its corresponding density. While for images space, high-frequencies indeed have lower entropy, for the latents, HF entropy exhibits a much flatter or an even slightly increasing profile indicating that they might be harder to model. We note that while higher entropy, intuitively, should lead to harder modeling of a distribution, it does not necessarily affect the quality of the final samples obtained from passing the latents through the decoder. Higher frequencies with smaller scale can still exhibit high entropy while our SE reduces the dependence on these frequencies.\n\n> Figure 5 does not show that SE preserves more content compared to the baseline.\n\nWe try being gentle about enforcing SE regularization not to affect the reconstruction quality. Figure 5 shows that our regularized AE does not introduce spurious high frequencies when they are chopped off in the latents. This also results in meaningful improvements in the corresponding reconstruction metrics as confirmed in Figure 8. Other examples (e.g., [[this one]](https://anonymous.4open.science/r/ae-diff-icml-rbtl-815F/figures/firetruck-freq.png)) can show this effect more noticeably.\n\n> [3 writing issues]\n\nWe fully agree with the remarks and will incorporate them in the manuscript.\n\n> Why are some amplitudes greater than 1 after normalization by D_{0,0} in Figures 2 and 3?\n\nD_{0,0} corresponds to the lowest frequency, which is basically the mean of all the values. It might not necessarily have the largest amplitude compared to other frequencies (though this rarely happens in natural signals). For example, [[this figure]](https://anonymous.4open.science/r/ae-diff-icml-rbtl-815F/figures/freqs01-example.png) shows an example of an image which has 0 amplitudes everywhere, except for the “[[0,1]]” spatial frequency.\n\n> Why do high-frequency components become more pronounced as bottleneck channels increase?\n\nThis is an interesting question. We hypothesize that increasing the autoencoder's bottleneck channels enables the model to better capture finer, high-frequency details. Initially, with limited capacity, the encoder prioritizes smoother, low-frequency information. As capacity grows, it encodes additional high-frequency details, which are unique and informative. However, without explicit regularization promoting frequency-based disentanglement, these high-frequency components distribute across channels in an unstructured manner. Thus, higher-dimensional bottlenecks enhance high-frequency representations but do not yield systematic frequency-specific disentanglement per channel. We will clarify this point in the revised manuscript.\n\n> Results for various numbers of steps.\n\nWe generated the Table 1 results using 256 steps (L#318). Additional plots for FluxAE/CogVideoX-AE/LTX-AE across step counts (16,32,64,128,256) for FID/DinoFID and FVD (images: 50K samples, videos: 10K samples) are provided [[here (see the neighboring folders as well)]](https://anonymous.4open.science/r/ae-diff-icml-rbtl-815F/num-steps-plots/image_flux_dit-xl/2_fid-dinov2.png). Regularized autoencoders consistently improve diffusability. We will include these results (+ DiT-XL/2 for CogVideoX-AE) in the final paper.\n\n> Extended KL influence discussion.\n\nWe provided an expanded KL discussion (noise injection, relevant literature, RGB+noise spectrum analysis) in response to Reviewer tijt.\n\nWe welcome any additional suggestions from the reviewer.", "claims_and_evidence": "The claim below lacks verification or literature support:\n\nWe also hypothesize that higher frequencies components are harder to model than lower frequency components for the following reasons and thus should be avoided: (i) they have higher dimensionality; (iii) they are more susceptible to error accumulation over time\n\nReasons: \n* High-frequency components of images usually have a lower entropy than the low-frequency components, thus image high-freq signals should be easier to model since they are intuitively close to 0 (this is also why JPEG removes high-freq signals for compression).\n* It is not clear to me why high-freq signals have higher dimensionality\n* It is not clear why high-freq signals are more susceptible to error accumulation", "methods_and_evaluation_criteria": "The authors proposed a regularization term (Scale Equivariance) on the vanilla VAE loss, which is easy to understand.\nBut, different from the explanation in the paper, I think the principle of SE is that it explicitly enhances the reconstruction of the image's low-frequency signals. \n\nThe evaluation criteria FID, FVD, PSNR, SSIM, and LPIPS are commonly used in the community.", "theoretical_claims": "no theoretical claim", "experimental_designs_or_analyses": "The experimental designs and analyses are overall sound and convincing.\nThe authors verify their methods on image generation, video generation, and autoencoder reconstruction.\n\nOne suggestion:\nit is better for the authors to add an experiment to show the effect of Scale Equivariance by training from scratch.\n(The authors only verified SE by fine-tuning the pre-trained VAE)", "relation_to_broader_scientific_literature": "The key contribution of this paper is improving the VAE used in image generation and video generation.\n\nSpecifically, SD-VAE, FluxAE, and CogVideoX-AE are basic components for high-resolution image generation and video generation.", "essential_references_not_discussed": "the references are well discussed", "other_strengths_and_weaknesses": "`Strengths`:\n\n* The paper is easy to follow. The analysis, motivation and method are overall sound and well-organized.\n\n* Experiments are well-designed, and the results are promising.\n\n* The proposed regularization method is potentially useful for the community.\n\n\n`Weaknesses`\n\n* It is not clear why reducing the high-frequency of the latent space can improve the generative modeling. The authors provide the hypothesis but lack sufficient verification. I encourage the authors to explore it from the entropy of the high-freq coefficients in both RGB space and DCT space, where the distribution of high-freq components in the RGB space should have low entropy.\n\n* It is better to show the effect of Scale Equivariance when training the VAE from scratch, as a supplement to fine-tuning.\n\n* It seems not obvious from Figure 5 that Scale Equivariance preserves more content compared to the baseline.\n\n* Presentation-wise: (1) It is better to explain in detail why stronger KL regularization leads to a larger high-freq latent expression (random noise of the latent codes). (2) CosmosTokenizer employs wavelet transform mainly for compression according to their paper; (3) Equation 2 should have a scaler for the term of Scale Equivariance.", "other_comments_or_suggestions": "already stated above", "questions_for_authors": "Q1: In equation 1, $A_{u, v}$ should be <=1 after normalizing by $D_{0, 0}$, but why there are some components still > 1 in Figure 2 and Figure 3?\n\nQ2: Why 'As the number of channels in the autoencoder’s bottleneck increases, high-frequency components become more pronounced'?\n\nQ3: What are the number of sampling steps used in Table 1? what are the results when using different sampling steps?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received initial mixed reviews but 3/4 reviewers increased their scores after the rebuttal, while the one who didn't had recommended acceptance from the start.

Reviewers agree that the paper introduces a novel contribution to an important but unexplored problem with potential practical applications.

There are a few concerns whether the hypothesis about the relation between latent space spectrum and generative performance is valid, but the paper is nonetheless an important step on investigating the connection between latent space properties and generative model behavior.
Decision: Accept (poster)