Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models
Authors: Junyu Chen
Han Cai
Junsong Chen
Enze Xie
Shang Yang
Haotian Tang
Muyang Li
Song Han
Venue: ICLR 2025 Poster (2025)
Keywords: efficient high-resolution diffusion models
Primary area: generative models

Abstract: We present Deep Compression Autoencoder (DC-AE), a new family of autoencoders for accelerating high-resolution diffusion models. Existing autoencodes have demonstrated impressive results at a moderate spatial compression ratio (e.g., 8x), but fail to maintain satisfactory reconstruction accuracy for high spatial compression ratios (e.g., 64x). We address this challenge by introducing two key techniques: (1) Residual Autoencoding, where we design our models to learn residuals based on the space-to-channel transformed features to alleviate the optimization difficulty of high spatial-compression autoencoders; (2) Decoupled High-Resolution Adaptation, an efficient decoupled three-phase training strategy for mitigating the generalization penalty of high spatial-compression autoencoders. With these designs, we improve the autoencoder's spatial compression ratio up to 128 while maintaining the reconstruction quality. Applying our DC-AE to latent diffusion models, we achieve significant speedup without accuracy drop. For example, on ImageNet 512x512, our DC-AE provides 19.1x inference speedup and 17.9x training speedup on H100 GPU for UViT-H while achieving a better FID, compared with the widely used SD-VAE-f8 autoencoder.

Published as a conference paper at ICLR 2025
DEEP COMPRESSION AUTOENCODER FOR EFFICIENT HIGH- RESHURESOOUTION DIFFUSION MODELS
Junyu Chen 1. Han Cai Junsong Chen', Enze Xie Shang Yang', Haotian Tang', Muyang Li', Song Han 1.3 MIT Tsinghua University NVIIDI
https::/ https://githbb....
ABSTRACT
We present Deep Compression Autoencoder (DC-AE), a new family of autoen- coders for accelerating high-resolution diffusion models. Existing autoencoders have demonstrated impressive results at a moderate spatial compression ratio (e.g.. 8x), but fail to maintain satisfactory reconstruction accuracy for high spatial com- pression ratios (e.g.. 64x). We address this challenge by introducing two key tech- niques: (1) Residual Autoencoding, where design models to learn resid- uals based on the space-to-chane -ocennnel transformed features to alleviate the optimiza- tion difficulty of high spatial
spatial-compressii compression autoencoders; (2) Decoupled High- Resolution Adaptation, an efficient decoupled three-phase training strategy for mitigating the generalization penalty of high spatial-compression autoencoders. With these designs, we improve the autoencoder's spatial compression ratio up to 128 while maintaining the reconstruction quality. Applying our DC-AE latent diffusion models, we achieve significant speedup without accuracy drop. For ex- ample, on ImageNet 512 x 512, our DC-AE provides 19.1x inference speedup and 17.9x training speedup on H100 GPU for UViT-H while achieving better FID, compared with the widely used SD-VAE-f8 autoencoder.
INTRODUCTION
Latent diffusion models (Rombach et al., 2022) have emerged as a leading framework and demon- strated great success in image synthesis (Labs, 2024; Esser et al., 2024). They employ an autoen- coder to project the images to the latent space to reduce the cost of diffusion models. For example, the predominantly adopted solution in current latent diffusion models (Rombach al., 2022; Labs, 2024; Esser et al., 2024; Chen et al., 2024b;a) is to use an autoencoder with a spatial compression ratio of (denoted as f8), which converts images of spatial size x W to latent features of spatial W. This spatial compression ratio is satisfactory for low-resolution image synthesis (c.g., size H. x 5
256 X 256). However, for high-resolution image synthesis (c.g., 1024 x 1024), further increasing the spatial compression ratio is critical, especially for diffusion transformer models (Peebles & Xie, 2023; Bao et al., 2023) that have quadratic computational complexity to the number tokens.
The current common practice for further reducing the spatial size is downsampling the diffusion model side. For example, in diffusion transformer models (Peebles & Xie, 2023; Bao et al., 2023), this is achieved by using a patch embedding layer with patch size p that compresses the latent features to H x W. tokens. In contrast, little effort has been made on the autoencoder side. The main Sp 5p
bottleneck hindering the employment of high spatial-compressio autoencoders is the reconstruction accuracy drop. For example, Figure (a) shows the reconstruction results of SD-VAE (Rombach et al., 2022) on ImageNet 256 x 256 with different spatial compression ratios. We can see that the rFID (reconstruction FID) degrades from 0.90 to 28.3 if switching from f8 to f64.
This work presents Deep Compression Autoencoder (DC-AE), new family of high spatial- compression autoencoders for efficient high-resolution image synthesis. By analyzing the un- derlying source of the accuracy degradation between high spatial-compression and low spatial-
Equual contribution. Junyu Chen is an intern at MIT during this work. Project lead. Conrepoondence to: head grort é coom aonghar @mf edu.

---

Published as a conference paper at ICLR 2025
w
w
W/8
W/64
Input Image
SD-VAE f8
Diffusion Model
Input Image
DC-AE f64
Diflusion Model
H
(a) Latent Diffusion Models with SD-VAE
(b) Latent Diffusion Models with DC-AE
Figure
1: DC-AE accelerates diffusion models by increasing autoencoder's spatial compression
ratio.
SD-VAE
DC-AE
SD-VAE-18p8 SD-VAE-132p2
SD-VAE-116p4 DC-AE-164p1
SD-VAE-18p2 DC-AE-164p1
100
64.4
10
28.3
100
95.93
88.06
FAD!
10
76.57
FDJ
19.1x speedup
0.90
FDD
65
3.55
0.1
35.96
3,500
7,000
18
f32
164
f128
30
Inference Throughput (image/s) 1
(c)
Figure 2: (a) Image Reconstruction Results ImageNet 256 256. denotes the spatial compres- sion ratio. When the spatial compression ratio increases, VAE has a significant reconstruction accuracy drop (higher rFID) while DC-AE does not have this issue. (b) ImageNet x12 Image Generation Results on UVIT-S with Various Autoencoders. p denotes the patch size. Shifting the token compression task the autoencoder enables the diffusion model to focus more on the denois- ing task, leading to better FID. (c) Comparison to SD-VAF-f8 on ImageNet 512x 512 with UVIT Variants. DC-AF-f644 provides 19. x higher inference throughput and 0.54 better ImageNet FII) than SD- on UVIT-H.
compression autoencoders, we find high spatial compresion autoencoders are more difficult to op- timize (Section 3.1) and suffer from the generalization penalty across resolutions (Figure 3 b). To this end, we introduce two key techniques to address these two challenges. First, we propose Resid- ual Autoencoding (Figure 4) to alleviate the optimization difficulty of high spatial-compression autoencoders. It introduces extra paraaarreetric shortcuts to the autoencoder to let the neural net- work modules learn residuals based on the space-to-channel operation. Second, propose De- coupled Higg-Reeoluttinn Adaptation (Figure 6) to tackle the other challenge. It introduces high-resolution latent adaptation phase and generalization penalty while maintaining low training cost.
resolutioiin local refinement phase to avoid the
With these techniques, we increase the spatial compression ratio autoencoders 32, 64, and 128 while maintaining good reconstruction accuracy (Table 2). The diffusion models can fully focus on the denoising task with our DC-AE taking over the whole token compression task, which delivers better image generation results than prior approaches (Table 3). For example, replacing SD-VAE-18 with our DC-AE-164, we achieve 17.9 x higher H100 training throughput and 19.1x x higher H100 inference throughput on UViT-H (Bao al., 2023) while improving the ImageNet 512 512 FID from 3.55 to 3.01. We summarize our contributions as follows:
We analyze the challenges of increasing the spatial compression ratio of autoencoders and provide insights into how to address these challenges.
We propose Residual Autoencoding and Decoupled High- Resslutesottion Adaptation that effectively improve the reconstruction accuracy of high Ppatial-compression autoencoders, making their construction accuracy feasible for use in latent diffusion models.
We build DC-AE, a new family of autoencoders based on our techniques. It delivers significant training and inference speedup for diffusion models compared with prior autoencoders.
2 RELATED WORK
Autoencoder for Diffusion Models. Training and evaluating diffusion models directly in high- resolution pixel space results in prohibitive computational costs. To address this issue, Rombach al. (2022) proposes latent diffusion models that operate in compressed latent space produced
2

---

Published as a conference paper at ICLR 2025
Without Residual Autoencoding
18
Without Decoupled High- Resh-Resolttonn Adaptation
With Decoupled High-Resolution
With Residual Autoencoding
132
100
164
Adaptation
7.40
28.3
3
10
(FID
2.04
0.90
0.51 0.49 0.50
0.51
0.49
0.50
0.19
0.180.17 0.20
0.1
0
18
132
164
256x256
1024x1024
1024x1024
(a)
(b)
autoencoders are more difficult to optimize. Even with the same latent shape and stronger learning capacity, it still cannot match the f8 autoencoder's rFID. (b)
Figure 3: (a) High
High spatial-compression autoencoders suffer from significant reconstruction accuracy drops when generalizing from low-resolution to high-resolution.
by pretrained autoencoders. The proposed autoencoder with 8x spatial compression ratio and latent channels has been widely adopted in subsequent works (Peebles & Xie, 2023; Bao et al., 2023). Since then, follow-up works mainly focus on enhancing the reconstruction accuracy of the f8 autoencoder by increasing the number of latent channels (Esser al., 2024; Dai al., 2023; Labs, 2024). Additionally, to improve the reconstruction quality, Zhu et al. (2023) leverages a heavier decoder and incorporates task-specific priors. In contrast to prior works, our work focuses on an orthogonal direction, increasing the spatial compression ratio of the autoencoders (c.g., f64). To the best of our knowledge, our work is the first study in this critical but underexplored direction.
Diffusion Model Acceleration. Diffusion models have been widely used for image generation and showed impressive results (Labs, 2024; Esser et al., 2024). However, diffusion models are computationally intensive, motivating many works to accelerate diffusion models. One representa- tive strategy is reducing the number of inference sampling steps by training-free few-step samplers (Song et al., 2021; Lu et al., 2022a;b; Zheng et al., 2023; Zhang & Chen, 2023; Zhang et al., 2023; Zhao al., 2024b; Shih et al., 2024; Tang et al., 2024) or distilling-based 2023; Salimans & Ho, 2022; Yin et al., 2024b;a; Song et al., 2023; Luo et al., 2023; Liu et al., 2023). Another representative strategy is model compression by leveraging sparsity (Li al., 2022: Ma et al., 2024b) or quantization (He et al., 2024; Fang et al., 2024; Li et al., 2023; Zhao et al., 2024a). Designing efficient diffusion model architectures (Li et al., 2024d; Liu et al., 2024; Cai et al., 2024) or inference systems (Li et al.. 2024b; Wang et al., 2024) is also an effective approach for boosting efficiency. In addition, improving the data quality (Chen et al., 2024b;a) can boost the training efficiency of diffusion models.
methods (Meng et al.,
All these works focus on diffusion models while the autoencoder remains the same. Our work opens up new direction for accelerating diffusion models, which can benefit both training and inference.
METHOD
In this section, we first analyze why existing high ppopial-oompeession autoencoders (e.g., SD-VAE- f64) fail to match the accuracy of low spatial-compression autoencoders (e.g., SD-VAE-I8). Then we introduce our Deep Compression Autoencoder (DC-AE) with Residual Autoencoding and De- coupled Rish--Resolution Adaptation to close the accuracy gap. Finally, we discuss the applications of our DC-AE to latent diffusion models.
3.1 MOTIVATION
We conduct ablation study experiments to get insights into the underlying source of the accuracy gap between high spatial-compression and low coopes--omppesssoon autoencoders. Specifically, consider three settings with gradually increased spatial compression ratio, from f8 to f64.
Each time the spatial compression ratio increases, stack additional encoder and decoder stages upon the current autoencoder. In this way, high spatial-compression autoencoders as sub-networks and thus have higher learning capacity.
autoencoders contain low
3

---

Published as a conference paper at ICLR 2025
Downsample Block
C, 2R
2C, R
Space to Channel
Channel Averaging
2C,
4C, R
(a) DC-AE Downsample Block with Residual Autoencoding
Upsample Block
4C,
Channel to Space
Channel Duplicating
C. 2R
2C, 2R
(b) DC-AE Upsample Block with Residual Autoencoding
Figure 4: Illustration of Residual Autoencoding. It adds non-parametric shortcuts to let the neural network modules learn residuals based on the space-to-champe operation. 'C' denotes the number of channels. 'R' denotes the image size.
Additionally, we increase the latent channel number to maintain the same total latent size across dif- ferent settings. We can then convert the latent a higher spatial compression ratio one by applying a space-to-channel operation (Shi et al., 2016): H x W x C + H W x p²C.
We summarize the results in Figure 3 (a, gray dash line). Even with the same total latent size and stronger learning capacity, we still observe degraded reconstruction accuracy when the spatial compression ratio increases. It demonstrates that the added encoder and decoder stages (consisting of multiple SD- D-E building blocks) work worse than a simple space-to-chanie operation.
Based on this finding, we conjecture the accuracy gap comes from the model learning process: while we have good local optimums the parameter space, the optimization difficulty hinders high spatial-compression autoencoders from reaching such local optimums.
3.2 DEEP COMPRESSION AUTOENCODER
Residual Autoencoding. Motivated by the analysis, we introduce Residual Autoencoding to ad- dress the accuracy gap. The general idea is depicted in Figure 4. The core difference from the conventional design is that we explicitly let neural network modules learn the downsample residu- als based the space-to-channel operation alleviate the optimization difficulty. Different from ResNet (He et al., 2016), the residual here not identity mapping, but space-to-channel mapping.
In practice, this is implemented by adding extra non-parametric shortcuts on the encoder's down- sample blocks and decoder's upsample blocks. Specifically, for the downsample block, the non- parametric shortcut is space-to-channel operation followed by non-parametric channel averaging operation to match the channel number. For example, assuming the downsample block's 's input fea- ture map shape is H W C and its output feature map shape shortcut is:
2C, then the added
HxWXC
space-so-channcl
H
X
W
X
4C
2
splil groups
H
W
X 2C,
H
X
W
x
2C1
H
X 2
W
X 2C.
2
2
2 2
channel overgingg
Accordingly, for the upsample block, the non- non-paramttric shortcut is channel-to-space operation followed by non- non-poramn ta--aarmetricc channel duplicating operation:
H
W X
x 2C
channel space
x W
x
C
2
2
duplicate
C HXWX H HXWX C 2
CANCHi
2
chinnel duplicating

---

Published as a conference paper at ICLR 2025
Without GAN Loss
FID 3.82
Fuiiiii with GAN Loss FiD:
Local Refinement FID 0.69
Figure 5: Autoencoder already learns to reconstruct content and semantics without GAN loss, while GAN loss improves local details and removes local artifacts. We replace the GAN loss full training with lightweight local refinement training which achieves the same goal and has lower training cost.
Reconstruction Loss GAN Loss
Reconstruction Loss
Reconstruction Loss
Decoder
Decoder
B
Decoder
Latent
Latent
Latent
Encoder
Encoder
B
Encoder
Resolution Images
Hiso-Resol Images
Low-Resolution Images
Phase 1: Low-Resolution Full Training
Phase 2: High-Resolution
Phase 3: wewRResotution
Latent Adaptation
Local Refinement
Figure 6: Illustration of Decoupled High-Resolution Adaptation.
In addition to the downsample and upsample blocks, also change the middle stage design fol- lowing the same principle (Figure 10 b, right).
Figure (a) shows the comparison with and without Residual Autoencoding ImageNet 256 x 256. We can see that Residual Autoencoding effectively improves the reconstruction accuracy of high spatial-compression autoencoders.
Decoupled High-Reson ReshluReoottnnn Adaptation. Residual Autoencoding alone can address the accuracy gap when handling low-resolution images. However, when extending to high-resolution images, find it not sufficient. Due to the large cost of high-resolution training, the common practice for high-resolution diffusion models directly using autoencoders trained on low-resolution images (e.g., 256 256) (Chen al., 2024b;a). This strategy works well for low sparial-compression
5

---

Published as a conference paper at ICLR 2025
ImageNet 512 CClassCoonditional
Diffusion Model
Autoencoder
Patch Size
#Tokens
FID (w/o CFG)
FID (w/ CFG) L
SD-VAE-18
8
64
125.08
95.93
SD-VAE-f16
4
64
115.32
88.06
UVIT-S [1]
SD-VAE-(32
2
107.33
76.57
DC-AE-164
1
64
67.30
35.96
Table Ablation Study on Patch Size and Autoencoder's Spatial Compression Ratio.
autoencoders. However, high ppatial-comprestie autoencoders suffer from significant accuracy drop. For example, in Figure (b), we can see that f64 autoencoder's s rFID degrades from 0.50 to 7.40 when generalizing from 256 x 256 to 1024 x 1024. In contrast, the f8 autoencoder's rFID improves from 0.51 to 0.19 under the same setting. Additionally, we also find this issue more severe when using higher spatial compression ratio. In this work, we refer this phenomenon as autoencoders. A straightforward solution to
the generalization penalty of high spatial-compresse address this issue conducting training on high-resolution images. However, suffers from large training cost and unstable high-resolution GAN loss training.
We introduce Decoupled High-Resolutee Adaptation to tackle this challenge. Figure demon- strates the detailed training pipeline. Compared with the conventional single-phase training strategy (Rombach et al., 2022), our Decoupled High-Resolution Adaptation has two key differences.
First, we decouple the GAN loss training from the full model training and introduce dedicated local refinement phase for the GAN loss training. In the local refinement phase (Figure 6, phase 3), we only tune the head layers of the decoder while freezing all the other layers, The intuition of this design is based on the finding that the reconstruction loss alone is sufficient for learning to reconstruct the content and semantics. Meanwhile, the GAN loss mainly improves local details and removes local artifacts (Figure 5). Achieving the same goal of local refinement, only tuning the decoder's head layers has lower training cost and delivers better accuracy than the full training.
Moreover, the decoupling prevents the GAN loss training from changing the latent space. This ap- proach enables conduct the local refinement phase on low-resolution images without worrying about the generalization penalty. This further reduces the training cost of phase and avoids the highly unstable high-resolution GAN loss training.
Second, we introduce an additional high-resolution latent adaptation phase (Figure 6, phase 2) that tunes the middle layers (i.e., encoder's head layers and decoder's input layers) to adapt the latent space for alleviating the generalization penalty. In our experiments, we find only tuning middle layers is sufficient for addressing this issue (Figure b) while having a lower training cost than high-resolution full training (memory cost: 153.98 GB - 67.81 GB)* (Cai et al., 2020).
3.3 APPLICATION TO LATENT DIFFUSION MODELS
Applying our DC-AE to latent diffusion models straightforward. The only hyperparameter to change the patch size (Peebles Xie, 2023). For diffusion transformer models (Peebles & Xie, 2023; Bao et al., 2023), increasing the patch size p is the common approach for reducing the number of tokens. is equivalent first applying the space-to-channel operation reduce the spatial size of the given latent by px and then using the transformer model with patch size of 1.
Since combining a low coatillcompressionn autoencoder (e.g., f8) with the space-to-chin to-channel oper- ation can also achieve a high spatial compression ratio, a natural question is how it compares with directly reaching the target spatial compression ratio with DC-AE
We conduct ablation study experiments and summarize the results in Table 1. We can see that directly reaching the target spatial compression ratio with the autoencoder gives the best results among all settings. In addition, also find that shifting the spatial compression ratio from the diffusion model to the autoencoder consistently leads to better FID.
Assuming the input resolution 1024 1024 and the hatch size is 12.

---

Published as a conference paper at ICLR 2025
ImageNet 256
Latent Shape
Autoencoder
FFID 4
PSNR
SSIM
LPIPS 4
SD-VAE [40]
2.64
22.13
0.59
0.117
f32c32
8x8x32
DC-AE
0.69
23.85
0.66
0.082
SD-VAE [40]
26.65
18.07
0.41
0.283
f64c128
4x4x 128
DC-AE
0.81
23.60
0.65
0.087
ImageNet 512x512
Latent Shape
Autoencoder
FFID 4
PSNR
SSIM
LPIPS 4
SD-VAB [40]
16.84
19.49
0.48
0.282
164c128
8x8x 128
DC-AE
0.22
26.15
0.71
0.080
SD-VAE [40]
100.74
15.90
0.40
0.531
f128c512
4x4x512
DC-AE
0.23
25.73
0.70
0.084
FFHQ 1024x1024
Latent Shape
Autoencoder
rFID
PSNR
SSIM
LPIPS 1
SD-VAR [40]
6.62
24.55
0.68
0.237
164c128
16x 16x 128
DC-AE
0.23
31.04
0.83
0.061
SD-VAE [40]
179.71
18.11
0.63
0.585
f128c512
8x8x512
DC-AE
0.41
31.18
0.83
0.062
Mapillary
Vasiss Vistas 2048 2048 Latent Shape
Autoencoder
rFID
PSNR
SSIM
LPIPS $
SD-VAR [40]
7.55
22.37
0.68
0.262
164c128
32x32x 128
DC-AE
0.36
29.57
0.84
0.075
SD-VAE [40] DC-AE
152.09
17.82
0.67
0.594
f128c512 12
16x 16x512
0.38
29.70
0.84
0.074
Table 2: Image Reconstruction Results.
EXPERIMENTS
4.1
SETUPS
Implementation Details. We use mixture of datasets train autoencoders (baselines and DC-AE), containing ImageNet (Deng et al., 2009), SAM (Kirillov et al., 2023), MapillaryVVstas (Neuhold al., 2017), and FFHQ (Karras et al., 2019). use the ImageNet training split train autoencoders and diffusion models. The model architecture similar to SD-VAE (Rombach et al., 2022) except for our new designs discussed in Section 3.2. In addition, use the original autoencoders instead of the variational autoencoders for our models, as they perform the same in our experiments and the original autoencoders are simpler. We also replace transformer blocks with EfficientVTT blocks (Cai et al., 2023) make autoencoders more friendly for handling high-resolution images while maintaining similar accuracy.
ImageNet experiments, we exclusively
For image generation experiments, apply autoencoders diffusion transformer models including DiT (Peebles & Xie, 2023) and UVIT (Bao et al., 2023). We follow the same training settings as the original papers. Additionally, we build USiT by combining UVIT (Bao al., 2023) with the SiT sampler (Ma et al., 2024a). The SiT and USiT models are trained for 500k iterations with batch size 1024. We consider three settings with different resolutions, including ImageNet (Deng et al., 2009) for 512 x 512 generation, FFHQ (Karras et al., 2019) and MJHQ (Li et al., 2024a) for 1024 x 1024 generation, and Mapillary Vistas (Neuhold al., 2017) for 2048 2048 generation.
Efficiency Profiling. We profile the training and inference throughput on the H100 GPU with PyTorch and TensorRT respectively. The latency is measured on the 3090 GPU with batch size 2. The training memory is profiled using PyTorch, assuming a batch size of 256. use fp16 for all cases.
4.2 IMAGE COMPRESSION AND RECONSTRUCTION
Table 2 summarizes the results of DC-AE and SD- SD-VAE (Rombach et al., 2022) under various settings (f represents the spatial compression ratio and e denotes the number of latent channels). DC-AE provides significant reconstruction accuracy improvements than SD-VAE for all cases. For example,
7

---

Published as a conference paper at ICLR 2025
Diffusion
Patch
Throughput (image/s) 1 Latency Memory Inference Training
FID w/o CFG
Autoencoder
NFE
Model
Size
(ms) 4 (GB)
CFG
Flux-VAE-18 [20]
2
250
54
0.83
7915
56.3
27.35
8.72
Asym-VAE-A [58]
2
250
54
0.85
7686
56.2
11.39
2.97
SD-VAE-8 [40]
250
54
0.85
7686
56.2
12.03
3.04
DIT-XI [38]
DC-AE-132
250
241
4.03
1958
20.9
9.56
2.84
DC-AE-f32
250
241
4.03
1958
20.9
6.88
Flux-Vos VAE-18 [20]]
2
30
55
5.82
913
54.2
30.91
12.63
Asym- VAE-18 [58]
2
30
55
5,85
914
54.1
11.36
3.51
SD-VAE-18 [40]
30
55
5.85
914
54.1
11.04
3.55
UVIT-H [T]
DC-AE-132
30
247
17.9%
27.03
29.1
x
246
18.6
9.83
2.53
-0.54
DC-AE-f64
30
984
111.77
104
10.6
13.96
3.01
DC-AE-164
30
984
111.77
105
10.6
12.26
2.66
Asym-VAE-FE [58]
2
30
27
2.62
2243
OOM
9.87
3.62
SD-VAE-8 [40]
30
27
2.62
2243
OOM
9.73
3.57
UVIT-2B [1]
DC-AE-f32
30
112
11.08
590
42.0
8.13
2.30
DC-AE-f64
30
450
45.55
258
30.2
7.78
DC-AF-f64
30
450
45.55
258
30.2
6.50
2.25
MAGVIT-<2 [51]
3.07
1.91
EDM2-XXL [17]
1.91
1.81
MAR-L [24]
2.74
1.73
SIT-XL [33]
DC-AE-132
241
20.9
7.47
2.41
USiT-H
DC-AB-132
247
18.6
3.80
1.89
USiT-2B
DC-AB-132
112
42.0
2.90
1.72
represents the model is trained for 4x training iterations (i.e., 500K 2,000K iterations). 1 represents the model is trained with 4x batch size (i.c., 256 - 1024). 'NFE' denotes the number of functional evaluations. The NFEs for SiT (Ma al., 2024a) and USiT models are left blank as they use an adaptive-step evaluation scheduler.
Table 3: Class-Conditional Image Generation Results on ImageNet 512x
Diffusion Model
Patch Size
Throughput (image/s) Latenny Memory Training
MJHQ 512x512
Autoencoder
NFE
Inference
(ms)
(GB) 4
FID + CLIP Score f
SD- VAE-18 [40] 2 20
43
7.81
742
60.45
6.3
26.36
PIXART-a
DC-AF-132
20
173
31.27
209
23.77
6.1
26.41
Table 4: Text-to-Image Generation Results.
on ImageNet 512 512, DC-AE improves the rFID from 16.84 to 0.22 for the f64c128 autoencoder
and 100.74 0.23 for the f128c512 autoencoder.
SD-VAE-f8p2
DC-AE-f64p1
In addition to the quantitative results, Figure 7 shows im- age reconstruction samples produced by SD-VAE and DC-AE. Reconstructed images by DC-AE demonstrate better visual quality than SD-VAE's reconstructed images. In particular, for the f64 and f128 autoencoders, DC-AE still maintains good visual quality for small text and the human face.
100
67.30
FID (w/o CFG)
13.96
9.73
10
11.04
7.78
transition point
4.3 LATENT DIFFUSION MODELS
We compare DC-AE with the widely used SD- VAE-18 autoen- coder (Rombach et al., 2022) various diffusion transformer models. For DC-AE, we always use a patch size (denoted pl). For SD- DAE-8,, we follow the common setting and use a patch size of 2 or (denoted as p2, p4). The results are summarized in Table Table 4, and Figure
0.0
0.8
1.6
Billion Params
Figure 9:
Model Scaling Re- sults on ImageNett 512x512 with UVIT. DC-AF-f64 benefits more from scaling up than SD- VAE-f8..
8

---

Published as a conference paper at ICLR 2025
f32
f64
f128
L P E D P E C F D F 2
20/30
L
D
20140
r I F
SD-VAE
20/30
FEL O Z DETFOTNO
20/26
Y FR Y
20/20
D0:100DE
303D
L P E D P E RECED F
L P
P E C
E
D F P
L P P E C F F C
E D
20150
20440
20/00
2040
DC-AE
ED F C Z P
20/30
E
D
F C
20/30
E
D
Z P
20/30
20/00
SD-VAE
DC-AE
Figure Autoencoder Image Reconstruction Samples.
ImageNet 512x512. As shown in Table 3, DC-AE-f32pl consistently delivers better FID than SD-VAE-18p2 on all diffusion transformer models. addition, has fewer tokens than SD- VAE-f8p2, leading to 4.5x higher H100 training throughput and 4.8 higher H100 inference throughput for DiT-XL, We also observe that larger diffusion transformer models seem to benefit more from our DC-AE (Figure 9). For example, DC-AE-f64pl has worse FID than SD-VAL ED-VAE-f8p2 on UViT-S but a better FID on UVIT-2B, We conjecture it is because DC-AE-f64 has larger latent channel number than SD-VAE-f8, thus needing
model capacity (Esser et al., 2024).
Applying DC-AE to USIT models, we achieve highly competitive results compared with prior lead- ing image generative models. For example, DC-AE DC-AE-132+USIT-2 achieves 1.72 FID on ImageNet 512x512, outperforming the SOTA diffusion model EDM2-XXI. and SOTA auto-regressive image generative models (MAGVTT-v2 and MAR-L.).
Text-to-Image Generation. Table 4 reports text-to-imant to-iimgee generation results. All models trained for 100K iterations from scratch. Similar to prior cases, we observe DC-AE-f32pl provides better FID and better CL.IP Score than SD- VA--APPf8 Figure 8 demonstrates samples generated

---

Published as a conference paper at ICLR 2025
DC-AE
aee
aign
dheslyying
pord Lnsty rendenig.
paletee
Pronpe thue Sesket HamDON maca
Promppt: A dose has person, digilll regared Prenpt: Beantfi scene
Han Gaa dicioon oms lemali, victor unmor utaaaniie, beat fadaev
characiar.
WagoNeet timber will woll, Holf, Canlo kpuu
ImageMett 538 come
Inagehot 140 nod scokid sandupor, CUri Erila AbTO
Figure 8: Images Generated by Diffusion Model using Our DC-AF.
by the diffusion models with our DC-AE, showing the capacity to synthesize high-quality images while being significantly more efficient than prior models.
5 CONCLUSION
We accelerate high-resolution diffusion models by designing deep compression autoencoders to reduce the number of tokens. We proposed two techniques: residual autoencoding and decoupled high-resolution adaptation address the challenges brought by the high compression ratio. The resulting new autoencoder family DC-AE demonstrated satisfactory reconstruction accuracy with spatial compression ratio of up to 128. DC-AE also demonstrated significant training and inference efficiency improvements when applied to latent diffusion models.
ACKNOWLEDGEMENTS
We thank NVIDIA for donating the DGX machines. We thank MIT-IBM Watson AI Lab, MIT and Amazon Science Hub, MIT Al Hardware Program, and National Science Foundation for supporting this research.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 22669-22679, 2023, 1, 2, 6, 7, 8, 16, 18
Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. Advances in Neural Information Processing Systems, 33:11285- 11297, 2020.
Han Cai, Junyan Li, Muyan Hu, Chuang Gan, and Song Han. Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 17302-17315 20233,
Han Cai, Muyang Li, Qinsheng Zhang, Ming-Yu Yu Liu, and Song Han. Condition-aware neural net- work for controlled image generation. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7194-7203, 2024.
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Weak-to-strong training diffusion transformer for 4k text-to-image generation. arXiv preprini arXiv:2403 04692, 2024a. 1, 3, 16
Junsong Chen, YU Jincheng. GE Chongjian, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In International Conference on Learning Representations, 2024b. 1, 5, 16
Xiaoliang Dai, Ji Hou, Chih- Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint woXiv:23091188/0 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024. 1, 3, 9, 16, 17, 18
Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016. 4
Yefei He, Luping Liu, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Ptqd: Accurate post- training quantization for diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020. 16
Phillip Isola, Yan-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattemn recognition, pp. 1125-115 1134, 2017. 15
Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards better evaluation metric for image generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9307- 9315, 2024. 17
11

---

Published as a conference paper at ICLR 2025
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019. 7
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyz- ing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24174-24184 2024. 8, 18
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015 4026, 2023.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019. 17
Black Forest Labs.
Online, 2024. black-Forest labs/ flux. 3, 16, 17, 18
Flux.
URL https: com/
Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground
v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402. 17245, 2024a. 7, 21
Muyang Li, Ji Lin, Chenlin Meng, Stefano Ermon, Song Han, and Jun- Yan Zhu. Efficient spa- tially sparse inference for conditional gans and diffusion models. Advances in neural information processing systems,
2022. 3
Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Kai Li, and Song Han. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 7183-7193, 2024b.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng. and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406. 18388, 2024c. 8, 18
Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7533-115455 2023. 3
Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang. Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds. Advances in Neural Information Processing Systems, 36, 2024d.
Songhua Liu, Weihao Yu, Zhenxiong Tan, and Xinchao Wang. Linfusion: 1 gpu, minute, 16k image. arXiv preprint arXiv:2409.02097, 2024. 3
Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. In The Twelfth International Conference on Learning Representations, 2023. 3
I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv: 2017. 15
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022a. 3, 16
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arX1v:2211.01095, 2022b.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthe- sizing high-resolution images with few-step inference. arXiy preprint arXiv:2310.04378, 2023.
12

---

Published as a conference paper at ICLR 2025
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden- and Sain- ing Xie. Sit: Exploring flow and diffusion- based generative models with scalable interpolant transformers. arXiv preprint arXiv:2401.088 404000000040, 2024a. 7, 8, 18
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepeache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15762-15772, 2024b. 3
Heusel Martin, Ramsauer Hubert, Unterthiner Thomas, Nessler Bernhard, and Hochreiter Sepp. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30:6626-661 2017. 17
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14297-14306, 2023. 3
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEEE international conference on computer vision, pp. 4990-4999, 2017. 7
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 2023. 1, 6, 7, 8, 16, 17, 18
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 17, 18
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022. 2, 6, 7, 8, 16, 17, 18
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference Learning Representations, 2022. 3
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016. 17
Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1874-1883, 2016. 4
Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, and Nima Anari. Parallel sampling of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, pp. 3221 132222. PMLR, 2023.
Zhiwei Tang, Jiasheng Tang, Hao Luo, Fan Wang, and Tsung Huing-HiiL Chang. Accelerating parallel sampling of diffusion models. In Forty-first International Conference on Machine Learning, 2024.
Jiannan Wang, Jiarui Fang, Aoyu Li, and PengCheng Yang. Pipefusion: Displaced patch pipeline parallelism for inference of diffusion transformer models. arXiv preprint arXiv:2405. 14430, 2024. 3
13

---

Published as a conference paper at ICLR 2025
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T Freeman. Improved distribution matching distillation for fast image synthesis. arXiv preprint ar 47Xiv:2405.14867, 2024a.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6613 6623, 2024b.
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023. 8, 18
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In The Eleventh International Conference on Learning Representations, 2023. 3
Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit models. In International Conference on Learning Representations, 2023.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings the IEEE conference computer vision and pattern recognition, pp. 586-595, 2018. 15
Tianchen Zhao, Tongcheng Fang, Enshu Liu, Wan Rui, Widyadewi Soedarmadji, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, Huazhong Yang, al. Vidit-q: Efficient and accurate quantiza- tion of diffusion transformers for image and video generation. arXiv preprint arXiv:2406.0255 02540, 2024a.
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: unified predictor- corrector framework for fast sampling of diffusion models. Advances in Neural Information Processing Systems, 36, 2024b. 3
Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Dpm-solver-v3: Improved diffusion ode solver with empirical model statistics. Advances in Neural Information Processing Systems, 36: 55502-55542, 2023. 3
Zixin Zhu, Xuelu Feng, Dongdong Chen, Jianmin Bao, Le Wang, Yinpeng Chen, Lu Yuan, and Gang Hua. Designing better asymmetric vqgan for stablediffusion. arXiv preprint arXiv:2306. 04632, 2023. 3, 8, 17, 18
14

---

Published as a conference paper at ICLR 2025
A
DC-AE ARCHITECTURE AND TRAINING DETAILS
Encoder Steges
Group Norm
SILU
3x3 Cony
1x1 Conv
Latent (H,W332)
1x1 Conv
Sx3 Conv
Decoder Stages
SD-VAE Model Architecture
Downsample Block
Space to Channel
Encoder Stages
3x3
Latent (H,W)3))
3x3 Channel Duplicating H,W, 1024
Decodir Stages
Upsample Block
Channel Averaging
Chennel-to-Space
1,241024
H,W,32
H,W,32
Architecture Residual Autoencoding
256 250 x
612 120
512x 64
x x
1084 16
DCAE Downsample
DCAE Downsample
DCAE Downsomple
DCAE Downsample
Input Stem
ResNetBlock
ReeNerBBook
ErficienVVIT Block
ErficientVIT Block
EfficientVT Blook
S
Inpur
(c) DC-AE Encoder Stages
64
258
EfficlentiI Block
DCAE Upsample
EtficlentVI
DCAE Upsample
EfticlentvIT
DCAE Upsample
ResNetBlook
DCAE Upsample
ResNetBlock
Output Head
3x622x 512
Block
Block
1
Lys
(d) DC-AE
Stages
Figure 10: Detailed Architecture of SD-VAE, DC-AE, DC-AE Encoder, and DC-AE Decoder Stages.
We present the detailed architecture of SD-VAE, DC-AE, DC-AE encoder, and DC-AE decoder stages in Figure 10 to complement Figure 4.
We use the AdamW optimizer (Loshchilov, 2017) 7) for all training phases.
constant learning rate of 6.4e-5 with weight decay of 0.1, and Adam betas of (0.9, 0.999). We use LI loss and LPIPS loss (Zhang et al., 2018).
In phase (low-resolution full training).
In phase 2 (high-resolution latent adaptation), decay of 0.001, and AdamW betas of (0.9, 0.999). We use the same loss as phase 1.
constant learning rate of 1.6e-5, weight
In phase 3 (low-resolution local refinement), betas of (0.5, 0.9). We use L1 loss, LPIPS loss (Zhang et al., 2018), and PatchGAN loss (Isola et al., 2017).
constant learning rate of 5.4e-5, and AdamW
B ABLATION STUDY ON TRAINING DIFFERENT NUMBERS OF LAYERS
Figure 11 presents the ablation study training different numbers of layers in phase 2 (high- resolution latent adaptation) and phase 3 (low-resolution local refinement).
C ADDITIONAL IMAGE RECONSTRUCTION RESULTS
Table reports the reconstruction results under the low spatial-compression ratio setting, DC-AE delivers slightly better results than VAE under this setting.
15

---

Published as a conference paper at ICLR 2025
27.8
27.8
1.5
1.5
27.7
27.7
(FDJ
Train Full
Natwork
27.6
Train Ful Nenwor
Train Rul
Network 0.5
Train Fu
Networ
0.6
30
Training Step Time (6)
Trairing Memory (GB)
Trairing Step Time (e)
Trairing Memory (GB) 1
Figure 11: Ablation Study Training Different Numbers of Layers in Phase 2 (Left) and Phase 3 (Right).
ImageNet 256
Latent Shape
Autoencoder [FII
PSNR
SSIM ? LPIPS
SD-VAE [40]
0.63
24.99
0.71
0.063
18c4
32x32x4
DC-AE
0.46
25.46
0.73
0.057
Table 5: Image Reconstruction Results under the Low Spatial-Compressiee Ratio Setting.
D LATENT SCALING AND SHIFTING FACTORS
Following the common practice (Rombach al., 2022; Peebles Xie, 2023; Bao et al., 2023; Esser et al., 2024; Labs, 2024; Chen et al., 2024b;a), we normalize the latent space of our autoencoders to apply to latent diffusion models. Given a dataset, we compute the root mean square of the latent features and use its multiplicative inverse as the scaling factor for our autoencoders. We do not use shifting factor for our autoencoders.
E DIFFUSION MODEL ARCHITECTURE DETAILS
In addition to existing UVIT models, scaled the model up to 1.6B parameters, with depth of 28, a hidden dimension of 2048, and 32 heads. We denote this model as UViT-2B.
F DIFFUSION SAMPLING HYPERPARAMETERS
For the DiT models, we use the DDPM (Ho et al., 2020) sampler from the DiT (Peebles Xic, 2023) codebase with 250 sampling steps and guidance scale of 1.3.
For the UVIT models, we use the DPMSolver (Lu et al., 2022a) sampler with 30 sampling steps and guidance scale of 1.5.
18p2 DC- AE rsop1 DIT-XL
SD-VAE-IBp2
VAE-18p2 DC AE-AEE 15ep1 DIT-XL
VAE-VAP-pp UMITH DC AE #64p1t UMTH
DC-AE 164p1t UWITH
FID1
FDL
2
2
1.3
1.4
1.5
1.6
1.4
1.5
1.8
1.7
10
20
30
40
50
10
20
30
40
50
Scale
Sampling Stops
Sampling Staps
Figure 12: Ablation Study on Diffussin Sampling Hyperparameters. We use the DPMSolver sampler for both DIT-XL and UViT-H. DC-AE provides significant speedup over the baseline mod- els while maintaining the generation performance under different diffusion sampling hyperparame-
ters.
G HEGOLRISN IMAGE GENERATION RESULTS
Apart from ImageNet 512x512, also test models for sigherrresolution image generation. As shown Table 6, we have similar finding where DC-AE-f32pl achieves better FID than SD-VAE- f8p2 for all cases.
16

---

Published as a conference paper at ICLR 2025
FFHQ 1024x 1024 (Unconditional) & MJHQ 1024x 1024 (Clas-Condiitinal))
Diffusion
Patch Size
Throughput (imagels)
Latency Memory (ms) 4 (GB) +
FFHQ FID
MJHQ FID
Autoencoder
NFE
Model
Training
Inference
w/o CFG
w/o CFG w/ CPG
SD3-VAE-18 [9]
2
250
83
1.63
3554
41.4
46.28
109.43
103.02
Flux-VAE-f8 [20]
2
250
83
1.63
3554
41.4
59.15
143.16
139.06
SDXL-VAB [39]
250
1.67
3530
41.2
49.00
39.21
Asym-VAB-R VAE-f8 [58)
250
1.67
3530
41.2
17.10
48.30
38.35
2
250
84
1.67
3530
41.2
16.98
48.05
38,19
DiT-S [38]
SD- VAE-18 [40]
250
470
11.13
632
10.7
23.81
60.94
51,29
DC-AR--32
250
475
11.15
634
10.7
13.65
34.35
27.20
DC-AF-1324
250
11.15
10.7
11.39
28.36
21.89
DC-AB-164
250
2085
50.26
230
26.88
61.30
53.38
Mapillary Vistas 2048 x 2048 (Unconditional)
Diffusion
Patch Size
Throughput (image/s) f ?|Latency Memory
Mapillary Vistas + w/o CFG
Autoencoder
NFE
Model
Training
Inference
(ms)
(GB)
SD- VAE-f8 [40]
4
250
84
1.64
3561
41.4
69.50
DÌT-S [38]
DC-AB-164
250
459
10.91
639
11.0
59.55
Table 1024 1024 and 2048x2044 Image Generation Results. $ represents the model trained with 4x batch size (i.e., 256
1024).
H IMAGE GENERATION RESULTS WITH OTHER EVALUATION METRICS
Table 7 presents a comprehensive evaluation of different diffusion models and autoencoders on Im- ageNet 512x512. The evaluation metrics include FID (Martin et al., 2017), inception score (IS) (Salimans et al., 2016), precision, recall (Kynkäänniemi et al., 2024). Our DC-AF consistently delivers significant efficiency improvements while maintain- ing the generation performance under different evaluation metrics.
al., 2019), and CMMD (Jayasumana
I ADDITIONAL SAMPLES
In Figure and 14, we provide additional image reconstruction samples produced by SD-VAE and DC-AE. Reconstructed images by DC-AE demonstrate better visual qualities than SD-VAE's reconstructed images, especially for the f64 and f128 autoencoders. Some samples are cropped for better visualization of details like human faces and small texts.
In Figure 15 and Figure 16, we show randomly generated samples on ImageNet 512x512 and MJHQ-30K 512x512 by the diffusion models using DC-AE.
17

---

Published as a conference paper at ICLR 2025
Diffasion Model
Autoencoder
Panch
NFE
Inferenze Throughput
FID wio CPG
Inception Score weo CPG
CPG
Piecision 1
CPG w/ WFG
CPG
Kecall f
w/CPG
CMMD I
w/CKG
[9]
FAuX-VAF-88 [20]
49.73
106.07
0.37
1.90
SDXI -VA--8 [30]
49.85
51.03
26.38
27:58
56.72
0.57
0.74
0.58
0.50
1.35
1.05
Asym: VAE-f8 [S]]
30
09.85
52.68
25.14
30.22
65:27
0.58
0.74
0.62
0.51
1.09
0.80
SD-VAE-IR [40]
49.85
$1.96
24.57
30.37
65.73
0.57
0.74
0.64
0.52
1.23
0.91
(1]
VAE-F16 140]
214.68
76.36
21.38
43.35
0.55
1.83
23.07
DC-AF-R2
214:17
46
0.76
0.56
1,00
DC-AE-164
30
806.23
67.
35.96
24.55
52.55 55
0.44
0.64
0.60
0.56
1.44
1.14
DC-AE-164
896.23
61,84
30.63
27.78
61.76
0.47
0.67
0.63
0.56
1.35
1.04
Fux VAB 48 [20]
2
250
0.83
27.35
8.72
53.09
130.20
0.68
0.83
0.61
0.48
0.54
0,30
Asy=-VAF-18 [S]]
250
0.85
11.30
2.97
108.70
10
0.75
0.83
0.65
0.53
0.37
0.20
SD-VAE-IX
12.08
[XT-XI,
DC-AE-122
250
4.03
6.88
2,41
141.07
263 56
0.76
0.82
0.63
0.56
0.29
0.18
VAX--V [X]]
3.82
30.91
12.63
56.72
127.93
0.76
0.49
0.50
AAy--VAE-8 [58]
30
3.85
11.36
3.51
124.24
249.21
0.75
0.83
0.61
0.53
0.32
0.20
LITHE
SD VAE:8 (40]
30
5.85
11.04
3.55
125.08
250.66
0.75
0.82
0.61
0.53
0.39
0.26
DC-AE-B2
27.03
9.83
121.91
255.07
0.83
0.54
BC-AE-154
111.77
13.96
99.30
229.16
DC-AB-164"
111.77
12:26
2.65
109.20
0.73
0.82
0.67
0.57
0.43
0.27
[20)
Asyas VAE-4 [S]|
2.62
9.87
3.62
131.95
258.63
0.76
0.83
0.59
0.52
0.30
0.19
SD-VAF-18 [4i]
30
2,62
9.73
3.57
132.86
260.50
0.76
0.83
0.59
0.52
0.37
0.24
UVIT-28 [1]
DC-AF-122
30
11.08
8.13
2.30
135. 44
272.73
0.76
0.83
0.66
0.56
0.30
0.17
DC-AE-161
43.55
7.78
2,47
138.11
280.49
0.77
0.81
0.54
0.35
0.22
DC-AF-164
45.55
6.50
152.35
293.45
0.77
0.83
0.56
0.31
MAGVIT-<2 [SI]]
3.07
213.1
324.3
EDMP-XXI.117]
1.91
MAR-L(224]
2.74
205.2
279.9
SIT-L [33]
AC-
7,47
2,41
131.37
237.71
0.77
0.82
0.65
0.58
0.36
0.23
USIT-H
DC-AE-02
3,80
1.89
174.55
252.35
0.78
0.82
0.64
0.60
0.24
0.18
UB
DC-AE-B2
2,90
1.72
187.68
248.10
0.82
0.61
0.17
Table Class-Conditional Image Generation Results on Image 512x512 with More Eval-
uation Metrics. represents the model is trained for 4x training iterations (i.e., 500K - 2,000K iterations). $ represents the model trained with 4x batch size (i.e., 256 > 1024). 'NFE' denotes the number of functional evaluations. The NFEs for SiT (Ma et al., 2024a) and USIT models are left blank as they use an adaptive-step evaluation scheduler.
18

---

Published as a conference paper at ICLR 2025
f32
f64
f128
SD-VAE
DC-AE
ИИПИады Жельсинии. 19 NOmVDUU Drunmn !
SD-VAE
импиадьь. Жельсинки. 19 ИИПИады. Xельсинки. 19 ИИпИадь Жельсинки. 19
DC-AE
Figure 13: Additional Autoencoder Image Reconstruction Samples.
19

---

Published as a conference paper at ICLR 2025
f32
f64
f128
SD-VAE
DC-AE
ENYGRPRISE ORBECE
SD-VAE
I TUALE
ENTERPRISE GREECE
ENTERPRISE GREECE
ENTERPRISE GREECE
DC-AE
INVEST &
INVEST &
INVEST
Figure 14: Additional Autoencoder Image Reconstruction Samples.
20

---

Published as a conference paper at ICLR 2025
Homy
vodlder Der to. frandur biakiio
dggs bateer yatl secc nauers lighing
bosk and dee talos can te daph mella
envinement mneccaing sammirs lnveran pien cade Mevadaree to bec ot wis ano highod
cnd deul
dig
acimsaii Eynk and aiac as tore Scaar
PDR Mar
unliace calyy to
con
the oldeaangee high wakkton tha (gt oftt and ligh amarii
Due vite pod
Karrec weach vo decusic hears and
Ploepe Mooue Ladggian
Figure 15: Random 512x512 Text-to-Image Samples. Prompts randomly drawn from MJHQ- 30K (Li et al., 2024a).
21

---

Published as a conference paper at ICLR 2025
3
100
5
REEEEE
IEOCINQ
HHEA
Figure 16: Random Generated Samples on ImageNet 512x512.
22


Reviews:
[{"review_id": "oweKmsgNXI", "reviewer": "Reviewer_jBHL", "summary": "This paper presents a family of autoencoders that achieve comparable or better reconstructions compared to the SD autoencoders while having a much stronger spatial compression rate, resulting in efficiency gains for downstream tasks such as latent diffusion training.\nUsing the techniques or residual autoencoding and decoupling high-resolution adaptation, the model is able to generalize well to higher resolutions while having a very large compression factor (x32, 64 vs 8 for SD-VAE).", "questions": "* While increasing the patch size is one method to reduce spatial resolution, one can also use a pixel shuffle operation in order to achieve a lower spatial dimension with a higher channel dimension. How does this simple operation, using SD-VAE compare with your method ?\n* Training LDMs requires a shift and scale factor to be applied to the latents before the diffusion process, how are these values computed in your experiments ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The paper is well written and easy to follow.\n* The method allows to significantly reduce the computational requirements to train large scale diffusion models at high resolution, making it more accessible as a research topic.\n* Evaluation of multiple metrics across 4 different datasets are reported.\n* Evaluations on downstream diffusion training is also provided.\n* Qualitative examples clearly showcase the improvements brought about by the proposed model.", "weaknesses": "* What parameters were used for sampling the diffusion models (sampler, number of sampling steps, guidance scale) ? A more through investigation on the impact on sampling quality would be useful to get a better grasp of the limitations of this method.\n* Missing ablations on the constituent parts of DC-AE.\n* In tables 3 and 4, missing comparisons with more recent autoencoders such as the one from SD-XL, SD3, and Asymetric Autoencoder.\n* In table 5. the PixArt model trained with DC-AE achieves a lower CLIP score than the one trained with SD-VAE while the text says otherwise (maybe a typo?). What if the memory usage is equalized with the SD-VAE by increasing the batch size, do the improvements get larger or do they stagnate ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XKBhV9D9xX", "reviewer": "Reviewer_rPFh", "summary": "This paper looks at the problem of scaling image autoencoders, used with generative models, to higher spatial compression rates without reducing visual quality. The authors identify a problem with existing AEs: visual quality degrades with higher spatial compression rates (Fig. 2) even when increasing the number of channels in the latent representation to maintain a fixed total size (see the \"latent shape\" column in Table 2). They also observe a quality degradation when generalizing from low resolutions (used for training) to higher-resolutions.\n\nThese observations imply that \"high spatial-compression autoencoders are more difficult to optimize\" so the paper introduces an architecture change (\"residual autoencoding\") and a training change (\"decoupled high-resolution adaptation\") to improve results with high spatial compression rates and to reduce generalization error across resolutions.\n\nResidual autoencoding (Fig. 4) replaces strided conv blocks used for up/downscaling with residual blocks that learn a residual added to space-to-depth (for downscaling) or space-to-depth (for upscaling) blocks. The model uses channel averaging to adjust the number of channels when there is a mismatch, and authors also use channel averaging for a new kind of residual block even when there is no up/downscaling.\n\nDecoupled high-resolution adaptation (DHRA) switched from a single-stage approach for training the AE to a three-stage appraoch (Fig. 6). In the standard approach, the AE is trained on low-res data using all reconstruction losses (typically mse + perceptual + adversarial). In DHRA, the stages are:\n 1. low-res images optimized without the adversarial loss\n 2. finetune with high-res images with only the \"inner\" layers (end of encoder, beginning of the decoder) trained. Other layers are frozen and, again, no adversarial loss is used.\n 3. finetune again with reconstruction and adversarial losses. Only the final layers of the decoder are trained.\n\nThe authors argue that this multi-stage approach is more efficient than training the full model on high-res images, and it prevents the adversarial loss from affecting the encoder which helps with generalization.\n\nThe authors validate their claims by training models and evaluating reconstruction quality (Table 2) and generation quality (Tables 3 and 4). They also provide some example images for qualitative evaluation (Fig. 7 and in the supplementary material).", "questions": "Regarding generalization across resolutions, I'm wondering how much changes to the training procedure can close the gap even with single-stage training. In particular, it makes sense that the statistics for a fixed-size patch (say 32x32 pixels) is very different between a 256x256 image and 1024x1024 pixel image of the same scene. You could train on 256x256 *crops* from a 1024x1024 dataset (rather than downscaling), but then you'd have a generalization gap in the other direction.\n\nIn other work, researchers try to avoid this problem by generating low-res training sets by randomly downscaling high-res sets. With enough training, this approach lets the model see the full range of statistics. Was this done for the experiments in the DC-AE paper? If not, do you think it's sufficient or perhaps complementary to DHRA? Either way, I still see potential benefits of DHRA, e.g., maybe it's good that the GAN doesn't influence the encoder (and thus the latent space), but this isn't investigated, in isolation, in the paper.\n\nYou can also frame higher spatial-compression rates as meaning that the model sees less training data (as measured by number of latents) per fixed-size image. This means that any negative affects from the boundary are exacerbated (e.g., if zero-padding is used for conv layers). In the extreme, imagine using f256 with 256x256 training data -- we certainly wouldn't expect this model to generalize to higher resolutions. Some discussion of these effects would help, and investigation of whether the generalization issue is mostly one of different pixel statistics or of boundary artifacts would strengthen the paper.\n\nDid you experiment training different numbers of layers in Stage 2 and Stage 3? A graph showing the impact (training speed vs. evaluation metrics) as the number of layers increases would be interesting.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The primary strength of this paper is the development of an effective and relatively simple solution to a real-world deficiency that impacts all research and products using autoencoders. Specifically, the use of residual autoencoding and DHRA allow either better visual quality (e.g., in Table 3 look at rows witht he same number of tokens like SD-VAE-f16 with patch-size=2 or SD-VAE-f32 with patch-size=1 to DC-AE-f32) or similar visual quality with lower latency and higher throughput (e.g., Table 3 comparing SD-VAE-f32 to DC-AE-f64).\n\nAnother strength is that the authors evaluate their method on both reconstruction and generation problems. Improvements on just reconstruction still have some value, but many improvements to reconstruction quality do not translate to gains on the generation side.", "weaknesses": "A more detailed summary of the DC-AE architecture would help. Fig. 4 provides a high-level overview, but I'm not sure how literal or complete it is, and the details of the \"Encoder/Decoder Stages\" are not provided. To be fair, the authors do provide code, but architecture and training details should be in the paper (probably in the appendix).\n\nThe conjecture in Section 3.3 that DC-AE helps because otherwise the \"diffusion model needs to simultaneously learn denoising and token compression when using a patch size > 1\" is pretty hand-wavy. For example, I can always set patch_size=1 and add add a space-to-depth layer to the encoder and claim that now the encoder alone is responsible for token compression. That said, I don't have a better explanation, but I think it's ok for papers to accept that the contribution is a better architecture justified with empirical evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jNm3KoHdv3", "reviewer": "Reviewer_9DQC", "summary": "The paper presents a new type of autoencoder for high resolution generative diffusion models. Autoencoders are used in generative diffusion models primarily to reduce the computation necessary for training and evaluation. However, there is some loss of representation capacity in mapping the image to the latent domain. The loss of representation capacity becomes more severe with higher downsampling ratios, which limits compute gains beyond those provided by 8X downsampling, as well as high-resolution generation.\n\nThe present paper aims to expand the possible range of downsampling ratios to as high as 64X. The paper achieves this first by analyzing the challenges of higher downsampling ratios, finding that staged architectures actually have worse performance than architectures with simple space-to-channel operations. Following this, the paper proposes a new architecture with residual space-to-channel and channel-to-space connections that effectively skip the information across network blocks, improving optimization High-resolution performance after adopting the residual space-to-channel operations still left some to be desired, so the paper further proposes a decoupled high-resolution adaptation procedure that finetunes the head of the decoder while finetuning the rest of the model. A second stage tunes the middle layers.\n\nIn numerical experiments the trained autoencoders exhibit better reconstruction accuracy than their counterparts from stable diffusion, which are commonly used in the community. The full diffusion models show promising FID numbers vs. stable diffusion with more efficient training in both quantitative and qualitative examples.", "questions": "My overall rating is due to the somewhat limited set of modifications and the limited experiments, which should be a topic for rebuttal. In addition, I also have the following questions:\n\n1. Did you adopt all the same loss functions and GAN architectures as in the stable diffusion paper?\n2. How often does the proposed architecture output artifacts?\n3. Are the images in all figures random-samples or selected? This should be made clear in the text.\n4. Did you consider other operations for space-to-channel other than averaging? Same question for channel duplication in the upsampling process.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The baselines using stable diffusion are well-known and recognized by the community.\n3. The qualitative examples seem compelling.\n4. The advantages with using less compute could be of significant impact.", "weaknesses": "1. The authors mostly use stable diffusion for all experiments, although several datasets are considered.\n2. A more detailed quantitative analysis of why standard staged training has difficulties with gradient propagation is not presented.\n3. The proposed modifications are somewhat incremental - this seems close to simply removing some skip connection convolutions + a reshape operation in the standard stable diffusion autoencoder. The extra fine-tuning procedure is more detailed, but is only necessary due to the autoencoder changes.\n4. Autoencoder comparisons in Table 2 are only done at high downsampling ratios - not sure how the proposed architecture performs with shallow downsamples.\n5. Only the FID metric (presumably with the Inception V3 backbone) is used to evaluate the models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y2eSXfMhtF", "reviewer": "Reviewer_pFgo", "summary": "The paper proposes a new VAE architecture for latent diffusion generative modeling along with a multi-step training recipe. More specifically, the proposed design combines space-to-channels with averaging to build deep autoencoders with a high spatial subsampling factor (up to 64x). In order to train these models effectively at high resolution, they are first trained at low resolution without GAN loss, then partially tuned at higher resolution, and then refined with a GAN loss. The more aggressive subsampling leads to lower spatial resolution/sequence length in the latent space than VAEs from the literature and hence leads to substantial speed-ups of about 4x to 16x while maintaining generation quality, in combination with different latent diffusion models.", "questions": "- What is the role of the averaging compared to simple space to channel? How would the model behave without the averaging? Would the channel dimension grow too fast?\n- How does the use of EfficientViT blocks affect the quality (reconstruction/generation) of the method? Does it lead to better behavior when trained with large subsampling factors compared to the baseline?\n- “..., we assume the number of sampling steps is 1.”: Does this also hold for inference throughput and latency? If so, I would suggest reporting the throughput/latency to sample a full image.\n- What is the resolution of the images in Figure 7? It would generally be helpful to indicate the resolution for the visual examples in the caption.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed architectural modifications are simple and sound, and the corresponding multi-step training recipe is relatively simple as well.\n- The speed-ups are quite substantial.\n- The method is tested on a broad range of data sets (including text-to-image modeling), and in combination with different latent diffusion models. So the results are quite comprehensive.", "weaknesses": "- The role of the space-to-channels-based autoencoder design and the multi-step training recipe are entangled. How does the baseline autoencoder (with more aggressive subsampling) perform, when it is trained with the multi-step recipe?\n- The paper seems to lack important experimental details, including optimizer, learning rate, schedule etc. These are important in particular given the multi-stage training with different losses (including a GAN loss). Also, what losses does the method use exactly? It looks like the low resolution stage uses a perceptual loss (Figure 5). \n\n\\\nMinor:\n- In the side-by-side comparisons it would be useful to have the original image as additional reference.\n- Some prior works discuss the relevance of space-to-channel in image enhancement/generation methods, probably worth mentioning (see e.g. Shi et al. \"Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.\" CVPR 2016)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q5midK4LKM", "reviewer": "Reviewer_iQFp", "summary": "The submission presents Deep Compression Autoencoder (DC-AE), a new series of autoencoders that enable highly efficient high-resolution image synthesis with latent diffusion models. The key architecture design is the adding residual connection with space-channel inter-play in both downsampling and upsampling layers, which alleviates the optimization difficulty significantly. They further propose decoupled high-resolution adaptation, a three-phase training strategy that mitigates the generalization penalty of high spatial-compression autoencoders. By improving the autoencoder's spatial compression ratio up to 128 while maintaining reconstruction quality, DC-AE provides significant speedups for both training and inference of latent diffusion models without sacrificing image generation accuracy. The authors demonstrate the effectiveness of DC-AE on various datasets and diffusion transformer architectures.", "questions": "-\tCould you please clarify the experimental settings and AE architectures in Section 3.1? By Line 161 ‘… convert the latent to higher spatial compression…’, do you mean that the f8 to f64 AEs have the same output latent size initially, yet f32 and f64 AEs further squeeze the space dimension into channel dimension?\n-\tIn Table 5, the proposed method shows a lower CLIP score with the Pixart diffusion transformer, is it a typo?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "-\tNovel autoencoder design for high compression ratios. The authors introduce additional non-parametric residual connections in the downsampling and upsampling layers, which facilitate the optimization of autoencoders in high spatial compression ratio scenarios.\n-\tThe authors carefully design a three-stage training recipe to mitigate the high-resolution generalization gaps by fine-tuning the trained AE on high-resolution data with minimal additional costs.\n-\tThe proposed DC-AEs show a substantial reduction in training and inference costs of latent diffusion models on various datasets.", "weaknesses": "-\tAlthough the VAE shows little FID loss as it progresses to a higher spatial compression ratio (DC-AE-f32 to DC-AE-f64), it does not translate to similar results in the corresponding latent diffusion models as shown in Table 3. The underlying reason is probably that a diffusion model with higher capacity is required when dealing with high compression ratio latents.\n-\tThe current submission lacks either theoretical analysis or illustrative toy examples on the proposed design choice of space-channel residual connection, e.g., comparisons on the loss landscapes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)