PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher
Authors: Dongjun Kim
Chieh-Hsin Lai
Wei-Hsiang Liao
Yuhta Takida
Naoki Murata
Toshimitsu Uesaka
Yuki Mitsufuji
Stefano Ermon
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Models; Distillation Models; Generative Models; Generative AI; Text-to-Image Generation; Generative Adversarial Networks (GAN); Variational Autoencoders (VAE)
Primary area: generative_models
TL;DR: We train one-step text-to-image generator that is progressively growing in its resolution. For that, we only need low-resolution diffusion models.

Abstract: The diffusion model performs remarkable in generating high-dimensional content but is computationally intensive, especially during training. We propose Progressive Growing of Diffusion Autoencoder (PaGoDA), a novel pipeline that reduces the training costs through three stages: training diffusion on downsampled data, distilling the pretrained diffusion, and progressive super-resolution. With the proposed pipeline, PaGoDA achieves a $64\times$ reduced cost in training its diffusion model on $8\times$ downsampled data; while at the inference, with the single-step, it performs state-of-the-art on ImageNet across all resolutions from $64\times64$ to $512\times512$, and text-to-image. PaGoDA's pipeline can be applied directly in the latent space, adding compression alongside the pre-trained autoencoder in Latent Diffusion Models (e.g., Stable Diffusion). The code is available at https://github.com/sony/pagoda.

PaGoDA Progressive Growing of One-Step Generator from a Low-Resolution Diffusion Teacher
Dongjun Kim" +
Chieh-Hsin Lai"
Stanford University CA, USA
Sony AI
Tokyo, Japan
dongjun@stand edu
chieh- waw.saay.lso...oo com
Wei-Hsiang Liao
Yuhta Takida Sony AI
Naoki Murata Sony AI
Toshimitsu Uesaka Sony Al
Sony AI
Yuki Mitsufuji Sony AI, Sony Group Corporation
Stefano Ermon
Stanford University
Abstract
The diffusion model performs remarkable in generating high-dimensional content but is computationally intensive, especially during training. We propose Progressive Growing of Diffusion Autoencoder (PaGoDA), novel pipeline that reduces the training costs through three stages: training diffusion on downsampled data, distill- ing the pretrained diffusion, and progressive super-resolution. With the proposed pipeline, PaGoDA achieves a 64x reduced cost in training its diffusion model on 8x downsampled data; while at the inference, with the single-stey step, it performs state-of-te -tate-of-the-arr art on ImageNet across all resolutions from 64 x 64 to 512 x 512, and text-to-image. PaGoDA's pipeline can be applied directly the latent space, adding compression alongside the pre-trained autoencoder in Latent Diffusion Models (e.g.. Stable Diffusion). The code available at https: : //github. thub. com/ sony pagoda.
Introduction
Diffusion Models (DM) [1. 2], which generate content through gradual denoising, have recently achieved high fidelity high-dimensional generation [3, 4J. While slow sampling has been improved by distilling trained DMs into single-step generators [5-7], DMs remain computationally intensive, especially high resolutions, requiring substantial data and GPU resources, thereby limiting large- scale training to few organizations [8, 9]. This highlights the need for more efficient pipeline to reduce both training and inference costs while maintaining the quality.
To address these challenges, we present Progressive Growing of Diffusion Autoencoder (PaGoDA), novel pipeline that significantly reduces costs while achieving competitive quality with one-step sampling. PaGoDA is built on simple yet effective idea: while diffusion distillation [6] is typically treated as a final stage of the whole pipeline, we explore have one more stage for the super- resolution after diffusion distillation. This approach led us to design PaGoDA with three distinct stages below.
"Equal contribution
This work was partially done during an internship Sony AI.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

DDTM Inversion with Pretrained DM
Downsample
Single-Step
Stago Super Resulution (4 Faiy
Stage Diffuson Protmaning
Stage
Diflusion Distillation (luze
Figure I:
Pipeline overview. PaGoDA deterministically encodes with downsampling followed by DDIM inversion, and constructs its decoder in progressively growing manner.
PaGoDA's 's Proposed Training Pipeline
Stage 1. (Pretraining) Train a DM on downsampled data.
Stage 2. (Distillation) Distill the trained DM with DDIM inversion to a one-step generator.
Stage 3. Super-Resolutton Progressively expand the generator for resolution upsampling.
By adding Stage for the super-resolution after the distillation phase, approach gains key advantage: training DM on a low-dimensional, downsampled space rather than directly the desired high-dimensional space. This dimensional reduction substantially lowers the computational demands of diffusion pretraining by orders of magnitude. For example, an 8 x 8 downsampling rate reduces the training computation by factor of 64 Moreover, the computational costs for the distillation and super-resolution stages are relatively minimal compared to the initial diffusion pretraining, making our pipeline highly efficient in terms of overall computation.
Figure provides an overview of our pipeline. We begin with DM trained at base resolution, and generate dataset of base-resolution data-latent pairs (x, where is real data and z is the latent representation of x, obtained by DDIM inversion [10]. In Stage 2, we train a decoder to map z back to completing the diffusion distillation [6]. In Stage 3, we add ResNet blocks [11] enhance sample resolution and progressively train these newly added upscaling networks, as visualized in Figure 2. The novel use of DDIM inversion in the distillation process, first introduced in PaGoDA, enables the decoder to be trained with the high-frequency signal from the real data at Stage 3. This integration of DDIM inversion establishes strong connections across stages, creating cohesive and unified framework.
Stage 2
Distilation
ROB TO
Train
Input Block
Middle Block
Output Block
Freeze
Carrectoo
Imput Block
& Middle
% Output
Output Block
To RGB
Block
Block
Stage Super Res
Figure 2: (Top) Stage PaGoDA learns the one-step generator base resolution. (Down) At Stage 3, PaGoDA progressively learns for super-resolution by adding additional network blocks.
In our experiments, we employed the progressively growing generator to upsample from the pre- trained diffusion model's 64 64 resolution to generate samples 512 512 resolution. Notably, PaGoDA achieved state-of-tt sfate-of-the-an the-hrt-tt (SOTA) Fréchet Inception Distances (FID) [12] on ImageNet across all resolutions from x 64 512 x 512. Additionally, we demonstrated PaGoDA's effectiveness in addressing inverse problems and facilitating controllable generation. However, PaGoDA's potential extends beyond its current application. As PaGoDA being a dimensional reduction technique that operates independently of Latent Diffusion Models (LDM) [3], PaGoDA could be directly applied into the latent space as-is, offering the possibility of further gain on training computes. We leave this exploration promising avenue for future research.

---

2 Preliminary
DM [1] samples from the data distribution Pdata through an iterative denoising process, beginning from a Gaussian prior distribution Pprior. This denoising process attempts to reverse [2] a forward diffussin process. If the forward process is defined by dx; = dl,, [13], the deterministic counterpart of the denoising (generation) process, known as the probability flow ordinary differential equation (PF-ODE) [2], DDIM [10], is expressed
dx. dt
-tV log (xx)) to (x,, t),
where Sob (x,,t) t) is a neural approximation of V Vlo (x(x). Consequently, (deterministic) sample generation from DM is equivalent to solving the PF-ODE (or DDIM) along the trajectory, formally,
J, 18 (xe: () dl,
DDIM (XT) 0
Pprior-
Modern solvers of the PF-ODE [10, 14] have significantly accelerated sampling speed, reducing the required network evaluations from hundreds to tens. To further speed up sampling, DMs are distilled with student model [6] Go R + IR to map from Xy to xg XDDIM (xx) by minimizing
Lasal Go) = E (xe)
(XDD - Go Go
(1)
We call this DDIM- approach as the noise-to-dee distillation.
3 Progressive Growing of Diffusion Autoencoder
3.1 Stage 1: Diffusion Models Trained on Downsampled Data
Training DMs for high-dimensional data generation is primarily feasible for a limited number of well-resourced organizations, largely due two factors: access large-scale datasets and substantial computational resources. This centralization of model development underscores the urgent need democratize access by significantly reducing resource demands during diffusion training. While several strategies [15, 3] have been proposed, our approach, PaGoDA, introduces paradigm shift by training the DM downsampled resolution in Stage rather than at the original full resolution. For instance, training on d-dimensional downsampled resolution requires approximately 4" times less computational budget compared to training in the full 4" didimensionaal space. In practical terms, when n = 3. this translates to training in an 8 downsampled space, effectively reducing training costs by a factor of 64x, thus making large-scale diffusion training more accessible to a broader range of researchers.
Although this paper does not extend PaGoDA's application the LDM such SD, training on a (say) 4x4 downsampled latent space could theoretically reduce the computational cost by 16x compared to full-resolution latent training. further emphasizing PaGoDA's potential for widespread adoption. In the case of generating 1024x 1024 images, PaGoDA requires training the diffusion model only 32x32 resolution, with Stage subsequently upscaling the full 128 x 128 latent space of conventional approaches [8, 9]. This progressive approach illustrates PaGoDA's s effectiveness in maintaining model quality while lowering the barriers to high-resolution diffusion training.
3.2 Stage 2: Diffusion Distillation Downsampled Data with DDIM Inversion
After pretraining DM on the downsampled space, PaGoDA distills DM to a one-step generator. For distillation, PaGoDA introduces Stage 3. In particular, we propose the reconstruction loss (compare it with Casl in Eq. of Section 2)
loss specifically designed for later usage in super-resolution
Lixe (Go) (x))
|xo
Go (x+ (XDDIM"
(xx))
(2)
DDDM where x (xo) is now the latent representation of xp, obtained from DDIM inversion, not from DDIM, i.e., the solution at time T of the PF-ODE starting from Xy in time forward, defined by
(xx) = Xo 1' 18 (x,, t) dt.
DDIM x
3

---

(a) Resolution Jump w/ Recon Loss
(b) Resolution Jump Recon Loss
Figure 3: Effect of the reconstruction loss in Stage 3. Without the reconstruction loss, the object
moves at each resolution jump.
Distillation using Lre maps latent representations real data, following data-to-noise distillation approach. While this method has the potential to improve real data aligmment compared to the traditional noise-to-data approach in Eq. 1, observe decline in generation quality over iterations. This decline stems from the prior hole problem [16], where the generator's input, x XPDIM derived from limited real data, fails to cover the entire prior manifold, leaving certain regions unexplored.
A straightforward strategy like early stopping could alleviate this issue, but it restricts the of Exponential Moving Average (EMA) in Stage 2. To fundamentally resolve this problem, we propose solution that maintains generation quality even during prolonged training. In Section 3.4, provide optimal and stability analysis of Stage 2, guaranteeing that our proposal is stable across training iterations. The key challenge effectively covering the prior manifold, which address by introducing an auxiliary adversarial loss, as defined below:
Ladv )+) = pua log (x)] Eppra(z) log (1 (1- D (Go())))
(3)
Here, Dus is discriminator that classifies the real and fake samples by maximizing the adversarial loss, and Ppeior(2) (z) is the prior distribution.
The second term in Lade, which involves (z) with z sampled from the prior, ensures that the entire support of the prior distribution during training. Overall, we train PaGoDA with the mini-max optimization of the following combined objective:
decoder is exposed
min max LPacioDA (Goo D = min Ge
(Go) + max Lat (Go, Do)
(4)
Gg D+
D+
While PaGoDA incorporates the adversarial loss, the reconstruction loss simultaneously guides the decoder to accurately reconstruct the entire training data. This combination allows the adversarial loss address underrepresented regions in the prior distribution effectively without compromising sample diversity. In our ImageNet experiments, we found that updating the reconstruction loss with little 1% of the data-latent pairs did affect sample quality and diversity. Exploring the impact of varying the number of data-latent pairs is left as a future work.
3.3 Stage 3: Progressively Growing Decoder for Super-Resolutee
Stage 3 trains the super-resolution generate higher-dimensional data from the downsampled resolution learned in the previous stages. As illustrated Figure 2, the resolution jump from Rd to R4dd is achieved by freezing most parameters of the distilled model from Stage 2 and training only final layers, which is augmented with additional upscaler network (of ResNet blocks [17]). In other words, within the base-resolution U-Net [18], we freeze its imput, middle, and output blocks except for the last few layers (previously highest resolution block) during Stage 3 training. Consequently, the unfrozen latter part of the network is trained for super-resolution. We suggest to progressively increasing the resolution by factor of 2x, though larger jumps by factors of 4x or 8x yield comparable performance.
Additionally, the last layer typically converts multi-channel (usually 128 or 256 channels) features to 3-channel RGB output. However, to minimize information loss, we retain these features and pass them directly to the next output block without converting them to 3 channels. This architectural choice, along with progressive training, is heavily inspired by Progressive Growing GAN [19].
In Stage 3. the reconstruction loss from Stage 2 is adapted
Exic (Xtigh)
||xxige - Go Go(xPi (XXDIM (appI-M(xx0) (xx

---

where Xo E R is the downsampled counterpart of Xhigh E I24"d 'hee adversarial loss in this stage is
Para (Miia)) [log D (Xnigh)] + E (p) log ( DG
Overall, both the reconstruction and adversarial losses are combined to guide training.
Stage 3 employs two key mechanisms to effectively cap- ture high-frequency details while maintaining training stabil- ity. First, the reconstruction loss is applied directly to high- dimensional real data, which was not feasible with earlier noise- to-data distillation methods with Eq. 1. As illustrated in Fig- ure 3, Lie stabilizes the upscaling process by preventing objects from shifting across resolutions, allowing the added neural net- work to focus solely on upsampling. Second, the adversarial loss operates directly in high- figh-dimensional space, enabled by the one-step generator trained in Stage 2. This generator is critical; without adversarial training in Stage 3 would be infeasible. As shown in Figure 4 tested on ImageNet, the adversarial loss pivotal for achieving effective upscaling performance.
StyleGAN-XL
PaGoDA
2.0
14
Resolution
Figure 4: The adversarial loss makes PaGoDA competitive with GAN-based super-resolution mod- els in Stage
3.4 Optimality Guarantee and Training Stability of PaGoDA Pipeline
When using the conventional Last for distillation, the optimal student becomes Go- (xp) = x DDIM (xr). meaning that the student's samples replicate those of DDIM. As result, the student's performance is heavily dependent on the teacher's performance. Consequently, the student's generative distribu- tion may diverge from the real data distribution, even when Lau is combined with adversarial loss. In contrast, by using the DDIM inversion-baas based reconstruction loss proposed in Stage we mathematically prove Theorem 3.1 that the optimal student's 's generative distribution aligns with the real data dis- tribution. As visualized in Figure 5, our PaGoDA Stage 2 (red) achieves robust performance even with weaker teacher, unlike traditional noise-to- to-se-to-data distillation loss Las of Eq. 1, which struggles despite the use of adversarial loss.
Ease + Eoay Cree Rodr
Teacher FID 2.44
Teacher FID 6.16
Figure 5: Comparison of Last and Lree, both combined with Lade, ing identical hyperparameters. Lve shows the robust performance, also supported by Theorem 3.1.
Theorem 3.1. Let X > O. Suppose D' (G) E arg max (G, D). If both PaGoDA's 's reconstruc- tion loss and adversarial loss share a common minimizer G then pG (x) Panna ()). Here, pa- is the generative distribution learned by optimizing Eq. (4).
Additionally, Theorem 3.2 shows that PaGoDA's training is stable with the help of reconstruction loss, even with adversarial training. We empirically observe that PaGoDA can be trained effectively without many of the techniques typically used to stabilize GANs [20, 21].
Theorem 3.2. [Informal] Let E be fixed deterministic encoder. Suppose that at the generator's 's equilibria G of Eq. (4), Pc- (x) Pámu(x),, and G (E(x)). Then, under conditions similar to those found in the stability literature for improving GAN [22, 21], training with Eq. (4) is stable (gradient descent locally converges to equilibria).
We refer to Theorems B.4 and B.9 for rigorous and extended versions of Theorems 3.1 and 3.2, respectively. All proofs can be found in Appendix B.
4 PaGoDA with Classifier-Free Guidance
In this section, we integrate Classifier-Free Guidance (CFG) [23, 4] into PaGoDA for Text-to- Any generation, with a focus on Text-2-Image. Incorporating CFG alters the sample distribution, necessitating adjustments to the loss functions for Stages 2 and 3. Since previous GAN literature [24- 27] has not addressed CFG integration, we introduce the classifier-free guided adversarial loss to accommodate this adaptation.
5

---

CFG guides the denoising process by adjusting the conditional score gradient V (x ()) into guided score V Pr (x,, c) + (w - 1)V log p(c)x.). This adjustment leads our distillation learning Paau(x|c, [x|c, w), defined by
target from Pdata (x ()) to Paata
Paata (x|c, w) Pdata (x|c)" Pxat(x)'
reflecting the influence of guidance strength w.
4.1 Classifier-Free Guided Adversarial Loss
To describe the classifier-free adversarial loss, we first consider the loss:
Lud (Go, D+))= (x)(())) log (x, w.w) + Epa, (xlo,w) log (1- Du D,((x,,w)))
where now both generator and discriminator incorporates w as an additional condition [28], see Eq. (3) for the comparison. From the standard GAN argument [29], this GAN loss guarantees the optimal generator to match to the data distribution, i.e., Pc (x(c,w) = Paaa (x|c, w). Hence, the classifier-free adversarial loss could be defined by
Laddy (Go, ( Dap)= (c)s(()) L (Go; D
(e log Du (xx (x,,c,,) w)] + (e) (xla.,) log (1 DD (1-Dwix, (xx,,w)))
A key challenge with L is that sampling from Pdata (xlc, is generally infcasible, making difficult to compute the first term of C To address this issue, we leverage the Bayes formula
Pasa (c) J (w) (dstu (xla) w) = Pauta (x, c)p(wlxx c),
where both representations two different ways decompose the joint distribution over (x, c, w)), with (w) being the prior distribution of the CFG scale w. From this formula, if we could predict the guidance weight w by observing and i.e., if we know p(w x, c), then sampling (x, c, w) from Pdaaa (c) m (w Paaaaaaaccc wi) can be alternatively achieved by: 1) sampling (x, c) from Paata (x, c), and 2) predicting most likely w using p(w[x, c).
We approximate p[w] x, c) with a U-Net encoder network with I-dimensional output, called CFG weight estimator wa. The input of w/ network is a single-channel matrix with (i.j)-th value as the multiplication of the i/j-th values of x/c CLIP embeddings, respectively. As this matrix is input the downsampled 64 x 64 x matrix the U-Net encoder. These CLIP embeddings are also used to condition the network. With DM pretrained at Stage 1, which is supposed to be sufficiently close the data distribution,
high-dimensionall
train the CFG weight estimator by minimizing E. Preu (au()))) [|| was (2, c, w), 1) where ((2 c, w) is a clean base-resolution sample drawn the teacher diffusion. Then, Wa( (x, c) -alue becomes the point estimation of p(w) x, c).
4.2 PaGoDA Pipeline with Classifier-F Guidance
We replace the adversarial loss in Stages 2 and 3 with the proposed classifier-free guided adversarial loss. In Stage 3, we shift the focus from x E R to Xhigh E Rg'dd to effectively capture high- frequeryy details. Additionally, both Stages and we replace the input of the generator in the reconstruction loss to be classifier-ff free guided DDIM inversion noise. To enhance text-sample alignment, we further regularize training with CLIP [30] similarity. For training, we use the ViT-L/14 [31] CLIP model pretrained on YFCC100M [32], while for evaluation, we use the ViT-g/14 CLIP model pretrained on LAION-2B [33], minimizing the risk of overfitting.
Experiments
5.1 PaGoDA Tested on ImageNet without CFG
We conduct experiments on ImageNet using PaGoDA without CFG to validate the core pipeline described in Section 3, utilizing the discrete time diffusion scheduling proposed by EDM [13]. Before training, we collect DDIM inversion latent representations for all ImageNet data using the Heun method [13] with 40 timesteps (79 NFE). Throughout the experiments, maintain the batch size

---

Figure Uncurated samples generated by PaGoDA at resolution 512 x 512 without CFG, Left: class 31 (tree frog); Right: class 33 (loggerhead turtle).
be 256 for both Lree and Ladv in Stages 2 and 3. We initialize our base resolution generator with the pre-trained diffusion U-Net. Following CTM [7], we implement adaptive weighting [34] with
I Cmell where l represents the last layer of the generator.
0.2
of La
For higher resolution generation, we double the previous resolution by adding two auxiliary ResNet blocks followed by one upsampler ResNet block. The previously trained generator remains frozen, except for the highest-resolution blocks, which are unfrozen. We then train these newly added blocks along with the unfrozen parts, using a fixed GAN weight of = 1.0. Appendix A. provides additional details. By freezing part of the trained generator, we achieve greater stability in super- resolution training without adaptive weighting. See Figure uncurated 512 x 512 random samples of ImageNet without CFG.
5.1.1 Quantitative Results
Table presents the performance of PaGoDA. Our model consistently outperforms all existing models across all resolutions, achieving SOTA FIDs without the need of CFG and any other stabilization tricks for GAN. Remarkably, PaGoDA's Inception Score (IS) [35] is on par with other diffusion and GAN models that employed CFG, which implies that PaGoDA samples are as distinctive CFG samples. Also, PaGoDA generates samples as diverse as the real data distribution, evidenced by diversity recall metric [36], where the PaGoDA reports 0.63 for 64 x 64 resolution (data's recall is 0.67). In contrast, StyleGAN-XL is far behind of PaGoDA in terms of the diversity metric, reporting 0.52 for 64 x 64 resolution. Note that we used StyleGAN-XL: discriminator in PaGoDA training, implying that the reconstruction loss significantly improves the sample diversity.
5.1.2 Discussion on Base Resolution
When applying PaGoDA pipeline, the choice of downsampled base resolution in Stage will be primarily determined by available computational resources. Thus, we investigate the impact of the base resolution at this section. To understand the impact, we conducted experiments at 32 x 32 and 64 x 64 resolutions, summarized in Table 2. Starting resolutions below 32 32 imposes excessive complicacy on the upscaling network, while higher resolutions significantly increase the computational costs the Stage 1. Therefore, analysis focuses on these two resolutions, balancing between computational efficiency and upscaling feasibility.
We utilized only H100 node with 8 GPUs for diffusion training 32 x 32 with 4096 batch size. Also, for 64 64 diffusion, we borrow a pretrained checkpoint [5], which used 2 32 A100 GPUs to train with 4096 batch size. Results in Table 2 demonstrate that the diffusion model trained in Stage I maintains robust performance across both resolutions. Interestingly, the one-step generator distilled
This is estimate.

---

Table 1: Experimental results of PaGoDA ImageNet.
Sampling NPE
Without CRG
With CRG
Without CPG
With CRG
Model
FID
IS
Recall
FID
IS
Recall
FTD
Recall
FID
1S
Recall
64
resolution
128 resolution
RIN [37]
250
1.23
66.5
2.75
144.1
simple Diffusion [38]
250
1.91
171.9
2.05
189.9
VDM+ [39]
79
1.43
63.7
1.75
171.1
1.78
190.5
StyleGAN-XL
1.51
82.35
0.52
1.81
200.55
0.55
CTM [7]
1.92
70.38
0.57
PaGoDA (ours)
76.47
0.63
1.48
174.36
256
256 resolution
512
512 resolution
DIT-XL [41]
250
9.62
121.5
2.27
278.2
12.03
105.3
3.04
240.8
simple Diffusion [38]
211.8
2.44
256.3
3.54
248.7
VDM+ [39]
250
2.40
225,3
2.12
267.7
2.99
232.2
2.65
278.1
EDM2-XXL [+2]
63
1.91
1.81
StyleGAN XL [40]
2.30
265.12
0.53
2.41
267.75
0.52
PaGoDA (ours)
1.56
259.61
0.59
1.80
251.31
0.58
Table 2: Ablation of base resolution.
Model
Upecaled
FID
Upscaled Res
Speed [s]
Teacher Diftusion
32 32
32 x 32
79
1.75
64 x 61
64 61
2.44
3.16s
296M
32 x 32
32 x 32
1
0.79
64 x
64
PaGoDA
32 32
128 x 128
1.61
64 x 64
128
128
1.48
0.041s
299M
32 x 32
256 x 258
L83
64 x 64
256
256
1.56
0.044s
301M
64 x 64
512 512
1.80
0.046s
302M
in Stage 2 consistently outperforms the teacher model, likely benefiting from the effectiveness of StyleGAN-XL, [40], combined with the reconstrcution loss. In Stage 3, the degree of upscaling from the base resolution emerges as the most influential factor for the quality, with upscaling up to 8x showing minimal performance degradation across both tested resolutions.
The upscaler in PaGoDA refines coarse samples generated lower resolutions, making the pipeline inherently aligned with the scaling laws of smaller resolutions. This design is advantageous, as scaling laws typically worsen with increasing resolution [43], while PaGoDA leverages the more favorable scaling dynamics at lower resolutions to maintain efficiency. Furthermore, the lightweight upscaling module introduces minimal additional latency, keeping inference times nearly identical to those the base resolution. This practical efficiency makes PaGoDA a promising solution for scalable diffusion model training across various computational settings.
5.1.3 Discussion on Upscaling Capability
In Stage we train the super-resolution module using combination of reconstruction and adversarial losses. As shown in Figures 3 and 4, we compare PaGoDA's performance to that of StyleGAN-XL. The comparison reveals key insights: 1) PaGoDA maintains consistent object alignment across resolution jumps, largely due to the reconstruction loss, and 2) its performance is strongly influenced by the GAN component, which plays a crucial role in capturing sigh-frequeccy details.
Other upsampling methods, such as SD and Cascaded Diffusion Models (CDM) [44] also target high-quality up- scaling. While PaGoDA, CDM, and SID share the same goal, they adopt different approaches, making them com- plementary rather than competing solutions. In fact, their strengths can be combined to enhance overall compres- sion and upscaling performance. For instance, CDM or PaGoDA can be applied to the latent space of SD, inte- grating their techniques for better results. Despite their compatibility, still essential to assess how these meth- ods compare in terms of their upscaling effectiveness. In the following analysis, we break down the upscaling capabilities of PaGoDA, CDM, and SD to understand their respective strengths and potential limitations.
Table 3: Comparison on upsampling.
Model
Resolution
Params
64" DM
1.18
63
1.33
EDM2
512" LDM
1.1B
63+1
1.96
64 DM (teacher)
0.3B
79
2.44
PaGoDA
64 + 642
0.3B
1.21
612 512
0.3B
1.80
8

---

3H1
S
(a) Inpainting
(b) Super- resolution
(c) Class transfer
(d) Latent interpolation
Figure 8: Controllable generation of PaGoDA with various tasks.
Since PaGoDA is experimented based on EDM [13], we adapted the experimental results from EDM2 [42] to facilitate direct comparison with PaGoDA in the upscaling performance. EDM2 presents results for both pixel DM and latent DM. In latent diffusion, a 512 x 512 x 3 image is compressed into a 64 x 64 X latent space for training DM, while pixel diffusion operates directly on 64 x 64 x 3 images, sharing the identical network architecture used in its latent DM. As reported Table 3, EDM2 shows a minor performance decline from 1.33 to 1.96.
Similarly, PaGoDA exhibits a comparable performance drop from 1.21 1.80 when upscaling from 64 64 512 512. This similarity suggests that PaGoDA's upscaling capacity aligns closely with that of the LDM framework. indicating minimal performance differences even when handling high- resolution data.
45
CDM
PaGoDA
8.0
20
10
Lastly, when comparing PaGoDA to CDM, we observe in Figure 7 that CDM encounters significant performance drops beyond certain dimensional thresholds (128 x 128), while PaGoDA maintains consistent performance across varying res- olutions. This robustness makes PaGoDA a reliable option for high-resolution generation, with its performance remaining steady even resolution increases.
64
Resolution
Figure Comparison between PaGoDA and CDM.
5.2 Discussion on Controllability
Once we have a trained PaGoDA generator Ggo we can utilize for solving inverse problems [45] and for controllable generation [46] in training-fires free manner [47].
Latent Optimization We consider the inverse problem: y A(x) + n, where y represents the observation, and A Rd + R with d 2 m is known operator. The restored data can be reconstructed by optimizing the latent. Specifically, if xt E arg min, |yy - A(Go, (z, c))ll then Go, (z", cc is the best possible estimate of the solution for the inverse problem. Figure 8-(a) displays the outcomes an inpainting task where latent optimization employed with Adam optimizer [48].
DDIM Inversion Specific tasks, such super-resolution illustrated in Figure 8-(b) and class transfer depicted in Figure 8-(c), can be effectively addressed without relying on latent optimization. For these tasks, apply DDIM inversion the downsampled observations, then map the DDIM latent back to RGB pixel by feeding the latent into the decoder. Generally, using DDIM inversion yields superior outcomes compared to latent optimization for these types of tasks.
Latent Interpolation Building on techniques from GAN research, we also explored latent in- terpolation for style mixing. Despite our model's latent dimension being larger than the typical 512-dimensional style vector used in GAN, our observations indicate that latent mixing by slerp operation [49, 20] achieves effective results, demonstrated in Figure 8-(d).

---

Table 4: Experimental results T2I. FID-5K is sured on MSCOCO-2017 [54] validation data. CLIP score measured by the ViT-g/14 backbone. Our
Table 5: Experimental results T2I. FID-
30K is based on MSCOCO-2014 [54] vali-
dation data. Speed is measured on A100.
model uses De as the pre-trained diffusion.
Model
Params
Speed [s]
FID 4
Model
Params
NPE
Speed [s]
FID 4
CLIP
<Diff-I [60]
9.1B
32.0s
6.95
SD1.5
0.9B
50+1
2.59s
19.1
31.3
LDM [3]
1,5B
9.4s
12.63
DeepFloyd-IF [53]
0.98
27
2.95s
22.3
28.1
Imagen [61]
3.0B
9.1s
7.27
Latent Distillation Models based SD1.5 [3]
SD1.5 [3]
0.98
2.9%
9.62
CAD [28]
0.98
8+1
0.34s
24.2
30.0
PixArt-a [62]
10.65
PD [55]
0.9B
4+1
0.21s
26.4
30.0
Scutt [59]
0.13s
12.22
LCM [56]
30.4
GigaGAN [24]
1.0B
0.13s
InstaHlow [57]
0.9B
1+1
0.09s
23.4
30.4
StyleCiA-TT [25]
1.0B
0.
13,90
UFOGen [58]
0.9B
1+1
0.09s
22.5
31.1
InstaFlow [57]
0.9B
0.09s
13.10
Scott [59]
21.9
UFOGen [58]
0.9B
0.09s
12.78
ADD [26]
0.98
1+1
0.09s
19.7
32.6
DMD [63]
0.9B
0.09s
11.49
Pixel Distillation Model based DeepFloyd-IF [S3]
LAFITE [64]
75M
0.02x
26.94
PaGoDAA (ours)
0.98
0.05s
20.4
31.2
PaGoDA (ours)
0.9B
0.05s
10.23
5.3 Text-to-Image Generation
We collect the data-latent pairs on the CC12M dataset [50] through DDIM inversion and utilize the filtered COYO-700M [51] dataset for adversarial training. The filtering criteria include only data with CLIP score (measured by ViT-B/32 [52]) higher than 32, and aesthetic score-v2 [33] higher than 5.0. Due concerns regarding sensitive content the open-sourced LAION dataset [33], we were unable to conduct large-scale diffusion training for Stage 1. This constraint led us to focus primarily on stages 2 and 3, leveraging pretrained open-source checkpoints. For the pretrained teacher diffusion, we used the DeepFloyd-IF model [53], trained on 64 x 64 pixel space. For further experimental details, see Appendix A.2.
Table 4 compares our PaGoDA mainly with the distilled models from SD v1.5 on 512 512. One notable observation from the table is that, even though the latent distilled model generates the latent representation in a single step, additional time is required for decoding this latent into image. In contrast, PaGoDA (on pixel teacher) eliminates such decoding step, thereby overcom- ing the time constraints associated with distilling SD models. For more detailed breakdown of the time taken by each com- ponent, see Figure 9.
0.00 T
Sampling Speed
Decoder (2-x)
2x Faster
0.66
Prag. Growing
0.08
64x84 Latent (z)
64x64 Pixel
0.00
PaGoDA
Figure 9: PaGoDA offers faster in- ference than the one-step LCM.
Returning to the performance results in the table, PaGoDA achieves performance comparable to that of the teacher model. This superior performance is also observed demonstrating PaGoDA's scalability on text-to-image tasks.
different test set shown in Table 5, further
Conclusion
PaGoDA introduces training pipeline that can democratize diffusion training by cutting training budget with orders of magnitudes. The pipeline is consisted of three stages: 1) we pretrain the diffusion models on the downsampled data, 2) we distill the teacher diffusion into one-step generator on the downsampled data, and 3) we train upsampler module until reach to the desired resolution.
Acknowledgement
This project was supported by Sony, ARO (W911NF-21-1-0125) ONR (N00014-23-1-2159)) and the CZ Biohub. Computational resource of AI Bridging Cloud Infrastructure (ABCI) provided by National Institute of Advanced Industrial Science and Technology (AIST) was used. We extend our special thanks our colleagues Takashi Shibuya from Sony AI and Yutong He from Carnegie Mellon University for their invaluable feedback.
10

---

References
[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
[2] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
[3] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[4] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems,
2021.
[5] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01466 2023.
[6] Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388, 2021.
[7] Dongjun Kim, Chich- Miin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. International Conference on Learning Representations, 2024.
[8] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
[9] Black-Forest. Flux. httpe://blaakfooeesstt wttp://ttaaaaaaa.annnaaaaaae ai/annonccing http://oonnnnnn---------eeeeeeee 2024.
[10] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference Learning Representations, 2020.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[12] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[13] Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.
[14] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
[15] Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Mare Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. In The Twelfth International Conference Learning Representations, 2023.
[16] Jyoti Aneja, Alex Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. Advances in neural information processing systems, 34:480-493, 2021.
[17] Sergey Zagoruyko and Nikos Komodakis. 4 2016.
Wide residual networks. arXiv preprint

---

[18] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assister intervention-MICCA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part 18, pages 234-241. Springer, 2015.
[19] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprini arXiv: 10196, 2017.
[20] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4401-4410, 2019.
[21] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning, pages 3481-3490. PMLR, 2018.
[22] Vaishnavh Nagarajan and Zico Kolter. Gradient descent gan optimization is locally stable. Advances in neural information processing systems, 30, 2017.
[23] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 202/ Workshop on Deep Generative Models and Downstream Applications, 2021.
[24] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10124-10134, 2023.
[25] Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis. In International conference on machine learning, pages 30105-301 PMLR, 2023.
[26] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311.10042 2023.
[27] Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. arXiv preprint arXiv:2403. 2024.
[28] Chenlin Meng, Robin Rombach, Ruiqi Gao. Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14297-14306, 2023.
[29] Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu. David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8744887733. PMLR, 2021.
[31] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[32] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec100m: The new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016.
[33] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278- 012:25778-5222 25294, 2022.
12

---

[34] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12888, 2021.
[35] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung. Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[36] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models, Advances in neural information processing systems, 32, 2019.
[37] Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. arXiv preprint arXiv:2212.11972. 2022.
[38] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, pages 13213- 13232. PMLR, 2023.
[39] Diederik P Kingma and Ruiqi Gao. Understanding diffusion objectives the elbo with simple data augmentation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[40] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
[41] William Peebles and Saining Xie. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[42] Tero Karras, Miik Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. arXiv preprint arXiv:2312.0266 2023.
[43] Tom Henighan, Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14..000101...1,,,, 2020.
[44] Jonathan Ho. Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1):2249-22 2281, 2022.
[45] Hyungjin Chung, Jeongsol Kim, Michael T Mecann, Marc L Klasky, and Jong Chul Ye. Diffu- sion posterior sampling for general noisy inverse problems. arXiv preprint 22XV:22209.16687 2022.
[46] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[47] Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, Zico Kolter, Ruslan Salakhutdinov, et al. Manifold preserving guided diffusion. In International Conference on Learning Representations, 2023.
[48] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xiv: 2014.
[49] Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques, pages 245-254, 1985.
[50] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
[51] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Sae- hoon Kim. Coyo-700m: Image-text pair dataset. https: / thub. com www.kaaaakbariii// coyo- dataset, 2022.
13

---

[52] Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agar- wal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICMI., 2021.
[53] DeepFloyd Lab. If by deepfloyd lab at stabilityai. https://githul http:/hh....ooodeeeplloo///////e 2023.
[54] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 750-755. Springer, 2014.
[55] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2021.
[56] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023.
[57] Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality diffusion-based text-to- im-ge generation. In The Twelfth International Conference on Learning Representations, 2023.
[58] Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. Ufogen: You forward once large scale text-to-image generation via diffusion gans. arXiv preprint arXiv:231 2023.
[59] Hongjian Liu, Qingsong Xie, Zhijie Deng, Chen Chen, Shixiang Tang, Fueyang Fu, Zheng- jun Zha, and Haonan Lu. Scott: Accelerating diffusion models with stochastic consistency distillation. arXiv preprint arXiv:2403.0 01500, 2024.
[60] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al. ediff-i: Text-toiimage diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.0133 2022.
[61] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494 2022.
[62] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint (xXxv:2310.00222 2023.
[63] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Free- man, and Taesung Park. One-step diffusion with distribution matching distillation. arXiv preprint arXiv:231 1. 18828, 2023.
[64] Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Towards language-free training for text-to-imagg image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17907-17917, 2022.
[65] Bosco Yung. Open-nsfw 2.
2021.
[66] Gant Laborde.
[67] Roman Inflianskas
[68] netterr/eeterrppooaaaa//ooaaao.p txt.
[69] Jamie
Dubs and Ryan 2377394710cccd4b4466..
Lewis.
https://gist.. http::///sstttithubbb teathub.com//ae wtt.hu...cmm/yanleiiss..
14

---

[70] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training cata-efficient image transformers & distillation through attention. In International conference machine learning, pages 10347-10357. PMLR, 2021.
[71] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-61 6114, 14. PMLR, 2019.
[72] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. Advances in neural information processing systems, 33:7559-7570, 2020.
[73] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv: 1908. 03265, 2019.
[74] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07/933 2023.
[75] James Betker, Gabriel Goh. Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. comfpupers/dallle-.. pdf. 2(3):8, 2023.
[76] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[77] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the JEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[78] Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via biock-wise quantization. arXiv preprint arXiv: 2rXiv:2110002866 10.02861, 2021.
[79] Adrien Saumard and Jon A Wellner. Log-concavity and strong log- concavit::;:: a review. Statistics surveys, 8:45, 2014.
[80] Wenpin Tang and Hanyang Zhao. Contractive diffusion probabilistic models, arXiv preprint 0rXiv:2401.13115 2024.
[81] Junlong Lyu, Zhitang Chen, and Shoubo Feng. Sampling is as easy keeping the consistency: convergence guarantee for consistency models. 2023.
[82] Xuefeng Gao, Hoang M Nguyen, and Lingjiong Zhu. Wasserstein convergence guarantees for general class of score-based generative models. arXiv preprint arXiv:2311.11003 2023.
[83] Bernard Asner, Jr. On the total nonnegativity of the hurwitz matrix. SIAM Journal on Applied Mathematics, 18(2):407-44 1970.
[84] Nam Parshad Bhatia and Giorgio P Szegö. Stability theory of dynamical systems. Springer Science & Business Media, 2002.
[85] Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. The numerics of gans. Advances in neural information processing systems, 30, 2017.
[86] David Balduzii, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Graepel. The mechanies of n-player differentiable games. In International Conference on Machine Learning, pages 354-3633 PMLR, 2018.
[87] Ian Gemp and Sridhar Mahadevan. Global convergence to the equilibrium of gans using 2018.
variational incqualities. arXiv preprint
[88] Chuang Wang, Hong Hu, and Yue Lu. A solvable high-dimensional model of gan. Advances in Neural Information Processing Systems, 32, 2019.
[89] Chongli Qin, Yan Wu, Jost Tobias Springenberg, Andy Brock, Jeff Donahue, Timothy Lillicrap, and Pushmeet Kohli. Training generative adversarial networks by solving ordinary differential cquations. Advances in Neural Information Processing Systems, 33:5599-5609, 2020.
15

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have made our abstract and introduction to accurately reflect the core contribution of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have created a separate "Limitations and Broader Impacts" section in the appendix to enumerate potential limitations of our methodology. including the algorithmic, theoretical, and experimental limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [Yes]
Justification: Although we omit some of assumptions in the main paper mainly due to page limit, provide full details of assumptions and complete proof in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose all experimental details in the main paper and the appendix, in- cluding the hyperaparameters used and the datasets used with their filetering methodologies. For further reproducibility, we plan release our code upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
17

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: In the reviewing process, we release our code to the reviewers to regenerate our experimental results. After the acceptance, plan to release the code to the public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nug cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips.
wwwcc/gguess/eeeee /aaeee/e/ooeeebmiiicc wwwsssoliicyyyoeeo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and basclines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specifyy all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We faithfully release our hyperparameters and experimental details in the appendix and the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We have not reported error bars mainly due the lack of computational resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are of range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer:
[Yes]
Justification: We explain how much compute resources we used for experiments in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics pettps//nnot h/terrip////r/p...../ cc cccpublic//eiiccccccccccececeeee wtwpehhithhiieeennnnneeeee
Answer:
[Yes]
Justification:
We faithfully follow the code of ethics, suggested by the link above.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the broader impacts separate section in the "Limitations and Broader Impacts" of the Appendix C.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: For the T2I checkpoint release, we plan to use HuggingFace to enroll every the system to control the downloaded user list. Additionally, we prohibited using the LAION dataset [33], which includes NSFW contents. Instead, we used the COYO- 700M [51] dataset, a large-scale text-to-image dataset that removes NSFW images by NSFW image detectors [65, 66] and texts that contain NSFW words [67-69].
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly credited the original owners of assets by citing them. In the code release, we comply the license and terms of the assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
20

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: In the supplementary, we include the the details of the dataset/cooeemmodll via structured templates.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22

---

Contents
1
Introduction
2
Preliminary
3
3 Progressive Growing of Diffusion Autoencoder
3.1 Stage I: Diffusion Models Trained on Downsampled Data
3.2 Stage 2: Diffusion Distillation on Downsampled Data with DDIM Inversion
3.3 Stage 3: Progressively Growing Decoder for Super-Resolution
3.4
Optimality Guarantee and Training Stability of PaGoDA Pipeline
4
PaGoDA with Classifier-Free Guidance
4.1
Classifier-Free Guided Adversarial Loss
4.2
PaGoDA Pipeline with Classifier-Free Guidance
5
Experiments
5.1
PaGoDA Tested on ImageNet without CFG
5.1.1
Quantitative Results
5.1.2
Discussion Base Resolution
5.1.3
Discussion on Upscaling Capability
5.2
Discussion on Controllabilityy
5.3
Text-to-lmage Generation
10
Conclusion
10
A
Experimental Details
24
A.I Conditional Generation with ImageNet
24
A.2
Text-to-Image Generation
25
B Theoretical Analysis
28
B.1
Convergence with PaGoDA's Reconstruction Loss
28
B.1.1 Preliminaries of Convergence Analysis
28
B.1.2
W2 Bound with PaGoDA's Reconstruction Loss
29
B.1.3
Wi Bound with PaGoDA's Reconstruction Loss
L...............................
33
B.2
Optimality analysis
34
B.3
Stability Analysis
35
B.3.1
Preliminaries of Dynamical System
35
B.3.2
Preliminaries for Analysis of PaGoDA Training
36
B.3.3
PaGoDA's Training is Stable
37
B.3.4
Literature on Stability Analysis of Adversarial Training
40
Limitations and Broader Impacts
42
23

---

STOP
Figure 10: text-to-image samples from PaGoDA.
A Experimental Details
A.1 Conditional Generation with ImageNet
Throughout the experiments, omit the class condition otherwise mentioned for notational simplicity.
Dataset Construction. We loaded Image dataset using center cropping and downsampling using the bicubic algorithm from the PIL python package. To augment the data, we applied horizontal random flip, and obtained each of latent representations by solving the EDM's 2nd-order ODE sampler (Heun's 's method) [13] with their suggested diffusion time scheduling and timestep selection. Consequently, in total, we processed approximately 2.5 million data instances forward in time using the PF-ODE to prepare for training. This computational cost of constructing the training dataset is comparable sampling equivalent volume of sample from pre-trained diffusion model.
GAN Details. We adopted the discriminator architecture from StyleGAN-XX Initially, We loaded DeiT-base [70] and EfficientNet-lice [71] as feature extractors, in line with StyleGAN-XILs setup. When processing real fake data through the discriminator, we first applied differentiable aug- mentation (DiffAugmel [72], incorporating three transformations: Translation, Cutout, and Color. Interestingly, we observed no performance differences between the unconditional and conditional discriminators. We hypothesize that this lack of disparity arises because the discriminator primarily updates the generator to refine high-frequency details, while preserving the wow-frequency global semantics due to the reconstruction power. Additionally, opted not to use additional techniques to tame the GAN training, such as RI regularization [21] or path length regularization [20] in our GAN training. PaGoDA's 's training generally remains stable due to its reconstruction loss, which is consistent with our theoretical expectation (Thcorem B.9).
We conducted tests GANs under two distinct scenarios. Initially, following the approach used in Stable Diffusion's VAE training, we introduced both the real data x and the reconstructed sample X (x) to the discriminator, training it to differentiate between the two while updating generator to maximize log Dpp (x). this setup, as the reconstruction only utilizes the latent representation Exx), the generation quality is not improved.
In the second scenario, adhering to the traditional GAN framework, trained the discriminator using randomly sampled real data alongside randomly generated fake data x Ge(z) from z ~ Pprior (z). Then, the endeavor of maximizing log D (x) now significantly improves the generation quality. Overall, we observed no performance degradation when both types of GAN training were applied to
'nttps://www image-manneo net. org/indez. php

---

Table 6: Comparison ImageNet 64 64. We evaluate scores, including Fréchet distance DINOv2 features [74], based on the statistics released by EDM2. The validation scores are measured by comparing 50k samples and 50k ImageNet validation data.
FiDioepiaaN3
FDDeNou2
61 61
Architecture
NFE
vs. Train
vs.
Train
Val
Val Data
1,05
13.86
SLyeGANN-X
GAN
220.47
CTM
ADM
1.69
2.88
159.67
165.96
EDM
ADM
2.93
112.17
EDM2-XL
EDM2
63
1.38
2.29
70.31
80.53
PaGoDA
ADM
1.01
2.10
70.04
78.44
Table 7: Comparison on ImageNet 512 x 512. From this result, would interesting to experiment PaGoDA EDM2 architecture for better performance. * the results of ImageNet 256 x 256.
312 512
Arch
DM
# Paramss
NFE
FiDineetions
FDpisos2
StyleGAN XL
GAN
0.2B
2.41
214.88*
EDM wio CPG (teacher)
ADM
Latent
0.3B
7.24
204.10
EDM2-S w/o CFG
EDM2
Latent
0.38
63
2,56
68.64
EDM2-S
0.38
EDM2 XXL w/ CFG
EDM2
Latent
1.1B
63
1.81
33.09
PaGoDA
Pixel
You are LLaVA, a large language and vision assistant trained by UW Madison WAIV Lab.
You are able to understand the visual content that the user provides, and assist the user with
variety of tasks using natural language.
You should follow the instructions carefully and explain your answers in detail.
Given the caption of this image "(text prompt)", describe this image in a very detailed manner
Figure 11: Input prompt of LLaVA to recaption the text-image paired data.
the generator. However, given our limited budget and the goal to develop a generative model rather than compression model, opted proceed solely with the second type of GAN setup.
Reconstruction Details. For the reconstruction loss, we train the generator Go by comparing the original data x ~ Putata (x) and its reconstructed counterpart Go (x(x,x c), c) at the data's resolution, where Exx, c) is the solution of the DDIM inversion. Since our training occurs in pixel space, we conduct this comparison in the feature space using the Learned Perceptual Image Patch Similarity (LPIPS) metric, and there is no need to develop a new feature extractor latent space. We experi- mented with features extracted from DeiT-base [70] and Efficcenttet-lite [71]: however, we observed no notable improvement from using LPIPS.
For the training, we use the RAdam [73] with learning rate of 8e-6 for the decoder and 2e-3 for the discriminator, and without weight decay. We use the EMA of 0.999, and all reported FIDs are based on the EMA checkpoint. Until 256 resolution, we use only 1 H100 node (with 80Gb memory) to train, and we use A100 nodes (with 40Gb memory, total 8 x 64 GPUs) to train the 5122 model. Throughout the experiments, we use the batch size of 256.
For the concerns on the overfitting, we provide additional results in Tables and 7.
A.2 Text-to-Image Generation
Dataset Construction. Due to the presence of inappropriate contents (CSAM) in the LAION dataset [33], we have decided to discontinue its use. Instead, we are now training our model using the CC12M [50] and a filtered version of COYO-700M, 700M [51] datasets. For COYO-700M, we apply filters to select only those text-image pairs that meet specific criteria: a CLIP score (measured by ViT- B/32 [52]) above 32.0 and an aesthetic score-v2 v2 [33] higher than 5.0. Additionally, we are enhancing the dataset quality by recaptioning the original text prompts from CC12M, adopting practices similar to those used in DallE-3 [75] and PixArt-a [62]. Specifically, employ LLaVA-7B [76], language
25

---

The image shows a nouple wekkngg through a field with small stone building, which appears small house hut. and woman walking towards house,
LLaVA
The suggests might ba old tracitional dwelling. presence of and the couple walking towords the house preato pasceful and serene stmeaphere, making it on interetingg and visusilyily appeoting scene.
seather
Watch your la you enter
Figure 12: Example of recaptioned image-text pair.
(a) DeepFloyd-IF Caption Generation
(b) PaGoDA Caption Generation
(c) DecpFloyd-IF Recaption Generation
(d) PaGoDA Recaption Generation
Figure 13: Caption vs. Recaption. From left right, CFG scale increases. The caption and its corresponding recaption are given by the exemplary case in Figure 12.
model with vision assistance, generate descriptions of the images based the text prompts, thereby ensuring more relevant and accurate text-image pairings.
The input prompt of LLaVA depicted in Figure 11, where put text prompt [text prompt). The output from this recaptioning process adheres to a consistent format, typically beginning with phrases like "This image features " or "This image shows ... To provide clear demonstration, Figure 13 displays several examples of original captions alongside their recaptioned counterparts.
Interestingly, the recaptioned samples generally outper- form the original caption samples. Notably, the recap- tioned samples exhibit sufficient quality, particularly when the CFG scale is small, shown in Figure 14. Therefore, to ensure balanced generation performance across vary- ing CFG scales,
DeepFloyd-IF (Caption) DeepFloyd-IF (Recaption)
28
generate samples from the original captions with the CFG scale uniformly sampled from the range [2, 10). For the recaptioned text, that follows a truncated Gaussian distribution on the range with scale of 3. Overall, incorpo-
26
24
CFG scale
[1, 10), centered rating these recaptioned texts into the PaGoDA training results in only marginal improvement in performance metrics such as FID and CLIP. However, it significantly enhances the actual quality of generation, particularly smaller CFG scales, because the recaptioning provides better-aligned training data.
CLIP
Figure 14: Effect of recaptioning.
Using LLaVA, recaption č(x, c) and obtain the DDIM latent representation, E(x, ē(x, c)), the entire CC12M dataset. Then, for the original text c, we have a triplet of (image, text, latent) of (x, E(x, c)) for one and another triplet (x, č(x, c), E(x, ě(x, c)) for recaptioned dataset. When computing Lres we mix these triplets and randomly sample from this mixed dataset.
GAN Details. Similar to the ImageNet case, we have adopted the discriminator architecture from StyleGAN-T. In line with StyleGAN-T, we utilize the DINO ViT-S/16 [77] as the feature extractor and apply DiffAugment [72], incorporating Translation, Cutout, and Color transformations, Building
26

---

DINO Encoder
CLIP
Augment &
Image Encoder
Discriminator
Tokenize
Transformer x
D2
Text Encoder
Transformer
D
Transtormer
High- qugii-uuittyiioo photo of Japanese Pe with background
D4
Transformer x
Dx
Scale (o))
Time Emb
Time Emb
Conv
DINO Feature
Linear
Conv
Linear
BatchNorm
BatchNorm
Text CLIP Emb
Lesky ReLU
L Linear
Leaky ReLU
Linear
Trainable
Affine
Conv
Pre-trained
Discriminator Head
Fixed-function
Per-token hinge loss
Figure 15: Discriminator architecture.
upon this architecture, we integrate w condition into each discriminator head, as illustrated in Figure 15. The inputs for each discriminator head include the DINO feature, text CLIP embedding. and the CFG scale which is scaled by factor of 100. We handle the CFG scale similarly to the time variable traditional diffusion U-Net models, incorporating the output CFG embedding into the existing components of the StyleGAN-T discriminator head. We assume both image x and text are related with the CFG scale, thus we designed the discriminator to incorporate w information into both modules, enhancing the relevance and contextuality of the discrimination process.
Reconstruction Details. In our text-to-ine eext-to-image training, we largely adhere to the protocols established for ImageNet training. However, notable modification involves the decoder network, which now incorporates w condition as an auxiliary input. Crucially, this w condition is processed in decoder in the same way as the time condition in diffusion models. We achieve this by scaling by factor of 100, thus aligning it with the existing time ranges. This method ensures a consistent treatment the parameter, integrating it smoothly into the established model architecture.
CLIP Details. Neither the reconstruction loss nor the GAN loss directly models or maximizes the text-image correlation. To address this, we introduce an additional text-image alignment metric to train our model. Specifically, we employ ViT-L/I4 [52] to assess the CLIP value. This regularization significantly enhances PaGoDA's performance, as evidenced in Figure 16 by improving both FID and CLIP scores. These enhancements suggest that not only is the sample quality improving, but also the alignment between text and images is becoming more accurate.
For the training, use the Adam W8bit optimizer [78] to minimize the required memory with learning rate of le-5 for both decoder and discriminator. Similar the
DeepFloyd-IF
PaGoDA w/ CLIP
30
PaGoDA w/o CLIP
ImageNet experiment, we do not apply the weight decay. In this text-to-image experiment, we do use EMA, following previous works [61]. In the base resolution, we
26
use the adaptive weighting with À = 4 |[V Emell
[[ C.s. Overall,
22
the DeepFloyd-IF-I1 model with 0.9B number of parameters.
CLIP
Figure 17 compares PaGoDA with the existing baselines.
Figure 16: Effect of CLIP regularization.
27

---

FRCADA
FeGROA and Racon
FaCEOAAN AND fhess
Woww
117 Shene
Palane
127 Sheto
11 5480
130-2 Steps)
130+2 steps
5180
335000
Pelcerse Par0221
Figure 17: Human evaluation result on T2I with CFG set to be 7 across models.
B Theoretical Analysis
In this section, we present rigorous statements and proofs of all theorems. The theorems are shown for the unconditional generation case (i.e., without the condition c), but the analysis can be extended to the conditional scenario.
B.1 Convergence with PaGoDA's Reconstruction Loss
In Section B.1.1, we introduce the necessary notations and preliminaries. In Sections B. 3.1.2 and B.1.3, we demonstrate that the Wasserstein-1 I and Wasserstein-2 discrepancies the learned density (with PaGoDA's reconstruction loss) from Pdata are upper bounded by PaGoDA's reconstruction loss and the pre-trained DM's training error. All results proved for unconditional generation (i.e., without as an input), but they can be easily generalized to the conditional case.
B.1.1 Preliminaries of Convergence Analysis
Consider OU process for & [0, 7]. where T'
dx; = x(()xxx dt g(t) dw;
Its associated PF-ODE is
dx; [- f(t)x; 2 (t) log) dt.
We consider f(t) and g(t) v2 for simplicity. That is,
dx, xx dt + /2dw,,
(5)
We recall that PaGoDA's reconstruction loss (unconditional case) is defined as:
Lree (0; Φb)) = Expa(())rs x) [|| G% (2)
Here, use Pdb. (z)x) denote the density obtained by solving the pre-trained teacher DM's empirical PF-ODE forward time from f to to
dx; = - 1000--9990000 (1)s (xe, 1)] dt,
where (x(x),)) () indicates the pre-trained DM. We remark that p (z|x) defines deterministic process.
We take Pptior = (1-
the prior distribution, and define PT GO Aatata the distribution obtained by solving the teacher-determined empirical PF-ODE forward in time. Let us consider the density obtained by sampling from PaGoDA (trained without GAN) po,0 = G -0 Peior also let CT denote the ground truth transition map from T to defined by the PF-ODE.
Conceptually, Theorems B.I and B.3 demonstrate that
W Wp(.0. Pdata) S Lice (0; Φ0) + EDM, p 1,2.
28

---

This implies that training with PaGoDA's reconstruction loss ensures the learned density Po.e Gp-iopprior is close to Pdata in Wasserstein distance sense. Moreover, improving the teacher DM to reduce the eттоr EDM way to further decrease the discrepancy between Po, and Pdata-
We remark that the differences between the two theorems primarily lie the distinct smoothness assumptions Pdata-
B.1.2 W2 Bound with PaGoDA's Reconstruction Loss
Assumption I-1.
m² : (x) ||x oc;
iii There is a ÉDSM > 0 so that supx,1 + s (x, - V (o) oMi
(ii) Go +0
is Lipschitz in x:
[G)
G (y))1 00,
A := sup x
|x y l12
for all O and T.
(iv) log Pdata is y-strongly concave with 7 3/2:
(x- y, V log Pdata (x) log Pdata (y)) -7 yll2
for all x and y. Theorem B.1. Given that Assumption I-1 holds, suppose ô is a positive constant such that ô <
3- 20, and let h(7, T) = 22 +y(1- 275 (1+8). where it is noted that h(y,, T) is also positive.
-2
Then
W2(po.0, Páaa)
Lm (0; (0) Eppa (xpe (x)x)
GT (2)
+ (A + e 1M(XTT)T (2(p,,
6DM
(1 h((,TTT) c mA.
228h(77 T)
In particular, if we assume Assumption 1-1 (ii) holds also for GD
W2(Po.0 Pdaa) S Lree (0; Φp)) + CDM + c m.. Here, we use S to absorb the dependence on the constants and y into the inequality.
We present an inequality which is essential for the proof of Theorem B.1.
Lemma B.2 (Proposition 3.5. in [79]). Let P and Q be two distributions on RRD. Suppose that log P yP-concave and log Q "in-concave. Then the convolution of log P Q a (1/7PP + 1/1q) concave distribution.
Proof of Theorem B.I. The proof of the theorem is inspired by [80, 81]. Define Pr GO-TT iPdata and Po. = GT GT ApT,oo From the triangle inequality, we have
W2(Po.0, Pamma S W2(Po.0 PP..P....Po.0)) W2 (tan))
(4)
(B)
For (A), let 7 Fly, 2) E II (ppiorr pr. be a coupling of Ppncr and pr Then
(A) = Wo (G% -G% :ppiior: (PT
E (yyy GZ (y) T (z) (z)
E Eyy.
G/
C% -0 (z) (z) 2
(Eryz) (z)
GT (z)
(A.1)
(A.2)
29

---

For (A.1), we can yield
(lyyaa>m l1y - 2|/2 2
(A.1) 1 A
min
Tell Prin.Pr.oo))
A AWal Pprior: PT.
AW2 (ptioor Pr) + AW2 (P:
S 6 [ (x))
|| )² + AW2 (pr,Pr,
(6)
Here, the last inequality is consequence of the following bound
W2(pprior: pr) - $ (xx))(x)) [|x0112)
which holds because Pprior taken as NN (1 -e 2T) II) and XT pr governed by Eq. (5) admits the expression
XT - x0 (-" e e +2, z z~ (O, - (1)
For (A.2), since pT, (0 (z) = Pooo (z)u(z)b (z|x (z)x) (x) dx, by applying Minkowski inequality we have
(A.2) = [B ()yx (2) (2)))))
(uppr.ss Gz GT (2)11)
(panl(X)rss (2/x)) || (Z) xx(22 1/2
(x)x)) (x1))))))
G -T
(0: Φ0)
[pentxipss (2(x)
GT GTT (2)1
(7)
The proof for (B) is motivated by [80]. Consider the following two reverse time PF-ODEs on the interval [0,
20,000 Sebo (Zr. T t), 20.00 PT,00
dt
and
24 + V log (o 20 ~ pr: d2.
dt
with coupling of 20. PT.do and àn pr that W2 E 120,000 20 2011
We notice that W (Po.00 Painaa I E || 2T,000 - 2 Thus, we need to obtain a upper bound of
E do 11. Let u(t) E 120,.00 20l12 Then
dt u(() = 2T
21. dt
(21.00 kx))
2u(t) + 2F
(2e.00 2,, Sobo (2t,000 T (logpr-((2))
2u(1) + 21E
(2e,00 2, S (2r,000 T V logg pr - (2),))))
(B.1)
+ 2E (21.00 - 2r, Pog ((21,0m)))
VlogpT
(8)
(B.2)
Let 8 > by applying Yang's inequality ab (V25a) a and and the Assumption 1-1, it becomes
da²
42
(B.1) for nonnegative
/28
25
2 (B.1) < 8u(t) +
48
30

---

We turn our attention to (B.2). Naively, (B.2) may be naively bounded above by Lip where Lip( V p-)) is the Lipschitz constant of V logpt() in x. However, will now derive sharper bound by incorporating assumptions on the data distribution.
We notice that (x,) = f (x (x ( dxu, where Pro(xx(Xo) = e (1 e=²()i) is transition kernel from 0 to determined by the forward SDE. Therefore, express- ing ps in convolution form, under Assumption I-1, and leveraging Lemma B.2, we deduce that log is 7/(( -2(2-1) + 7(1 - e 2-2(T ) -strongly concave distribution (see [82]). Hence,
(B.2) < 2 7(1 - e 2 00) u(t).
(10)
With the inequalities (9) and (10), we deduce from Eq. (8) that
('( S a(t)u(t) + By applying Grönwall's inequality, we obtain
BMM 28
where a(t) = (2 + 25
e-2(7-1) + 7(1 c 6-2(T-11) -1)-
24
c
2
T
EE 12 2 e A(T 110.,00
220122
DD
e A(())-A(t) dt,
20
TM
T
= =ATO (PT pr) +
A(T)-A(() dt.
(11)
28
0
where A(t) = So ((s) ds.
We aim to find an upper bound for inequality (1)) that decays exponentially with respect to T. In a(t),
27(1 1)e -(T-1)
b(t) == -- +x( (21--1)) as function of has the derivative (e-2(T-)) (00---20---
This implies when 7 2 1, b's minimum occurs at b(0) -4--2 (1-))) which implies a(t) S
2(1 + d b(0) for all [o, Setting 6 < 3-e-2r, which implies 1 ( then
-2T
y 2/4 =
1-- 3 (notice that (1 + -)ee-t -2r - > 20), we can deduce that
a(l) 11+5- 2-2 711 2-2 O.
7
Let h(),))) := - 27 (1+ 8) > O. Then we establish that a(t) S -h(,,T), A(T) s
-h(7, T),, and A(T) A(1) < -h(7, T) which implies 1e" A(T))A(()) di < 1 - e -(,,T)T By
applying the above bounds and inequality (11), (B) becomes
(B) s EE |6
2r 112)*
x e h(T.T)T W (pr pr) +
(1 ---M.NT
28h(7, T)
Se e +4(,,T)T W2 PT)
/28h((,, (1 e -G..T)
CDM
(12)
Here, the last inequality is from a simple inequality Va+b S va + bb for nonnegative a and b. By combining inequalities (6), (7), and (12), obtain
W W2(Po.e, Pamaa Se & (Wo)) || (|x0112) 113) $ A + AW2 (PT, PT.
+ LPaCODA (b) + Bpau(x)no (2/xx) xx
G2)
e (ionTTTTW(pTT..o PT) (28h(?,T))
CDM
e e- ehln.TyT 3
-oco (0: (b0) + [ Epan(x))p (21x) xx GTT ||2]
31

---

+ (A - ) (2
CDM
(1-c e (:::122)) $ te 2 mA.
228h(r,
This shows the first inequality in Theorem B.1.
Now, we show the second inequality in the statement of Theorem B.1. First, we establish an upper bound for (x)p (=/x)) | (2) à
in terms of CDM Let Goo denote the transition map defined by the empirical PF-ODE defined by the teacher Pæb (x|z), and G denote the ground
truth transition map defined by the PF-ODE from O to T. Then we have x G - (x)) for and
all E
E Pa (x/x) ||x (2- 1/2
1/2
E (xan GG
- GT 1 (x))
A (pau()) G (x) - Go. (x) 1/2
1/2
(13)
Here, we utilize the assumption that Assumption 1-1 (iii) also holds for GT
Consider the following two forward-time PF-ODEs on the interval [0, T'. both starting from Xo Pdata
dx; -2
dx -x,,0 Spo (xt.000 t)-
v log (xt,;
dt
df
By subtracting them and integrating from to f,, we obtain
x0.0e L// ||xx xx + (V logp:(x+) Spp (x+. 7)) dt
|x X1. 2ll S x
- L x, X su d7 dr + +DMTT.
By applying Grönwall's inequality,
||: - xx.xoo ll2 < Te eDM-
(14)
Combining the above inequality with inequality (13), implies
1/2
E
(2)
S ATe EDM- CDM-
(15)
Next, we derive upper bound for W2(PT, Pr) related to CDM. Let π ((, z) be coupling between 20-+T 2 ~ Pr. = 2o PPaata and z PT = G0-77ipdaas
. Pr) W2 (G (G0 Tppaaa -0 Ex(,,)) || 211 (TeTeDM)²
(16)
where the last inequality is derived from the inequality (14).
Therefore, with the first conclusion of Theorem and inequalities (15) and (16), we derive W2(po,e, e. Panaa S (io Φo) + EDM + e - mA.
The proof can be easily extended in two directions: (1) more general (VP)-SDE:
dx; = (t)x, dt gt) dw;
with II p= (o))) lg -oll=<r..0.] < oc, and (2) truncation the least time 8 (instead of t 0),
with an additional argument based on [81]
W2(ps, Pdan)) - (Epan(xs)) E Pren (E) (1 e "0)xx + v vI- e e- 2
< ((1 - e 6m + - e -2))
(VDVm)Vo
where ps GT-
32

---

B.1.3 W1 Bound with PaGoDA's Reconstruction Loss
Assumption II-1.
ii) m : (x) (x) |xx < oo;
(ii) There is a CDM > 0 so that s (x, t) V lo pe (x)
GMM
(iii G# Lipschitz in x:
G (x)
7y 0c,
A sup x
Xx - y
for all and T.
(iv) V (logp(()) is Lipschitz in x with integrable Lipschitz constant:
V
log pi (x) logpr(y)ll < 00,
(t) == sup x
x yll2
and Ag is an initegaaee function on (0,00).
In the following proposition, we prove a variant of Theorem B.1 which does not assume log-concavity of the data density (i.e., Assumption I-1 (iv)).
Theorem B.3 (Variant Theorem B.1). Assume that Assumption II-1 holds. Let v be either the
oracle data distribution Paata or an empirical distribution PaaaaN == & Ei=1 8xx where x; Pdata for = 1.... N. Let the PaGoDA's reconstruction loss starting from v be defined as
Lrec 0b) (x(x)px |||
Go. (2)2())
Then we have
Wi(po.0.V) S Lmo (20) Ex(x) (21x) ||x- GJ GT (2))11 CrTenD
+ (Cr + AAW (WW A)T + - (= | 1|x0112)
In particular, if we assume Assumption 1-1 (iii) holds also for GT for T 0 logg sufficiently large, we have
(mm) Exzu
Wi Wi(po.e. Pdosa) E (0 (b)) EDM-
Here, we use S to absorb the dependence on the constants T and y into the inequality.
Proof of Theorem B.3. inequality, have
Define Pr. = GO-+T tv, and Po. GT GT-+0ppr.oo From the triangle
W Wi < (Po.o..Po.oo)
Wilpo,ou: v). (1)
(1)
For (A), by following the similar argument as in Theorem B.I, we can obtain
(A) < T (Byan(x))
|ا0ا|| )A + AW1 (pr.Pr.ee pr. + CPacIDDA Φo) + Ex (m/x) ||x GT (2)11))
(17)
For (B), by subtracting the following equations and integrating over t from 0 to T.
24.000 + Sobo (24.000 t), 20.00 ~ Pr.
de da, dr
2, + V log Pr ( 20 ~ Pr:
we will obtain
2T. do. 2r (20,00 20) 1' (Zr.o. T- V (logpT-((2)) du.
33

---

Now let u(t) == E 22,.00 - 20l12- Then
u(t)= - u(0) + E /'' || (o: -) - V VlogPr- dr
S u() + / E SSp (2: Vlog log (2r
+ E V Vog PT Po
(Br,
1ox)pr-(22)l1
$ ((0) + DM + [' A.(F))(T)dr,
where (t) is the Lipschitz constant of Vlogp(()) in By applying integral form of Grönwall's inequality, we get
(B) < E (2T 2rl12 S Cy.lE 120.00 - 2012 + CTTEDM = CTWi(pT. + CTTEDM-
(18)
where := exp (Jo (1 d() and the last equality follows from choosing a coupling of 20.00
Pr, and 2g pr so that W1 (PT, Pr) = E 120. 20l12
By combining inequalities (17) and (18), obtain
(P(Po...V) S Lee (Opi Φo) + Eu(x) (x/xx) |x GT- [2] + CTTTEM
+ (Cr + A)W1 (PT, Pr) + (B (xx) ||x 11 )A.
A similar argument to Theorem B.1 can be applied to obtain the second inequality in the statement of Theorem B.3.
B.2 Optimality analysis
In this section, we compare the optimality of the learned distributions resulting from PaGoDA's training and distillation- based training loss, incorporating GAN [7, 6].
PaGoDA's 's Loss
We recall PaGoDA's training objective LPAGODA
LPECODA (Go, D+) = GGo( + AL (Go. D+))
leverages the reconstruction loss
Cro(GGo = E (x)n(()
(xx) ||x (G(
and adversarial loss
Catv (Go, D+)) = E (x)) log (x) + E Epprr(z) log (1 (I-DD(GG(z)))
In the realm of knowledge distillation (KD) methods for DMs, approaches like local consistency [5], global consistency [6], soft consistency [7] utilized to learn the noise-to-data trajectory of the teacher DM. Let us consider the global consistency loss as case study (similar arguments can apply other distillation objectives). where the teacher's trajectory is obtained by solving its empirical PF-ODE from T to O. The long jump along the trajectory is represented Grecher (z), where denotes the initial point (noise), T signifies the initial time, and 0 denotes the final time. The output of GT corresponds to the estimation of clean data, starting from z.
Knowledge Distillation Loss
(RD(Go) = (eyonn(t)
Glasto (2) Go Gz)2)11)
In this context, we abuse the notation Gz(z) denote the generator for KD.
The training of KD also incorporate adversarial loss for enhanced performance [7, 27]. We represent the combined loss as:
CKD.GAN(Ge, Dv) = (kD(Ge) + Lady (Ge, D
34

---

Theorem B.4. Let Peu be the density determined the teacher DM. Suppose that GAN admits an optimal discriminator D
In PaGoDA, assume that the network parametrized generator class {Go} is expressive enough so that it can simultaneously optimize both GG) and Cado ( Go: D*) with a same
minimizer. Namely, arg ming (ere (Go)) arg ming Ladd (Go; D² F 0. Then
pe PaGoDA == .
.PoGoDA Pprtorr = Pdltt-
In contrast, suppose that Pñt + Paltas then under similar conditions for KD+GAN where arg ming (Ga) arg ming Lads (Go; D*)3 + 0, there is no minimizer O* that pa- .D-GAN == . GANiGanpirror = Pdata-
The first part of the proof of the theorem follows from the following Lemma. Lemma B.5. If arg ming ( [f(0) } n arg (ggmin(g(()))
0, then argming (00 + g(0)}
arg ming (f()) n arg ming (g(0)}
prove the relationship arg (f)) + .(0)} 2 arg [f()} n arg ming {g))) Indeed, holds without additional assumption. Suppose that 0" E arg f ( n arg ming(g(0)). Then for any 0, have f(0) 2 f(0") and g(O) 2 g
Proof. First,
which implies f(0) + (()) 2 f(0)) + g(0)). That is, 0 E arg arg (f()) + +(0)).
On the other hand, suppose that 0" E arg arg ( + (())). We want to prove that 0* arg ming { narg ( Let 0% E arg ming (O) narg ming (g))). where we notice that the existence of 0% is guaranteed by the assumption. In particular, we have f(0") 2 S(0)) and g(0") 2 9(0)). Then for any 0, we have
min min(s(()) + ( = f(0)) + g(0") 2 ((0)) + ((0A) 2 min min (f(0) + ((0)
(f(0) m Thus, mino
|(0)} = f(0)) + 9(0 = f(0%) + ((0)) and
[f(0") - f(e))) + [o(0)) - o(0))] = O. This implies f(0)) f(0;) ming ((0)) and g(0)) g(0)) ming (g(0)) as the individual terms are nonnegative. Therefore, 0" E arg ming (0)) narg min, (0)) proof.
))), which concludes the
Proof of Theorem B.4. With the lemma above, let 0" arg ming LRaGaDA D Conse- quently, 0" should also simultaneously minimize both free and Ladv- Minimizing Lie implies that pø- PaGoDA Gg- PaGoDA where PT. represents the density derived from solving the teacher's empirical PF-ODE forward, starting from Pdata- On the other hand, optimizing Ladv implies that pø- PPGGDDA
Paaa by applying Theorem [29]. This establishes the first part of the theorem.
In the second part, suppose on the contrary that there is a minimizer 0 of LKD+GAN such that pø- KD+GAN Pdata Again, by applying the above lemma, we infer that 0 should also minimize LKD (and Ladv). ). This implics that Po KD=GAN = Pæo However, this contradicts our assumption that Pdma # Pdpo- Thus, such minizer does not exist and the second part of the theorem is proven.
We remark that (1) optimality of CGAN(0) may not be unique in 0, and that (2) the first part of the theorem can be directly extended scenarios involving downsampling in the encoder.
B.3
Stability Analysis
B.3.1 Preliminaries of Dynamical System
To study its convergence and stability, we first introduce the prerequisites for Lyapunov stability [83, 84] in general setup. Let F: E + E be continuously differentiable operator (that is, 86' operator), where Q R. We consider the discrete iteration dynamical system defined by
fx+1 F(Ex) with Eo Ω.
35

---

Namely, Ex+1 (Eo) = Fo....oF(uu). A point e- i is called fixed point or equilibrium
k-copies
(we use the terms interchangeably) of F g' = F(5*). The stability and convergence analysis
focuses on how the dynamical system F( approaches a fixed point as iterations are sufficiently large.
Definition B.I. (Stability [84)) Let 6* be an equilibrium of the 6 operator F: ? + Ω. The equilibrium 5* is said be
stable if for every c > O there is a > O so that whenever ||e - - < 6, we have (E) - < e for all & E NU (0).
asymptoticallyy stable if g" is stable, and there is ā 8.30000 >> 0 that whenever || <0 6, we have lim (() (E) 1= O.
exponentially stable if g* is asymptotically stable, and there is 6 0 and ax >> that whenever | 1r 8, we have (()((((E)) 1- S a a - e- 6k for all & E N U {0). The largest 3 0 that satisfies the inequality for exponential stability is referred to as the rate of convergence.
Let I be subset of the set of all equilibria. We say the dynamical system FF() locally converges T if F(() is exponentially stable at any point in r.
The intuitions of the above stability notions are
A stable equilibrium indicates that if an initialization is within some 9-neighborhood of the equilibrium, the iterations starting from that initialization will always remain within an é- t-neighborhood of the equilibrium, for any arbitrarily chosen
An asymptotically stable equilibrium indicates that iterations starting near the equilibrium not only remain close but ultimately converge to the equilibrium.
An asymptotically stable equilibrium indicates that the iterations not only converge but do so at a rate no slower than the rate -Ok with respective to iteration step
Analyzing the eigenvalues of the Jacobian VeF((*) of the operator F at equilibrium &* is crucial tool for studying stability. principle [83, 84], we can ensure that the Jacobian of F at some equilibrium has only eigenvalues with strictly negative real parts, then the dynamical system (=() is asymptotically stable at that equilibrium. In particular, we refer to matrix as a Hurwitz matrix if all its eigenvalues have strictly negative real parts.
In the following lemma, we present a necessary condition to ensure that a special class of matrices will be Hurwitz.
Lemma B.6, (Necessary condition for a Hurwitz matrix [21]) Consider the following matrix J e M)N+M)x(M (NN with NxN Q RERM MMMMMM and B R M
P
-B7
J=
B
Q
Suppose that B is full rank. Then all eigenvalues of have negative real part, either (1) P is negative definite and Q is negative semi-definite, or (2) P is negative semi-definite and Q is negative definite.
B.3.2 Preliminaries for Analysis of PaGoDA Training
We consider PaGoDA's training, integrating reconstruction and adversarial losses with a weight
n >
C(O, +b) Eppn(()) (x) [n |x Go f(Dx + Epc, (x) [f(-
(19)
Epunixx (x) (n |x (I ((Dx(x)) + Epper()) [f(- [C
(20)
Here, f: R - R. is continuous differentiable function. In the vanilla GAN [29], the ffuuccton
is taken f(u) = - log (1 + exp(-u)), where fu) exp(-u)/ + exp(-u)) > 0 and
36

---

f"(u) - exp((-u)/ exp(-u)) < 0 for all u E R. We maintain the generality of f and will prove the training stability of PaGoDA across wide class of f.
The velocity field v(0, vb) corresponding to the gradient descent update
- Vo vb) V v. 2/)
v(0, 2/)
Gradient descent is a special case fixed-point iteration. Now, we specify the operator F as an alternative gradient descent operator. That is, consider Fn == FD.b Fa, with learning rate > 0. Here,
(0, 0
/VoC(0, bb) and FD.,(0,
0 1b + 1V C( bb)
4pb
point (0", 2b²) is called an equilibrium of the system defined by v if v(0", 2b)) = 0 (equivalently, ( 2 =0). We analyze the learning dynamic via the Jacobian matrix of v(O, 26) which is defined as the following:
- V)C(0, J(O. v)) = ((0,
-(0,
V C(0, 2b)
The following proposition relates Lemma B.8 to the stability of the gradient descent operator Fu. serving as the main tool prove the training stability PaGoDA in Theorem B.9.
Lemma B.7. (Locally stable manifold modification of (21]) Suppose that the gradient descent operator Fú. Fi(v, (1, w) is a 66 mapping. Assume that there is neighborhood Qi of so that FR admits equilibrium (v² x Q:
(w="w") w²) be an equilibrium (fxed point) of Fu-
Full" =)( (w²",, for all w E QQ.
If all the eigenvalues J == V ou.Fi ,w") have negative real parts, then for a sufficiently small learning rate h, the gradient descent iteration defined by Fu. locally converges on I == {(v" ,w) E Q} with a rate of convergence \maae) Here, Amax denotes the eigenvalue of I with the largest absolute value.
Proof of Lemma B.7
This proposition followed by Lemma A.5. and Theorem A.3. of [21].
B.3.3 PaGoDA's Training is Stable
Proving PaGoDA's stability involves two steps: First, derive the components of First, deriving the components of J (0", Second, verify that these components satisfy Lemma B.6. After these, we can apply Lemma B.7 to conclude PaGoDA's training stability whenever the learning rate h 0 is sufficiently small.
Assumption III-1.
(i) E is not identity map.
(ii) At 0", Po- Pdltas and Go- (E(x)) for a.e. supp(pamu).
(iii) ub², (x) = and Dx (x) = for E supp(pama).
Lemma B.8. Suppose that Assumption III-1 holds for equilibrium (0", x²²). Then the Jacobian at the equilibrium can be computed as
KcG Kna
-KTG
J(0"
Knn
Here, and
KCG 27 pam (x) V (EG()))())) VoGoo (E(x))
(0) E. VoGo- (z)T. De- VoGo (z)).
KDG - -0) DoLE (x)
V D's (x) 0=0-
KDD 20) (x) V (x) v D (x) (x)
37

---

Proof of Lemma B.8. We first compute the gradients of L in terms of and sb where we utilize the formulations Eqs. (19) and (20), respectively.
VoL(O, xb) = Byl (x) [xx VoG
(ppra(z)) (.. voGo(z)).
(21)
VyL(0, x)- Ex)m(()) ( Pa (x) (f (( (22)
VgL(0,1)) 1/) 2n Bean (x)
2n (oo((() ([
Go(F(xx)),
1 (xs
VoCo(z) (2Dg(G((Z) VoGo(z)
ppeee (x) [f ( VoGooz)) ( VoGo(z)]
Eppor( (x) If (Go VxDx VoGo(z)).
According to Assumption III-1 (ii) and (iii), we have
v/L(0", ,,""00"""") pu (x) (VoGo (E(x)) VoGg.
(0))) p VoGoo Deo VoGo
Thus, obtain
KGG = V2r(O", 25
21)lBp VoGoo VaGo (E(x)) + (0) VoGoo(z)) (VCD (o)o
To compute KDG, we first derive Vo from Eq. (19)
VoL(O, x/) = - 2m (x) (x Go(E(x)), (oGx(E()))) + VoLPo, (20) (x0 [f(-
Thus, we can compute
V ((0,b)) = -VOL (x) (f(D=((x)) (x)),
and hence,
KDG = C(0', = - pes (x) Ve (x)
0=0-
To compute KDD, we can obtain from Eq. (20) that
C(0, 1p)
Bpau(x) [f" (x) Dx(x). (x
Dx
+ Eygp (x) (
(x))
+ Exun(x) xx) (x) Dzx (x)) - Epc, (x) (=
(x)
Hence, by using Assumption III-1 (ii) and (iii), we get
KDD = C00 (()) = 2f (mn) (x)) Dx- (x) (x
We consider the following sets
MG = {8 po = Piaa: x Go((())) for a.. x E supp (pans)
MD = S(bb) = 0}.
where S(1b) := Ex 2x)) [x> Also, we let Tw-Ma denote the tangent
space of Mp at 2b"
Assumption 111-2. (0) > 0 and (0)
(i) The second continuously differentiable function f: R + satisfies:
< 0.
iii There is a 8 > so that Mg n (s(00) and MD n (Bbb))) are 8 manifolds.
38

---

(ii) (oGx-(E(x))) FoGo-(E(x))
is positive definite, for all x supp (puma)
(iv) 0.1((1p)) # 0 for any w E Tp where h(ep) = V VoE (x) (x)(
O=--
w D.D (x)w 2 0. for all w & Te-Ma and supp (P Remark. Two special cases are either (v-1) D (x) = 0 for x E supp(Pdan)) or (v-2) w" V''2D'' (x)w O, for all w & Mg and x E supp(Psam).
Theorem B.9. Suppose that Assumptions III-1 and III-2 hold for an equilibrium ,0*,"*) and 1j > is sufficiently large. Then the alternative gradient descent iteration Fú, described in Section B.3.2 is locally convergent on Mg x MD for a sufficiently small learning rate h > 0.
Proof of Theorem B.9. The argument is motivated by [21]. We notice that MM x MD is a subset of all equilibria of the operators Fh (or v@0, .))). This because that for any (0, +b) E .M x Ma. we have Pe = Pdata. Go (x))) D (x) 0. and V D = O for x E supp From
Eqs. (21) and (22), we then can obtain VoC(e, vb) C equilibrium.
0, meaning (0. 1/) is an
Now, we show that the alternating gradient descent converges locally on MM MD by verifying Lemma B.8 is fulfilled, and hence, Lemma B.7 can be applied. Let (0", x6)) E MM x MD- There is a E. -diifecomorphim Vr that transforms a neighborhood of (0", xb") onto an open set in R (N+M) due Assumption III-2 (ii). More precisely, we can compute the relation of Fa and after the V-reparametrization. Let Ç = (0, w/), and
F (C) = V o Fr. o v (C' v((c) (0 2b) (vo v (C)).
Then
(C) Vo.v 3²) Vo v²) Vo VO0 x")-1 V (5") = Vo. (00 1²) Vo. w²") Vo. (0" "---
We remark that similar matrices have identical ranks and spectrum. Therefore, without loss of the generality, we can assume that (0 //- = (On. 0m) E RN x RM and
MG = Mo = No)No x N-No
MD = MD Mo x MM
We write the new parameterizations as 0 = (VG. wG) R R N- and (vp. wp) RMp x M-MPp For simplicity, we write v(O, +b) == v(ug, WG, UD WD). To apply Lemma B.7, aim to show that V ((veun)) v6" ab²) only admits eigenvalues with negative real parts. From Lemma B.8,
KoG -KTa Kna
V ( Φb) =
KnD
Here, KeG, KDG. and K DD represent submatrices of Keg, KDG, and K DD. respectively, with coordinates (vG, UD), indicating the Jacobian of v with derivatives taken along the vg and vp directions.
First of all, we show that KDD is generally negative semi-definite. Let & (N++M) be any vector. Then
<D = 20) (x) E De. Ve (x)
= (0) pum [(V 10 D (x) Dx) S0
because f*)(0) < from Assumption III-2 (i). Thus, for any Èa E No and ED E R MD
consider E = (Éc. ED) in (vG, up) ccoordiaat,,,
e kT = = KDp 00,
39

---

where & = (Éc. ON-No: MD) E ((N+M)
Next, we demonstrate that DD is full rank. We observe that ED & 0 if and only if & Mo. Then, according Assumption III-2 (iv), deduce that if ED
K = (0) r (x) V Du (x)
0--- = =- #0.
The elements of K D corresponding to the -o-corddinates are represented by D while those corresponding to the wD- ccoordinatsss are 0. Therefore, we conclude that DDGED # O. Consequently, by the rank-nullity theorem, DG is full-rank.
Finally, by using similar arguments by selecting (ve, up)-coordinate,, without loss of generality, only need to show Kag is negative definite. By applying Assumption III-2 (i) and (v), the following lemma concludes that if 7) > 0 is sufficiently large, we can conclude the negative definiteness of Vgc(o" 25*) under Assumption III-2 (v-2).
Lemma B.10. Lel A be positive definite, and B be positive semi-definite. Then there is a Ajmin so that A + B is negalive definite for all 1j >> Omin
The lemma holds because, for positive (semi-) definite matrix X, we generally have
Amax ()) [ w Xw 2 Amin (X) ||
for all w. Here, Amax (X) and Amia (X) denote the maximum and minimum eigenvalues of X, respec- tively. Thus if select 7)
Amne (BB then for any * O, have
(ni
(A)
w (- B)w - Aw + w" Bw s (- 11)Amin (A) + Amax (B)) | l| 0.
By applying Lemma B.6, we know that V (wo.w)) v(0", 2b*) only has eigenvalues with negative real parts. Therefore, with sufficiently small learning rate 0, Lemma B.7 guarantees the locally convergence of Fn on MG x MD.
B.3.4 Literature on Stability Analysis of Adversarial Training
Studying the stability of GAN training from a dynamical systems perspective has been a popular approach [85, 22, 21, 86-89]. Generally, proving or disproving whether adversarial training is stable challenging. However, [21] provides an example (Dirac-GAN) showing that, in general, GANs are not stable unless additional conditions imposed.
As result, researchers have explored additional conditions to stabilize GAN training, Essentially, the goal is to impose extra regularizations the GAN loss (G x/) = Expn(x) [D)
VoLGAN (0, +b)
Enc, (x) [f(- (x))] or its velocity field VGAN ( xb) V LGAN (0, +b)
to ensure that the resulting Jacobian is Hurwitz. To elaborate further, we revisit the Jacobian JaAN of the vanilla GAN, given by VGAN(0. 4b):
CGCGAN(O, xb) JGAN (0, +b) v2 V. CaAN
-V Can, ve LaNN(( xb)
KGG KnG
-KT DG K
Here, we slightly abuse the notation from Section B.3.3 by using Kij, i,j E (D, G}. to denote the corresponding components in JGAN. By similar argument of Lemma B.8, we can obtain (indeed, 7 0 in Lemma B.8) that
Kaa = f') V GoGo-(2) (Go VoGo (z)).
K = - =f (x) V Dw- (x)
0=0-
KDD = (0) (x)))))))))))))))))))))))))))))) Da (x) Vap (x)"1.
Conceptually [83, 84], we can ensure that the Jacobian at some equilibrium has only eigenvalues with strictly negative real parts, then the gradient descent iteration of LGAN is asymptotically stable
40

---

Table 8: Comparison of various assumptions on stability analysis.
Method
KGG
KDD
[2]]'s Vanilla GAN
Both Pdata and pe covers the whole space R."
Additional technical assumptions (difficult to verify).
[2]]'s Vanilla GAN
De-- (x) V Dx) on supp (Ptaa))
V2D y- (x) positive definite.
No further assumptions.
This implies KGg is negative definite.
[21]'s Regularized GAN
De-((x) Vx (x) =0 supp(pamaa.
By introducing regularizer modify the vector field v and obtaining new vector field v, they can determine an LDD so that KDD == KDD LDD is negative definite. Therefore, is not vanilla GAN anymore.
Di w. (x) = 0 on supp(Pdata)
This simply implies Koa =0
PaGoDA
Dx-(x)) = V (x) -0 supp(pamaa.
V (x) just need to be positive semi-definite on supp(Ptaaa.
No further assumptions.
Then with v 0 chosen be sufficiently large in PaGoDA, KGG is negative definite.
at that equilibrium. Therefore, the objective of many studies [85, 22, 21, 89] is to find conditions to verify Lemma B.6. We focus discussing the conditions for Kog and Knn to be negative (semi-)definite, as this distinguishes PaGoDA's Theorem B.9 from the existing literature.
Under Assumption III-2 (i) that (0) < O, is worth noting that K KDD is generally negative semi- definite without additional conditions. Hence, studies [85, 22, 21] attempted to impose additional regularizers JGAN YGAN to ensure that either KpD is negative definite (as in [22, 21J) Koa negative definite (as in [21]). In Table 8, we provide a comparison the various assumptions, at high-level, drawn from the literature.
We emphasize that PaGoDA does not require D2 (x) to strictly positive definite, thanks to PaGoDA'ss reconstruction loss. Specifically, it accommodates the scenario where V2 D (x) on supp() It's noteworthy that this capability enables PaGoDA to address cases where the instability of GAN is demonstrated, as exemplified by examples provided by [21].
41

---

C Limitations and Broader Impacts
Limitations. Algorithmically, the reconstruction loss is incompatible with the classifier-free guidance. which requires us to adopt the original distillation loss. However, as reconstruction loss directly uses the real data, it provides additional merit to decoder training, resulting better performance as evidenced in the experiments. Theoretically, some theoretical assumptions of PaGoDA are challenging to verify practice, For example, Theorems B. and B.3 require certain Lipschitz continuity of the score functions. This assumption is difficult to maintain at = 0 due to the potential concentration of the data manifold in lower-dimensional space, causing singularity. However, by truncating the PF-ODE solving at = 8 (for some >> 0), which is common practice, this singularity is avoided, making the Lipschitz continuity assumption more feasible. In addition, Theorem B.4's assumption of the existence of common minimizer can be difficult to verify empirically. However, with proper neural network parametrization and effective optimization, this assumption becomes more feasible. At last, verifying Assumptions IIII1 and III-2 concerning the optimal properties of the generator and discriminator (Ge. D is challenging in practice. These assumptions, essential for general (Lyapunov) stability analysis, difficult to validate empirically. However, they appear reasonable based on our experimental observations. Last, empirically, PaGoDA's T2I generation capability relies heavily the scale and quality of the training dataset.
Broader Impacts. PaGoDA, as a general media generative model, carries the risk of producing harmful or inppropprite content, such deepfake images, graphic violence, offensive material. To mitigate these risks, we avoid using the LAION dataset [33] in our model training, but robust content filtering and moderation mechanisms are essential to additionally prevent the generation of unethical or harmful media.
42


Reviews:
[{"review_id": "2qybyEFEYx", "reviewer": "Reviewer_TnXh", "summary": "This paper introduces a method to distill a diffusion model into a one-step generator. The training process combines several loss functions. First, the authors use DDIM inversion to transform real images into latent noise, which is then fed into the generator. The generated images are supervised with a reconstruction loss. Second, an additional GAN loss is applied to the generated images, distinguishing them from real images. For text-to-image generation, the authors present a novel approach to enable classifier-free guidance in the GAN loss by using an auxiliary guidance predictor. Other auxiliary losses, such as knowledge distillation loss and CLIP loss, are also employed to enhance performance. The final method are evaluated across various benchmarks.", "questions": "Most of my concerns are outlined in the weaknesses section. To reiterate, I am particularly concerned about the utilization of a pretrained classifier as a discriminator and the CLIP loss, which bias the quantitative evaluation. Additionally, the overall novelty is moderate, and some components might be unnecessarily complex.", "limitations": "yes.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. The presented approach performs well on common benchmarks while achieving good inference efficiency.\n\nS2. The authors plan to open-source the method, which will be beneficial for further research.", "weaknesses": "W1. Some loss components are questionable and may mislead readers about the actual performance of the proposed approach. For instance, it is well-known that using a GAN loss with a discriminator pretrained on ImageNet significantly biases the FID metric [1]. This issue also applies to the CLIP loss, which biases the evaluation of FID/CLIP scores when used during training. A substantial portion of the strong results presented might be attributed to these misleading numbers, making it difficult to draw accurate conclusions about effectiveness. This is a common issue in some previous works but needs to be addressed now. For example, using GANs is good, but it might be better to avoid using a pretrained classifier as the discriminator, as suggested by recent explorations in diffusion GAN literature [2]. For text-to-image results, it is recommended to conduct human evaluations to assess real image quality, prompt alignment, and diversity.\n\nW2. Many of the proposed components have been well explored in previous literature, but the connections to these works are not sufficiently discussed in the related works section. For instance, the combination of a regression loss and a GAN loss has been explored for text-to-image generation in several studies, including [3, 4]. Additionally, the use of distillation loss and CLIP loss are common practices [5, 6]. This reduces the perceived novelty of the paper to a moderate level.\n\nW3. The introduction of a classifier-free GAN objective might overcomplicate the problem. Previously, GAN loss with classifier-free guidance (CFG) was enabled by generating fake samples using an original diffusion model with guidance, as applied in [2], and then utilizing the standard GAN loss. This method is simpler than training an auxiliary classifier and has a similar computational cost, since the current classifier training also requires generating samples with CFG using the original diffusion model.\n\n\n[1] Kynkäänniemi, Tuomas, et al. \"The role of imagenet classes in fr\\'echet inception distance.\" arXiv preprint arXiv:2203.06026 (2022).\n\n[2] Sauer, Axel, et al. \"Fast high-resolution image synthesis with latent adversarial diffusion distillation.\" arXiv preprint arXiv:2403.12015 (2024).\n\n[3] Lin, Shanchuan, Anran Wang, and Xiao Yang. \"Sdxl-lightning: Progressive adversarial diffusion distillation.\" arXiv preprint arXiv:2402.13929 (2024).\n\n[4] Song, Yuda, Zehao Sun, and Xuanwu Yin. \"SDXS: Real-Time One-Step Latent Diffusion Models with Image Conditions.\" arXiv preprint arXiv:2403.16627 (2024).\n\n[5] Sauer, Axel, et al. \"Adversarial diffusion distillation.\" arXiv preprint arXiv:2311.17042 (2023).\n\n[6] Kang, Minguk, et al. \"Scaling up gans for text-to-image synthesis.\" CVPR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness 1.** To address the concerns about fair evaluation, we provide additional results below.\n\n**W1-1. GAN's discriminator pretrained on ImageNet biases the FID metric.**\n\n**Ans.**\n\n**[Frechet Distance (FD)]** We agree with the reviewer that the use of ImageNet pretrained discriminator could bias the model towards achieving a good FID. Following EDM2 practice, we evaluate FD using DINOv2 features. Tables in the attached PDF imply that FID and FD trends are consistent in ImageNet 64x64, but not for high-resolution, which may indicate FID bias. Despite this, PaGoDA outperforms its teacher EDM at both resolutions on FD. We will update the manuscript to include a full set of results and discussions. Please also refer to R2-W9 and R3-W3.\n\n**W1-2. CLIP regularizer may also bias the evaluation.**\n\n**Ans.**\n\n**[Human Evaluation]** We conduct human evaluation to fairly measure the effect of CLIP loss at Figure A of the attached PDF. According to the figure, the adoption of CLIP regularization yields positive influence on prompt alignment. Also, to minimize the undesirable bias in FID/CLIP scores, we utilized different CLIP models for training (ViT-L/14) and evaluation (ViT-g/14). These models were trained on different datasets: ViT-L/14 with YFCC100M and ViT-g/14 with LAION-2B.\n\n**Weakness 2. To address the novelty concerns, we clarify our contributions below.**\n\n**W2-1. The combination of a regression loss and a GAN loss has been explored for T2I generation in several studies.**\n\n**Ans.**\n\n**[DDIM vs. DDIM Inversion]** Although the reviewer classifies both reconstruction loss (from DDIM inversion) and distillation loss (from DDIM) as regression loss, they function differently. Previous distillation models [3, 4] adopt DDIM for distillation, starting from the prior latent and obtaining synthetic data, which may harm model performance if the synthetic data is of low-quality. In contrast, DDIM inversion in PaGoDA, starting from the real data and obtaining its latent, can benefit from the high-quality of real data.\n\n**[Theoretically Grounded GAN Integration]** Additionally, we show that existing methods combining GAN loss and DDIM-distillation loss do not optimally learn the data distribution, as noted in Theorem B.4 (extended Theorem 3.1) in the appendix. In contrast, Theorem 3.1 demonstrates that PaGoDA's objective ensures optimal data distribution—making it the first diffusion distillation approach to guarantee this theoretically. This advantage stems from using real input data $x$ to construct $(x,z)$ pairs for distillation. Empirically, Figure 5-(d) shows that PaGoDA's loss (recon+GAN) maintains performance, while KD+GAN loss degrades as the teacher model worsens (imperfect teacher model).\n\n**[GAN Stability]** Moreover, previous works have not adequately addressed GAN stability concerns. Our Theorem 3.2 provides the first guarantee that GAN can be stable if combined with DDIM inversion. Addressing these questions on optimality and stability offers new insights into the distillation community.\n\n**W2-2. The use of distillation loss and CLIP loss are common practices [5, 6].**\n\n**Ans.**\n\nPlease note that we are not claiming CLIP regularization as our contribution. Instead, we advocate using the CLIP regularizer as an option to achieve better text alignment.\n\n**W2-3. Proposed components explored in literature reduce the novelty of the paper.**\n\n**Ans.**\n\n**[DDIM Inversion + Progressive Growing = Upsampling]** We respectfully request the reviewer to re-evaluate our contribution. We are not complicating matters by adding well-established losses. Instead, we argue that DDIM inversion not only guarantees learning the data distribution (see Theorem 3.1) but, when combined with our progressively growing technique, offers an alternative to the upsampling methods used by Stable Diffusion and Cascaded Diffusion Models.\n\n**Weakness 3.** We provide explanations to address concerns about methodological complexity.\n\n**W3-1. Using the standard GAN loss is simpler than the proposed classifier-free GAN, which complicates the objective.**\n\n**Ans.**\n\n**[Upsampling]** We would like to address the reviewer's concern by connecting GAN with upsampling. Suppose $d$ is the base resolution where DM has been trained. To double the resolution, the GAN's real/fake data should also be doubled. Otherwise, the GAN cannot generate $2d\\times 2d$ images with full details.\n\n\n**[Problem of Standard GAN]** A problem arises when using standard approach of putting $\\omega$-CFG teacher sample as GAN's real input. Since the teacher sample is of $d\\times d$ dimension, without learning independent upsampling models like Cascaded Diffusion Models, there is no efficient way to upsample the teacher sample to $2d\\times 2d$. \n\n**[Use Real-world Data]** Our proposal is to use high-resolution real-world data. We put real-world data (downsized to $2d$ resolution) into GAN's real part instead of using teacher samples. We compare the standard approaches, and our proposal as follows:\n- Standard way: generate $\\omega$-CFG teacher sample and feed it to GAN's real part.\n    - (-) Student resolution not *extendable*\n    - (-) Reach to *teacher* quality at optimum\n- Our proposal: retrieve real data, downsize it, predict $\\omega$ of it, and put it into GAN's real part.\n    - (+) Student resolution *extendable*\n    - (+) Reach to *data* quality at optimum\n\n**[Use Released Checkpoint]** We recognize that our approach may involve higher initial costs due to the need to train the classifier. However, once the classifier is publicly available, the training cost will be similar to the standard approach. We kindly ask the reviewer to focus on the qualitative benefits of our proposal.\n\n**[Concurrent Works]** The literature [2-6], released publicly within 2-3 months before the NeurIPS submission, should generally be considered concurrent work to ours. We could add a section in our revision, but we think it is unfair to evaluate our method based on them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jNEgC51Tcn", "reviewer": "Reviewer_FHu2", "summary": "The paper introduces a novel GAN-based diffusion distillation approach that leverages ideas from PG GAN for effective high-resolution synthesis.", "questions": "* Please address the questions and concerns raised in \"Weaknesses\".\n\n* Intuitively, the reconstruction and distillation losses are most beneficial for the base-resolution training. Did the authors try to deactivate these losses for SR stages? \n\n* Did the authors consider using a pretrained DM as a discriminator, following the ideas from [6]? It would be interesting to explore if noise-level feedback might be valuable for learning better generators. \n\n* PaGoDa approaches very low FID values on ImageNet. Could the performance gains be a result of overfitting the training data?", "limitations": "The authors addressed the limitations and potential negative social impact in the appendix.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* PaGoDA demonstrates state-of-the-art single-step generation performance on ImageNet, competing with or outperforming more expensive alternatives, including the teacher DMs. \n\n* The proposed classifier-free GAN is an interesting modification for better conditional generation.\n\n* The authors justify most components of the proposed method by providing valuable discussions and ablation studies.\n\n* The paper includes thorough theoretical analysis, as well as many experimental and implementation details.", "weaknesses": "* PaGoDa is designed for pixel-based DMs only and hence is not applicable to most state-of-the-art T2I models, e.g., SDXL, SD3. Also, the motivation behind this (L42) seems a bit arguable. For example, one could try to adapt PaGoDa to LDM by replacing the AE decoder with a shallow one mapping a 64x64 latent variable to a 64x64 image. Then, PaGoDA can be applied as is. Do the authors have any thoughts on this?\n\n* The method requires collecting the dataset of a large amount of (x, z) and ($\\hat{x}_w$, z) pairs. Likely, this significantly increases the training costs compared to sampling-free alternatives. Could the authors provide the data collection vs training costs? How does the overall training time compare to other distillation approaches?\n\n* The KD objective is used for the guided generators (i.e., using the reverse process for sampling ($\\hat{x}_w$, z) pairs). This contradicts the motivation behind the reconstruction loss (L89). While the ablation study in Tab.7 aims to address this, could the authors clarify the inference CFG scale used in this experiment? How does the role of the reconstruction loss change with different CFG scales? \n\n* PaGoDa uses CLIP regularization in the T2I setup. I believe this makes the CLIP score evaluation incorrect. Moreover, other distillation methods, e.g., LCM, DMD, or ADD, do not use it. This raises concerns about the performance of the core approach in the T2I setting. \n\n* For T2I evaluation, automated metrics have been shown to correlate poorly with human preference (e.g., [1,2]). It would be highly beneficial to conduct a human study comparing PaGoDA to other distillation approaches and T2I diffusion models. If a reliable human evaluation setup is unavailable, I would suggest evaluating a metric learned to mimic human preference, e.g., [3,4,5].\n\n* The CFG details for T2I inference are missing. What scale is in the final t2i results? How does the model perform with different CFG scales?\n\n* The paper does not provide qualitative T2I results demonstrating the diversity of generated images for a given prompt.\n\n* The ablation study lacks evaluation of the proposed approach without adversarial objective. \n\n\n[1] Podell et al. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis.\n\n[2] Li et al. Playground v2.5: Three Insights towards Enhancing\nAesthetic Quality in Text-to-Image Generation.\n\n[3] Xu et al. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation\n\n[4] Kirstain et al. Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation.\n\n[5] Wu et al. Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis.\n\n[6] Sauer et al. Fast High-Resolution Image Synthesis with\nLatent Adversarial Diffusion Distillation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the reviewer for helpful reviews. Below, we faithfully answer the raised concerns, but we would be happy to provide additional experiments upon reviewer's request in our final revision.\n\n**Weakness 1. Can we integrate PaGoDA with SD?**\n\n**Ans.**\n\nYes, we can integrate PaGoDA in the latent space of SD. If $d$ (say 512) is the latent resolution of SD (data resolution could be up to 2K or 4K), we can train a DM on $d/2^n$ (say 64) dimension (which is much cheaper than training DM on full 512 latent dimension) and then distill/upsample with PaGoDA up to $d$. Although we focus on the core algorithm with pixel DM to pursue a fundamental understanding of core components, we are interested to integrate PaGoDA with SD in future to examine the degree of compression this integration can offer.\n\n**Weakness 2. Discuss the training costs compared to sampling-free alternatives.**\n\n**Ans.**\n\n**[Data Collection is Optional]** While we utilized collected data-latent pairs mainly to save budget for model building, once training configurations are set, training data pairs can be obtained online during training.\n\n**[Minimal Overhead]** In T2I, convergence is achieved around 20k iterations, requiring about 1 day with 8xH100, including latent calculation. This overhead is minimal compared to the total DM pretraining cost.\n\n**[Safer Training]** We further address that sampling-based methods are safety-secure. During training, a classifier can filter out problematic samples related to privacy, NSFW, and copyright. Moreover, maintaining clear records of training data aids in post-mortem analysis.\n\n**Weakness 3. Adoption of KD objective in T2I is contradictory to reconstruction loss's motivation.**\n\n**Ans.**\n\n**[KD for High CFG]** Figure 4 highlights that the latents from DDIM inversion are out-of-prior at high CFG scales. Therefore, a model trained with reconstruction loss performs poorly at these scales. To resolve this problem, we impose the KD loss. However, using KD-only limits student upper bounded by teacher.\n\n**[Effect of Recon+KD]** When we add reconstruction loss, the FID-CLIP curve is shifted right (higher CLIP) and below (better FID) with the optimal CFG (w.r.t FID) falls from 2.5 to 1.15. We also report Figure A of the attached PDF of human evaluation, resulting in better image quality and prompt alignment.\n\n**[Eliminate KD]** We are interested in eliminating KD objective by substituting CFG with CFG++ [A], a recent method with improved DDIM and DDIM inversion on guidance scale less than 1. On these scales, DDIM latent falls into in-distribution that facilitates model training without KD. We originally planned to explore this as future works but would be happy to investigate on this if reviewer requests in our final revision.\n\n[A] CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models\n\n**Weakness 4. Using CLIP regularization makes the CLIP score evaluation incorrect.**\n\n**Ans.**\n\nFigure A in the attached PDF shows that CLIP regularization is advantageous for prompt alignment under human judgement. To minimize potential bias in evaluation, we utilized different CLIP models for training (ViT-L/14 trained on YFCC100M) and evaluation (ViT-g/14 trained on LAION-2B).\n\n**Weakness 5. Clarify the CFG details for inference.**\n\n**Ans.**\n\nThe CFG scale in Tab. 5 and 6 is 1.15. As shown in Figure 14, CFG in PaGoDA exhibits a similar trend to teacher diffusion.\n\n**Weakness 6. Does PaGoDA work without adversarial objective?**\n\n**Ans.**\n\n**[Existence of Prior Holes]** PaGoDA requires a GAN loss. Since PaGoDA's encoder is a deterministic mapping, it avoids posterior collapse but suffers from prior holes. These holes are unseen during training with recon-only loss, causing decoder to struggle in creating high-quality samples in those areas. Therefore, we need GAN loss to cover the entire prior manifold during training.\n\n**[Uniform Encoding]** DDIM inversion encodes data uniformly into the prior manifold, as illustrated in Figure B of the attached PDF. This uniformity constrains generator, preventing mode collapse by densely regulating its output within the prior space through reconstruction loss.\n\n**[Well-Grounded GAN]** Moreover, Theorem 3.1 proves that the optimal generator of a GAN-augmented distillation model matches data distribution, unlike previous works where optimal generator is heavily influenced by pretrained DM. Theorem 3.2 confirms the stability of combining GAN with reconstruction loss.\n\n**Question 7. Can we deactivate reconstruction/distillation losses for SR stages?**\n\n**Ans.**\n\nDeactivating recon/distill losses causes model to solely depend on GAN. As seen in Figure 6, relying solely on GAN for SR results in object location shifted across resolution. This unwanted shift hinders the ability to control samples in base resolution. By activating reconstruction loss, the object's position remains consistent in resolution changes.\n\n**Question 8. Can we use a pretrained DM as a discriminator feature extractor?**\n\n**Ans.**\n\nYes, but note that feature extractors behave differently. According to [6], discriminative features prioritize local details over global shape, while generative features do the opposite. Since reconstruction loss strongly penalizes errors in shaping the global context, we used features from discriminative models to prioritize capturing local details with GAN loss.\n\n**Question 9. Could the ImageNet performance a result of overfitting?**\n\n**Ans.**\n\nTable A of the attached PDF shows the Frechet Distance on DINOv2 feature space, comparing 50k samples with 50k val data. The degree of overfitting (FD val - FD train) in PaGoDA is comparable to baselines, implying PaGoDA is as robust as baselines in overfitting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mA8agjtbCo", "reviewer": "Reviewer_1Kk6", "summary": "The paper proposes PaGoDA, an adversarial distillation method to support single-step generation of image resolutions higher than the teacher diffusion model. PaGoDA first solves the forward PF-ODE (data to noise) to collect noise from the dataset. Then, it gradually adds upsampling layers to the model and trains only those layers. The training objectives include reconstruction loss and GAN loss, and for text-to-image experiments, additional distillation loss and CLIP regularization are introduced to support classifier-free guidance. PaGoDA demonstrates strong one-step generation performance on ImageNet and COCO benchmarks and shows its superiority through various ablation studies.", "questions": "- What classifier-free guidance scale did PaGoDA use in Table 6? Is the point that achieved the best performance in Figure 14 reported in Table 6?\n\n- Why were different datasets used when collecting latents (CC12M) and during GAN training (COYO-700M)?\n\n- Does the DDIM inversion in Section 5.4 needs a diffusion teacher for inversion?", "limitations": "Appendix C explains well the gap between PaGoDA's theoretical assumptions and its practice.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "**Originality**\n\n- This is the first study to support higher resolution than the teacher diffusion model when distilling a diffusion model into a single-step generator.\n\n- It is also the first study to use the forward PF-ODE to collect noise-data pairs for distillation.\n\n**Significance**\n\n- Despite using GAN loss, the training is stable due to the progressive growing strategy, as shown by the good performance at various resolutions in Table 2.\n\n- By using the forward PF-ODE for distillation, it learns the true data distribution and achieves higher sample diversity compared to GANs, as evidenced by the recall performance in Table 2.\n\n- The PaGoDA is shown to be applicable to the pixel-space text-to-image model, DeepFloyd.\n\n**Quality**\n\n- The paper is well-written and includes detailed implementation details in the appendix for reproduction. \n\n- Various ablation studies demonstrate the superiority of each loss and the upsampling method of this approach compared to the VAE decoder of latent diffusion models.", "weaknesses": "- PaGoDa is claimed to have a faster sampling speed compared to LCM, but wouldn't the sampling speed be faster with **smaller VAE** [A]? Wouldn't LCM be more efficient than PaGoDa if the VAE decoder has less layers than PaGoDa's additional upsampling layers?\n\n- Additionally, smaller VAE only requires training of a VAE, while PaGoDa involves sequential training stages depending on the resolution, making the training pipeline somewhat inconvenient. \n\n- In Table 4, PaGoDa performs better than the diffusion teacher, and in Table 1, PaGoDa outperforms EDM2-XXL. This may be because **the discriminator uses EfficientNet and DeiT pre-trained on ImageNet**, leading to better FID scores, which use an Inception network also trained on ImageNet. This issue has been discussed in [B]. There are two potential solutions to this problem. The first is to compare **Fréchet distances using DINOv2**, following the EDM2 paper. The second is to **train the discriminator for GAN loss from scratch**. If PaGoDa's GAN loss is truly stable, it should be possible to train without a pre-trained network.\n\n[A] Tiny AutoEncoder for Stable Diffusion: https://github.com/madebyollin/taesd \n\n[B] Kynkäänniemi et al., The Role of ImageNet Classes in Fréchet Inception Distance, ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer’s constructive feedback.\n\n**Weakness 1. LCM with lighter VAE may be more efficient than PaGoDA's upsampling blocks?**\n\n**Ans.**\n\n**[Lightweight PaGoDA]** The smaller VAE [A] decoder can be integrated into PaGoDA's super-resolution framework. If LCM's VAE is made more lightweight, PaGoDA's network could be similarly lightweighted. The degree of simplification for heavy decoder (or upsampling network) is determined by the function complexity. As PaGoDA’s upsampling function is simpler than VAE's nonlinear transformation, PaGoDA is more suitable for model distillation. Also, a lightweight decoder sometimes fail at high-quality generation, illustrated in Figure C in the attached PDF.\n\n**Weakness 2. A smaller VAE requires single-stage training, while PaGoDA involves multiple sequential stages.**\n\n**Ans.**\n\n**[Smaller VAE is Sequential Training]** The smaller VAE [A] separately distills a heavy encoder and decoder with lightweight counterparts to match latent distributions. In knowledge distillation, lightweight models distilled from heavy teachers tend to perform better than those trained without teacher. Likewise, the smaller VAE [A] requires to have the original VAE to minimize the performance sacrifice, which falls into a sequential training.\n\n**[PaGoDA Skips Intermediate Stages]** PaGoDA's progressive upsampling can be compressed. Instead of doubling resolution jumps, they can be increased by factors of 4x or 8x, allowing the super-resolution network to be trained all at once. It was observed that even with an 8x jump, model remained stable and performance was retained.\n\n**Weakness 3. FID evalutaion could be biased due to the use of EfficientNet and DeiT pre-trained on ImageNet.**\n\n**Ans.**\n\n**[Frechet Distance]** Following reviewer's suggestion, we additionally report FD using DINOv2 in Tables A and B of the attached PDF. On ImageNet 64x64, we have three implications:\n- vs. EDM & CTM: All three models (PaGoDA, EDM, CTM) share the same architecture. CTM performs better than teacher EDM only in FID, a signal of FID bias. However, PaGoDA consistently outperforms teacher EDM in both FID and FD.\n- vs. StyleGAN-XL: Direct comparison between PaGoDA and StyleGAN-XL is challenging due to differences in architectures and training details. Nonetheless, PaGoDA outperforms StyleGAN-XL in all metrics.\n- vs. Validation Data: Given that PaGoDA's FID is better than the validation data while its FD is worse, FID may no longer be a reliable metric for model evaluation.\n\nOn ImageNet 512x512, we compare PaGoDA as follows:\n- vs. EDM: PaGoDA significantly surpasses EDM in both FID and FD, even with the same architecture (EDM is trained on latent space).\n- vs. EDM2: EDM2 outperforms PaGoDA in FD, regardless of the number of parameters used. However, please note that EDM2 requires $63$ steps to achieve good FDs, while PaGoDA requires only a single step.\n\nBased on these findings, we will focus on comparing PaGoDA using FD in our paper revision. We plan to explore whether applying PaGoDA on EDM2 architecture would bring us a better quality in future work.\n\n**Question 4. CFG scale used in table 6 for PaGoDA: is the best performance point in Fig. 14 reported in Table 6?**\n\n**Ans.**\n\nIt is correct that we used the best performance in Figure 14, with a CFG scale of 1.15, to report the score in Table 6. Adopting reconstruction loss reduced the optimal CFG scale from 2-3 to 1-2.\n\n**Question 5. Why were different datasets used when collecting latents (CC12M) and during GAN training (COYO-700M)?**\n\n**Ans.**\n\n**[Prevent Overfitting]** We used different datasets to mitigate overfitting and memorization issues. Let's suppose that $z$ is a latent of $x$ from CC12M. Then, the reconstruction loss pushes $G(z)$ towards $x$. If we utilize the same CC12M for GAN training, then the discriminator will push $G(z')$ towards CC12M, where $z'$ is a neighborhood of $z$. Therefore, to alleviate $G(z')$ being too close to $x$ in CC12M, we provide an effect of $G(z')$ diverging from CC12M by training discriminator with COYO-700M.\n    \n**Question 6. Does the DDIM inversion in Section 5.4 needs a diffusion teacher for inversion?**\n\n**Ans.**\n\nYes, we utilized teacher diffusion in Section 5.4 for controllable generation.\n\n\n**<Final comments to Reviewer 3>**\n\nWe respectfully request the reviewer to re-evaluate our paper with an emphasis on its upsampling capabilities. The components, though seemingly separate, are integrated to achieve efficient super-resolution generation.\n\n**[Upsampling: DDIM Inversion for Recon Loss]** DDIM inversion allows obtaining a latent $z$ from high-dimensional input data $x_{\\text{high}}$ by downsampling it to DM's resolution. By training a map from $z$ to $x_{\\text{high}}$, PaGoDA provides upsampling capabilities alternative to SD's VAE and Cascaded Diffusion Models. When using DDIM, we cannot train such an upsampling map.\n\n**[Upsampling: Classifier-Free GAN for Adversarial Loss]** Previous diffusion distillation methods used standard GANs with $\\omega$-CFG teacher samples, limiting student performance and resolution extension. An easy fix could be using appropriately resized real-world data into GAN's real part, but this way is incompatible to T2I because the optimal student follows $p_{\\text{data}}(x|c)$, not $p_{\\text{data}}(x|c,\\omega)$. Classifier-Free GAN solves this by predicting $\\omega(x,c)$ and feeding it into the student network, ensuring alignment with the desired distribution $p_{\\text{data}}(x|c,\\omega)$ (see also L157-L159).\n\n**[Upsampling = Progressive Growing + DDIM Inversion + Classifier-Free GAN]** By integrating three proposed components altogether, we achieved efficient super-resolution generation in both training and sampling, providing a new and effective alternative to Stable Diffusion and Cascaded Diffusion Models. We kindly ask the reviewer to reconsider our paper with this perspective in mind.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all the reviewers for their constructive and helpful feedback. For a clearer evaluation, we would like to highlight a high-level overview of the contributions in this paper.

**[Switch DDIM to DDIM Inversion]** PaGoDA proposes to utilize DDIM inversion for distillation. Unlike DDIM, which starts from Gaussian prior to synthesize fake data for generator training, DDIM inversion begins from real data to its latent representation for generator training. This approach allows PaGoDA to benefit from the high quality of real data, whereas distillation from DDIM depends on the quality of synthetic data, which can be less reliable.

**[DDIM Inversion for Upsampling]** DDIM inversion, starting from real data, enables the generator to learn up to the resolution of the real data. In contrast, DDIM alone, without additional upsamplers like Cascaded Diffusion Models, cannot easily extend generator's resolution to higher dimensions.

**[Switch Standard GAN to Classifier-Free GAN]** PaGoDA introduces a GAN loss tailored for super-resolution generation and compatible with CFG. While standard GAN distillation puts $\omega$-CFG teacher samples on the GAN's real part, this approach, similarly to DDIM, limits resolution due to reliance on synthetic data. Instead, we use real data in the GAN's real part, aligning with the motivation of using DDIM inversion. To ensure the student learns the $\omega$-conditioned data distribution $p_{\text{data}}(x|c,\omega)$ rather than $p_{\text{data}}(x|c)$, we predict $\omega(x,c)$ for the real data and use this in the student network evaluation, completing the design of  Classifier-Free GAN.

**[Upsampling = Progressive Growing + DDIM Inversion + Classifier-Free GAN]** High-dimensional generator can be achieved by incorporating three proposed core components altogether: progressive growing (decoder architecture), DDIM inversion (reconstruction loss), and Classifier-Free GAN (adversarial loss). With no single component, PaGoDA is not successfully working, necessitating all three components as essential ones. This high-dimensional generator, trained from low-dimensional teacher DM, provides a new and effective alternative to Stable Diffusion and Cascaded Diffusion Models. We, therefore, kindly ask the reviewer to reconsider our paper with this perspective in mind.

We attach one-page of additional PDF here for the reviewer's information.

Meta-review: The paper introduces the model for the adversarial distillation method for single-step-image generation. Progressive growing GAN inspired the proposed method and combined the benefits of GAN's upscaling and diffusion models. The proposed model solves an essential problem of efficient conditioned image generation in the viral domain. The authors highlight the benefits of the solution compared to other efficient approaches like LCM. 

The reviewers expressed quite a lot of concerns. The major issues were focused on using CLIP both as a regularizer and evaluator, human-based evaluation for T2I scenarios, application to some existing models, like SD or novelty, and relation to the most recent works. After the rebuttal stage, the major concerns were addressed, and a consensus among the authors was achieved regarding accepting the paper. In my opinion, the proposed approach is novel and represents a promising new direction in the field of generative model distillation applications. Therefore, I recommend this paper for acceptance.
Decision: Accept (poster)