Title: PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher
Venue: NeurIPS 2024
Accepted: yes
Keywords: Diffusion Models; Distillation Models; Generative Models; Generative AI; Text-to-Image Generation; Generative Adversarial Networks (GAN); Variational Autoencoders (VAE)
Abstract: The diffusion model performs remarkable in generating high-dimensional content but is computationally intensive, especially during training. We propose Progressive Growing of Diffusion Autoencoder (PaGoDA), a novel pipeline that reduces the training costs through three stages: training diffusion on downsampled data, distilling the pretrained diffusion, and progressive super-resolution. With the proposed pipeline, PaGoDA achieves a $64\times$ reduced cost in training its diffusion model on $8\times$ downsampled data; while at the inference, with the single-step, it performs state-of-the-art on ImageNet across all resolutions from $64\times64$ to $512\times512$, and text-to-image. PaGoDA's pipeline can be applied directly in the latent space, adding compression alongside the pre-trained autoencoder in Latent Diffusion Models (e.g., Stable Diffusion). The code is available at https://github.com/sony/pagoda.

PaGoDA : Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher

Abstract

The diffusion model performs remarkable in generating high-dimensional content but is computationally intensive, especially during training. We propose P rogressive G rowing o f D iffusion A utoencoder ( PaGoDA ), a novel pipeline that reduces the training costs through three stages: training diffusion on downsampled data, distilling the pretrained diffusion, and progressive super-resolution. With the proposed pipeline, PaGoDA achieves a 64 × reduced cost in training its diffusion model on 8 × downsampled data; while at the inference, with the single-step, it performs state-of-the-art on ImageNet across all resolutions from 64 × 64 to 512 × 512 , and text-to-image. PaGoDA's pipeline can be applied directly in the latent space, adding compression alongside the pre-trained autoencoder in Latent Diffusion Models (e.g., Stable Diffusion). The code is available at https://github.com/sony/pagoda .

1 Introduction

Diffusion Models (DM) [1, 2], which generate content through gradual denoising, have recently achieved high fidelity in high-dimensional generation [3, 4]. While slow sampling has been improved by distilling trained DMs into single-step generators [5-7], DMs remain computationally intensive, especially at high resolutions, requiring substantial data and GPU resources, thereby limiting largescale training to a few organizations [8, 9]. This highlights the need for a more efficient pipeline to reduce both training and inference costs while maintaining the quality.

To address these challenges, we present P rogressive G rowing o f D iffusion A utoencoder ( PaGoDA ), a novel pipeline that significantly reduces costs while achieving competitive quality with one-step sampling. PaGoDA is built on a simple yet effective idea: while diffusion distillation [6] is typically treated as a final stage of the whole pipeline, we explore to have one more stage for the superresolution after diffusion distillation. This approach led us to design PaGoDA with three distinct stages as below.

Stage 1.

Stage 2.

Diffusion Distillation (

)

Stage 3.

Super-Resolution

(

)

DDIM Inversion with Pretrained DM

Downsample

Single-Step

香

PaGoDA's Proposed Training Pipeline

Stage 1. (Pretraining) Train a DM on downsampled data.

Stage 2. (Distillation) Distill the trained DM with DDIM inversion to a one-step generator.

Stage 3. (Super-Resolution) Progressively expand the generator for resolution upsampling.

By adding Stage 3 for the super-resolution after the distillation phase, our approach gains a key advantage: training DM on a low-dimensional, downsampled space rather than directly in the desired high-dimensional space. This dimensional reduction substantially lowers the computational demands of diffusion pretraining by orders of magnitude. For example, an 8 × 8 downsampling rate reduces the training computation by a factor of 64 × . Moreover, the computational costs for the distillation and super-resolution stages are relatively minimal compared to the initial diffusion pretraining, making our pipeline highly efficient in terms of overall computation.

Figure 1 provides an overview of our pipeline. We begin with DM trained at base resolution, and generate a dataset of base-resolution data-latent pairs ( x , z ) , where x is real data and z is the latent representation of x , obtained by DDIM inversion [10]. In Stage 2, we train a decoder to map z back to x , completing the diffusion distillation [6]. In Stage 3, we add ResNet blocks [11] to enhance sample resolution and progressively train these newly added upscaling networks, as visualized in Figure 2. The novel use of DDIM inversion in the distillation process, first introduced in PaGoDA, enables the decoder to be trained with the high-frequency signal from the real data at Stage 3. This integration of DDIM inversion establishes strong connections across stages, creating a cohesive and unified framework.

Input

Block

Middle

Block

Output

Block

To RGB

Train

Freeze

Input

Block

Middle

Block

To RGB

Output

Block

Output

Block

Skip Connection

Stage 2

Distillation

Stage 3

Super-Res

In our experiments, we employed the progressively growing generator to upsample from the pretrained diffusion model's 64 × 64 resolution to generate samples at 512 × 512 resolution. Notably, PaGoDA achieved state-of-the-art (SOTA) Fréchet Inception Distances (FID) [12] on ImageNet across all resolutions from 64 × 64 to 512 × 512 . Additionally, we demonstrated PaGoDA's effectiveness in addressing inverse problems and facilitating controllable generation. However, PaGoDA's potential extends beyond its current application. As PaGoDA being a dimensional reduction technique that operates independently of Latent Diffusion Models (LDM) [3], PaGoDA could be directly applied into the latent space as-is, offering the possibility of further gain on training computes. We leave this exploration as a promising avenue for future research.

2 Preliminary

DM[1] samples from the data distribution p data through an iterative denoising process, beginning from a Gaussian prior distribution p prior. This denoising process attempts to reverse [2] a forward diffusion process. If the forward process is defined by d x t = √ 2 t d w t [13], the deterministic counterpart of the denoising (generation) process, known as the probability flow ordinary differential equation (PF-ODE) [2], or DDIM [10], is expressed as

where s ϕ 0 ( x t , t ) is a neural approximation of ∇ log p t ( x t ) . Consequently, (deterministic) sample generation from DM is equivalent to solving the PF-ODE (or DDIM) along the trajectory, formally,

Modern solvers of the PF-ODE [10, 14] have significantly accelerated sampling speed, reducing the required network evaluations from hundreds to tens. To further speed up sampling, DMs are distilled with a student model [6] G θ : R d → R d to map from x T to x DDIM 0 ( x T ) by minimizing

We call this DDIM-based approach as the noise-to-data distillation.

3 Progressive Growing of Diffusion Autoencoder

3.1 Stage 1: Diffusion Models Trained on Downsampled Data

Training DMs for high-dimensional data generation is primarily feasible for a limited number of well-resourced organizations, largely due to two factors: access to large-scale datasets and substantial computational resources. This centralization of model development underscores the urgent need to democratize access by significantly reducing resource demands during diffusion training. While several strategies [15, 3] have been proposed, our approach, PaGoDA, introduces a paradigm shift by training the DM at a downsampled resolution in Stage 1, rather than at the original full resolution. For instance, training on a d -dimensional downsampled resolution requires approximately 4 n times less computational budget compared to training in the full 4 n d -dimensional space. In practical terms, when n = 3 , this translates to training in an 8 × 8 downsampled space, effectively reducing training costs by a factor of 64 × , thus making large-scale diffusion training more accessible to a broader range of researchers.

Although this paper does not extend PaGoDA's application to the LDM such as SD, training on a (say) 4 × 4 downsampled latent space could theoretically reduce the computational cost by 16 × compared to full-resolution latent training, further emphasizing PaGoDA's potential for widespread adoption. In the case of generating 1024 × 1024 images, PaGoDA requires training the diffusion model at only 32 × 32 resolution, with Stage 3 subsequently upscaling it to the full 128 × 128 latent space of conventional approaches [8, 9]. This progressive approach illustrates PaGoDA's effectiveness in maintaining model quality while lowering the barriers to high-resolution diffusion training.

3.2 Stage 2: Diffusion Distillation on Downsampled Data with DDIM Inversion

After pretraining DM on the downsampled space, PaGoDA distills DM to a one-step generator. For distillation, PaGoDA introduces a new loss specifically designed for later usage in super-resolution at Stage 3. In particular, we propose the reconstruction loss (compare it with L dstl in Eq. 1 of Section 2)

where x DDIM -1 T ( x 0 ) is now the latent representation of x 0 , obtained from DDIM inversion, not from DDIM, i.e., the solution at time T of the PF-ODE starting from x 0 in time forward, defined by

64 × 64

128 × 128

256 × 256

512 × 512

(a) Resolution Jump w/ Recon Loss

64 × 64

128 × 128

256 × 256

512 × 512

(b) Resolution Jump w/o Recon Loss

Distillation using L rec maps latent representations to real data, following a data-to-noise distillation approach. While this method has the potential to improve real data alignment compared to the traditional noise-to-data approach in Eq. 1, we observe a decline in generation quality over iterations. This decline stems from the prior hole problem [16], where the generator's input, x DDIM -1 T , derived from limited real data, fails to cover the entire prior manifold, leaving certain regions unexplored.

A straightforward strategy like early stopping could alleviate this issue, but it restricts the use of Exponential Moving Average (EMA) in Stage 2. To fundamentally resolve this problem, we propose a solution that maintains generation quality even during prolonged training. In Section 3.4, we provide optimal and stability analysis of Stage 2, guaranteeing that our proposal is stable across training iterations. The key challenge is effectively covering the prior manifold, which we address by introducing an auxiliary adversarial loss, as defined below:

Here, D ψ is a discriminator that classifies the real and fake samples by maximizing the adversarial loss, and p prior ( z ) is the prior distribution.

The second term in L adv, which involves G θ ( z ) with z sampled from the prior, ensures that the decoder is exposed to the entire support of the prior distribution during training. Overall, we train PaGoDA with the mini-max optimization of the following combined objective:

While PaGoDA incorporates the adversarial loss, the reconstruction loss simultaneously guides the decoder to accurately reconstruct the entire training data. This combination allows the adversarial loss to address underrepresented regions in the prior distribution effectively without compromising sample diversity. In our ImageNet experiments, we found that updating the reconstruction loss with as little as 1% of the data-latent pairs did not affect sample quality and diversity. Exploring the impact of varying the number of data-latent pairs is left as a future work.

3.3 Stage 3: Progressively Growing Decoder for Super-Resolution

Stage 3 trains the super-resolution to generate higher-dimensional data from the downsampled resolution learned in the previous stages. As illustrated in Figure 2, the resolution jump from R d to R 4 n d is achieved by freezing most parameters of the distilled model from Stage 2 and training only the final layers, which is augmented with an additional upscaler network (of ResNet blocks [17]). In other words, within the base-resolution U-Net [18], we freeze its input, middle, and output blocks except for the last few layers (previously highest resolution block) during Stage 3 training. Consequently, the unfrozen latter part of the network is trained for super-resolution. We suggest to progressively increasing the resolution by a factor of 2 × , though larger jumps by factors of 4 × or 8 × yield comparable performance.

Additionally, the last layer typically converts multi-channel (usually 128 or 256 channels) features to 3-channel RGB output. However, to minimize information loss, we retain these features and pass them directly to the next output block without converting them to 3 channels. This architectural choice, along with progressive training, is heavily inspired by Progressive Growing GAN [19].

In Stage 3, the reconstruction loss from Stage 2 is adapted as

where x 0 ∈ R d is the downsampled counterpart of x high ∈ R 4 n d . The adversarial loss in this stage is

Overall, both the reconstruction and adversarial losses are combined to guide training.

Stage 3 employs two key mechanisms to effectively capture high-frequency details while maintaining training stability. First, the reconstruction loss is applied directly to highdimensional real data, which was not feasible with earlier noiseto-data distillation methods with Eq. 1. As illustrated in Figure 3, L rec stabilizes the upscaling process by preventing objects from shifting across resolutions, allowing the added neural network to focus solely on upsampling. Second, the adversarial loss operates directly in high-dimensional space, enabled by the one-step generator trained in Stage 2. This generator is critical; without it, adversarial training in Stage 3 would be infeasible. As shown in Figure 4 tested on ImageNet, the adversarial loss is pivotal for achieving effective upscaling performance.





























3.4 Optimality Guarantee and Training Stability of PaGoDA Pipeline

When using the conventional L dstl for distillation, the optimal student becomes G θ ∗ ( x T ) = x DDIM 0 ( x T ) , meaning that the student's samples replicate those of DDIM. As a result, the student's performance is heavily dependent on the teacher's performance. Consequently, the student's generative distribution may diverge from the real data distribution, even when L dstl is combined with adversarial loss. In contrast, by using the DDIM inversion-based reconstruction loss proposed in Stage 2, we mathematically prove in Theorem 3.1 that the optimal student's generative distribution aligns with the real data distribution. As visualized in Figure 5, our PaGoDA Stage 2 (red) achieves robust performance even with a weaker teacher, unlike traditional noise-to-data distillation loss L dst of Eq. 1, which struggles despite the use of adversarial loss.





















+

adv

rec

+

adv

Theorem 3.1. Let λ > 0 . Suppose D ∗ ( G ) ∈ arg max D L adv ( G,D ) . If both PaGoDA's reconstruction loss and adversarial loss share a common minimizer G ∗ , then p G ∗ ( x ) = p data ( x ) . Here, p G ∗ is the generative distribution learned by optimizing Eq. (4).

Additionally, Theorem 3.2 shows that PaGoDA's training is stable with the help of reconstruction loss, even with adversarial training. We empirically observe that PaGoDA can be trained effectively without many of the techniques typically used to stabilize GANs [20, 21].

Theorem 3.2. [Informal] Let E be a fixed deterministic encoder. Suppose that at the generator's equilibria G ∗ of Eq. (4), p G ∗ ( x ) = p data ( x ) , and x = G ∗ ( E ( x )) . Then, under conditions similar to those found in the stability literature for improving GAN [22, 21], training with Eq. (4) is stable (gradient descent locally converges to its equilibria).

We refer to Theorems B.4 and B.9 for rigorous and extended versions of Theorems 3.1 and 3.2, respectively. All proofs can be found in Appendix B.

4 PaGoDA with Classifier-Free Guidance

In this section, we integrate Classifier-Free Guidance (CFG) [23, 4] into PaGoDA for Text-toAny generation, with a focus on Text-2-Image. Incorporating CFG alters the sample distribution, necessitating adjustments to the loss functions for Stages 2 and 3. Since previous GAN literature [2427] has not addressed CFG integration, we introduce the classifier-free guided adversarial loss to accommodate this adaptation.

CFG guides the denoising process by adjusting the conditional score gradient ∇ log p t ( x t | c ) into a guided score ∇ log p t ( x t | c ) + ( ω -1) ∇ log p ( c | x t ) . This adjustment leads our distillation learning target from p data ( x | c ) to p data ( x | c , ω ) , defined by

reflecting the influence of guidance strength ω .

4.1 Classifier-Free Guided Adversarial Loss

To describe the classifier-free adversarial loss, we first consider the loss:

where now both generator and discriminator incorporates ω as an additional condition [28], see Eq. (3) for the comparison. From the standard GAN argument [29], this GAN loss guarantees the optimal generator to match to the data distribution, i.e., p G ∗ ( x | c , ω ) = p data ( x | c , ω ) . Hence, the classifier-free adversarial loss could be defined by

A key challenge with L c ,ω adv is that sampling from p data ( x | c , ω ) is generally infeasible, making it difficult to compute the first term of L CFG adv . To address this issue, we leverage the Bayes formula

where both representations are two different ways to decompose the joint distribution over ( x , c , ω ) , with π ( ω ) being the prior distribution of the CFG scale ω . From this formula, if we could predict the guidance weight ω by observing x and c , i.e., if we know p ( ω | x , c ) , then sampling ( x , c , ω ) from p data ( c ) π ( ω ) p data ( x | c , ω ) can be alternatively achieved by: 1) sampling ( x , c ) from p data ( x , c ) , and 2) predicting most likely ω using p ( ω | x , c ) .

We approximate p ( ω | x , c ) with a U-Net encoder network with 1-dimensional output, called CFG weight estimator ω ϕ . The input of ω ϕ network is a single-channel matrix with ( i, j ) -th value as the multiplication of the i / j -th values of x / c CLIP embeddings, respectively. As this matrix is high-dimensional, we input the downsampled 64 × 64 × 1 matrix to the U-Net encoder. These CLIP embeddings are also used to condition the network. With DM pretrained at Stage 1, which is supposed to be sufficiently close to the data distribution, we train the CFG weight estimator by minimizing E p prior ( z ) p data ( c ) π ( ω ) [ ∥ ω -ω ϕ ( ˆ x ( z , c , ω ) , c ) ∥ 2 2 ] , where ˆ x ( z , c , ω ) is a clean base-resolution sample drawn the teacher diffusion. Then, ω ϕ ( x , c ) -value becomes the point estimation of p ( ω | x , c ) .

4.2 PaGoDA Pipeline with Classifier-Free Guidance

We replace the adversarial loss in Stages 2 and 3 with the proposed classifier-free guided adversarial loss. In Stage 3, we shift the focus from x ∈ R d to x high ∈ R 4 n d to effectively capture high-frequency details. Additionally, in both Stages 2 and 3, we replace the input of the generator in the reconstruction loss to be classifier-free guided DDIM inversion noise. To enhance text-sample alignment, we further regularize training with CLIP [30] similarity. For training, we use the ViT-L/14 [31] CLIP model pretrained on YFCC100M [32], while for evaluation, we use the ViT-g/14 CLIP model pretrained on LAION-2B [33], minimizing the risk of overfitting.

5 Experiments

5.1 PaGoDA Tested on ImageNet without CFG

We conduct experiments on ImageNet using PaGoDA without CFG to validate the core pipeline described in Section 3, utilizing the discrete time diffusion scheduling proposed by EDM [13]. Before training, we collect DDIM inversion latent representations for all ImageNet data using the Heun method [13] with 40 timesteps (79 NFE). Throughout the experiments, we maintain the batch size

to be 256 for both L rec and L adv in Stages 2 and 3. We initialize our base resolution generator with the pre-trained diffusion U-Net. Following CTM [7], we implement adaptive weighting [34] with λ = 0 . 2 ∥∇ θ l L rec ∥ 2 2 ∥∇ θ l L adv ∥ 2 2 , where θ l represents the last layer of the generator.

For higher resolution generation, we double the previous resolution by adding two auxiliary ResNet blocks followed by one upsampler ResNet block. The previously trained generator remains frozen, except for the highest-resolution blocks, which are unfrozen. We then train these newly added blocks along with the unfrozen parts, using a fixed GAN weight of λ = 1 . 0 . Appendix A.1 provides additional details. By freezing part of the trained generator, we achieve greater stability in superresolution training without adaptive weighting. See Figure 6 for uncurated 512 × 512 random samples of ImageNet without CFG.

5.1.1 Quantitative Results

Table 1 presents the performance of PaGoDA. Our model consistently outperforms all existing models across all resolutions, achieving SOTA FIDs without the need of CFG and any other stabilization tricks for GAN. Remarkably, PaGoDA's Inception Score (IS) [35] is on par with other diffusion and GAN models that employed CFG, which implies that PaGoDA samples are as distinctive as CFG samples. Also, PaGoDA generates samples as diverse as the real data distribution, evidenced by diversity recall metric [36], where the PaGoDA reports 0.63 for 64 × 64 resolution (data's recall is 0.67). In contrast, StyleGAN-XL is far behind of PaGoDA in terms of the diversity metric, reporting 0.52 for 64 × 64 resolution. Note that we used StyleGAN-XL's discriminator in PaGoDA training, implying that the reconstruction loss significantly improves the sample diversity.

5.1.2 Discussion on Base Resolution

When applying PaGoDA pipeline, the choice of downsampled base resolution in Stage 1 will be primarily determined by available computational resources. Thus, we investigate the impact of the base resolution at this section. To understand the impact, we conducted experiments at 32 × 32 and 64 × 64 resolutions, as summarized in Table 2. Starting at resolutions below 32 × 32 imposes excessive complicacy on the upscaling network, while higher resolutions significantly increase the computational costs at the Stage 1. Therefore, our analysis focuses on these two resolutions, balancing between computational efficiency and upscaling feasibility.

We utilized only 1 H100 node with 8 GPUs for diffusion training on 32 × 32 with 4096 batch size. Also, for 64 × 64 diffusion, we borrow a pretrained checkpoint [5], which used ≥ 32 3 A100 GPUs to train with 4096 batch size. Results in Table 2 demonstrate that the diffusion model trained in Stage 1 maintains robust performance across both resolutions. Interestingly, the one-step generator distilled

in Stage 2 consistently outperforms the teacher model, likely benefiting from the effectiveness of StyleGAN-XL [40], combined with the reconstrcution loss. In Stage 3, the degree of upscaling from the base resolution emerges as the most influential factor for the quality, with upscaling up to 8x showing minimal performance degradation across both tested resolutions.

The upscaler in PaGoDA refines coarse samples generated at lower resolutions, making the pipeline inherently aligned with the scaling laws of smaller resolutions. This design is advantageous, as scaling laws typically worsen with increasing resolution [43], while PaGoDA leverages the more favorable scaling dynamics at lower resolutions to maintain efficiency. Furthermore, the lightweight upscaling module introduces minimal additional latency, keeping inference times nearly identical to those at the base resolution. This practical efficiency makes PaGoDA a promising solution for scalable diffusion model training across various computational settings.

5.1.3 Discussion on Upscaling Capability

In Stage 3, we train the super-resolution module using a combination of reconstruction and adversarial losses. As shown in Figures 3 and 4, we compare PaGoDA's performance to that of StyleGAN-XL. The comparison reveals key insights: 1) PaGoDA maintains consistent object alignment across resolution jumps, largely due to the reconstruction loss, and 2) its performance is strongly influenced by the GAN component, which plays a crucial role in capturing high-frequency details.

Other upsampling methods, such as SD and Cascaded Diffusion Models (CDM) [44] also target high-quality upscaling. While PaGoDA, CDM, and SD share the same goal, they adopt different approaches, making them complementary rather than competing solutions. In fact, their strengths can be combined to enhance overall compression and upscaling performance. For instance, CDM or PaGoDA can be applied to the latent space of SD, integrating their techniques for better results. Despite their compatibility, it is still essential to assess how these meth-

ods compare in terms of their upscaling effectiveness. In the following analysis, we break down the upscaling capabilities of PaGoDA, CDM, and SD to understand their respective strengths and potential limitations.

S

(d) Latent interpolation

Since PaGoDA is experimented based on EDM [13], we adapted the experimental results from EDM2 [42] to facilitate a direct comparison with PaGoDA in the upscaling performance. EDM2 presents results for both pixel DM and latent DM. In latent diffusion, a 512 × 512 × 3 image is compressed into a 64 × 64 × 4 latent space for training DM, while pixel diffusion operates directly on 64 × 64 × 3 images, sharing the identical network architecture used in its latent DM. As reported in Table 3, EDM2 shows a minor performance decline from 1.33 to 1.96.

Similarly, PaGoDA exhibits a comparable performance drop from 1.21 to 1.80 when upscaling from 64 × 64 to 512 × 512 . This similarity suggests that PaGoDA's upscaling capacity aligns closely with that of the LDM framework, indicating minimal performance differences even when handling highresolution data.

Lastly, when comparing PaGoDA to CDM, we observe in Figure 7 that CDM encounters significant performance drops beyond certain dimensional thresholds ( 128 × 128 ), while PaGoDA maintains consistent performance across varying resolutions. This robustness makes PaGoDA a reliable option for



































high-resolution generation, with its performance remaining steady even as resolution increases.

5.2 Discussion on Controllability

Once we have a trained PaGoDA generator G θ 0 , we can utilize it for solving inverse problems [45] and for controllable generation [46] in a training-free manner [47].

Latent Optimization We consider the inverse problem: y = A ( x ) + η , where y represents the observation, and A : R d → R m with d ≥ m is a known operator. The restored data x can be reconstructed by optimizing the latent. Specifically, if z ∗ ∈ arg min z ∥ y -A ( G θ 0 ( z , c )) ∥ 2 2 , then G θ 0 ( z ∗ , c ) is the best possible estimate of the solution for the inverse problem. Figure 8-(a) displays the outcomes of an inpainting task where latent optimization is employed with Adam optimizer [48].

DDIM Inversion Specific tasks, such as super-resolution illustrated in Figure 8-(b) and class transfer depicted in Figure 8-(c), can be effectively addressed without relying on latent optimization. For these tasks, we apply DDIM inversion to the downsampled observations, then map the DDIM latent back to RGB pixel by feeding the latent into the decoder. Generally, using DDIM inversion yields superior outcomes compared to latent optimization for these types of tasks.

Latent Interpolation Building on techniques from GAN research, we also explored latent interpolation for style mixing. Despite our model's latent dimension being larger than the typical 512-dimensional style vector used in GAN, our observations indicate that latent mixing by slerp operation [49, 20] achieves effective results, as demonstrated in Figure 8-(d).

5.3 Text-to-Image Generation

We collect the data-latent pairs on the CC12M dataset [50] through DDIM inversion and utilize the filtered COYO-700M [51] dataset for adversarial training. The filtering criteria include only data with CLIP score (measured by ViT-B/32 [52]) higher than 32, and aesthetic score-v2 [33] higher than 5.0. Due to concerns regarding sensitive content in the open-sourced LAION dataset [33], we were unable to conduct large-scale diffusion training for Stage 1. This constraint led us to focus primarily on stages 2 and 3, leveraging pretrained open-source checkpoints. For the pretrained teacher diffusion, we used the DeepFloyd-IF model [53], trained on 64 × 64 pixel space. For further experimental details, see Appendix A.2.

Table 4 compares our PaGoDA mainly with the distilled models from SD v1.5 on 512 × 512 . One notable observation from the table is that, even though the latent distilled model generates the latent representation in a single step, additional time is required for decoding this latent into image. In contrast, PaGoDA (on pixel teacher) eliminates such decoding step, thereby overcoming the time constraints associated with distilling SD models. For a more detailed breakdown of the time taken by each component, see Figure 9.

Returning to the performance results in the table, PaGoDA achieves performance comparable to that of the teacher model.





















z

x













z



 

This superior performance is also observed on a different test set as shown in Table 5, further demonstrating PaGoDA's scalability on text-to-image tasks.

6 Conclusion

PaGoDA introduces a training pipeline that can democratize the diffusion training by cutting training budget with orders of magnitudes. The pipeline is consisted of three stages: 1) we pretrain the diffusion models on the downsampled data, 2) we distill the teacher diffusion into a one-step generator on the downsampled data, and 3) we train an upsampler module until we reach to the desired resolution.

References

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2020.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469 , 2023.

Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388 , 2021.

Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. In International Conference on Learning Representations , 2024.

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning , 2024.

Black-Forest. Flux. https://blackforestlabs.ai/announcing-black-forest-labs/ , 2024.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations , 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems , 35:26565-26577, 2022.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems , 35:5775-5787, 2022.

Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Marc Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. In The Twelfth International Conference on Learning Representations , 2023.

Jyoti Aneja, Alex Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. Advances in neural information processing systems , 34:480-493, 2021.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146 , 2016.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 , pages 234-241. Springer, 2015.

Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 , 2017.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019.

Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning , pages 3481-3490. PMLR, 2018.

Vaishnavh Nagarajan and J Zico Kolter. Gradient descent gan optimization is locally stable. Advances in neural information processing systems , 30, 2017.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications , 2021.

Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10124-10134, 2023.

Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis. In International conference on machine learning , pages 30105-30118. PMLR, 2023.

Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311.17042 , 2023.

Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. arXiv preprint arXiv:2403.12015 , 2024.

Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14297-14306, 2023.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems , 27, 2014.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM , 59(2):64-73, 2016.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems , 35:25278-25294, 2022.

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12873-12883, 2021.

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems , 29, 2016.

Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems , 32, 2019.

Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. arXiv preprint arXiv:2212.11972 , 2022.

Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning , pages 1321313232. PMLR, 2023.

Diederik P Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings , pages 1-10, 2022.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4195-4205, 2023.

Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. arXiv preprint arXiv:2312.02696 , 2023.

Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701 , 2020.

Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research , 23(1):2249-2281, 2022.

Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Marc L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209.14687 , 2022.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3836-3847, 2023.

Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, J Zico Kolter, Ruslan Salakhutdinov, et al. Manifold preserving guided diffusion. In International Conference on Learning Representations , 2023.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques , pages 245-254, 1985.

Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR , 2021.

Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. https://github.com/kakaobrain/ coyo-dataset , 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , 2021.

DeepFloyd Lab. If by deepfloyd lab at stabilityai. https://github.com/deep-floyd/IF , 2023.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations , 2021.

Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378 , 2023.

Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. In The Twelfth International Conference on Learning Representations , 2023.

Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. Ufogen: You forward once large scale text-to-image generation via diffusion gans. arXiv preprint arXiv:2311.09257 , 2023.

Hongjian Liu, Qingsong Xie, Zhijie Deng, Chen Chen, Shixiang Tang, Fueyang Fu, Zhengjun Zha, and Haonan Lu. Scott: Accelerating diffusion models with stochastic consistency distillation. arXiv preprint arXiv:2403.01505 , 2024.

Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 , 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022.

Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixartα : Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426 , 2023.

Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. arXiv preprint arXiv:2311.18828 , 2023.

Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Towards language-free training for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17907-17917, 2022.

Bosco Yung. Open-nsfw 2. https://github.com/bhky/opennsfw2 , 2021.

Gant Laborde. https://github.com/GantMan/nsfw_model .

Roman Inflianskas. https://github.com/rominf/profanity-filter/blob/master/ profanity_filter/data/en_profane_words.txt .

Jaclyn Brockschmidt. https://github.com/snguyenthanh/better_profanity/blob/ master/better_profanity/profanity_wordlist.txt .

Jamie Dubs and Ryan Lewis. https://gist.github.com/ryanlewis/ a37739d710ccdb4b406d .

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning , pages 10347-10357. PMLR, 2021.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105-6114. PMLR, 2019.

Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. Advances in neural information processing systems , 33:7559-7570, 2020.

Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.03265 , 2019.

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023.

James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf , 2(3):8, 2023.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2024.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. arXiv preprint arXiv:2110.02861 , 2021.

Adrien Saumard and Jon A Wellner. Log-concavity and strong log-concavity: a review. Statistics surveys , 8:45, 2014.

Wenpin Tang and Hanyang Zhao. Contractive diffusion probabilistic models. arXiv preprint arXiv:2401.13115 , 2024.

Junlong Lyu, Zhitang Chen, and Shoubo Feng. Sampling is as easy as keeping the consistency: convergence guarantee for consistency models. 2023.

Xuefeng Gao, Hoang M Nguyen, and Lingjiong Zhu. Wasserstein convergence guarantees for a general class of score-based generative models. arXiv preprint arXiv:2311.11003 , 2023.

Bernard A Asner, Jr. On the total nonnegativity of the hurwitz matrix. SIAM Journal on Applied Mathematics , 18(2):407-414, 1970.

Nam Parshad Bhatia and Giorgio P Szegö. Stability theory of dynamical systems . Springer Science & Business Media, 2002.

Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. The numerics of gans. Advances in neural information processing systems , 30, 2017.

David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Graepel. The mechanics of n-player differentiable games. In International Conference on Machine Learning , pages 354-363. PMLR, 2018.

Ian Gemp and Sridhar Mahadevan. Global convergence to the equilibrium of gans using variational inequalities. arXiv preprint arXiv:1808.01531 , 2018.

Chuang Wang, Hong Hu, and Yue Lu. A solvable high-dimensional model of gan. Advances in Neural Information Processing Systems , 32, 2019.

Chongli Qin, Yan Wu, Jost Tobias Springenberg, Andy Brock, Jeff Donahue, Timothy Lillicrap, and Pushmeet Kohli. Training generative adversarial networks by solving ordinary differential equations. Advances in Neural Information Processing Systems , 33:5599-5609, 2020.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have made our abstract and introduction to accurately reflect the core contribution of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have created a separate 'Limitations and Broader Impacts' section in the appendix to enumerate potential limitations of our methodology, including the algorithmic, theoretical, and experimental limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Although we omit some of assumptions in the main paper mainly due to page limit, we provide full details of assumptions and complete proof in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We disclose all experimental details in the main paper and the appendix, including the hyperaparameters used and the datasets used with their filetering methodologies. For further reproducibility, we plan to release our code upon acceptance.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: In the reviewing process, we release our code to the reviewers to regenerate our experimental results. After the acceptance, we plan to release the code to the public.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We faithfully release our hyperparameters and experimental details in the appendix and the main paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We have not reported error bars mainly due to the lack of computational resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We explain how much compute resources we used for experiments in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We faithfully follow the code of ethics, suggested by the link above.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impacts as a separate section in the 'Limitations and Broader Impacts' of the Appendix C.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: For the T2I checkpoint release, we plan to use HuggingFace to enroll every users to the system so to control the downloaded user list. Additionally, we prohibited using the LAION dataset [33], which includes NSFW contents. Instead, we used the COYO700M [51] dataset, a large-scale text-to-image dataset that removes NSFW images by NSFW image detectors [65, 66] and texts that contain NSFW words [67-69].

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have properly credited the original owners of assets by citing them. In the code release, we comply the license and terms of the assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: In the supplementary, we include the the details of the dataset/code/model via structured templates.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Contents

STOT

A Experimental Details

A.1 Conditional Generation with ImageNet

Throughout the experiments, we omit the class condition c otherwise mentioned for notational simplicity.

Dataset Construction. We loaded ImageNet2014 4 dataset using center cropping and downsampling using the bicubic algorithm from the PIL python package. To augment the data, we applied a horizontal random flip, and obtained each of latent representations by solving the EDM's 2nd-order ODE sampler (Heun's method) [13] with their suggested diffusion time scheduling and timestep selection. Consequently, in total, we processed approximately 2.5 million data instances forward in time using the PF-ODE to prepare for training. This computational cost of constructing the training dataset is comparable to sampling an equivalent volume of sample from a pre-trained diffusion model.

GAN Details. We adopted the discriminator architecture from StyleGAN-XL. Initially, We loaded DeiT-base [70] and EfficientNet-lite [71] as feature extractors, in line with StyleGAN-XL's setup. When processing real or fake data through the discriminator, we first applied differentiable augmentation (DiffAugment) [72], incorporating three transformations: Translation , Cutout , and Color . Interestingly, we observed no performance differences between the unconditional and conditional discriminators. We hypothesize that this lack of disparity arises because the discriminator primarily updates the generator to refine high-frequency details, while preserving the low-frequency global semantics due to the reconstruction power. Additionally, we opted not to use additional techniques to tame the GAN training, such as R1 regularization [21] or path length regularization [20] in our GAN training. PaGoDA's training generally remains stable due to its reconstruction loss, which is consistent with our theoretical expectation (Theorem B.9).

We conducted tests on GANs under two distinct scenarios. Initially, following the approach used in Stable Diffusion's VAE training, we introduced both the real data x and the reconstructed sample ˜ x = G θ ( E ( x )) to the discriminator, training it to differentiate between the two while updating the generator to maximize log D ψ (˜ x ) . In this setup, as the reconstruction only utilizes the latent representation E ( x ) , the generation quality is not improved.

In the second scenario, adhering to the traditional GAN framework, we trained the discriminator using randomly sampled real data alongside randomly generated fake data ˜ x = G θ ( z ) from z ∼ p prior ( z ) . Then, the endeavor of maximizing log D ψ (˜ x ) now significantly improves the generation quality. Overall, we observed no performance degradation when both types of GAN training were applied to

-You are LLaVA, a large language and vision assistant trained by UW Madison WAIV Lab.

-You are able to understand the visual content that the user provides, and assist the user with a variety of tasks using natural language.

-You should follow the instructions carefully and explain your answers in detail.

-Given the caption of this image "{text prompt}", describe this image in a very detailed manner

the generator. However, given our limited budget and the goal to develop a generative model rather than a compression model, we opted to proceed solely with the second type of GAN setup.

Reconstruction Details. For the reconstruction loss, we train the generator G θ by comparing the original data x ∼ p data ( x ) and its reconstructed counterpart G θ ( E ( x , c ) , c ) at the data's resolution, where E ( x , c ) is the solution of the DDIM inversion. Since our training occurs in pixel space, we conduct this comparison in the feature space using the Learned Perceptual Image Patch Similarity (LPIPS) metric, and there is no need to develop a new feature extractor in latent space. We experimented with features extracted from DeiT-base [70] and EfficientNet-lite [71]; however, we observed no notable improvement from using LPIPS.

For the training, we use the RAdam [73] with learning rate of 8e-6 for the decoder and 2e-3 for the discriminator, and without weight decay. We use the EMA of 0.999, and all reported FIDs are based on the EMA checkpoint. Until 256 2 resolution, we use only 1 H100 node (with 80Gb memory) to train, and we use 8 A100 nodes (with 40Gb memory, in total 8 × 8 = 64 GPUs) to train the 512 2 model. Throughout the experiments, we use the batch size of 256.

For the concerns on the overfitting, we provide additional results in Tables 6 and 7.

A.2 Text-to-Image Generation

Dataset Construction. Due to the presence of inappropriate contents (CSAM) in the LAION dataset [33], we have decided to discontinue its use. Instead, we are now training our model using the CC12M [50] and a filtered version of COYO-700M [51] datasets. For COYO-700M, we apply filters to select only those text-image pairs that meet specific criteria: a CLIP score (measured by ViTB/32 [52]) above 32.0 and an aesthetic score-v2 [33] higher than 5.0. Additionally, we are enhancing the dataset quality by recaptioning the original text prompts from CC12M, adopting practices similar to those used in DallE-3 [75] and PixArtα [62]. Specifically, we employ LLaVA-7B [76], a language

LLaVA

The image shows a couple walking through a field with a

small stone building, which appears to be a small house

or  a  hut.  The  man  and  woman are walking towards the

house, and they are both wearing coats, suggesting that

the weather is cold or chilly. The house is made of stone,

which adds to its rustic charm and suggests that it might

be an old or traditional dwelling. The presence of a field

and  the  couple  walking  towards  the  house  create  a

peaceful

and

serene

atmosphere,

making

it

an

interesting and visually appealing scene.

田

Watch your head as you enter the <PERSON> house

Figure 13: Caption vs. Recaption. From left to right, CFG scale increases. The caption and its corresponding recaption are given by the exemplary case in Figure 12.

model with vision assistance, to generate descriptions of the images based on the text prompts, thereby ensuring more relevant and accurate text-image pairings.

The input prompt of LLaVA is depicted in Figure 11, where we put text prompt to { text prompt } . The output from this recaptioning process adheres to a consistent format, typically beginning with phrases like ' This image features ... ' or ' This image shows ... '. To provide clear demonstration, Figure 13 displays several examples of original captions alongside their recaptioned counterparts.

Interestingly, the recaptioned samples generally outperform the original caption samples. Notably, the recaptioned samples exhibit sufficient quality, particularly when the CFG scale is small, as shown in Figure 14. Therefore, to ensure balanced generation performance across varying CFG scales, we generate samples from the original captions with the CFG scale uniformly sampled from the range [2 , 10] . For the recaptioned text, we use a CFG scale that follows a truncated Gaussian distribution on the range [1 , 10] , centered at 2 with a scale of 3. Overall, incorporating these recaptioned texts into the PaGoDA training results in only a marginal improvement in performance metrics such as FID and CLIP. However, it significantly



































enhances the actual quality of generation, particularly at smaller CFG scales, because the recaptioning provides better-aligned training data.

Using LLaVA, we recaption ˜ c ( x , c ) and obtain the DDIM latent representation, E ( x , ˜ c ( x , c )) , on the entire CC12M dataset. Then, for the original text c , we have a triplet of ( image , text , latent ) of ( x , c , E ( x , c )) for one set, and another triplet ( x , ˜ c ( x , c ) , E ( x , ˜ c ( x , c )) for recaptioned dataset. When computing L rec, we mix these triplets and randomly sample from this mixed dataset.

GAN Details. Similar to the ImageNet case, we have adopted the discriminator architecture from StyleGAN-T. In line with StyleGAN-T, we utilize the DINO ViT-S/16 [77] as the feature extractor and apply DiffAugment [72], incorporating Translation , Cutout , and Color transformations. Building

Image

Encoder

High-quality photo of a

Japanese PaGoDA with

mount Fuji in background

Text

Encoder

CLIP

Augment & crop

Tokenize

Transformer x 3

Transformer x 3

Transformer x 3

Transformer x 3

𝐷1

𝐷2

𝐷3

𝐷4

𝐷5

DINO Encoder

Conv 1

BatchNorm

Conv 9

BatchNorm

Conv 1

DINO Feature

Linear

Time Emb

SiLU

Linear

Linear

Time Emb

SiLU

Linear

CFG Scale (

𝜔

)

Text CLIP Emb

＋

⋅

＋

＋

＋

Discriminator

Trainable

Pre-trained

Fixed-function

Discriminator

Head

Per-token hinge loss

upon this architecture, we integrate a ω condition into each discriminator head, as illustrated in Figure 15. The inputs for each discriminator head include the DINO feature, text CLIP embedding, and the CFG scale ω , which is scaled by a factor of 100. We handle the CFG scale similarly to the time variable in traditional diffusion U-Net models, incorporating the output CFG embedding into the existing components of the StyleGAN-T discriminator head. We assume both image x and text c are related with the CFG scale, thus we designed the discriminator to incorporate ω information into both modules, enhancing the relevance and contextuality of the discrimination process.

Reconstruction Details. In our text-to-image training, we largely adhere to the protocols established for ImageNet training. However, a notable modification involves the decoder network, which now incorporates a ω condition as an auxiliary input. Crucially, this ω condition is processed in decoder in the same way as the time condition in diffusion models. We achieve this by scaling ω by a factor of 100, thus aligning it with the existing time ranges. This method ensures a consistent treatment of the ω parameter, integrating it smoothly into the established model architecture.

CLIP Details. Neither the reconstruction loss nor the GAN loss directly models or maximizes the text-image correlation. To address this, we introduce an additional text-image alignment metric to train our model. Specifically, we employ ViT-L/14 [52] to assess the CLIP value. This regularization significantly enhances PaGoDA's performance, as evidenced in Figure 16 by improving both FID and CLIP scores. These enhancements suggest that not only is the sample quality improving, but also the alignment between text and images is becoming more accurate.

For the training, we use the AdamW8bit optimizer [78] to minimize the required memory with learning rate of 1e-5 for both decoder and discriminator. Similar to the ImageNet experiment, we do not apply the weight decay. In this text-to-image experiment, we do not use EMA, following previous works [61]. In the base resolution, we use the adaptive weighting with λ = 4 ∥∇ θ l L rec ∥ 2 2 ∥∇ θ l L adv ∥ 2 2 . Overall, we use the DeepFloyd-IF-I model with 0.9B number of parameters.

Figure 17 compares PaGoDA with the existing baselines.







































0

20

40

60

80

100

Preference Rate(%)

PaGoDA

(1 Step)

PaGoDA

(1 Step)

PaGoDA

(1 Step)

PaGoDA

(1 Step)

PaGoDA

(1 Step)

PaGoDA

(1 Step)

PaGoDA w/o CFG

(1 Step)

PaGoDA w/o Recon

(1 Step)

PaGoDA w/o CLIP

(1 Step)

DeepFloyd

(27 Steps)

SD1.5

(50+1 Steps)

LCM (SD1.5)

(4+1 Steps)

Image Quality

0

20

40

60

80

100

Preference Rate(%)

PaGoDA

(1 Step)

PaGoDA

(1 Step)

PaGoDA

(1 Step)

PaGoDA

(1 Step)

PaGoDA

(1 Step)

PaGoDA

(1 Step)

PaGoDA w/o CFG

(1 Step)

PaGoDA w/o Recon

(1 Step)

PaGoDA w/o CLIP

(1 Step)

DeepFloyd

(27 Steps)

SD1.5

(50+1 Steps)

LCM (SD1.5)

(4+1 Steps)

Prompt Alignment

B Theoretical Analysis

In this section, we present rigorous statements and proofs of all theorems. The theorems are shown for the unconditional generation case (i.e., without the condition c ), but the analysis can be extended to the conditional scenario.

B.1 Convergence with PaGoDA's Reconstruction Loss

In Section B.1.1, we introduce the necessary notations and preliminaries. In Sections B.1.2 and B.1.3, we demonstrate that the Wasserstein-1 and Wasserstein-2 discrepancies of the learned density (with PaGoDA's reconstruction loss) from p data are upper bounded by PaGoDA's reconstruction loss and the pre-trained DM's training error. All results are proved for unconditional generation (i.e., without c as an input), but they can be easily generalized to the conditional case.

B.1.1 Preliminaries of Convergence Analysis

Consider OU process for t ∈ [0 , T ] , where T > 0 :

We consider f ( t ) ≡ 1 and g ( t ) ≡ √ 2 for simplicity. That is,

We recall that PaGoDA's reconstruction loss (unconditional case) is defined as:

Here, we use p ϕ 0 ( z | x ) to denote the density obtained by solving the pre-trained teacher DM's empirical PF-ODE forward in time from t = 0 to t = T :

where s ϕ ( x t , t ) indicates the pre-trained DM. We remark that p ϕ 0 ( z | x ) defines a deterministic process.

We take p prior := N ( 0 , (1 -e -2 T ) I ) as the prior distribution, and define p T, ϕ 0 := G 0 → T ϕ 0 ♯p data as the distribution obtained by solving the teacher-determined empirical PF-ODE forward in time. Let us consider the density obtained by sampling from PaGoDA (trained without GAN) p 0 , θ := G T → 0 θ ♯p prior . We also let G T → 0 denote the ground truth transition map from T to 0 , defined by the PF-ODE.

Conceptually, Theorems B.1 and B.3 demonstrate that

Its associated PF-ODE is

This implies that training with PaGoDA's reconstruction loss ensures the learned density p 0 , θ = G T → 0 θ ♯p prior is close to p data in Wasserstein distance sense. Moreover, improving the teacher DM to reduce the error ϵ DM is a way to further decrease the discrepancy between p 0 , θ and p data .

We remark that the differences between the two theorems primarily lie in the distinct smoothness assumptions on p data .

B.1.2 W 2 Bound with PaGoDA's Reconstruction Loss

Assumption I-1.

(ii) There is a ϵ DSM > 0 so that sup x ,t ∥ s ϕ 0 ( x , t ) -∇ log p t ( x ) ∥ 2 2 ≤ ϵ 2 DM ;

(iii) G T → 0 θ is Lipschitz in x :

for all θ and T .

(iv) log p data is γ -strongly concave with γ > 3 / 2 :

for all x and y .

Theorem B.1. Given that Assumption I-1 holds, suppose δ is a positive constant such that δ < e -2 T 3 -e -2 T , and let h ( γ, T ) := γ e -2 T + γ (1 -e -2 T ) -(1 + δ ) , where it is noted that h ( γ, T ) is also positive. Then

In particular, if we assume Assumption I-1 (iii) holds also for G T → 0 ,

Here, we use ≲ to absorb the dependence on the constants T and γ into the inequality.

We present an inequality which is essential for the proof of Theorem B.1.

Lemma B.2 (Proposition 3.5. in [79]) . Let P and Q be two distributions on R D . Suppose that log P is γ P -concave and log Q is γ Q -concave. Then the convolution of log P ∗ Q is a (1 /γ P +1 /γ Q ) -1 -concave distribution.

Proof of Theorem B.1. The proof of the theorem is inspired by [80, 81]. Define p T, ϕ 0 := G 0 → T ϕ 0 ♯p data, and p 0 , ϕ 0 := G T → 0 ♯p T, ϕ 0 . From the triangle inequality, we have

For (A), let π ( y , z ) ∈ Π ( p prior , p T, ϕ 0 ) be a coupling of p prior and p T, ϕ 0 . Then

̸

For (A.1), we can yield

Here, the last inequality is a consequence of the following bound

which holds because p prior is taken as N ( 0 , (1 -e -2 T ) I ) , and x T ∼ p T governed by Eq. (5) admits the expression

For (A.2), since p T, ϕ 0 ( z ) = ∫ p ϕ 0 ( z | x ) p data ( x ) d x , by applying Minkowski inequality we have

The proof for (B) is motivated by [80]. Consider the following two reverse time PF-ODEs on the interval [0 , T ]

and

with a coupling of ˆ z 0 , ϕ 0 ∼ p T, ϕ 0 and ˆ z 0 ∼ p T so that W 2 2 ( p T, ϕ 0 , p T ) = E ∥ ˆ z 0 , ϕ 0 -ˆ z 0 ∥ 2 2 . We notice that W 2 2 ( p 0 , ϕ 0 , p data ) ≤ E ∥ ˆ z T, ϕ 0 -ˆ z T ∥ 2 2 . Thus, we need to obtain a upper bound of E ∥ ˆ z T, ϕ 0 -ˆ z T ∥ 2 2 . Let u ( t ) := E ∥ ˆ z t, ϕ 0 -ˆ z t ∥ 2 2 . Then

Let δ > 0 , by applying Yang's inequality ab = ( √ 2 δa )( b √ 2 δ ) ≤ δa 2 + b 2 4 δ to (B.1) for nonnegative a and b , and the Assumption I-1, it becomes

We turn our attention to (B.2). Naively, (B.2) may be naively bounded above by Lip ( ∇ log p t ( · ) ) u ( t ) , where Lip ( ∇ log p t ( · ) ) is the Lipschitz constant of ∇ log p t ( · ) in x . However, we will now derive a sharper bound by incorporating assumptions on the data distribution.

We notice that p t ( x t ) = ∫ p t | 0 ( x t | x 0 ) p data ( x 0 ) d x 0 , where p t | 0 ( x t | x 0 ) = N ( x t ; e -t x 0 , (1 -e -2 t ) I ) is a transition kernel from 0 to t determined by the forward SDE. Therefore, expressing p t in convolution form, under Assumption I-1, and leveraging Lemma B.2, we deduce that log p T -t is a γ/ ( e -2( T -t ) + γ (1 -e -2( T -t ) ) ) -strongly concave distribution (see [82]). Hence,

With the inequalities (9) and (10), we deduce from Eq. (8) that

By applying Grönwall's inequality, we obtain

where A ( t ) := ∫ t 0 a ( s ) d s .

We aim to find an upper bound for inequality (11) that decays exponentially with respect to T . In a ( t ) , b ( t ) := γ e -2( T -t ) + γ (1 -e -2( T -t ) ) as a function of t has the derivative as 2 γ ( γ -1) e -2( T -t ) ( e -2( T -t ) + γ (1 -e -2( T -t ) ) ) 2 . This implies when γ ≥ 1 , b 's minimum occurs at b (0) = γ γ + e -2 T (1 -γ ) , which implies a ( t ) ≤ 2 ( 1 + δ -b (0) ) for all t ∈ [0 , T ] . Setting δ < e -2 T 3 -e -2 T , which implies 1 2 > δ (1+ δ ) e -2 T -δ , then γ > 3 2 = 1 + 1 2 > 1 + δ (1+ δ ) e -2 T -δ (notice that (1 + δ ) e -2 T -δ > 2 δ ), we can deduce that

Let h ( γ, T ) := γ e -2 T + γ (1 -e -2 T ) -(1 + δ ) > 0 . Then we establish that a ( t ) ≤ -h ( γ, T ) , A ( T ) ≤ -h ( γ, T ) T , and A ( T ) -A ( t ) ≤ -h ( γ, T ) t which implies ∫ T 0 e A ( T ) -A ( t ) d t ≤ 1 -e -h ( γ,T ) T . By applying the above bounds and inequality (11), (B) becomes

Here, the last inequality is from a simple inequality √ a + b ≤ √ a + √ b for nonnegative a and b . By combining inequalities (6), (7), and (12), we obtain

This shows the first inequality in Theorem B.1.

Now, we show the second inequality in the statement of Theorem B.1. First, we establish an upper bound for [ E p data ( x ) p ϕ 0 ( z | x ) ∥ ∥ x -G T → 0 ( z ) ∥ ∥ 2 2 ] 1 2 in terms of ϵ DM. Let G 0 → T ϕ 0 denote the transition map defined by the empirical PF-ODE defined by the teacher p ϕ 0 ( x | z ) , and G 0 → T denote the ground truth transition map defined by the PF-ODE from 0 to T . Then we have x = G T → 0 ( G 0 → T ( x )) for all x ∈ supp ( p data ) , and

Here, we utilize the assumption that Assumption I-1 (iii) also holds for G T → 0 .

Consider the following two forward-time PF-ODEs on the interval [0 , T ] , both starting from x 0 ∼ p data :

By subtracting them and integrating from 0 to t , we obtain

By applying Grönwall's inequality,

∥

x

t

-

x

t,

ϕ

0

∥

2

Te

ϵ

DM

.

Combining the above inequality with inequality (13), it implies

Next, we derive an upper bound for W 2 ( p T, ϕ 0 , p T ) related to ϵ DM. Let π (ˆ z , z ) be a coupling between ˆ z ∼ p T, ϕ 0 = G 0 → T ϕ 0 ♯p data and z ∼ p T = G 0 → T ♯p data .

W 2 2 ( p T, ϕ 0 , p T ) = W 2 2 ( G 0 → T ϕ 0 ♯p data , G 0 → T ♯p data ) ≤ E π (ˆ z , z ) ∥ ˆ z -z ∥ 2 2 ≤ ( Te T ϵ DM ) 2 , (16) where the last inequality is derived from the inequality (14).

Therefore, with the first conclusion of Theorem B.1 and inequalities (15) and (16), we derive

The proof can be easily extended in two directions: (1) a more general (VP)-SDE:

with ∥ f ∥ L ∞ ( t ;[0 ,T ]) , ∥ g ∥ L ∞ ( t ;[0 ,T ]) < ∞ , and (2) truncation at the least time t = δ (instead of t = 0 ), with an additional argument based on [81]

where p δ = G T → δ ♯p prior .

T

(14)

B.1.3 W 1 Bound with PaGoDA's Reconstruction Loss

Assumption II-1. (i) m := E p data ( x ) ∥ x ∥ 2 < ∞ ;

(ii) There is a ϵ DSM > 0 so that sup x ,t ∥ s ϕ ( x , t ) -∇ log p t ( x ) ∥ 2 2 ≤ ϵ 2 DM ;

(iii) G T → 0 θ is Lipschitz in x :

for all θ and T .

(iv) ∇ log p t ( · ) is Lipschitz in x with integrable Lipschitz constant:

̸

and Λ s is an L 1 -integrable function on (0 , ∞ ) .

In the following proposition, we prove a variant of Theorem B.1 which does not assume log-concavity of the data density (i.e., Assumption I-1 (iv)).

Theorem B.3 (Variant of Theorem B.1) . Assume that Assumption II-1 holds. Let ν be either the oracle data distribution p data or an empirical distribution ˆ p data ,N := 1 N ∑ N i =1 δ x i , where x i ∼ p data for i = 1 , · · · , N . Let the PaGoDA's reconstruction loss starting from ν be defined as

Then we have

In particular, if we assume Assumption I-1 (iii) holds also for G T → 0 , then for T = O ( log ( m Λ ϵ DM ) 2 ) is sufficiently large, we have

Here, we use ≲ to absorb the dependence on the constants T and γ into the inequality.

Proof of Theorem B.3. Define p T, ϕ 0 := G 0 → T ϕ 0 ♯ν , and p 0 , ϕ 0 := G T → 0 ♯p T, ϕ 0 . From the triangle inequality, we have

For (A), by following the similar argument as in Theorem B.1, we can obtain

For (B), by subtracting the following equations and integrating over t from 0 to T ,

we will obtain

̸

Now let

where Λ s ( t ) is the Lipschitz constant of ∇ log p t ( · ) in x . By applying integral form of Grönwall's inequality, we get

where C T := exp ( ∫ T 0 Λ s ( t ) d t ) and the last equality follows from choosing a coupling of ˆ z 0 , ϕ 0 ∼ p T, ϕ 0 and ˆ z 0 ∼ p T so that W 1 ( p T, ϕ 0 , p T ) = E ∥ ˆ z 0 , ϕ 0 -ˆ z 0 ∥ 2 .

By combining inequalities (17) and (18), we obtain

A similar argument to Theorem B.1 can be applied to obtain the second inequality in the statement of Theorem B.3. ■

B.2 Optimality analysis

In this section, we compare the optimality of the learned distributions resulting from PaGoDA's training and distillation-based training loss, incorporating GAN [7, 6].

PaGoDA's Loss We recall PaGoDA's training objective L PaGoDA

leverages the reconstruction loss

and adversarial loss

Knowledge Distillation Loss In the realm of knowledge distillation (KD) methods for DMs, approaches like local consistency [5], global consistency [6], or soft consistency [7] are utilized to learn the noise-to-data trajectory of the teacher DM. Let us consider the global consistency loss as a case study (similar arguments can apply to other distillation objectives), where the teacher's trajectory is obtained by solving its empirical PF-ODE from T to 0 . The long jump along the trajectory is represented as G T → 0 teacher ( z ) , where z denotes the initial point (noise), T signifies the initial time, and 0 denotes the final time. The output of G T → 0 teacher corresponds to the estimation of clean data, starting from z .

In this context, we abuse the notation G θ ( z ) to denote the generator for KD.

The training of KD can also incorporate adversarial loss for enhanced performance [7, 27]. We represent the combined loss as:

Theorem B.4. Let p ϕ 0 be the density determined the teacher DM. Suppose that GAN admits an optimal discriminator D ∗ .

In PaGoDA, assume that the network parametrized generator class { G θ } is expressive enough so that it can simultaneously optimize both L rec ( G θ ) and L adv ( G θ ; D ∗ ) with a same minimizer. Namely, arg min θ {L rec ( G θ ) } ∩ arg min θ {L adv ( G θ ; D ∗ ) } ̸ = ∅ . Then

̸

In contrast, suppose that p ϕ 0 = p data, then under similar conditions for KD+GAN where arg min θ {L KD ( G θ ) } ∩ arg min θ {L adv ( G θ ; D ∗ ) } ̸ = ∅ , there is no minimizer θ ∗ so that p θ ∗ , KD+GAN := G θ ∗ , KD+GAN ♯p prior = p data .

The first part of the proof of the theorem follows from the following Lemma.

̸

Lemma B.5. If arg min θ { f ( θ ) } ∩ arg min θ { g ( θ ) } = ∅ , then arg min θ { f ( θ ) + g ( θ ) } = arg min θ { f ( θ ) } ∩ arg min θ { g ( θ ) } .

Proof. First, we prove the relationship arg min θ { f ( θ ) + g ( θ ) } ⊇ arg min θ { f ( θ ) } ∩ arg min θ { g ( θ ) } . Indeed, it holds without additional assumption. Suppose that θ ∗ ∈ arg min θ { f ( θ ) } ∩ arg min θ { g ( θ ) } . Then for any θ , we have f ( θ ) ≥ f ( θ ∗ ) and g ( θ ) ≥ g ( θ ∗ ) , which implies f ( θ ) + g ( θ ) ≥ f ( θ ∗ ) + g ( θ ∗ ) . That is, θ ∗ ∈ arg min θ { f ( θ ) + g ( θ ) } .

On the other hand, suppose that θ ∗ ∈ arg min θ { f ( θ ) + g ( θ ) } . We want to prove that θ ∗ ∈ arg min θ { f ( θ ) }∩ arg min θ { g ( θ ) } . Let θ ∗ ∩ ∈ arg min θ { f ( θ ) }∩ arg min θ { g ( θ ) } , where we notice that the existence of θ ∗ ∩ is guaranteed by the assumption. In particular, we have f ( θ ∗ ) ≥ f ( θ ∗ ∩ ) and g ( θ ∗ ) ≥ g ( θ ∗ ∩ ) . Then for any θ , we have

Thus, min θ { f ( θ ) + g ( θ ) } = f ( θ ∗ ) + g ( θ ∗ ) = f ( θ ∗ ∩ ) + g ( θ ∗ ∩ ) and

This implies f ( θ ∗ ) = f ( θ ∗ ∩ ) = min θ { f ( θ ) } and g ( θ ∗ ) = g ( θ ∗ ∩ ) = min θ { g ( θ ) } , as the individual terms are nonnegative. Therefore, θ ∗ ∈ arg min θ { f ( θ ) } ∩ arg min θ { g ( θ ) } , which concludes the proof.

Proof of Theorem B.4. With the lemma above, let θ ∗ ∈ arg min θ L PaGoDA ( G θ , D ∗ ) . Consequently, θ ∗ should also simultaneously minimize both L rec and L adv. Minimizing L rec implies that p θ ∗ , PaGoDA = G θ ∗ , PaGoDA ♯p T, ϕ 0 , where p T, ϕ 0 represents the density derived from solving the teacher's empirical PF-ODE forward, starting from p data. On the other hand, optimizing L adv implies that p θ ∗ , PaGoDA = p data by applying Theorem 1 in [29]. This establishes the first part of the theorem.

̸

In the second part, suppose on the contrary that there is a minimizer θ ∗ of L KD+GAN such that p θ ∗ , KD+GAN = p data. Again, by applying the above lemma, we infer that θ ∗ should also minimize L KD (and L adv). This implies that p θ ∗ , KD+GAN = p ϕ 0 . However, this contradicts our assumption that p data = p ϕ 0 . Thus, such a minizer does not exist and the second part of the theorem is proven.

■

We remark that (1) optimality of L GAN ( θ ) may not be unique in θ , and that (2) the first part of the theorem can be directly extended to scenarios involving downsampling in the encoder .

B.3 Stability Analysis

B.3.1 Preliminaries of Dynamical System

To study its convergence and stability, we first introduce the prerequisites for Lyapunov stability [83, 84] in a general setup. Let F : Ξ → Ξ be a continuously differentiable operator (that is, C 1 operator), where Ω ⊂ R N . We consider the discrete iteration dynamical system defined by

Namely, ξ k +1 = F ( k ) ( ξ 0 ) := F ◦ · · · ◦ F ︸ ︷︷ ︸ k-copies ( ξ 0 ) . A point ξ ∗ ∈ Ω is called a fixed point or equilibrium

(we use the terms interchangeably) of F if ξ ∗ = F ( ξ ∗ ) . The stability and convergence analysis focuses on how the dynamical system F ( k ) ( ξ 0 ) approaches a fixed point as iterations k are sufficiently large.

Definition B.1. (Stability [84]) Let ξ ∗ be an equilibrium of the C 1 operator F : Ω → Ω . The equilibrium ξ ∗ is said to be

stable if for every ϵ > 0 there is a δ > 0 so that whenever ∥ ξ -ξ ∗ ∥ 2 < δ , we have ∥ ∥ F ( k ) ( ξ ) -ξ ∗ ∥ ∥ 2 < ϵ for all k ∈ N ∪ { 0 } .

asymptotically stable if ξ ∗ is stable, and there is a δ > 0 so that whenever ∥ ξ -ξ ∗ ∥ 2 < δ , we have lim k →∞ ∥ ∥ F ( k ) ( ξ ) -ξ ∗ ∥ ∥ 2 = 0 .

exponentially stable if ξ ∗ is asymptotically stable, and there is a δ > 0 and α, β > 0 so that whenever ∥ ξ -ξ ∗ ∥ 2 < δ , we have ∥ ∥ F ( k ) ( ξ ) -ξ ∗ ∥ ∥ 2 ≤ α ∥ ξ -ξ ∗ ∥ 2 e -βk for all k ∈ N ∪ { 0 } . The largest β > 0 that satisfies the inequality for exponential stability is referred to as the rate of convergence .

Let Γ be a subset of the set of all equilibria. We say the dynamical system F ( k ) locally converges on Γ if F ( k ) is exponentially stable at any point in Γ .

The intuitions of the above stability notions are

A stable equilibrium indicates that if an initialization is within some δ -neighborhood of the equilibrium, the iterations starting from that initialization will always remain within an ϵ -neighborhood of the equilibrium, for any arbitrarily chosen ϵ .

An asymptotically stable equilibrium indicates that iterations starting near the equilibrium not only remain close but ultimately converge to the equilibrium.

An asymptotically stable equilibrium indicates that the iterations not only converge but do so at a rate no slower than the rate e -βk with respective to iteration step k .

Analyzing the eigenvalues of the Jacobian ∇ ξ F ( ξ ∗ ) of the operator F at an equilibrium ξ ∗ is a crucial tool for studying stability. In principle [83, 84], if we can ensure that the Jacobian of F at some equilibrium has only eigenvalues with strictly negative real parts, then the dynamical system F ( k ) is asymptotically stable at that equilibrium. In particular, we refer to a matrix as a Hurwitz matrix if all its eigenvalues have strictly negative real parts.

In the following lemma, we present a necessary condition to ensure that a special class of matrices will be Hurwitz.

Lemma B.6. (Necessary condition for a Hurwitz matrix [21]) Consider the following matrix J ∈ R ( N + M ) × ( N + M ) with P ∈ R N × N , Q ∈ R M × M , and B ∈ R M × N .

Suppose that B is full rank. Then all eigenvalues of J have negative real part, if either (1) P is negative definite and Q is negative semi-definite, or (2) P is negative semi-definite and Q is negative definite.

B.3.2 Preliminaries for Analysis of PaGoDA Training

We consider PaGoDA's training, integrating reconstruction and adversarial losses with a weight η > 0 .

Here, f : R → R is a continuous differentiable function. In the vanilla GAN [29], the f -function is taken as f ( u ) := -log ( 1 + exp( -u ) ) , where f ′ ( u ) = exp( -u ) / ( 1 + exp( -u ) ) > 0 and

f ′′ ( u ) = -exp( -u ) / ( 1 + exp( -u ) ) < 0 for all u ∈ R . We maintain the generality of f and will prove the training stability of PaGoDA across a wide class of f .

The velocity field v ( θ , ψ ) corresponding to the gradient descent update is

Gradient descent is a special case of fixed-point iteration. Now, we specify the operator F as an alternative gradient descent operator. That is, we consider F h := F D,h ◦ F G,h with a learning rate h > 0 . Here,

A point ( θ ∗ , ψ ∗ ) is called an equilibrium of the system defined by v if v ( θ ∗ , ψ ∗ ) = 0 (equivalently, F h ( θ ∗ , ψ ∗ ) = 0 ). We can analyze the learning dynamic via the Jacobian matrix of v ( θ , ψ ) which is defined as the following:

The following proposition relates Lemma B.8 to the stability of the gradient descent operator F h , serving as the main tool to prove the training stability of PaGoDA in Theorem B.9.

Lemma B.7. (Locally stable on manifold - modification of [21]) Suppose that the gradient descent operator F h = F h ( υ , ω ) is a C 1 mapping. Let ( υ ∗ , ω ∗ ) be an equilibrium (fixed point) of F h . Assume that there is a neighborhood Ω of ω ∗ so that F h admits equilibrium on { υ ∗ } × Ω :

If all the eigenvalues of J := ∇ υ F h ( υ ∗ , ω ∗ ) have negative real parts, then for a sufficiently small learning rate h , the gradient descent iteration defined by F h locally converges on Γ := { ( υ ∗ , ω ) ∣ ∣ ω ∈ Ω } with a rate of convergence | λ max | . Here, λ max denotes the eigenvalue of J with the largest absolute value.

Proof of Lemma B.7. This proposition is followed by Lemma A.5. and Theorem A.3. of [21]. ■

B.3.3 PaGoDA's Training is Stable

Proving PaGoDA's stability involves two steps: First, derive the components of First, deriving the components of J ( θ ∗ , ψ ∗ ) . Second, verify that these components satisfy Lemma B.6. After these, we can apply Lemma B.7 to conclude PaGoDA's training stability whenever the learning rate h > 0 is sufficiently small.

Assumption III-1. (i) E is not an identity map.

(ii) At θ ∗ , p θ ∗ = p data, and x = G θ ∗ ( E ( x )) for a.e. x ∈ supp ( p data ) .

(iii) At ψ ∗ , D ψ ∗ ( x ) = 0 and ∇ x D ψ ∗ ( x ) = 0 for x ∈ supp ( p data ) .

Lemma B.8. Suppose that Assumption III-1 holds for an equilibrium ( θ ∗ , ψ ∗ ) . Then the Jacobian at the equilibrium can be computed as

Here, and

Proof of Lemma B.8. We first compute the gradients of L in terms of θ and ψ , where we utilize the formulations Eqs. (19) and (20), respectively.

According to Assumption III-1 (ii) and (iii), we have

Thus, we obtain

To compute K DG , we first derive ∇ θ L from Eq. (19) as

Thus, we can compute

and hence,

To compute K DD , we can obtain from Eq. (20) that

Hence, by using Assumption III-1 (ii) and (iii), we get

We consider the following two sets

where S ( ψ ) := E p data ( x ) [ | D ψ ( x ) | 2 + ∥∇ x D ψ ( x ) ∥ 2 2 ] . Also, we let T ψ ∗ M D denote the tangent space of M D at ψ ∗ .

Assumption III-2. (i) The second continuously differentiable function f : R → R f ′ (0) > 0 and f ′′ (0) < 0 .

(ii) There is a δ > 0 so that M G ∩ B δ ( θ ∗ ) and M D ∩ B δ ( ψ ∗ ) are C 1 manifolds.

satisfies:

(iii) ∇ θ G θ ∗ ( E ( x )) T · ∇ θ G θ ∗ ( E ( x )) is positive definite, for all x ∈ supp ( p data ) .

̸

(iv) ∂ w h ( ψ ∗ ) = 0 for any w / ∈ T ψ ∗ M D , where h ( ψ ) := ∇ θ E p G θ ( x ) [ D ψ ( x ) ] ∣ ∣ ∣ θ = θ ∗ .

(v) w T ∇ 2 x D ψ ∗ ( x ) w ≥ 0 , for all w / ∈ T θ ∗ M G and x ∈ supp ( p data ) . Remark. Two special cases are either (v-1) ∇ 2 x D ψ ∗ ( x ) = 0 for x ∈ supp ( p data ) , or (v-2) w T ∇ 2 x D ψ ∗ ( x ) w > 0 , for all w / ∈ T θ ∗ M G and x ∈ supp ( p data ) .

Theorem B.9. Suppose that Assumptions III-1 and III-2 hold for an equilibrium ( θ ∗ , ψ ∗ ) and η > 0 is sufficiently large. Then the alternative gradient descent iteration F h described in Section B.3.2 is locally convergent on M G ×M D for a sufficiently small learning rate h > 0 .

Proof of Theorem B.9. The argument is motivated by [21]. We notice that M G ×M D is a subset of all equilibria of the operators F h (or v ( θ , ψ ) ). This is because that for any ( θ , ψ ) ∈ M G ×M D , we have p θ = p data , x = G θ ( E ( x )) , D ψ ( x ) = 0 , and ∇ x D ψ ( x ) = 0 for x ∈ supp ( p data ) . From Eqs. (21) and (22), we then can obtain ∇ θ L ( θ , ψ ) = ∇ ψ L ( θ , ψ ) = 0 , meaning ( θ , ψ ) is an equilibrium.

Now, we show that the alternating gradient descent converges locally on M G ×M D by verifying Lemma B.8 is fulfilled, and hence, Lemma B.7 can be applied. Let ( θ ∗ , ψ ∗ ) ∈ M G ×M D . There is a C 1 -diffeomorphism Ψ that transforms a neighborhood of ( θ ∗ , ψ ∗ ) onto an open set in R ( N + M ) due to Assumption III-2 (ii). More precisely, we can compute the relation of F h and v after the Ψ -reparametrization. Let ζ := Ψ( θ , ψ ) , and

Then

We remark that similar matrices have identical ranks and spectrum. Therefore, without loss of the generality, we can assume that ( θ ∗ , ψ ∗ ) = ( 0 N , 0 M ) ∈ R N × R M , and

We write the new parameterizations as θ := ( υ G , ω G ) ∈ R N G × R N -N G and ψ := ( υ D , ω D ) ∈ R M D × R M -M D . For simplicity, we write v ( θ , ψ ) := v ( υ G , ω G , υ D , ω D ) . To apply Lemma B.7, we now aim to show that ∇ ( υ G , υ D ) v ( θ ∗ , ψ ∗ ) only admits eigenvalues with negative real parts. From Lemma B.8,

Here, ˆ K GG , ˆ K DG , and ˆ K DD represent submatrices of K GG , K DG , and K DD , respectively, with coordinates ( υ G , υ D ) , indicating the Jacobian of v with derivatives taken along the υ G and υ D directions.

First of all, we show that K DD is generally negative semi-definite. Let ξ ∈ R ( N + M ) be any vector. Then

because f ′′ (0) < 0 from Assumption III-2 (i). Thus, for any ˆ ξ G ∈ R N G and ˆ ξ D ∈ R M D if we consider ˆ ξ := ( ˆ ξ G , ˆ ξ D ) in ( υ G , υ D ) -coordinate,

Next, we demonstrate that ˆ K DG is full rank. We observe that ˆ ξ D = 0 if and only if ξ / ∈ T ψ ∗ M D . Then, according to Assumption III-2 (iv), we deduce that if ˆ ξ D = 0

̸

̸

̸

The elements of K DG ξ corresponding to the υ D -coordinates are represented by ˆ K DG ˆ ξ D , while those corresponding to the ω D -coordinates are 0 . Therefore, we conclude that ˆ K DG ˆ ξ D = 0 . Consequently, by the rank-nullity theorem, ˆ K DG is full-rank.

Finally, by using similar arguments by selecting ( υ G , υ D ) -coordinate, without loss of generality, we only need to show K GG is negative definite. By applying Assumption III-2 (i) and (v), the following lemma concludes that if η > 0 is sufficiently large, we can conclude the negative definiteness of ∇ 2 θ L ( θ ∗ , ψ ∗ ) under Assumption III-2 (v-2).

Lemma B.10. Let A be positive definite, and B be positive semi-definite. Then there is a η min > 0 so that -η A + B is negative definite for all η > η min .

The lemma holds because, for positive (semi-) definite matrix X , we generally have

̸

for all w . Here, λ max ( X ) and λ min ( X ) denote the maximum and minimum eigenvalues of X , respectively. Thus if select η > λ max ( B ) λ min ( A ) , then for any w = 0 , we have

By applying Lemma B.6, we know that ∇ ( υ G , υ D ) v ( θ ∗ , ψ ∗ ) only has eigenvalues with negative real parts. Therefore, with a sufficiently small learning rate h > 0 , Lemma B.7 guarantees the locally convergence of F h on M G ×M D .

■

̸

B.3.4 Literature on Stability Analysis of Adversarial Training

Studying the stability of GAN training from a dynamical systems perspective has been a popular approach [85, 22, 21, 86-89]. Generally, proving or disproving whether adversarial training is stable is challenging. However, [21] provides an example (Dirac-GAN) showing that, in general, GANs are not stable unless additional conditions are imposed.

As a result, researchers have explored additional conditions to stabilize GAN training. Essentially, the goal is to impose extra regularizations on the GAN loss L GAN ( θ , ψ ) := E p data ( x ) [ f ( D ψ ( x )) ] +

E p G θ ( x ) [ f ( -D ψ ( x )) ] , or its velocity field v GAN ( θ , ψ ) := -∇ θ L GAN ( θ , ψ ) ∇ ψ L GAN ( θ , ψ ) to ensure that the resulting Jacobian is Hurwitz. To elaborate further, we revisit the Jacobian J GAN of the vanilla GAN, given by v GAN ( θ , ψ ) :

Here, we slightly abuse the notation from Section B.3.3 by using K ij , i, j ∈ { D,G } , to denote the corresponding components in J GAN. By similar argument of Lemma B.8, we can obtain (indeed, η = 0 in Lemma B.8) that

Conceptually [83, 84], if we can ensure that the Jacobian at some equilibrium has only eigenvalues with strictly negative real parts, then the gradient descent iteration of L GAN is asymptotically stable

at that equilibrium. Therefore, the objective of many studies [85, 22, 21, 89] is to find conditions to verify Lemma B.6. We focus on discussing the conditions for K GG and K DD to be negative (semi-)definite, as this distinguishes PaGoDA's Theorem B.9 from the existing literature.

Under Assumption III-2 (i) that f ′′ (0) < 0 , it is worth noting that K DD is generally negative semidefinite without additional conditions. Hence, studies [85, 22, 21] attempted to impose additional regularizers on J GAN or v GAN to ensure that either K DD is negative definite (as in [22, 21]) or K GG is negative definite (as in [21]). In Table 8, we provide a comparison of the various assumptions, at a high-level, drawn from the literature.

We emphasize that PaGoDA does not require ∇ 2 x D ψ ∗ ( x ) to be strictly positive definite, thanks to PaGoDA's reconstruction loss. Specifically, it accommodates the scenario where ∇ 2 x D ψ ∗ ( x ) = 0 on supp ( p data ) . It's noteworthy that this capability enables PaGoDA to address cases where the instability of GAN is demonstrated, as exemplified by examples provided by [21].

C Limitations and Broader Impacts

Limitations. Algorithmically, the reconstruction loss is incompatible with the classifier-free guidance, which requires us to adopt the original distillation loss. However, as reconstruction loss directly uses the real data, it provides additional merit to decoder training, resulting in better performance as evidenced in the experiments. Theoretically, some theoretical assumptions of PaGoDA are challenging to verify in practice. For example, Theorems B.1 and B.3 require certain Lipschitz continuity of the score functions. This assumption is difficult to maintain at t = 0 due to the potential concentration of the data manifold in a lower-dimensional space, causing singularity. However, by truncating the PF-ODE solving at t = δ (for some δ > 0 ), which is common in practice, this singularity is avoided, making the Lipschitz continuity assumption more feasible. In addition, Theorem B.4's assumption of the existence of a common minimizer can be difficult to verify empirically. However, with proper neural network parametrization and effective optimization, this assumption becomes more feasible. At last, verifying Assumptions III-1 and III-2 concerning the optimal properties of the generator and discriminator ( G θ , D ψ ) is challenging in practice. These assumptions, essential for general (Lyapunov) stability analysis, are difficult to validate empirically. However, they appear reasonable based on our experimental observations. Last, empirically, PaGoDA's T2I generation capability relies heavily on the scale and quality of the training dataset.

Broader Impacts. PaGoDA, as a general media generative model, carries the risk of producing harmful or inappropriate content, such as deepfake images, graphic violence, or offensive material. To mitigate these risks, we avoid using the LAION dataset [33] in our model training, but robust content filtering and moderation mechanisms are essential to additionally prevent the generation of unethical or harmful media.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes PaGoDA, an adversarial distillation method to support single-step generation of image resolutions higher than the teacher diffusion model. PaGoDA first solves the forward PF-ODE (data to noise) to collect noise from the dataset. Then, it gradually adds upsampling layers to the model and trains only those layers. The training objectives include reconstruction loss and GAN loss, and for text-to-image experiments, additional distillation loss and CLIP regularization are introduced to support classifier-free guidance. PaGoDA demonstrates strong one-step generation performance on ImageNet and COCO benchmarks and shows its superiority through various ablation studies.\n\nSTRENGTHS:\n**Originality**\n\n- This is the first study to support higher resolution than the teacher diffusion model when distilling a diffusion model into a single-step generator.\n\n- It is also the first study to use the forward PF-ODE to collect noise-data pairs for distillation.\n\n**Significance**\n\n- Despite using GAN loss, the training is stable due to the progressive growing strategy, as shown by the good performance at various resolutions in Table 2.\n\n- By using the forward PF-ODE for distillation, it learns the true data distribution and achieves higher sample diversity compared to GANs, as evidenced by the recall performance in Table 2.\n\n- The PaGoDA is shown to be applicable to the pixel-space text-to-image model, DeepFloyd.\n\n**Quality**\n\n- The paper is well-written and includes detailed implementation details in the appendix for reproduction. \n\n- Various ablation studies demonstrate the superiority of each loss and the upsampling method of this approach compared to the VAE decoder of latent diffusion models.\n\nWEAKNESSES:\n- PaGoDa is claimed to have a faster sampling speed compared to LCM, but wouldn't the sampling speed be faster with **smaller VAE** [A]? Wouldn't LCM be more efficient than PaGoDa if the VAE decoder has less layers than PaGoDa's additional upsampling layers?\n\n- Additionally, smaller VAE only requires training of a VAE, while PaGoDa involves sequential training stages depending on the resolution, making the training pipeline somewhat inconvenient. \n\n- In Table 4, PaGoDa performs better than the diffusion teacher, and in Table 1, PaGoDa outperforms EDM2-XXL. This may be because **the discriminator uses EfficientNet and DeiT pre-trained on ImageNet**, leading to better FID scores, which use an Inception network also trained on ImageNet. This issue has been discussed in [B]. There are two potential solutions to this problem. The first is to compare **Fréchet distances using DINOv2**, following the EDM2 paper. The second is to **train the discriminator for GAN loss from scratch**. If PaGoDa's GAN loss is truly stable, it should be possible to train without a pre-trained network.\n\n[A] Tiny AutoEncoder for Stable Diffusion: https://github.com/madebyollin/taesd \n\n[B] Kynkäänniemi et al., The Role of ImageNet Classes in Fréchet Inception Distance, ICLR 2023.\n\nQUESTIONS:\n- What classifier-free guidance scale did PaGoDA use in Table 6? Is the point that achieved the best performance in Figure 14 reported in Table 6?\n\n- Why were different datasets used when collecting latents (CC12M) and during GAN training (COYO-700M)?\n\n- Does the DDIM inversion in Section 5.4 needs a diffusion teacher for inversion?\n\nLIMITATIONS:\nAppendix C explains well the gap between PaGoDA's theoretical assumptions and its practice.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel GAN-based diffusion distillation approach that leverages ideas from PG GAN for effective high-resolution synthesis.\n\nSTRENGTHS:\n* PaGoDA demonstrates state-of-the-art single-step generation performance on ImageNet, competing with or outperforming more expensive alternatives, including the teacher DMs. \n\n* The proposed classifier-free GAN is an interesting modification for better conditional generation.\n\n* The authors justify most components of the proposed method by providing valuable discussions and ablation studies.\n\n* The paper includes thorough theoretical analysis, as well as many experimental and implementation details.\n\nWEAKNESSES:\n* PaGoDa is designed for pixel-based DMs only and hence is not applicable to most state-of-the-art T2I models, e.g., SDXL, SD3. Also, the motivation behind this (L42) seems a bit arguable. For example, one could try to adapt PaGoDa to LDM by replacing the AE decoder with a shallow one mapping a 64x64 latent variable to a 64x64 image. Then, PaGoDA can be applied as is. Do the authors have any thoughts on this?\n\n* The method requires collecting the dataset of a large amount of (x, z) and ($\\hat{x}_w$, z) pairs. Likely, this significantly increases the training costs compared to sampling-free alternatives. Could the authors provide the data collection vs training costs? How does the overall training time compare to other distillation approaches?\n\n* The KD objective is used for the guided generators (i.e., using the reverse process for sampling ($\\hat{x}_w$, z) pairs). This contradicts the motivation behind the reconstruction loss (L89). While the ablation study in Tab.7 aims to address this, could the authors clarify the inference CFG scale used in this experiment? How does the role of the reconstruction loss change with different CFG scales? \n\n* PaGoDa uses CLIP regularization in the T2I setup. I believe this makes the CLIP score evaluation incorrect. Moreover, other distillation methods, e.g., LCM, DMD, or ADD, do not use it. This raises concerns about the performance of the core approach in the T2I setting. \n\n* For T2I evaluation, automated metrics have been shown to correlate poorly with human preference (e.g., [1,2]). It would be highly beneficial to conduct a human study comparing PaGoDA to other distillation approaches and T2I diffusion models. If a reliable human evaluation setup is unavailable, I would suggest evaluating a metric learned to mimic human preference, e.g., [3,4,5].\n\n* The CFG details for T2I inference are missing. What scale is in the final t2i results? How does the model perform with different CFG scales?\n\n* The paper does not provide qualitative T2I results demonstrating the diversity of generated images for a given prompt.\n\n* The ablation study lacks evaluation of the proposed approach without adversarial objective. \n\n\n[1] Podell et al. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis.\n\n[2] Li et al. Playground v2.5: Three Insights towards Enhancing\nAesthetic Quality in Text-to-Image Generation.\n\n[3] Xu et al. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation\n\n[4] Kirstain et al. Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation.\n\n[5] Wu et al. Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis.\n\n[6] Sauer et al. Fast High-Resolution Image Synthesis with\nLatent Adversarial Diffusion Distillation\n\nQUESTIONS:\n* Please address the questions and concerns raised in \"Weaknesses\".\n\n* Intuitively, the reconstruction and distillation losses are most beneficial for the base-resolution training. Did the authors try to deactivate these losses for SR stages? \n\n* Did the authors consider using a pretrained DM as a discriminator, following the ideas from [6]? It would be interesting to explore if noise-level feedback might be valuable for learning better generators. \n\n* PaGoDa approaches very low FID values on ImageNet. Could the performance gains be a result of overfitting the training data?\n\nLIMITATIONS:\nThe authors addressed the limitations and potential negative social impact in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method to distill a diffusion model into a one-step generator. The training process combines several loss functions. First, the authors use DDIM inversion to transform real images into latent noise, which is then fed into the generator. The generated images are supervised with a reconstruction loss. Second, an additional GAN loss is applied to the generated images, distinguishing them from real images. For text-to-image generation, the authors present a novel approach to enable classifier-free guidance in the GAN loss by using an auxiliary guidance predictor. Other auxiliary losses, such as knowledge distillation loss and CLIP loss, are also employed to enhance performance. The final method are evaluated across various benchmarks.\n\nSTRENGTHS:\nS1. The presented approach performs well on common benchmarks while achieving good inference efficiency.\n\nS2. The authors plan to open-source the method, which will be beneficial for further research.\n\nWEAKNESSES:\nW1. Some loss components are questionable and may mislead readers about the actual performance of the proposed approach. For instance, it is well-known that using a GAN loss with a discriminator pretrained on ImageNet significantly biases the FID metric [1]. This issue also applies to the CLIP loss, which biases the evaluation of FID/CLIP scores when used during training. A substantial portion of the strong results presented might be attributed to these misleading numbers, making it difficult to draw accurate conclusions about effectiveness. This is a common issue in some previous works but needs to be addressed now. For example, using GANs is good, but it might be better to avoid using a pretrained classifier as the discriminator, as suggested by recent explorations in diffusion GAN literature [2]. For text-to-image results, it is recommended to conduct human evaluations to assess real image quality, prompt alignment, and diversity.\n\nW2. Many of the proposed components have been well explored in previous literature, but the connections to these works are not sufficiently discussed in the related works section. For instance, the combination of a regression loss and a GAN loss has been explored for text-to-image generation in several studies, including [3, 4]. Additionally, the use of distillation loss and CLIP loss are common practices [5, 6]. This reduces the perceived novelty of the paper to a moderate level.\n\nW3. The introduction of a classifier-free GAN objective might overcomplicate the problem. Previously, GAN loss with classifier-free guidance (CFG) was enabled by generating fake samples using an original diffusion model with guidance, as applied in [2], and then utilizing the standard GAN loss. This method is simpler than training an auxiliary classifier and has a similar computational cost, since the current classifier training also requires generating samples with CFG using the original diffusion model.\n\n\n[1] Kynkäänniemi, Tuomas, et al. \"The role of imagenet classes in fr\\'echet inception distance.\" arXiv preprint arXiv:2203.06026 (2022).\n\n[2] Sauer, Axel, et al. \"Fast high-resolution image synthesis with latent adversarial diffusion distillation.\" arXiv preprint arXiv:2403.12015 (2024).\n\n[3] Lin, Shanchuan, Anran Wang, and Xiao Yang. \"Sdxl-lightning: Progressive adversarial diffusion distillation.\" arXiv preprint arXiv:2402.13929 (2024).\n\n[4] Song, Yuda, Zehao Sun, and Xuanwu Yin. \"SDXS: Real-Time One-Step Latent Diffusion Models with Image Conditions.\" arXiv preprint arXiv:2403.16627 (2024).\n\n[5] Sauer, Axel, et al. \"Adversarial diffusion distillation.\" arXiv preprint arXiv:2311.17042 (2023).\n\n[6] Kang, Minguk, et al. \"Scaling up gans for text-to-image synthesis.\" CVPR 2023\n\nQUESTIONS:\nMost of my concerns are outlined in the weaknesses section. To reiterate, I am particularly concerned about the utilization of a pretrained classifier as a discriminator and the CLIP loss, which bias the quantitative evaluation. Additionally, the overall novelty is moderate, and some components might be unnecessarily complex.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}