Accelerated Diffusion using Closed-form Discriminator Guidance

Authors: Aadithya Srikanth, Siddarth Asokan, Nishanth Shetty, Chandra Sekhar Seelamantula
Primary area: generative models
Keywords: diffusion models, GANs, time-step-shifted sampling, discriminator guidance

Abstract:
Diffusion models are a state-of-the-art generative modeling framework that transform noise to images via Langevin sampling, guided by the score, which is the gradient of the logarithm of the data distribution. Recent works have shown empirically that the generation quality can be improved when guided by classifier network, which is typically the discriminator trained in a generative adversarial network (GAN) setting. In this paper, we propose a theoretical framework to analyze the effect of the GAN discriminator on Langevin-based sampling, and show that in IPM GANs, the optimal generator matches {\it score-like} functions, involving the flow-field of the kernel associated with a chosen IPM constraint space. Further, we show that IPM-GAN optimization can be seen as one of smoothed score-matching, where the scores of the data and the generator distributions are convolved with the kernel associated with the constraint. The proposed approach serves to unify score-based training and optimization of IPM-GANs. Based on these insights, we demonstrate that closed-form discriminator guidance, using a kernel-based implementation, results in  improvements (in terms of CLIP-FID and KID metrics) when applied atop baseline diffusion models. We demonstrate these results by applying closed-form discriminator guidance to denoising diffusion implicit model (DDIM) and latent diffusion model (LDM) settings on the FFHQ and CelebA-HQ datasets. We also demonstrate improvements to accelerated time-step-shifted diffusion, when coupled with a wavelet-based noise estimator for latent-space image generation.

Decision: reject
Reviewer scores: 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1729602488615

thread_type:
official_review

<thread_object>
creator: reviewer_qdum

<content>
summary:
This paper derives a closed form expression for the IPM-GAN discriminator, whose gradient is then used to perform Langevin sampling or guide diffusion generation.

strengths:
**Originality** : this paper is original in the aspect that it uses closed-form expressions for the optimal discriminator instead of discriminators parametrized by neural nets. It is also, to the best of my knowledge, the first paper to use IPM-GAN discriminators for diffusion sampling and guidance.

**Quality** : this paper provides some theoretical grounds for using IPM-GAN discriminators over f-GAN discriminators for diffusion. Some experiments on toy and image datasets are also provided as evidence that the proposed discriminator guidance improves sample quality.

**Clarity** : motivation as well as main theoretical results are presented clearly in the main text. In particular, in Section 3.1, the authors do a good job of explaining how IPM-GAN discriminator guidance could be better than f-GAN discriminator guidance.

**Significance** : this paper provides some theoretical grounds for how diffusion and GANs could be used synergistically to enhance generative performance.

weaknesses:
**Missing theory** : there is no guarantee that the proposed discriminator-based generator process correctly samples from the data distribution.

**Missing baselines** : while there may be other relevant baseline methods, the paper is missing comparison with the most important baseline - discriminator guidance [1]. The authors *must* compare the proposed method with [1] if they wish to support the claim that IPM-GAN discriminator guidance provides improvement over f-GAN discriminator guidance. In addition, if the authors wish to claim that the proposed method accelerates diffusion sampling, they must compare with additional relevant baselines such as [2,3].

**Missing ablation** : there is no ablation w.r.t. number of data samples used when approximating the optimal discriminator. Is this because a large number of samples is needed to obtain reasonable results?

[1] Refining Generative Process with Discriminator Guidance in Score-based Diffusion Models, ICML 2023

[2] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, NeurIPS 2022

[3] Fast ODE-based Sampling for Diffusion Models in Around 5 Steps, CVPR 2024

questions:
**Q1** : Why does the proposed method accelerate diffusion sampling? My understanding is that IPM-GAN discriminator guidance mitigates the mismatch between model distribution and the true data distribution at each time $t$. Sampling acceleration is related to minimizing the truncation error during numerical integration of the diffusion SDE/ODE as we reduce the number of function evaluations.

**Q2** : At lines 254-255, what do the authors mean by *the kernel $\kappa$ is the Green's function to the differential operator governing the optimal discriminator*? Please provide a concrete equation of $\kappa$ in relation to $D$.

**Q3** : Does the generative process defined by closed form discriminator gradients generalize beyond training data? In other words, how do we know whether the generative model is not simply memorizing the training dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730112232758

thread_type:
official_review

<thread_object>
creator: reviewer_9w6g

<content>
summary:
**Disclaimer**: I am not familiar with the theorem of optimal GANs. While I briefly reviewed some related works referenced in this paper, I am still not confident in assessing the significance or complete accuracy of the conclusions for the GANs. My rating is primarily based on the experimental results of using this form in diffusions. I am open to discussing this further with the author or other reviewers.

This paper proves the analytical form for optimal generators in IPM GANs. The authors then connect this form to score matching. Then, the authors develop a closed-form discriminator guidance for diffusion models.

strengths:
1. This paper extends previous conclusions on $f$-GAN and on IPM GAN discriminators to IPM GAN generators. 
2. The optimal IPM GAN generators naturally connect to diffusion models.
3. The proposed discriminator guidance is plug-and-play. Unlike previous discriminator guidance, it does not require additional training and seems compatible with different DM frameworks.

weaknesses:
As I disclaimed, I am not confident in assessing the significance or complete accuracy of the conclusions for the GANs. Therefore, my questions and concerns will mainly be related to experiments.

1. Experimental results are not fully convincing to me. I did not see gains on CIFAR-10 and ImageNet-64 datasets using the proposed discriminator guidance. 
The authors claimed that the proposed guidance could accelerate convergence, leading to fewer time steps in image generation. However, by looking at Fig 15 and 16, it is very obvious that the results of the guidance are much worse than the baseline.
I think a better way to illustrate the effectiveness is to compare the generated samples obtained with the same budget (NFEs). Also, visually checking the results is not enough; FID or other metrics need to be provided for a fair comparison.


2. Experimental settings are not fully clear. If I understand it correctly, the score is estimated by the MC estimators. However, I think I missed the details on the number of samples. Also, the MC estimator requires a batch of generated samples. I did not find details of how this is obtained. I guess it is only possible to draw a batch of samples (instead of 1) at each sampling time. Is this correct? In this way, we can use the samples at the last time step in this batch for the MC estimator.
If this is the case, I would be curious how the number of samples influences the performance. Especially in large-res applications like text-2-image generation, requiring a large batch for each prompt seems too much.

3. I am also a bit confused on the ablation. In Table 2, it seems that the more steps you apply discriminator guidance, the worse result you are getting. So what is the results when $T_D=0$ (w/o any discriminator guidance)? I saw there is one line saying that there is a surge around $T_D=10$. However, seeing this in plots or tables would be more convincing.
Also, why Table 2 only shows different $T_D$ for WANDA. What about LDM+DG?

questions:
1. Line 297-299: 
>From Lemma 3, we see that the gradient field of the kernels convolved with the density difference, and the data score $\nabla ln p_d(x)$, serve similar purposes, which is to output an arbitrarily large value at data sample location, and low values elsewhere.

I am quite confused about this: I did not see the data score in Lemma 3?

2. Line 374-376: 
>Since the proposed approach suggests the interoperability of the score and the discriminator-kernel gradient in Langevin flow, we also consider discriminator-guided Langevin sampling on the CIFAR-10 and ImageNet-64 datasets, considering EDMs as the baseline

Do you mean just adding the discriminator-kernel gradient and the score together (with some weighting for the guidance strength)? 
Also, for guidance, how do you choose the guidance strength?


3. what does the (RGB Color online) mean in the caption in figures?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646867504

thread_type:
official_review

<thread_object>
creator: reviewer_z719

<content>
summary:
**Summary**  
This paper seeks to extend the concept of discriminator guidance to the Integral probability metric(IPM) GAN model, with the ultimate goal of applying this framework to diffusion models.   
This paper derives a more general solution to the GAN-generator optimality using mathematical proof from the **Theorem 1**(optimal f-GAN generator). From this general solution, this paper aims to achieve the goal of **Closed-form Discriminator Guidance**. Following the mathematical approaches, this paper presents a series of comprehensive experiments.  

**Contribution**  
In addressing generative models, the concepts from GAN have been applied to diffusion model. While prior attempt focused on applying the discriminator of standard GAN to diffusion, this paper expands on that by applying the discriminator from the IPM-GAN framework to diffusion models.

strengths:
**Originality & Significance**  
- This paper derive the more general solution to optimal GAN generator. 
- Make connection between the solution to optimal IPM-GAN generator and diffusion model.

**Quality**
- This paper demonstrates superior performance in terms of metrics and training time.

**Clarity**
- This paper employs mathematical proof to substantiate its claims.
- In **5.1 ablations** and **Appendix**, the paper rigorously designs and presents robust experiments that substantiate its claims.

weaknesses:
1. It would be better if you could give the explanation for transition from GAN generator optimality to discriminator optimality.  
: This paper discusses GAN's alternating optimization and the optimality of the GAN generator. Subsequently, it mentions the transfer of generator optimality to discriminator optimality, but lacks sufficient explanation.

2. In lines 519 (520) - 521, how can this paper assert that 'there is a stark jump initially of about ___10 or so steps___ via the ~'?  
: Further explanation or experimental results are required for those figures.

3. Need more ablation for ___DG*___.  
In lines 454 (455) - 457 (458), where it states, 'Motivated by the fact ~ not accurate.' the paper asserts that ___DG*___ is not always useful. Subsequently, in section 5.1, the paper presents results for the first $T_D$ = {50, 100, 200} steps using the discriminator.  
Why does the paper focus only on the first $T_D$ steps? What about the later stages of training? Could you provide some toy examples or explanations to clarify this?


**Typos**
1. In line 199 : $\epsilon$ -> $\epsilon_t$

questions:
1. In line 479, what is the meaning of $w_{dg, 0}$?  
Table 2 may need to be revised for clarity.  


2. In the last paragraph of section 5, it appears that the ablation study for $T_D$ is conducted with respect to WANDA.  
However, how is this approach applied to LDM-DG? What is the justification for selecting $T_D = 50 $ for LDM-DG$^*$?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732506477460

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A Summary of our Responses and Revision

comment:
We would like to thank the various reviewers for taking the effort to review our paper! We would like to highlight the work's strengths identified by the reviewers:

1. Reviewer Z719 and QDuM both appreciated the **Originality and Significance** of the derived optimal GAN generator solution.
2. **All Three Reviewers praise the derived natural connections to diffusion models**, and the mathematics proofs provided to substantiate our claims. 
3. Reviewer Z719 remarks that the "*the paper rigorously designs and presents robust experiments that substantiate its claims*."
4. Reviewer 9w6G also remarks positively on the **plug-and-play** nature of the proposed novel discriminator guidance, which "*does not require additional training and seems compatible with different DM frameworks.*"
5. Reviewer QDuM remarks that, to the best of their knowledge, this submission is "*the first paper to use IPM-GAN discriminators for diffusion sampling and guidance.*"
6. Reviewer QDuM finds the paper well motivated and clearly presented.


In light of the reviewers’ comments, we carried out various ablations/experiments to further substantiate our claims. We summarize the changes we have incorporated as a result of the feedback received from the reviewers:

1. We have included additional *rigorous* ablations on the choice of the transition time $T_D$ and the guidance weight *w_{dg,t}* for WANDA with the LDM sampler (Tables 5-6 of the revision) which, on CelebA-HQ Dataset, result in state-of-the-art **improvements over baseline samplers with DG**$^*$ **applied for only 20\% of the overall iterations.**
2. We provide ablation on the time step jump obtained by the WANDA samples, and show that **DG**$^*$ **provides a jump of 2-10\% of the total steps.** 
3. We also include new experiments on sampling with the LDM sampler on the **LSUN-Churches dataset** to strengthen our results and show that **LDM+DG**$^*$ **once again outperforms the baseline**.
4. We include experiments on the DPM sampler (As suggested by Reviewer QDuM) and show that using **DG**$^*$ **over DPM improves FID on the same number of sampling steps, and archives comparable FID with 25% fewer steps (reduction from 20 to 15)**. (Table 4) on CelebA-HQ and similar gains on pixel-space CIFAR-10.
5. For completeness, we also included a summary of the proof that shows that the generator distribution models the data distribution in the proposed setting (Appendix D.1)

We request that the reviewers consider reverting with additional questions, if any, or consider raising their score in case their questions were resolved.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733300294280

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Rebuttal Discussions

comment:
We would like to thank the reviewers for taking the effort to actively engage during the rebuttal phase. In addition to the revisions summarized in the global response, we would like to make our closing arguments and highlight the discussions that took place as part of the rebuttal:

1. **Reviewer Z719** appreciated the *Originality and Significance* of the derived optimal GAN generator solution, the *superior performance in terms of metrics and training time*, and the rigorously designed and presented, robust ablations that validate the claims. Post the rebuttal, the reviewer **had no additional concerns** regarding the submission.

2. **Reviewer 9w6G** appreciated the extension of conclusions on $f$-GAN to IPM-GAN discriminators and IPM-GAN generators, the connections between the optimal IPM GAN generators and diffusion models, and the *plug-and-play* nature of the proposed discriminator guidance, which does not require additional training and is compatible with different diffusion model frameworks. 

     Post the rebuttal, Reviewer 9w6G **increased their score** and was satisfied with the additional experimental results, ablations, and figures of generated images presented on CIFAR-10 dataset. We have also satisfactorily addressed the reviewer’s  concerns on scalability, by emphasizing our experimental results of the proposed DG$^*$ to generate high-dimensional images by means of the LDM and DPM samplers on CelebA-HQ, FFHQ, and LSUN-Churches.

3. ⁠**Reviewer QDuM** praised the *originality* of using a closed-form expressions for the optimal discriminator, remarking that, *to the best of their knowledge, this is the first paper to use IPM-GAN discriminators for diffusion sampling and guidance*. The reviewer also commended the *quality, clarity of presentation and motivation of the theoretical results* and the significance of the proposed approach. 

    Post rebuttal, Reviewer QDuM also **increased their score** and found that we were able to address all concerns regarding experiments. The reviewer also appreciated inclusion of the convergence proof as part of the rebuttal, and found that it strengthens the proposed method significantly. We also satisfactorily addressed the reviewer’s concerns on (a) incorporating these results into the revision; (b) quantifying the acceleration demonstrated by the proposed approach using a momentum type argument; and (c) application of the derived lemma on convergence to strengthen the proposed WANDA setting. 

Thanks to the intellectually stimulating and positive interaction during the rebuttal phase, the acceptability of the manuscript has increased significantly. We sincerely hope that both reviewers and Area Chairs take cognizance of this fact and enhance their scores and find our submission worthy of acceptance to ICLR.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper derives a closed-form expression for the IPM-GAN discriminator, whose gradient is then used to perform Langevin sampling or guide diffusion generation.

While this is an interesting result, the current form of the work is not ready for publication due to the following reasons:
1. Structural changes to the paper are needed to incorporate the extra theory and experiments to address the concerns;
2. The discrepancy between theory and practice on the acceleration effect of discriminator-guided sampling has not been properly addressed.
3. The improvement in experimental results is not convincing enough.
4. The ethical concerns have not been properly addressed.