Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors argue that the Lévy dynamics sampling method is preferable to sampling based on Langevin dynamics in score-based generative models (SGM). The reason is based on the hypothesis that since Lévy dynamics has a heavy-tailed distribution, it can produce both large jumps and small roaming in the search for the sampling space, and thus can provide better sampling results than Langevin dynamics, which is not expected to produce large jumps. On the other hand, since exact numerical simulation of Lévy dynamics is difficult, the authors propose to use Gaussian mixture noises as an approximate method. Specifically, the authors propose Gaussian mixture SGM (GM-SGM) as a new type of SGM that learns to denoise Gaussian mixture noises.
strength_and_weaknesses: Strength:
1. Using Levy dynamics allows sample points to generate large jumps to pass through low-probability regions, making it less likely that they will remain at local minima.

2. Because of the high density near the zero center in Lévy dynamics, there is an expectation that local regions can be searched efficiently.

3. Experimental results show that a mixture of sampling from Gaussian distributions, which are computationally less expensive to approximate, instead of sampling from a Lévy distribution, still captures the desired behavior.

4. GM-SGM can automatically select large jumps or small roaming when sampling.


Concerns:
1. How much computation time/cost is needed for inference at each step t compared to SGM?

2. If GM-SGM can achieve high performance with a small number of iterations, can it achieve a better score with more iterations, or is the limit the current FID score (around 3.5)?
clarity,_quality,_novelty_and_reproducibility: The paper has a clear description of the idea and the issues involved. It is a new approach to introduce Levy dynamics to the score-based generative model. The reproducibility is also fine.
summary_of_the_review: The use of Lévy dynamics to create large jumps is a major contribution for achieving highly accurate image generation with a small number of steps. A discussion of the computation time would be helpful.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Based on heavy-tailed Levy dynamics can produce both large jumps and small roaming to explore the sampling space, resulting in better sampling results than Langevin dynamics with a lacking of large jumps, the authors explore a new class of score-based generative models (SGMs) with sampling based on the Levy dynamics. However, the exact numerical simulation of the Levy dynamics is significantly more challenging and intractable. The authors propose an approximation solution by leveraging Gaussian mixture noises during training to achieve the desired large jumps and small roaming properties.
strength_and_weaknesses: Strength: replacing Gaussian with Gaussian mixtures seems interesting, but I am not sure since I have never done any research in this area.


Weakness: 

1. Why can we use Gaussian mixtures to approximate the Levy distribution? How accurate the approximation is? I wonder if there is any theoretical guarantee. Intuitive, Gaussian mixtures are still lighted-tailed. I would like to see some rigorous analysis of these.


2.  It seems to me that by doing parallel tempering of Langevin dynamics and related techniques, we can also achieve both small roaming mode and large jump mode.

3. The stationary distribution of Langevin dynamics is known. However, when replacing the Gaussian noise with the Levy distribution or even Gaussian mixtures, the stationary distribution will be changed completely, and actually, we do not know what is the stationary distribution anymore.


Question to the author:

After reading the paper, I am still not sure how to perform the reverse procedure when the Gaussian mixture is applied in the forward procedure. Can authors comment on this?


clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is easy to follow.

Quality: I have never done any research in this area, but I believe the quality and novelty of this paper is high.

Reproducibility: Unknown.
summary_of_the_review: It seems to me that this work simply replaces Gaussian noise with Gaussian mixtures in SGMs. However, there are some questions that are not clear to me. 1) Why Gaussian mixtures can approximate heavy-tail distributions? This seems invalid to me. 2) Do we really need to replace Gaussian noise in the Langevin dynamics with others to improve sampling? As far as I am aware, the mainstream technique is to improve sampling techniques, e.g., adaptive bias force, parallel tempering, etc.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a class of score-based generative models motivated by Levy dynamics to produce large jumps and small roaming to facilitate exploration. Since the exact numerical simulation of Levy dynamics is quite challenging and intractable, the authors proposed a vanilla mixture formulation to approximate Levy dynamics; the authors relate such an update with the probabilistic graphical model for illustration purposes. The authors also empirically verified that the exploration property has appealing properties in generative models.
strength_and_weaknesses: 

Pros: the insight of exploring the mixture of Gaussian distributions to approximate heavy tail distributions is quite appealing. I like this idea and the local trap problem in generative tasks is always a critical concern. There are various popular methodologies to tackle this issue, but no one has studied the important Levy dynamics before. As such, it is worthwhile to study related algorithms and how such an idea performs in applications such as generative models.

Cons: 

1. My biggest concern is that the mixture formulation is not general enough; why it has to be 50% and 50%, just because it is easier to implement (n mod 2 ==0)? Can we achieve it in a better way, say probabilistically to achieve more flexible settings?

2. the transition from Levy process to Gaussian mixture is a bit fast. For certain types of Levy process, what is the optimal Gaussian mixture under some formulations to approximate it?

3. In section 4.2, the authors said the mixture update is more robust, but not enough proof can support that.

4. The mixture formulation is first presented in section 3.2 and then becomes continuous in section 4.1, but then goes to discrete again, which looks confusing.
clarity,_quality,_novelty_and_reproducibility: Clarity: i) The results of SGM in Tables 1 and 2 seem too strange to me, there seems to be a cut-off phenomenon for SGM according to the authors. However, I am not convinced why this happens; ii) The connection to the probabilistic graphical model is not detailed enough.



Novelty: The idea of exploring heavy tail distributions for exploration is interesting; Gaussian mixture approximation is acceptable, but the implemented method is not clever enough. 

Quality: The method itself is not comprehensive enough; e.g. at least the authors should clarify what is the **optimal mixture formulation given a Levy process of a certain degree.**

Reproducibility: methodology seems simple enough to implement; didn't check the code. I am not convinced why there is a cut-off phenomenon for SGM in table 1 and 2.

Minor: define GM before using it in abstract.
summary_of_the_review: The idea of adopting Levy process to facilitate exploration is interesting; the Gaussian mixture approximation is acceptable; however, the method itself is not good enough in my opinion. As such, I tend to be conservative in my ratings.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The work presented in this paper stems from the observation that sampling from a target distribution can benefit from heavy-tailed Levy dynamics, whereby the stochastic process is allowed to implement large jumps, thus avoiding local minima and allowing the process to explore the distribution space more efficiently.

Thus, the idea is to extend score-based generative models, a la Song et al.21b, such that the SDE dynamics describing the forward and backward diffusion processes can benefit from a Levy-like behavior, and allow large jumps in the corruption and denoising steps.

However, the inversion of an alpha-stable Levy process is intractable, and the authors suggest approximating the Levy process by inducing a Gaussian mixture model on the Brownian motion term of diffusion of the vanilla score-based SDE model. The hope is that this approximation faithfully reproduces the properties of an alpha-stable Levy distribution. It is important to note that while training the proposed model requires the Gaussian mixture of noise variances, samping from it is instead done using the vanilla approach of simulating the reverse SDE with a single, time-varying Gaussian noise distribution.

A series of experiments complement the methodological contribution of this work, whereby the authors compare their model to the vanilla score-based diffusion model, using three popular datasets for the task. In summary, the results indicate that the proposed method outperforms generative quality (in terms of FID score) of vanilla score-based models, at a fraction of the number of sampling steps.

strength_and_weaknesses: The strengths of the paper are:
* Improving the efficiency of score-based generative models is an important topic of research, as it is well known that baselines are very costly both in terms of training and sampling

* The experimental results show that the proposed method achieves superior quality of generated images when compared to a vanilla score-based generative model

* This work attempts at porting the body of literature on sampling with alpha-stable Levy noise distributions to diffusion models

The weaknesses of the paper are:
* There is no explicit motivation behind this work! In absence of it, my understanding is that the goal of this paper (based on the emphasis given on the results, and on the concluding remarks) is to accelerate sampling for diffusion models. I think that the authors should clearly state early on in their paper what is the problem they are trying to solve, and why current existing solutions are not sufficient.

* Based on the assumption above, if the goal is sampling acceleration, then the experimental campaign can be improved, by comparing the proposed method to other methods that aim at accelerating sampling, or in general making diffusion models more efficient. For example, several works (for example, [1]) propose adaptive integration schemes to simulate the backward SDE of a score-based diffusion model. I suggest to compare the proposed method to comparable approaches in the literature, instead of using a vanilla baseline which in this case would be “easy to beat”.

[1] https://arxiv.org/abs/2105.14080 “Gotta Go Fast When Generating Data with Score-Based Models”

* The quality of mathematical notation and in general mathematical rigor can be improved. Starting from eq (1), the notation is borrowed from the work on Langevin dynamics from “Bayesian Learning via Stochastic Gradient Langevin Dynamics”, Welling and Teh. But then the subsequent notation comes from early work on score-based generative models, to end up (in sec 3) with the notation and formalism of SDEs. Now, the problem is that step size, noise terms, noise schedule, the diffusion term, etc… use a similar if not the same variable name.  This is very confusing, and should be cleaned up. 
I have nothing against presenting the background in reverse, starting from the discretized SDEs, to continuous time SDEs expressions of the diffusion process, but at least the notation should be coherent. Also, some more details (in the background section) could be useful, as for example stating why you focus on variance exploding SDEs, and not variance preserving.

* I understand the definition of an alpha-stable Levy diffusion process is proposed as an original contribution, but I would kindly ask the authors to compare their proposal with similar work that question the use of Gaussian noise only, and proposed to use a different distribution, such as [2]

[2] https://arxiv.org/abs/2110.05948, “Denoising Diffusion Gamma Models”

* Due to the intractability of the reverse Levy process, which requires solving a fractional derivative (for which approximate algorithms exist) and, more importantly, requires access to the data distribution, which is exactly what we are aiming to sample from, the proposed approach is to “approximate” the Levy walk with a Gaussian mixture noise distribution. However, even if the simple 1-D example given in Fig.2 shows some similarities between the trajectories of the proposed approximation and a true Levy walk, in principle the approximation does not guarantee the same properties. There are no large jumps, and the basic principle of “escaping from a local minimum” of a Levy walk cannot be ensured.

* Experiments. In general, experiments are appropriate, as comparing the method to vanilla score-based diffusion is a valid approach to make sure the proposed method works. Nevertheless, the weakness of the empirical evaluation is that it does not compare the proposed method to existing methods from the literature that aim at accelerating sampling. In my view, as there are no clear motivations in the context of score-based generative models, to model the diffusion process as an alpha-stable Levy diffusion, the main take home message I get from the experiments is that it produces better samples faster than the vanilla model. Then, it makes sense to ask how the proposed method fares when compared to, say, better SDE integrators.

clarity,_quality,_novelty_and_reproducibility: This work is sufficiently clear, even though the mathematical exposition can be improved and be more rigorous, especially with notation. The quality of this article is good in general, but I really miss a problem statement, motivation and a list of contributions as seen by the authors. 

Concerning the novelty, considering alternative distributions to Gaussian to model the noise component has been attempted before (see [2] above). To the best of my knowledge, alpha-stable Levy distributions have not been considered before. So the idea outlined in section 3.1 seems novel to me. Nevertheless, I am not convinced that the proposed approach, although it works in practice, is a good approximation of the Levy diffusion.

I couldn’t check the code for this work, but the Gaussian mixture approximation does not seem to pose particular implementation challenges, and it only affects the training process, whereas sampling is the same as for standard score-based models. As a consequence, I think this work can be reproduced by reading the paper.

summary_of_the_review: This work presents an interesting perspective, although it lacks a clear motivation. The attempt to solve the intractability of the proposed alpha-stable Levy diffusion process is not convincing in my opinion. The proposed approach achieves good empirical results, when compared to a simple baseline, but it has not been compared to alternative methods to accelerate sampling. There are many other weaknesses in the submitted paper, stemming from lack of rigor in the mathematical description of the model, as well as in the proposed approximation of the Levy diffusion process.


=============================

Post rebuttal comment

=============================

I've read the rebuttal, all reviews and their comments. I am willing to slightly raise my score, but I am not convinced this submission deserves to be accepted at this time. I hope the Authors will find comments useful to further improve their work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Inspired by biological systems, this paper studies heavy-tailed Levy dynamics for diffusion models to encourage both large jumps and small roaming during sample generation. However, since simulating Levy dynamics is computationally expensive and intractable, they are approximated using Gaussian mixture distributions with shared mean and different variances in this paper. 

Pros:
- Introducing new stochastic processes for diffusion models is interesting and novel
- The paper is overall written nicely and clearly. The mathematical notation could be improved slightly as pointed out by vUWR

Cons:
- Demonstrating and motivating why diffusion models need large jumps is missing
- The discussion in Sec 4.2 does not provide theoretical justification for why the proposed method is more robust. Additionally, this analysis does not show that the generative score model actually will decide on large jumps vs small roaming as the final model will be the average of all denoising directions across all possible sampled noise scales (betas) given an x_t. 
- The approximation of Levy dynamics with a Gaussian mixture is not theoretically analyzed
- Comparisons against accelerated sampling techniques from diffusion models (both ODE and SDE solvers) would strengthen the position of this paper
justification_for_why_not_higher_score: Although the proposed idea of approximating the Levy dynamics using Gaussian mixtures in diffusion models seems interesting and novel, the reviewers have raised concerns regarding the key motivations of this submission, the quality of approximations, comparisons against accelerated sampling techniques, and the impact of multiple noise scales per time step on the robustness of the score model. Given these concerns, the paper does not appear to be ready for publication at ICLR.
Decision: Reject