Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper establishes a connection between diffusion models and serial reproduction. The main result is that as long as the distributions of images along the diffusion time remain approximately stationary, then the perfect solution for minimizing the KL between the sampling process and the real process leads to samples from the correct (clean) distribution. This holds independent of the corruption type, which explains to a certain degree the success of recent diffusion models with more general corruptions.
strength_and_weaknesses: I think this is an interesting paper. The main take-away (which is fairly intuitive) is that the corruption should be smooth -- and that's pretty much all that matters to sample from the correct distribution at the end. The main issue I have is that the mathematical statements yielding this result are not very precise, i.e. they do not consider how error propagates in Eq. 15 and the result only holds when the learning is perfect. 

Strengths:
* the paper is well-written.
* The simulation experiments are really interesting. They show that for toy-distributions (that we can characterize their density all the way), the corruption type is not really important to reconstruct the proper distribution. I am wondering if that's true for *any* corruption with smooth transitions.
* The topic of the paper is relevant since there have been many recent attempts to generalize diffusion models.


Weaknesses:
* the mathematical statements in the paper are not precise. While I appreciate the fact that the authors try to build intuition, it would be cool to have an analysis of how the errors propagate in Equation 15. This "approximately stationary" statement might completely blow-up the distribution we are sampling from once you integrate over all diffusion steps. How "approximately" is needed to get a reasonable bound?
* the analysis seems to be held only in the case where the solution gives 0 KL. How do learning errors propagate when that's not the case?
* There is no characterization of the conditions under which "approximately stationary" is not too far off. How is this connected to the smoothness of corruption? Are there toy models that satisfy this property? Are there toy models that don't satisfy this property and for them the learning fails?
* the connection to serial reproduction is interesting, yet kind of indirect. For me, the main point is that independent of the corruption type, you still sample from the correct final distribution as long as the condition of Eq. 15 is satisfied. It is true that in serial reproduction you also sample from the priors, independent of the transition kernel and I do appreciate the connection. However, I think it is an overstatement to say that this connection yields many insights on how diffusion models work.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. The connection to serial reproduction is novel. The mathematical analysis seems new, but it is not precise.
summary_of_the_review: I think this is a paper in an interesting direction. However, I am concerned because of the lack of clarity in the mathematical statements.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Inspired by a cognitive science paradigm, known as "serial reproduction", the authors show how the diffusion model approximates a Bayesian posterior at every denoising step, with the diffusion kernel as likelihood and the forward marginal as prior. This re-interpretation is enough to explain two properties of diffusion models that have been empirically observed: i) a diffusion model can approximate
the true data distribution irrespective of the choice of noise family for a fine enough noise schedule; and ii) the noise parameter has to diminish in order to reduce reconstruction error. The paper supports the theoretical insights with illustrative experiments, first on a setup where the noising and denoising distributions can be computed analytically and different noise schemes; and second, with MNIST images. One final insight provided by these experiments is that the sample accuracy saturates after some number of steps, and based on the theoretical insights, the authors suggest this is due to the forward process having converged to its stationary distribution.
strength_and_weaknesses: **Strength**
- The paper establishes a novel and insightful connection with cognitive neuroscience, making it accessible and hopefully inspire other researchers.
- The insights derived from this connection are relevant, as they explain two empirically observed properties, especially the robustness against the noise family, which made previous formulations that assumed Gaussian noise incomplete.

**Weaknesses**
- See comments on clarity below.
clarity,_quality,_novelty_and_reproducibility: **Clarity**
- My only negative comment on clarity is regarding the use of the term "sampling". I think it is common to distinguish between "learning/training" and "sampling", the latter referring to the denoising process. However, in the introduction, it is said that "sampling is then done using a sequential procedure whereby an input signal is iteratively *corrupted by noise and then denoised*". I found including noising in the sampling unnaturally trying to resemble the explanation on serial reproduction. 
- Apart of the previous comment, the paper is well organised, generally well written and easy to read.

**Quality.**
Following up with my previous comment on "sampling", the same idea of adding noise during sampling is used formally in Sec. 3, when describing sampling process as a Markov chain, with terms like, $x_t \rightarrow  \hat{x}_t$. What are these terms? Are they just artefacts for the explanation, meaning that $x_t$ is drawn from some noise kernel? Or it is assumed that $x_t$ is actually corrupted with noise before every denoising step? Please be explicit in the text if this is an artefact to avoid confusion.

**Novelty.** 
The paper establishes a novel and insightful connection.

**Reproducibility.** 
The paper includes a final section on reproducibility pointing to the code in the supplementary material. While I agree that making the code available is the gold standard for reproducibility, I encourage the authors to describe all the missing details also in the appendix, especially regarding the deep learning experiments for completeness.
summary_of_the_review: #### Meta-review
The authors have addressed my concern on the definition of the denoising process, explaining that it follows the latest research on more effective sampling for the more general cold-diffusion processes, and showing an equivalence with the original DDPM derivation.

---
#### First review
The paper provides a novel and insightful formulation of diffusion models, shedding light on two empirical observations that had not clear explanation with previous formulations. This is an interesting contribution that improves our understanding of these powerful models. 
My current recommendation is based on the understanding that my comment on quality is just a matter of clarity, and not a misunderstanding or variation of the denoising process.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper builds connections between existing diffusion models such as DDPM with serial reproduction paradigm. One major conclusion is that, current diffusion models can be explained as a natural consequence of this connection correspondence. Simulations on the MNIST dataset shows that the connection can be utilized for richer noise distribution family such as bimodal and fade.
strength_and_weaknesses: Strong:
1. The analysis and connection of between diffusion models and serial reproduction paradigm;

2. Simulations on datasets to show the robustness of applying different types of noise distributions.


Weak:
1. Larger scale datasets and richer experiments can help improve the quality of this paper;

2. The connection between diffusion models and serial reproduction paradigm seems to have less novelty in terms of methodology and it is difficult to say that new knowledge is obtained after reading this article.

clarity,_quality,_novelty_and_reproducibility: This paper provides a new way of understanding why diffusion models work so well. There are several equations can be modified to align with existing DDPM mathematical equations. 
The clarity and quality are average.
The novelty is limited and can be further improved.
The reproducibility of this paper is high since it is based on existing open-source code.

Detailed questions and comments:

1. Equation (5), better attach $\theta$ for $p$ to indicate that it is trainable with parameter set and differs with $q$ in equation (4).

2. Equation (8), $q$ better with a given sample $x_0$, e.g., from $q(x_{t-1}|x_t)$ to $q(x_{t-1}|x_t, x_0)$, to express the posterior distribution in the forward diffusion process.

3. Do you have more results of simulation on larger datasets besides Mnist dataset? Figure 4 shows some results – they are largely influenced by the sampling methods you use as well, such as Euler-Maruyama, the predictor-corrector and ODE sampling. Currently, the T=50 vs. T=500 is significantly different and based on my own experiments, T=50 could yield quite good results with better sampling methods and without changing anything of the trained model.

summary_of_the_review: Given current strengthes and weaknesses, it is difficult to score a high recommendation to this paper. Some equations can be updated, richer datasets are preferred and more sampling algorithms can be used to perform rich ablation study.

--

I would like to rank this paper higher, after reading their detailed responses.
Most of my questions were well answered.
Again, if any larger scale datasets' results are reported or shared, this paper worth an even higher score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work discusses the connection between the sampling process of diffusion probabilistic models and a cognitive paradigm known as serial reproduction. By building the correspondence, the work tries to provide new theoretical understandings of DPMs and intuitions of improvement of key features of DPMs.
strength_and_weaknesses: Strength:

- Theoretical understanding of diffusion probabilistic models is of great research interest recently. 
- The connection between the concept of serial reproduction and DPMs is also interesting.

Weakness:
- Contribution: It is not very clear what exactly is the contribution of the paper. The paper introduces a connection between the sampling process of DPMs and serial reproduction. Drawing inspiration from this connection, some key features of diffusion models such as noise scheduling can be principally improved. However, the conclusion that "the noise schedule should be chosen such that it alters the input distribution gradually" is not new (Dhariwal & Nichol, 2021 https://arxiv.org/abs/2105.05233).
- Technical soundness:
    * On page 4, between Eq.(9) and Eq.(10), the paper describes the sampling process of a DPM as $x_t \rightarrow \hat{x}_t  \rightarrow  \cdots$. Why a noisy version of $x_t$, i.e., $\hat{x}_t$ is needed in the sampling process? This seems not true for the well-understood DPMs where the sampling process involves only the optimal posterior distribution (as in Eq.(5)).
    * Latter derivations all depend on the above-mentioned assumption of the sampling process. So it should be crucial to clarify whether the standard sampling process resembles serial reproduction and what Eq.(10) actually means.

Minors:
- In section 2.2, the statement "the noise distribution $q_n$ is stationary" is not always true. For variance exploding diffusion (Song et al. 2020), the noise distribution is not stationary since the variance does not converge.
clarity,_quality,_novelty_and_reproducibility: - Clarity: The notation used in the paper is a little bit difficult to follow. For example, the mathematical form of $T_{q_n}(x_t|x_{t-1},\beta_t)$ is not explained in details and $\beta$ is mentioned as diffusion parameters and strength parameter in different place.
- Quality: I have raised some concerns about technical soundness. Please see my comments above.
- Novelty: The connection discovered by the paper is novel.
- Reproducibility is ok.
summary_of_the_review: In conclusion, I have raised several concerns regarding the contribution and technical soundness of the proposed method, which prevents me from giving this paper a higher rating now. I will consider increasing my rating if the authors address my concerns and clarify my misunderstandings.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: By observing that diffusion models bear a resemblance to a concept of "serial reproduction" in cognitive science, the paper draws insights into why diffusion models turn out to be robust to the choice of noise family, and what role noise level scheduling plays for effective sampling. 

Strengths: Understanding why diffusion models are so effective, and how they are influenced by noise families and noise scales, is a question of significant current interest.  The sim experiments are interesting as they confirm that precise nature of corruption is not really that important. 

Weaknesses: The deeper contributions and novelty of this paper remained unclear at the end of the review process. The conclusion that "the noise schedule should be chosen such that it alters the input distribution gradually" is not new. That the connection to serial connection yields many new insights was not entirely convincing in terms of changing the core algorithmic backbones of diffusion models. The reviewers make several suggestions to improve technical presentation and experimental analysis, e.g. some mathematical statements could be made more precise. 


justification_for_why_not_higher_score: See weaknesses above.
Decision: Reject