Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper introduces a non-linear drift component to the SDE formulation of diffusion-based generative modelling. Earlier work (DDPM) used a fixed linear drift component. This paper uses a trainable neural network as the drift component, and another one to estimate the score. Both these neural networks are trained by minimizing the KL divergence between the forward and backward trajectories, while accounting for their SDE formulations. It is found that this trained network performs reasonably well compared to prior works, and achieves better sample quality and shorter trajectories compared to DDPM as well as Continuous Normalizing Flows (FFJORD).
main_review: The paper is clear enough to understand, although there are some typos in a few places (mentioned below).

The "novel" contribution of making the drift component non-linear is relevant in the context of this line of research : discrete Markov chains (DDPM [1], etc.), to equivalent continuous formulations with the help of SDEs [2], to a variety of SDEs where the drift component is learnt as well instead of being prefixed.

Relevant examples have been shown in 2D data as well as image sampling to highlight the advantages of this method compared to the previous version (DDPM). Although the continuous formulation (SDE) is mentioned, the paper converts it back to a discrete Markov chain and trains it accordingly.

Figures 5, 8 and 9 illustrate the relevance of the current method compared to DDPM and FFJORD. They make a convincing argument for the use of a trainable forward process. Lines 182-184 make a convincing argument for why this particular type of 2D data was chosen. It is all the more encouraging that even though finer details are better, it is not at the expense of likelihood/NLL (in many cases).

A major significance is that the number of sampling steps can be quite low compared to the original formulation (Table 4) and yet the quality of generated samples does not significantly deteriorate as much as the baseline (DDPM).

Table 3 compares the NLL and FID of previous relevant methods for CIFAR10. DDIM [3] is missing from the list, as it is also a discrete formulation like DDPM but uses a different sampling scheme. It is a relevant comparison since DDIM also aims to reduce the number of training steps. Although, the method proposed is significantly different from the motivation of DDIM, in that DDIM preserves the DDPM formulation and only changes the sampling scheme, while the paper changes the forward process itself. 

The appendix mentions the resources used to complete each experiment. It needs to be seen whether computational efficiency could be improved.

Code has been provided to reproduce the experiments.

[1] Denoising Diffusion Probabilistic Models, NeurIPS 2020
[2] Score-based generative modeling through stochastic differential equations, ICLR 2021
[3] Denoising Diffusion Implicit Models, ICLR 2021

Line 87 says x(0) = z and x(T) = x, but lines 121-122 say x(T) = x and x(0) = z. I assume the latter, it is good if this is clarified.

Line 55 : the forward process from x* to z* ?

Line 166 : (to add) in Figure 3*

Line 192 : noi*sing

Line 198 : to*pological?

Figure 5 : Frac*tal tree

Line 239 : using*

---

Update : I stand by my original evaluation of accepting this paper:

The "novel" contribution of making the drift component non-linear is relevant in the context of this line of research
I found their methodology to be well motivated
The pros and cons of each design step have been mentioned
Enough experiments have been performed to explore the method, with the pros and cons
Code has been provided for reproduceability
Computational resources used for each experiment have been mentioned
Reviewers' comments have been responded to sufficiently and honestly
If the other reviewers feel there are still some unanswered questions, or comments that have not been responded to, please let us know. I think this paper deserves to be accepted.
limitations_and_societal_impact: The appendix describes the computational resources used in training the models. The figures and tables clearly illustrate the limitations of the method. It could be mentioned that lower sampling steps could alleviate computational burden significantly.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes to lean a more expressive forward process in a diffusion model by using a neural network drift function.
main_review: I think it is a nice insight that diffusion models can be thought of as relaxed normalizing flows. The method to learn a more expressive forward process is new, although it is straightforward and does not really resolve key issues in learning more expressive forward processes. For example, it is well known that the affine forward processes allow for sampling of $q(x_t | x_0)$ for arbitrary $t$, allowing for fast training by picking random $t$; furthermore, variance of training and evaluation can be reduced by Rao-Blackwellizing the variational lower bound into $x_0$ MSE prediction losses. Both properties do not seem true with the presented method when f is a neural network, and the computational complexity tradeoffs should be studied more thoroughly (e.g. what is the quantitative benefit of slower training time vs better sample quality or log likelihood?)

While it is interesting to learn more expressive forward processes, it is not clear why the method needs to be presented as a combination of diffusion and normalizing flows: it seems clear enough to present the method as learning a more expressive forward process for diffusion models. Also, I found it difficult to understand the description of time discretization in Section 3.3.

In Section 4.1, I am surprised that DDPM "blurs density details." Shouldn't it be possible to resolve this by carefully tuning the noise schedule? What hyperparameters were used for this experiment?

I find Table 4 slightly misleading. It is true that the FID scores are better than DDPM for small N, but for large N, the FID scores of DDPM are around 3, and it seems that the presented method does not approach that score.

Why does the table omit log likelihoods and FID scores from published methods that perform better, e.g. from Score-Based Generative Modeling through Stochastic Differential Equations by Song et al? The paper claims state-of-the-art NLL (line 246), but it is not.

The main limitation of the paper is that as it is presented, the results do not significantly improve upon the baseline enough to warrant the more complex training procedure and model construction, there is no adequate study of the training time (or FLOPS) vs NLL or sample quality, and in some cases, the results are presented in a slightly misleading way (e.g. Table 4).
limitations_and_societal_impact: Limitations of the paper are not adequately addressed, see the main review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper under review introduce the Diffusion Normalizing Flow, which combines the diffusion model method and the normalizing flow method in their algorithm. The idea is clearly presented and the main part of the paper is focused on the experiment implementation of the algorithm. 
main_review: Positive.

1. The idea seems to be new. Indeed, the proposed DiffFlow is one type of the Neural SDEs, where the volatility only depends on time variable. 

2. The idea and the comparison to diffusion model and normalizing flow is clearly presented in the figure and one theoretical result is provided to link the three different methods.


limitations_and_societal_impact: Negatives.

1. The model itself seems to lack theoretical justification. To be precise, The proposed diffusion model with equation (5), (6) does not come naturally from the cited references [34,16]. It seems that the original form of the backward diffusion process is first introduced in [37]. This is not clearly stated, if the reviewer did not miss other key references. 

2. Furthermore, the score function $\mathbf s(x,t,\theta)$ appears in the backward process without derivation or justification. The one in [37] follows from the results in Anderson 1982.  The reviewer is curious about the original sources of equation (6), or can the author provides the derivation of this backward process.  The term "stochastic adjoint" is used in the paper, but the adjoint process is not well explained in the paper, together with its link to equation (6).

3. Once equation (5),(6) are introduced, the author then claim to consider the DiffFlow by letting $f(t,x)$ to be $f(t,x,\theta).$ Is there any reason to do so? Why the diffusion function $g(t)$ is fixed in this case. Does this change keep the property of the original backward process? From the reviewer's point of view, the algorithm is rather intuitive, is there any derivation of the backward or adjoint process here after $f(t,x,\theta)$ being used.

4. It seems that the algorithm is very similar to the recent paper [Efficient and Accurate Gradients for Neural SDEs]. Can the author do some comparison in this case? (e.g. memory, efficiency, etc..)
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes "Diffusion Normalizing Flow", which generalizes current diffusion-based models to have a trainable inference process. This allows the model to transform structured data into an unstructured prior in a more efficient manner, thus resulting in faster sampling.
main_review: ### Significance and Limitation

The paper serves as a natural extension of [1] and [2], which have a fixed inference process that slowly transforms an arbitrary data distribution into an unstructured prior and learn to revert the dynamic. This paper proposes to parameterize the inference part so that the transformation of the data into an unstructured noise can be more efficient (as illustrated by Fig 4). 

Two limitations of the proposed methods:

1. Computational time: Despite the generality this new framework provides, I suspect training will take much more time since one would have to backprop through the entire trajectory. This is in contrast to the proposal of a recent concurrent work [3], which, due to the constraint on the form of the inference process, still allows exact sampling of the perturbed data at any time step. I see this as a limitation since having to backprop through the entire trajectory means it will make it harder to scale up to larger datasets. 
2. Memory: Backpropagation through the entire path also raises another problem — the increase in memory. This work proposes an "adjoint" method and claims to have constant memory consumption (in the number of integration steps). However, I found it to be not entirely true, as the algorithm requires caching all intermediate states $x_i$, which means it is still technically $O(N)$. A real constant-memory adjoint method would not require storing all the intermediate states, such as neural ODE, and another recent work [4], which suggests using the stochastic adjoint sensitivity method proposed by [5]. 

[1] Deep unsupervised learning using nonequilibrium thermodynamics

[2] Denoising diffusion probabilistic models

[3] Variational Diffusion Models

[4] A Variational Perspective on Diffusion-Based Generative Models and Score Matching

[5] Scalable gradients for stochastic differential equations

Presentation and clarity also need some work; see the following points for more detail. 

### More technical questions and criticisms:

1. I don't think we can simply drop all the constants as claimed in L142. For example, the normalizing constants of the Gaussians will depend on the variance, which is the diffusion coefficients of the SDEs, unless these coefficients are either constant or not trained. Overall I find the derivation in Section 3.1 very confusing; perhaps having a detailed derivation in the appendix will help.  
2. Is there a runtime comparison between DDPM and the proposed method in training? e.g. how much time does it take to make one parameter update, fixing the model size? 
3. It is claimed that DiffFlow is a combination of flows and diffusion models (e.g. L179). It's not very clear to me how NF is integrated into this framework. Do you compose flows with diffusion models? Or it's just a matter of interpretation?
4. Aside from the use of the equivalent ODE for density evaluation (which is originally proposed by [6], is there other reasons to introduce the more general marginal-equivalent SDEs, i.e. eq (17)? Also, why not report the ELBO of the SDE?
5. What is the concept of "layer" within this paper? e.g. in L231, it says DiffFlow uses no more than 5 layers. Is it the size of the network that parameterizes f, g, and s? 
6. The statement of Theorem 1 is very handwavy, and so is the proof. "is equivalent to", under what condition? In the proof it seems it requires fixing $p_B(x_i|x_{i-1})=p_F(x_i|x_{i-1})$, what does it mean by it's possible when g is small, e.g. a Gaussian approximation will suffice? It doesn't feel like a rigorous proof. 

[6] Score-based generative modeling through stochastic differential equations

### Typos, and other suggestions:

- L36: I'd put "continuous-time" normalizing flows, instead of just normalizing flows
- L100: $f:\mathbf{R}^d\times \mathbf{R}\rightarrow \mathbf{R}^d$
- L103 $p_F$ is defined to be the distribution over trajectories, but then in L106 it's used as the score, which is the gradient of the marginal density. This is not accurate.
- Eq (10): why does it hold? Is it the data processing inequality? Please clarify.
- L192: noising transformation?
- In Appendix A, [39] is cited, but is not displayed in the paper.
limitations_and_societal_impact: See above.
review_title: null

Meta-review: There was significant discussion of this paper, with one reviewer increasing their score. The score range was unusually wide, even after discussion.

Reviewers felt that the idea presented was both novel and interesting. I agree with this assessment -- I think this paper draws a very nice and interesting link between normalizing flows and diffusion models. It's impressive to cast them into the same theoretical framework, and I think this will inspire followup work.
However, reviewers also had remaining concerns. Most seriously that:
- the computational tradeoffs during training that this approach required weren't clearly acknowledged or discussed. I believe this is a fair criticism. Please include discussion of this, and the training time tradeoffs you discussed in your rebuttal, in any camera ready.
- that the memory discussion around the adjoint method is misleading. I am unable to fully judge this concern. I do think something like logarithmic checkpointing should also be able to prevent memory overhead from being an issue though, so it's not clear to me that memory usage was ever a critical problem to start with.
- that the performance table left out a number of results with better FID and log likelihood scores than the presented method. This is definitely true, and the additional rows **must** be added for any camera ready. The paper only claimed "competitive performance", not SOTA, so I do not believe this is a fatal flaw.
- there were also some concerns about clarity, and I encourage the authors to address the specific aspects that reviewers found challenging.

Due to these legitimate concerns, I struggled with my recommendation for this paper, and read the paper myself to make a more informed decision. Because of the novelty of the approach, and because of my judgement of the relative importance of novelty to scientific progress, I am recommending acceptance.
Decision: Accept (Poster)