Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors expend the linear diffusions of SDEs to a data-adaptive trainable nonlinear diffusion. They propose a new model, implicit Nonlinear Diffusion Models (INDM) to train its forward diffusion SDE. The proposed model constructs nonlinearity of data diffusion by transforming a linear diffusion process on the latent space back to the data space, where the transformation is implemented with a normalizing flow.
strengths_and_weaknesses: 1. Nonlinear diffusion is taken into account in diffusion models.
2. The proposed diffusion model is able to learn forward diffusion process out of the variational family of inference path measures, and the corresponding optimization is able to provide MLE training and a good diffusion bridge.
3. Experiments are carefully conducted and analyzed.

questions: Are there other generative models that integrate normalizing flow to be considered as baselines in the experiments?
limitations: 1. The motivations of integrating normalizing flow into the diffusion model is not convinced. Can we integrate other component rather than normalizing flow into the diffusion model?
2. The motivations of why invertibility is important is not clear, although the authors provide a figure and tried to illustrate this.
3. Normalizing flow has been widely used in generative model such as normalizing flow-based VAE. Thus, the idea of integrating normalizing flow into diffusion model is not novel, as diffusion model is categorized into generative model.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a variant of latent diffusion models using invertible flows, this enables likelihood-based training. By training a linear diffusion model in the latent space, the corresponding diffusion model in the data space will be nonlinear. Training the flow allows one to learn both drift and diffusion volatility in the forward data-space diffusion.

The authors demonstrate excellent generative performance on standard datasets, as measured by FID scores and show how the method may be adapted for interpolation.
strengths_and_weaknesses: 
Strengths:
- The authors propose an elegant method to train latent diffusion models using flows as encoder/ decoder, this permits efficient training of nonlinear diffusions via linear diffusion in the latent space and gives an explicit nonlinear diffusion in the data space
- The method exhibits excellent generative performance measured by FID scores 

Weaknesses:
- The paper would benefit from an algorithm detailing the exact training procedure
- In the appendices, line 599, it is stated that the diffusion model is trained without a flow (for 5 days), then is fine-tuned with the flow. This is a very important detail and should be included in the main text. Does the proposed method not work without pretraining? The authors should include the performance for the diffusion without the flow first to indicate what the benefit is. Similarly, given 5 days is a significant amount of time would this not mean the training cost is a lot more than regular diffusions? Hence Table 1 may be misleading.
- OT section is not clear and seems misleading:
    - It appears there is no theoretical justification for being close to OT, am I right in thinking this is purely an empirical observation?
    - Line 275 references Appendices C.4, however, there is no Appendix C.4 in the supplementary material
    - Line 264, is this the Wasserstein 2 distance being compared? How is it computed? Hence is the ground cost the squared Euclidean distance? It is important to note that DDPM uses a discretized Ornstein Uhlenbeck process (VPSDE) which will result in an approximate OT distance with different ground cost to the squared Euclidean, one would need to use a Brownian motion (VESDE) for fair comparison, see e.g. section 3.1 of [1]. Hence the OT trajectory for different ground costs may not be a straight line, see e.g. [2], hence the diagrams and discussion may be misleading e.g. line 247. 
    - FBSDE, as advocated in [18]'s code, trains a forward and backward network to completion and then fine tunes with a small number of steps. If not training each network to completion, the final networks will not result in an SB approximation as it will not be following the IPF. In order to get a reasonable OT computation, one needs to train the network to completion at each IPF iteration.
    - Perhaps comparing OT with closed form solutions such as between 2 Gaussian measures might be easier to justify, see e.g. Figure 2 of [1] (arxiv version)
- Figure 4 uses a checkpoint from FBSDE [18], is this where the first iteration is using score-matching with a long time horizon and the forward diffusion in the first IPF iteration, then fine-tuned in subsequent IPF iterations? If so then visually looking at the diffusion time for non linearity may be misleading if most of the diffusion is close to Gaussian, see e.g. [2] the double well experiment for non-linear trajectories


- Minor
    - Line 179, "Few works [18] in SBP detour the issue on the slow training by constructing an experience replay memory ", this technique for Diffusion SB was first introduced in [1], see page 53 Technique 3, and Algorithm 3. This paper was around 5 months before the reference cited.
- Line 229, again [1] describes continuous time SB in section 3.5, and implements a discretized version


Summary:

The paper proposes some nice ideas, and the method demonstrates excellent performance. The idea to use flows to convert a diffusion to latent is a good contribution.

Unfortunately the authors focus too much on connections to OT (which appears theoretically ungrounded, not explained well, with some apparent misunderstandings) and not enough on explaining the actual method, algorithm, and training procedure, omitting key details from the main text such as the pretraining regime.


[1] Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling
Valentin De Bortoli, James Thornton, Jeremy Heng, Arnaud Doucet, https://arxiv.org/abs/2106.01357
[2] Solving Schrödinger Bridges via Maximum Likelihood, Vargas, https://arxiv.org/abs/2106.02081

questions: - It is not clear what figure 7 is showing, shouldn't the end points lie along the same support for each method in data space if the diffusion model is working?
- Does the interpolation loss require taking a gradient through time in the diffusion simulation? If so what is the memory requirement, does this break for larger images?
- See questions on SB in weaknesses above
- Is the extensive pretraining regime required? 
- What is the total train time including pretraining vs other baselines (without pretraining)? 
- The latent is the same dimension as the data, is there any computational savings?
limitations: The authors briefly comment on training time in the final section, however, do not detail the extensive pretraining regime required.

It would benefit the paper to address some issues in the discussion regarding OT, and move this to the appendices, then instead spend more time discussing the training procedure, provide an algorithm and discuss the importance of the pretraining regime which will be of significant importance to practitioners.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a trainable non-linear diffusion process as a generative models. This work is an expansion of previous successful works that use linear diffusion processes and showed state-of-the-art results on sample generation.
This non-linearity is applied implicitly, helping avoid the alternative of an explicit non-linearity in the drift and diffusion coefficients, which leads to an intractable probability term that causes long training time.
The key idea of the presented method is to transform each data point to a latent variable using a non-linear invertible transformation, and applying a linear diffusion process on that latent variable, where in each point in time one can retrieve the relevant diffused image from a diffused latent variable.
The authors showed that their method also constructs a generative process that is easy to sample from.
strengths_and_weaknesses: Strength:
	- The problem is known to be valid and challenging, and the proposed idea is novel.
	- The method and the idea behind it are well explained in the paper.
	- The authors provide a motivation for non-linear diffusion process by presenting very nice figures of diffused data and its vector field during diffusion, that shows a clear advantage of non-linear diffusion over the linear version.

Weaknesses:
	- The quantitative results are not the best in all , but the gaps between them and the SOTA results are not too big, especially when considering the number of parameters required for the training.
It would be nice to see qualitative results on the CelebA dataset, as the gap in FID seems to be significant. 
questions: See my comments in the Weaknesses section.

limitations: The authors have addressed their method's limitations.
The described work has no potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to combine normalizing flows and denoising diffusion models in which the data distribution is first transformed into a latent space by normalizing flows and then the latent distribution is modeled by generative diffusion models. By leveraging Ito's theorem, it is proved that the resulting diffusion process is non-linear in the original data space. Therefore, the authors call it implicit nonlinear diffusion models (INDM). Experimentally, the proposed INDM achieves zero variational gap and state-of-the-art FID on the CelebA dataset.
strengths_and_weaknesses: **Strengths**

The application of Ito's theorem is exactly interesting. It shows a strong connection between diffusion models and normalizing flows. And I think this connection is intuitive since the diffusion models can be considered a special case of flow models. Moreover, the reduction of the variational gap is also intriguing by applying normalizing flows.


**Weaknesses**

- The benefit of normalizing flows is unclear. Although the authors try to explain by the means of optimal transport and relative energy, it still lacks an intuitive understanding of why flows+diffusion makes sense, say does the transformed latent space make it easier for diffusion models to learn? One virtue of normalizing flows is to transform complex data distribution into a simple form, then one could learn a model on the simple manifold. However, it is unclear what is learned by flow models in INDM and how flow models increase the efficiency of learning and sampling.

- It seems that the authors try to hind something. I think the main contribution of this paper is to show that the combination of normalizing flows can improve the expressivity of diffusion models. However, the authors do not discuss the benefit of normalizing flows thoroughly, but use a lot of vague discussions to make readers suck.

    - Sec6.1 shows a better converge on NLL and NELBO. But as mentioned before, it's unclear why the normalizing flow benefits this. The author tries to contribute this to the "bidirectional attraction". But after reading the main paper, I can't understand how the flow models can be used to reduce the variational gaps, say NLL=NELBO, and make the learning curve of INDM more MLE.
    
    - Sec6.2 shows the diffusion process of INDM is nonlinear. However, I don't think this illustration deserves to be shown in the main paper because Ito's theorem has told us everything. The appearance of sec6.2 just traps the reader in the details of the eigenvalues of the covariance matrix.

    - Sec6.3 extremely distracts my focus. I don't think the connection of optimal transport makes good contribute to the INDM. I must admit that I simply ran out of energy and gave up on close reading by the time that I got to the description of optimal transport in section 6.3. It is okay to show this connection, although it is still intuitive as I understand. But it is worth appearing in the appendix. Understanding this part in the main paper just makes readers feel distracted.

    - It seems that the relative energy is a good way to show the benefit of normalizing flows. This part deserves further discussion. However, I find the discussion on it is also quite vague. To be honest, I cannot fully understand smaller relative energy contributes to faster sampling. I think the authors should explain this in simple and concise language and give only the necessary supporting evidence for their claims.

- INDM seems an incremental improvement of LSGM. The diffusion and generative processes are similar. The only difference lies in that LSGM is based on the car but INDM is based on the normalizing flows. Just push back, the benefit of learning diffusion models on the latent space transported by flows is unclear, considering that the benefit of learning on latent space of LSGM is apparent, which learns an expressive prior for VAE and simple latent space for diffusion models.

questions: I like figure 9. It tells me to some extent what is learned in the latent space. Could you provide more cases like that, say the results on the datasets like two moons, pinwheel, checkerboard ... Also, an illustration of Cifar and CelebA is also important. From figure 3, it seems that the flow models just learn some color transformation. In this regard, the flow models make tiny contributions to data fitting. Could you show more cases on that?
limitations: 
- The model archiecture is similar to LSGM and the benefit of flows is unclear.

- The results look convincing enough. Though, the onerous discussion make me hesitate about its practical value. The authors should explain their approach in simple and concise language.
review_title: null

Meta-review: metareview: The paper presents a novel non-linear diffusion model that transforms a linear diffusion in a latent space using normalizing flows. After the discussion period, three reviewers rated the paper a weak accept and one as accept. Overall, reviewers found the main idea of training flows in this latent space to be novel and interesting, and felt that the method was supported by experiments showing very good performance. Beyond that, there was a lot of discussion between reviewers and authors. Some of this focused on specific details about the training method; while quite a lot focused on other characterizations and discussions presented by the authors in the paper, which the reviewer’s found confusing or distracting (especially about optimal transport). The author response clarified several points, which prompted two reviewers to raise their scores. The authors promised to remove the discussion about optimal transport. Overall, there is support to accept the paper; the authors are encouraged to take the reviewer feedback and discussions into account to clarify the final version of the paper.

confidence: Certain
Decision: Accept