Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to combine Normalizing Flows with generative Diffusion Models in such a way that the target data is first non-linearly transformed via the flow and then the distribution over latent embeddings is modeled with the diffusion model. The authors call their model *Parametrized Diffusion Model (PDM)*. Since the flow is invertible and defined by a deterministic function, it is possible to formally relate the combined flow and diffusion process to a non-linear diffusion directly in data space, leveraging Ito's Lemma. The paper validates the PDM on image modeling benchmarks (CIFAR-10 and CelebA 64x64). The experiments show that the method leads to reduced variational gaps between the data likelihood and the evidence lower bound that is used for training the model, compared to several baselines.
main_review: **Strengths**:
- Combining a Normalizing Flow with a Diffusion Model as proposed in the paper has not been done before. 
- The model allows to calculate exact data log-likelihoods using the diffusion model's probability flow ODE together with the tractable log-likelihood of the Normalizing Flow component.
- The deterministic transformation defined by the Normalizing Flow component allows to formally relate the combined flow and diffusion model to a non-linear diffusion directly in data space via Ito's Lemma.
- It is interesting that the method results in small variational gaps between log-likelihoods and evidence lower bounds used for training.

**Weaknesses**:
- It is mathematically elegant that we can formally relate the method to a non-linear diffusion in data space via Ito's Lemma. But what is the practical value of that? Both training and sampling still decompose the model into its separate flow and diffusion components and do not leverage this connection. Also, the experimental results suggest that the model only reduces variational gaps, but does not improve the actual likelihood itself (for example, *DDPM++ (IS)* vs. *PDM (NLL, dec)* in Table 3, also see points below).
- The method is very incremental compared to LSGM [1]. It is essentially equivalent with the only difference being that LSGM uses a VAE framework with separate, non-invertible encoder and decoder, while PDM uses an invertible neural network, so encoder and decoder are simply the inverse of each other. However, this has multiple downsides: (a) Invertible neural networks have significantly lower expressivity than free neural networks due to the invertibility constraint. Hence, PDM has less modeling power than LSGM. (b) Using free neural networks for encoder and decoder, LSGM allows to use smaller or larger latent spaces, which can affect sampling speed, and it also allows to tailor the encoder and decoder better to different data types. No such things are possible with Normalizing Flows. Compared to LSGM, the only advantage of the invertible and deterministic encoder/decoder networks (deterministic in contrast to VAEs, where encoder and decoder formally define distributions) is that we can use Ito's Lemma to define a non-linear diffusion in data space. However, as mentioned, this advantage is not practically leveraged in any way, it seems, and the model is still trained separating the regular flow and the regular linear latent-space diffusion.
- Related to that, I think the statement "LSGM cannot provide the diffused data learning at the middle of the process because LSGM does not induce the diffusion process on the data space due to the lack of VAE’s invertibility" is a bit misleading. While it is true that the encoder/decoder are not formally exact inverses of each other, in practice they do almost perfectly invert each other as indicated by the high reconstruction quality of modern VAEs and behave almost like deterministic functions. Therefore, also a model like LSGM can easily take the latent variable from anywhere along the diffusion process and map it back to data space with the decoder, just like PDM. In fact, such an experiment is demonstrated in Figure 15 in [1] (therefore, also the strict comparison between the pipelines in Figure 2 and 3 is a bit questionable).
- I think the paper misses some important baselines on the CIFAR-10 experiments: Recently, [2] achieved state-of-the-art log-likelihoods using diffusion models. This work should be cited and would also be an appropriate baseline. Furthermore, when comparing to [1], the authors choose only one setup that was tailored to very high FID at the cost of likelihood. However, PDM not only focuses on FID, but primarily at NLL, lower bounds, and smaller variational gaps. Hence, it would be more appropriate to also compare to *LSGM (NLL)* and *LSGM (balanced)* (Table 2 in [1]).
- The experimental results are not overly strong. Considering these further baselines, the strongest competitive diffusion-based works [1,2,3] all achieve likelihoods or lower bounds around 2.90 or lower, while PDM's best value is 3.11. Also the FID is significantly higher than recent works in the literature. I don't think that only demonstrating small variational gaps is a significant result, as long as the actual performance is not competitive. Note that I am giving more weight to the CIFAR-10 experiments, since this is a widely used benchmark within the relevant literature (i.e. diffusion models), while CelebA 64x64 is not that widely used. 

**Questions**:
- The NLL for *PDM (FID, dec)* is actually better than the NLL of *PDM (NLL, dec)* in Table 3. Why is that? Wouldn't we expect to achieve better likelihoods when optimizing with the "likelihood" weighting $\lambda(t)=g^2(t)$?
- Why does *PDM (NLL, dec)* achieve so much worse NLL/NELBO than *DDPM++ (IS)*? My understanding is that PDM uses a similar DDPM++ backbone, which is trained first (after this training stage, it should have similar NLL/NELBO as the pure diffusion *DDPM++ (IS)* baseline). Now, we additionally train the flow. Shouldn't it only further improve the results?
- Based on Figure 7 and as discussed in the text, it seems the flow component's main job is to adjust colour saturation of the synthesized images. I am wondering whether this is related to the training strategy: In the appendix, it is mentioned that first the diffusion component is trained and only then the flow is added. Hence, the flow will afterwards only improve or "clean up" whatever the diffusion model missed and this seems to be related to colour saturation. What if we trained the other way around and first trained the flow to transform the data distribution into a Normal distribution (standard flow training), and then trained the diffusion model on the actually achieved embedding distribution? We may see different results here. Also, what would happen if we do not do stage-wise training but directly trained all components end-to-end simultaneously from scratch?
- I am not sure I am fully understanding the value of Theorem 2: If we assume that our score model, i.e. diffusion model, is flexible enough to cover any path measure, then why do we require a flow component in the first place? With this assumption, the diffusion model should be able to model the data distribution itself (diffused for all $t$) perfectly without requiring an additional flow component, I would think?

**Minor Comments**:
- Equation 4 seems to be the negative ELBO, not ELBO.
- In Section 3.2, the paper discusses the variational gap and refers to previous works on diffusion models. Previous work showed that the generative distribution defined by the probability flow ODE and the stochastic SDE formulation are not identical, unless the model has learnt the ground truth score perfectly, which is not the case in practice (see [3]). Note that the base for the KL derivations is the SDE formulation, while the NLL calculation in practice relies on the probability flow ODE formulation. I think we are essentially assuming here that ODE and SDE-based models are similar. I believe this subtlety should be highlighted.
- Page 8, last sentence: Did the authors mean "destabilize", rather than "stabilize"?
- Section 9: "prodive" -> "provide".

[1] Vahdat et al., "Score-based Generative Modeling in Latent Space", 2021.

[2] Kingma et al., "Variational Diffusion Models", 2021.

[3] Song et al., "Maximum likelihood training of score-based diffusion models", 2021.
summary_of_the_review: It is interesting that the flow transformation allows for a formal definition of a non-linear diffusion in data space via Ito's Lemma, but it seems this cannot be leveraged for improved modeling power, as directly training this non-linear diffusion is not possible, unfortunately. Hence, the paper resorts to training a regular flow together with a regular "linear" diffusion, separating the two components. With this in mind, I think methodologically the paper is fairly incremental compared to LSGM [1]. It is basically the same with the encoder/decoder replaced with invertible neural networks and the invertibility does not provide any practical advantages. The paper's experimental results are not impressive and I think some relevant baselines are missing. Therefore, in conclusion I recommend rejection.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes parameterized diffusion model (PDM) that combines a normalizing flow on top of a linear diffusion model, effectively modeling a nonlinear diffusion. Empirically, such nonlinearity results in a tighter variational gap compared to the linear diffusion and other baselines.
main_review: The idea of injecting nonlinearity to a linear diffusion using normalizing flow is very interesting, although somewhat similar to [Song et al. 2021]. I find the experiments section quite convincing. In particular, the variational gaps obtained using the proposed method are a lot lower than the alternatives.

I find the exposition of the paper very hard to follow. Many sentences do not make sense to me, and notations are used without being defined properly.

Detailed comments:
- In Eq (3), some reference about how $\mathcal{L}$ appears would be helpful. Also, mention again how $\mathcal{L}$ is defined in Sec 3.2 where it is used. Throughout, the role of $\mathcal{L}$ is really mysterious to me.
- Above Eq (6), "To connect this VAE ELBO to the diffusion loss, ..." How is (6) related to diffusion? I found the connection a bit far-fetched, if it's just regarding $q(x|z)$ and $p(z|x)$ are "forward" and "backward" directions respectively.
- In Sec 3.2, how does the last equation follow?
- In the last paragraph of Sec 3.2, I don't understand how the variational gap has to be strictly positive for diffusion models. Why couldn't $\nu_\theta$ be arbitrarily close to $\mu$ as we optimize $\theta$? How is this really different from the nonlinear case Eq(10)?
- In the last sentence in Sec 4.2, "from the prior to $z_t^\theta$" I think it should be $z_0^\theta$.
- Before Theorem 2, it claims "this variational gap converges to zero as we train ...". This does not seem correct. The theorem mentions nothing about convergence, but only that at optimal the gap becomes zero. The actual convergence is hard to guarantee due to the non-convexity of the optimization.



summary_of_the_review: The paper proposes an interesting way to model nonlinear diffusion using normalizing flows, and presents convincing experiments results to support the main claims, although the clarity of the writing can be improved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The core contribution of the paper is to show how a nonlinear diffusion model may be constructed by cascading a normalizing flow with a linear diffusion in the resulting latent domain. The paper comes with a strong theoretical flavour and the proposed approach is well grounded.
Applications in classic image generation tasks are provided.
main_review: This paper is set up in the emerging toping of generative diffusion models, that is gaining some important momentum. Such models define a forward and a backward pass, that respectively consist in going from the signal domain to noise and back from noise to the signal domain.
Applications of the backward diffusion as a generative model is a very effective and popular way to achieve generative modeling.

As highlighted by the authors, current applications of such diffusion models come with linear dynamics, since those were the only ones so far to allow for close-form expressions for the required densities as a function of the time step. Their point is that such linear dynamics are limiting the expressive power of such models.

As a convenient way to circumvent this limitation, their solution is fundamentally quite simple: they propose to jointly train a normalizing flow from the data space, from which a classical linear diffusion is performed. The key point is that the method actually turns out to be equivalent to a nonlinear diffusion that would take place in the data domain, and they explicitly derive its dynamics by exploiting the Ito lemma and the fact that normalizing flows are invertible. The paper comes with much theoretical derivations that should be sufficient to support their claims.

As can be seen, I tend to think that this paper does have merits and that the study definitely may be inspiring to colleagues working on diffusion models. Although the experiments are limited, they are sufficiently convincing to me.

However, I must also say that all these remarkable developments come with an unacceptably low quality of english usage. A very important number of sentence from this paper comes with typos or with awkward usage of the wrong english words. 
* The most annoying systematic typo is the systematically wrong usage of the word "the". That word is very often written at incorrect places, which reads very awkward. I am  talking of something like a hundred of occurrences.
* Then, there are other more punctual typos or awkward sentences, that I shortly review later below.

For the reason that this english usage is not acceptable I am sorry to say that I must recommend rejection. I however strongly encourage the authors to have the paper (heavily) proofread by a native english speaker and then submit it again somewhere else.  

Comments on the go:

* "by maintaining the static linear diffusion": have not innovated on the topic of the diffusion mechanism that was always kept linear.
* You don't define your acronyms VE VP, SDE, MLE, etc
* Practically : in practice
* In equation 3, you should explain what P_{0t}(xt|x0) is and how you compute it
* more close to: closer to
* To be concrete: in practice
* All the suggested SDEs: As far as we know, all previously proposed
*  worth to note: worth noting
* in equation (7), how do you write and compute P_{0t}(z_t|z_0)
* sentences before 4.4 awkward.
* your ethics statements are largely exagerated. It reads like you are proposing a super dangerous technology, whereas you are basically proposing some variation over a quite established technology. Please strongly tune down. ("destroy the evidence-based justice system", come on...)


summary_of_the_review: very interesting paper but with a very questionable english usage
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The author proposed a framework that combines normalizing flow with diffusion models, allowing non-linear diffusions to be used for inference and generation. Specifically, the author first discussed the similarity between VAE and diffusion models to motivate the importance of using non-linear diffusion and the reducing variational gap. Then, the author proposed to decouple the non-linear diffusion into an invertible transformation with the normalizing flow and typical linear diffusions on the transformed variables. A training objective based on variational lower bound was also proposed. Further, the author also showed that the variational gap between the lower bound and log-likelihood can be reduced with the proposed non-linear diffusions. 
main_review: ## Strength
The paper is clearly written and easy to understand. The overall idea and relationship to VAE are also discussed at the very beginning, which helps the understanding of the proposed method. For theoretical correctness, I have briefly checked the derivation, which seems to be correct. However, I am still a bit confused about some of the notations, which I will elaborate on later. Empirically, the author compared the proposed method with various diffusion models and demonstrated better NLL/ smaller variational gap and stable training. 

## Weakness and potential improvement
I found the intuition behind the proposed method interesting and easy to follow, but I still have some concerns. First, if I understand correctly, the structure of PDM is very similar to LSGM mentioned in the related work. The only difference is the replacement of the encoder and decoder with normalizing flows. What are the actual empirical advantages? From the experiment, it seems that LSGM produces lower FID scores compared to PDM. Does PDM reduce the required number of generation steps like LSGM compared to typical linear diffusions?

For the evaluation metric, FID score is a good metric for overall sample quality, which captures both diversity and fidelity. However, since the PDM produces better NLL, I guess it can produce better distribution coverage than typical linear diffusions. So, if it does not trouble too much, **Precision** and **Recall** [1] are good metrics to separate the sample fidelity and diversity, which can provide more information than just FID.

From the generated images before and after normalizing flow, it seems that the normalizing flow only learns to colourize the image. Thus, most of the image details are generated by linear diffusions in latent space. My concern is that is it possible that the normalizing flow only learns trivial mappings? Why does normalizing flow not contribute to image details? 

What is $\lambda$ in Eq.3?

At the beginning of section 4.2, I don't think all non-linear diffusions can be decoupled into normalizing flow + linear diffusion. Thus,  it is only fair to say that normalizing flow + linear diffusion produce non-linear diffusion. 

What is $\sigma^2(t)$ in section 4.4?

In section 4.5 on page 7, I am still confused about why optimizing $\phi$ reduces the variational gap. I understand that the gap can be expressed as the difference in KL w.r.t. latent space path measures, but the variational gap also has the term $D_{KL}(p_\phi||p_\theta)$ which is also related to $\phi$. So why optimizing $\phi$ w.r.t. NELBO alone can reduce this gap?

[1] Tuomas Kynkaanniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models.  
 
summary_of_the_review: The paper is clearly written and I am also interested in the actual advantages of the proposed method compared to other diffusion models. If the author successfully addresses the above concerns, I can improve my rating of the paper. 

---
I have read the revised version of the paper and the author's responses. It addressed most of my concerns. However, although I understand the difference between PDM and LSGM from a theoretical point of view, I think demonstrating it empirically is also important because the inter-changeability between latent space and data space during inference is mainly due to the merit of normalizing flow, not the PDM framework. Since concatenating the flow and diffusion model is straightforward, the resulting model (from a structure point of view) has limited novelty. On the other hand, I think the argument of reducing the variational gap is interesting, and along with extensive empirical evaluations, I will raise my score to 6. But I still suggest the author to further argue about the novelty of PDM from both theoretical and empirical points of view.
review_title: null

Meta-review: This paper presents a simple approach called PDM for composing non-linear and complex normalizing flows with score-based generative models. Since score-based models can be considered as a special form of continuous-time normalizing flows, PDM corresponds to a composition of different classes of normalizing flows. 

Pros: 
* Combining generic normalizing flows with score-based models is an interesting direction as they have different characteristics and can be complementary to each other.
* Using Ito's lemma to show that the model learns a non-linear SDE in data space is valuable. 
* The authors show that the variational gap can be reduced using normalizing flows.

Cons: 
* The proposed method does not exhibit a clear advantage compared to the diffusion baseline without the normalizing flow component. On the CIFAR10 dataset, the best NLL and FID results are obtained by the diffusion baseline.

* Theorem 2 makes a very unrealistic assumption that a flow network is flexible enough to transform $p_r$ to any arbitrary distribution. If this holds, we wouldn't need the score-based generation model anymore. We could simply train the normalizing flow to map the input data distribution to a Normal distribution.

* This submission chooses to discuss differences with the recent LSGM framework. However, in doing so, several inaccurate claims are made. The lack of inference data diffusion in LSGM is mentioned as one of its drawbacks. However, it is not clear what is the value of having such a mechanism and what implications it may have on the expressivity of the model. Note that mapping from data space to latent space in VAEs can be considered as a stochastic inversion rather than an exact inversion. Ito's lemma does not require invertibility and it can be easily applied to the forward and generative diffusion in LSGM. The authors argue that applying it to the forward diffusion in LSGM will result in $\hat{p_{r}}\ne p_{r}$. But, $\hat{p_{r}}$ would be only considered for visualization of the forward diffusion and it is not used for training or any other purposes. LSGM, the proposed PDM, and score-based models are all trained with a reweighting of ELBO (see [here](https://arxiv.org/abs/2106.02808)). It is not clear if the drawback mentioned above has an impact on the training or expressivity of the model.

* The presentation in the paper requires improvement. The motivation on why invertibility plays a key role is not clear beyond generating the visualization in Figure 2. 

In summary, the paper proposes an interesting idea and explores directions very relevant to the current focus in generative learning. However, given the concerns above, we don't believe that the paper in its current form is ready for presentation at ICLR.
Decision: Reject