Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes to train continuous normalizing flows by specifying a continuum of distributions using an interpolation function. The neural network model is trained to estimate the expected velocity of the interpolation function though MSE minimization. This training approach does not require numerical simulation of ODEs, and is therefore much more efficient compared to conventional maximum likelihood approaches. The method can be viewed as a generalization of score-based diffusion models to non-Gaussian noise. Experiments on toy datasets and tabular datasets confirm that the resulting continuous normalizing flows obtain competitive performance.
strength_and_weaknesses: ## Strength

1. The proposed formulation is very clear and easy to understand. The proof is easy to follow.
2. The results of Wasserstein bounds offer valuable insights.

## Weaknesses

I have major concerns regarding novelty and the discussion of relevant work in this paper:

1. The motivation of training continuous normalizing flows without requiring numerical ODE simulation is not new. The same challenge has been tackled in many previous works, such as [1][2][3]. In particular, the closest work to this paper is perhaps ScoreFlow [3], which leverages score-based diffusion models to design a simulation-free approach for maximum likelihood training of continuous normalizing flows. This paper needs to include a comprehensive discussion on what sets this work apart from existing ones. For example, is there anything that was impossible before now becomes tangible with the new method? Does the performance improve in comparison to other simulation-free approaches? Additional comparisons with prior methods, such as ScoreFlow in [3], need to be included in the experimental section (Figure 2, Table 1,2 and Figure 4).

2. The proposed stochastic interpolant method is not entirely new. It can actually be considered as a special case of the result in [4], where SDEs are replaced by ODEs (by setting the diffusion coefficient to zero). Specifically, the interpolation function $I_t(x_0, x_1)$ in this work defines an ODE connecting $x_0$ and $x_1$, with the form $x'_t = \partial_t I_t(x_0, x_1)$. By averaging this ODE over the joint distribution of $(x_0, x_1)$, we can obtain another ODE that smoothly converts $\rho_0$ to $\rho_1$. This ODE can be derived with the techniques in [4], given by $x'_t = E[\partial_t I_t(x_0, x_1) | I_t(x_0, x_1)]$, which is equivalent to the velocity field $v_t( I_t(x_0, x_t))$, the target to be learned in this work.

3. The proposed stochastic interpolant approach was also implied by the work [5] (see Appendix D). This paper needs to clarify the connection.

## References

[1] Rozen, Noam, et al. "Moser flow: Divergence-based generative modeling on manifolds." Advances in Neural Information Processing Systems 34 (2021): 17669-17680.

[2] Ben-Hamu, Heli, et al. "Matching normalizing flows and probability paths on manifolds." arXiv preprint arXiv:2207.04711 (2022).

[3] Song, Yang, et al. "Maximum likelihood training of score-based diffusion models." Advances in Neural Information Processing Systems 34 (2021): 1415-1428.

[4] Peluchetti, Stefano. "Non-Denoising Forward-Time Diffusions." (2021).

[5] Salimans, Tim, and Jonathan Ho. "Progressive distillation for fast sampling of diffusion models." arXiv preprint arXiv:2202.00512 (2022).

clarity,_quality,_novelty_and_reproducibility: * Clear writing. Easy to follow.
* Novely is limited, as detailed above.
* Experiments can be improved. For 2D density estimation, need to have additional baselines (such as ScoreFlow and FFJORD). For tabular data and computational efficiency, need to compare wit ScoreFlow and other works mentioned above. Current experiments cannot show clear advantage over existing simulation-free approaches.
summary_of_the_review: Overall nice discussion of a clever idea for training continuous normalizing flows (generative neural ODEs) without ODE simulation. However, the method is not entirely novel since it is a special case of an existing paper. Experiments are not comprehensive and more comparisons are needed to understand the trade-offs with existing simulation-free approaches.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a continuous-time normalizing flow which does not need (i) to propagate gradients through ODE dynamics for training or (ii) to formulate the flow using a diffusion. The method is tested empirically on 2D data and standard tabular data benchmarks. 
strength_and_weaknesses: **Strengths**
The fundamental idea proposed in the paper is excellent. Diffusion models showed how to design normalizing flows which bridge the data and base distribution by definition, and how to train them efficiently without taking gradients through ODE or SDE dynamics. This paper takes that idea one step further and shows that such a flow can be designed without resorting to a diffusion process, and that training amounts to minimizing a simple quadratic loss analogously to how denoising score-matching facilitates score-matching without access to the ground-truth score. Moreover, the paper relates the quadratic objective to the Wasserstein-2 distance between the target distribution and the distribution defined by transporting the base under the model, and the authors also demonstrate that the proposed flows can bridge between any two distributions from which we have samples, not just a target distribution and tractable base distribution such as a Gaussian. These are all strong contributions. 

I'd also like to highlight a key point made by the paper, namely 'By specifying an interpolant density, the method therefore separates the tasks of minimizing the objective from discovering a path between the base and target densities. This is in contrast with conventional maximum likelihood (MLE) training of flows where one is forced to couple the choice of path in the space of measures to maximizing the objective.' I think this is a significantly underappreciated point in the modern generative modeling literature, and explains many of the issues with vanilla flows and VAEs which have been to some extent addressed by diffusion models. I am glad the paper explicitly notes the importance of this decoupling. 

**Weaknesses**
Unfortunately, the experimental evaluation is lacking. 

The 2D experiments are useful as a sanity check, but are not convincing enough to warrant their own section. Moreover, the fact that the proposed method can formulate a flow between two arbitrary but is again only tested on 2D data is unfortunate. For example, a compelling experiment which immediately jumped to mind is image upsampling. Diffusion-based image upsamplers transport a Gaussian to the high resolution image conditioned on a naively upsampled image using e.g. bilinear interpolation. In the case of the proposed method, a flow could be formulated to bridge between the naively upsampled image and the ground-truth directly. An experiment like this would be a strong demonstration of the advantage of the interpolation approach. 

In terms of the tabular data experiments, I appreciate that this benchmark of 5 datasets has been standard in the literature, but it was originally introduced in 2017 and is now quite outdated. I would question its usefulness beyond a sanity check for the method, similar to the toy experiments. For example, BSDS is a dataset of 8x8 black-and-white image patches whose train split contains 1 million examples. Diffusion models have been fit to datasets consisting of hundreds of millions or even billions of images at higher resolution. While it's not possible for all new methods to be tested at this scale, it's still necessary to compare on a reasonable-scale problem, ImageNet. Alternatively, design a new benchmark for tabular data. I appreciate this is a lot to ask, but at some point benchmarks become outdated and are no longer useful to track progress in modeling. 

Finally, and related to the previous points, the comparison to FFJORD is worthwhile as a baseline, but surely the primary mode of comparison should be flows trained in the diffusion model setting? These are much more competitive, and can even be trained by maximum likelihood without the need to backprop through ODE dynamics. This would provide a much stronger baseline, and a more convincing argument that the proposed method provides an advantage. 


clarity,_quality,_novelty_and_reproducibility: **Clarity**
The paper is clear and well written. The method is cleanly motivated and described. 

**Originality**
The method as proposed seems novel to me, and I'm not aware of previous work. In fact, the approach neatly takes lessons from efficient training of flows through the diffusion model formulation and circles back to make possible the training of flows directly and efficiently without the notion of a diffusion process. The connection to the Wasserstein-2 distance and the ability to build a flow between two arbitrary distribution for which we have samples is also novel. 
summary_of_the_review: I'm torn between the compelling idea presented in the paper, and the lacking empirical evaluation. With more thorough experimental evaluation, this would be an excellent submission. As it stands, I would still lean towards acceptance because I think the idea deserves to be communicated.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, we authors propose a new way to interpolate between two probability distributions. This is done by constructing a stochastic interpolant between two points $x_0$ and $x_T$ distributed w.r.t. $\pi_0$ and $\pi_T$ respectively. It turns out that the amortized dynamics obtained by integration w.r.t. to the product measure $\pi_0 \pi_T$ satisfies a continuity equation and can also be related to an Ordinary Differential Equation (ODE). The drift of this ODE can be learned in a manner akin to the Implicit Score Matching (ISM) in diffusion models. The authors elaborate on the links with diffusion models by showing that for a specific choice of interpolants the estimated velocity is related to the score of a diffusion model. They evaluate the performance of their approach on several toy and tabular datasets.
strength_and_weaknesses: STRENGHTS:
* I think this is a nice idea that showcases how the manipulation of (Fokker-Planck) PDEs can yield new algorithms. I thought the paper was well written with the main contribution clearly presented. It extends denoising diffusion models in a new direction by focusing on the issue of the speed in these models.
* The theoretical results regarding the control in Wasserstein 2 are new. 

WEAKNESSES:
* The experiments are quite weak. Considering that the presented approach extends diffusion models I would have expected the authors to compare their approach to diffusion models at some point. This impression is reinforced by the fact that "The choice of interpolant for experimentation was universally selected to be that of (5)" (where (5) highlights the link with diffusion models). It is not clear to me that the current approach brings any benefit compared to diffusion models. I think that the authors miss the opportunity to compare and evaluate different interpolants. Indeed, they introduce a whole new framework but end up only using one interpolant, which is underwhelming. 

* I find a bit misleading that bold numbers correspond to only the best methods among continuous normalizing flows (CNF) in Table 1, since we can find that other normalizing flows yield better results for each considered task. I have read that the authors "primary point of comparison is to other continuous time models, so we sequester them in benchmarking" but no explanation is given as to why CNFs perform worse than NFs for tabular data and what could be done to remedy this situation. Also, confidence intervals are missing in Table 1. 

* The authors do not discuss some related literature. In particular, one work that seems to be particularly relevant is [1] which considers a similar approach to the one of the author. Indeed in [1] the author considers a stochastic interpolant between $x_0$ and $x_T$ given by a bridge (in this paper a Stochastic Differential Equation (SDE)). Then the obtained diffusion is amortized with respect to some coupling between $\pi_0$ and $\pi_T$. The author of [1] shows that the obtained dynamics can also be described by an SDE whose drift can be learned. Finally, I think that the authors miss to discuss extension of denoising diffusion models which also consider finite-time dynamics like [2,3,4]. Is there any link between the proposed approach and control and optimal transport?

* Regarding the result in Wasserstein 2. Am I correct assuming that the exponential bound should depend on the finite time $T$? The current bound is correct because the authors restrict themselves to the $[0,1]$ interval but any rescaling of the time would harm the bound. This exponential dependency comes from the use of a Gronwall lemma and should be discussed.

OTHER COMMENTS/QUESTIONS:

* What limits the authors to consider an amortization w.r.t. to the independent coupling in Equation (6)? One could imagine other couplings and I think that the obtained results would still be true. Am I correct assuming that? In that case, a link with diffusion models can be drawn in a more explicit fashion by considering the coupling given by $X_0 \sim \pi_0$ and $X_T = e^-T X_0 + (1- e^{-2T})^{1/2} Z$ with $Z$ a Gaussian random variable. For $T= +\infty$ we recover the independent coupling but diffusion models are always run up to a finite time $T$.

* The current work is limited to ODEs. From a generative modeling point of view it has been shown [3] that ODEs and deterministic methods in general yield worse results than their SDEs counterpart. Can the theory developped by the authors be extended to the SDE setting? I am guessing that in that case the authors would recover derivations similar to [1]. 

[1] Peluchetti -- Non-Denoising Forward-Time Diffusions

[2] Vargas, Thodoroff, Lawrence, Lamacraft - Solving Schrödinger Bridges via Maximum Likelihood

[3] De Bortoli, Thornton, Heng, Doucet - Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling

[4] Chen, Liu, Theodorou - Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory

[5] Karras, Aittala, Aila, Laine - Elucidating the Design Space of Diffusion-Based Generative Models
clarity,_quality,_novelty_and_reproducibility: The writing is clear and the main ideas are clearly exposed.

As emphasized in the previous comment, the approach is novel with the caveat that one important and very related work is not discussed.

Regarding the quality of the work. I believe the methodology and theory developed in the paper to be interesting. However the experiments are a weak spot of the paper and more experimental investigation is required.

No anonymous repository is shared but the authors provide enough details about the experiments so that I believe them to be reproducible
summary_of_the_review: To conclude, I think that the approach developed by the author is interesting. 

The methodology and theory introduced in that paper are coherent and represent a nice extension of diffusion models.
However, I think that more investigation (in particular numerical) is required to make the case that this approach can be used 1) in a large-scale setting 2) outside of the diffusion models setting.

For these reasons I think that the work is currently not mature enough.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an alternative learning objective for a continuous normalizing flow. Instead of standard maximum likelihood training, which requires backpropagating through the ODE solver, authors define a continuous sequence of intermediate densities between $p_0$ and $p_1$, and train the dynamics of the continuous flow to match this sequence. The intermediate densities are defined using a differentiable interpolation function. The dynamics of the flow are trained to minimize an objective function that only requires samples from the base density, samples from the target density, and samples of time $t$. Authors show that this objective has a unique minimum, which is achieved when the _true_ dynamics are recovered. In addition, authors derive a bound on the Wasserstein distance between the learned density and the true target density, as well as link the proposed model to score-based diffusion models. Numerical experiments are undertaken which demonstrate the effectiveness of the method on real tabular data.
strength_and_weaknesses: Strengths:
- An idea that echoes prior work (pre-defining a sequence of densities and fitting it), yet is sufficiently novel and required a non-trivial amount of mathematical reasoning.
- Further theoretical study of the method, deriving a bound on the density fit, as well as showing a connection to the important class of generative models (diffusion).
- Experimental results: clear improvement on competing continuous flows on tabular data. Sensible results on synthetic datasets.
- High-quality, polished write-up with crisp notation.
- An in-depth discussion and analysis of the computation efficiency of the method.

Weaknesses/questions:
- Motivation for the interpolation function chosen in the paper: why have authors not started with a simpler, e.g. linear, interpolation function? Are there any preliminary experiments with alternative interpolation functions to discuss, and especially whether the method is sensitive to this choice?
- Diffusion-based models are known for their excellent sample quality. Given the shown connection between two classes of models, have authors tried fitting their model to higher-dimensional image datasets, and evaluating the samples?
- Fundamentally, the method aims to pre-define a continuous sequence of densities, and recover the dynamics from this sequence. Have authors considered alternative ways of constructing this sequence? Does the section 2.2 suggest that we could use a diffusion process instead?
clarity,_quality,_novelty_and_reproducibility: All four aspects are excellent: clear, high-quality write-up, a novel idea, and plenty of experimental details throughout the main text.
summary_of_the_review: I strongly recommend accepting the paper. The proposed method draws inspiration from a set of methods in the field, yet is sufficiently novel, and is analyzed in-depth, both theoretically and empirically (albeit the empirical part could be even stronger, see suggestions above). Experimental results are strong, and the manuscript itself is excellent.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents an alternative learning objective for a continuous normalizing flow between any pair of base and target distributions. The core idea is to train a neural network to estimate the expected velocity of the interpolation function with an MSE loss, thereby bypassing the costly numerical simulation of ODEs used in maximum likelihood training approaches. 
 
Strength: 
- Multiple reviewers praised the well-motivated method and clear polished writing. 
- The results of Wasserstein bounds offer valuable insights.
 
Weakness: 
- As Reviewer xdhs pointed out, the challenges of simulation-free training of continuous normalizing have been tackled in several prior works. The discussions of the proposed method in the context of these previous method needs improvement.
- Comparison with simulation-free approaches is missing.

The authors provided a detailed response to these concerns. Unfortunately, the reviewers do not engage with the discussions. The AC took a close look at the responses and believes that many of the concerns have been addressed or at least alleviated (e.g., discussions with respect to relevant prior work for clarifying the contribution of this work, additional comparison results). The AC thinks the paper has sufficient merits and recommends to accept. 

justification_for_why_not_higher_score: Most reviewers find the approach of this paper interesting. The additional results from the authors demonstrate the results can scale. However, as multiple reviewers pointed out that the approach is not intrinsically novel (e.g., multiple existing methods also presented simulation-free approaches), the AC believes this justifies a poster presentation. 
justification_for_why_not_lower_score: Including the new results and added discussions, the AC thinks the paper has become stronger and worthy of publication.
Decision: Accept: poster