Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper discusses ascent regularization for training continuous normalizing flows (CNFs). This is motivated from Wasserstein gradient flows and results in an interesting regularization that encourages the learned model to be similar to the target distribution around a large interval of time values. I find this to be an interesting alternative to optimal transport-based approach for reducing the compute cost to transport samples between a base distribution and a target distribution. Experiments include maximum likelihood on tabular data and as a variational inference model within a VAE.
strength_and_weaknesses: Strengths:
  - The proposed approach is interesting, as a way to mimic Wasserstein gradient flow paths. Using the time evolution of the score function is interesting as well.
  - The algorithm works well at regularizing the path and seems to help accelerate convergence towards the target distribution. This is shown across both MLE and VI settings, as well as in a toy setting where it is shown to improve upon MCMC in terms of the number of function evaluations.

Weaknesses:
  - I found the writing poor and can be significantly improved. Specifically, there seems to be multiple concepts being introduced \tilde{p}_t (Eq 4) and V (Eq 7) that do not appear in the experiments. See clarity section below for more comments. If I am mistaken, please correct me.
clarity,_quality,_novelty_and_reproducibility: Novelty:
  - Combining a score function estimator with the Wasserstein gradient flow for regularizing CNFs seems novel and works well in practice.

Quality:
  - ACNF compares favorably to CNF and RNODE (an OT-based regularization for CNFs) on a variety of settings. 
  - Being able to get close to the T=1 distribution with a shorter time interval is interesting. 

Clarity:
  - Eq (4), the "estimated log-likelihood" warrants more clarification. From what I understand, it is an approximation to p_0(x) where instead of integrating all the way from 0 to T, this integrates only up to t and directly assumes the base distribution at time t. So it is equal to p_0(x) when t=T, but otherwise is not really related to p_t. The sentence before Eq (4) makes it seem like \tilde{p}_t is approximating p_t, but this isn't the case right? 
  - It also isn't clear why \tilde{p}_t is being introduced. The sentence before Eq (4) states this is to avoid extra integration steps, but both Algorithms 1 and 2 seem to integrate all the way from 0 to T regardless. And the \tilde{p}_T in the training objective can be replaced by p_0. Can the authors clarify the importance of introducing \tilde{p}_t ?
  - Eq (7) seems out of place. I did not understand how this equation contributes to the paper but this takes up half of the section on ACNF. It might be better to have a dedicated Related Works section to discuss connections to other works. On the contrary, Eq (8) is interesting and should be emphasized more strongly I feel.
  - How are the NFEs computed? Is the ODE being solved with an adaptive solver?
  - Why is the model regularized for t > T? I did not understand why this is the case, since the model does not regularize beyond [0, T]. Also, is regularizing t > T useful for anything?
summary_of_the_review: I think this paper is interesting and has potential. However, I feel hesitant to recommend acceptance in its current form, particularly since multiple concepts and equations are introduced but I did not understand how these connect with the final training algorithm. My understanding is that the main contribution seems to be using the time evolution of the score function for regularizing the ODE to model the steepest descent direction on the KL, but the connection to \tilde{p}_t is unclear to me. If the authors can clarify this, I would be willing to change my rating.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a way to regularize continuous normalizing flows by regularizing them to have monotonically decreasing KL divergence wrt to the base distribution along the integration path. In doing so the paper builds on previous work of NODE, RNODE and CNFs and adds to the collection of papers around stabilizing continuous normalizing flows. Through a bunch of experiments on tabular datasets and some experiments on MNIST images the paper shows that their method can indeed lead to flows that converge faster (less NFEs) and are more stable.

strength_and_weaknesses: Strengths:

- The authors have formulated a novel way of regularizing continuous flows that not only makes the flow stable but also improves efficiency (in terms of NFEs) and final performance (in terms of NLL)
- The authors have backed their regularization functional with ample explanations and theoretical justification and followed it up with experiments that reflect the contributions they claim.

Weaknesses:
- While not specific to this work but continuous flows papers in general should discuss an important limitation: scaling to high-dimensional datasets like ImageNet or FFHQ. The authors show some results on MNIST but I think the stronger case should be made on high-dimensional images. If not, commenting on a possible roadmap to get there might also be useful.
- I am unsure how the computational cost compares wrt to previous work. From Algorithms 1 and 2 it seems like integrating augmented states for z_t and then for log(p_t(z_t)) should incur a higher computational cost. It would be better if the authors could discuss the overall compute cost as well and not just NFE.
- Another thing to discuss is more theoretical: why should there exist a map from the data distribution to a base distribution that has the ascent continuous property? The topology of the support of the data distribution may not be homeomorphic to a ball in high-dimensional space (in case of Gaussian) ? Moreover even if it is homeomorphic, it is not obvious to me why should an ascent-continuous map always exist.

clarity,_quality,_novelty_and_reproducibility: - The paper is written well and the ideas seem novel.
- The authors share some code but no unit tests on how to run yet. Hopefully, this will be fixed in the final submission to allow for reproducibility.

summary_of_the_review: I think the paper makes a good contribution to the bijective normalizing flows literature. It supports the claimed contribution with theory and appropriate empirical analysis.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors  propose a new class of continuous Normalizing flows (NFs), ascent continuous normalizing flows (ACNFs), that makes a base distribution converge faster to a target distribution.  The learned ACNFs demonstrate faster convergence towards the target distributions, therefore, achieving better density estimations, unbiased sampling and variational approximation at lower computational cost. 
strength_and_weaknesses: Pros:The paper proposed ACNFs, a new class of CNFs, that define flows with monotonic convergence toward a target distribution. The authors derive the dynamics for the steepest ACNF and propose a practical implementation to learn parameteric ACNFs via ascent regularization. The learned ACNFs illustrate three beneficial behaviors: 1) faster convergence to the target distribution with less computation; 2) self-stabilization to mitigate performance deterioration; 3) insensitivity to flow training length T . Experiments on both toy distributions and real-world datasets demonstrate the effectiveness of ascent regularization on learning ACNFs for various purposes.

Cons: The definition of \tilde p_t, at least in the main body of the paper, say in page 3, is not very clear. I would suggest the authors to make the derivation part before more clear, say those around prop. 1. 
clarity,_quality,_novelty_and_reproducibility: Most part of the paper is well-written with hight quality. The idea from this paper is too totally new, for instance similar ideas are in Stein Variational Gradient Descent method or other variational method. No problem in the reproducibility part. 
summary_of_the_review: This paper is above the average. I would suggest to accept this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a way to regularize continuous normalizing flows by ensuring they have monotonically decreasing KL divergence wrt to some base distribution along the integration path. The paper empirically shows the proposed method leads to flows that converge faster and are more stable on tabular datasets and MNIST images 

Overall the reviewers and the AC agree that the paper would be interesting to the community.
justification_for_why_not_higher_score: The experiments are limited to simple datasets, hence it remains unclear to what extent the advertised benefits translate to more complex data. 
justification_for_why_not_lower_score: The proposed method is of interest to the community of normalizing flows, it proposes a sound and novel method for improving training of continuous normalizing flows.
Decision: Accept: poster