Learning Continuous Normalizing Flows For Faster Convergence To Target Distribution via Ascent Regularizations
Authors: Shuangshuang Chen
Sihao Ding
Yiannis Karayiannidis
Mårten Björkman
Venue: ICLR 2023 poster (2023)
Keywords: normalizing flows
gradient flows
density estimation
unbiased sampling
variational inference

Abstract: Normalizing flows (NFs) have been shown to be advantageous in modeling complex distributions and improving sampling efficiency for unbiased sampling. In this work, we propose a new class of continuous NFs, ascent continuous normalizing flows (ACNFs), that makes a base distribution  converge faster to a target distribution. As solving such a flow is non-trivial and barely possible, we propose a practical implementation to learn flexibly parametric ACNFs via ascent regularization and apply it in two learning cases: maximum likelihood learning for density estimation and minimizing reverse KL divergence for unbiased sampling and variational inference. The learned ACNFs demonstrate faster convergence towards the target distributions, therefore, achieving better density estimations, unbiased sampling and variational approximation at lower computational costs. Furthermore, the flows show to stabilize themselves to mitigate performance deterioration and are less sensitive to the choice of training flow length $T$.


Published as a conference paper at ICLR 2023
LEARNING CONTINUOUS NORMALIZING FLOWS FOR FASTER CONVERGENCE To TARGET DISTRIBUTION VIA ASCENT REGULARIZATIONS
Shuangshuang Chen Sihao Ding
Yiannis Karayiannidis Márten Björkman' 1
1 School of Electrical Engineering and Computer Science, Royal Institute of Technology
2 Al Lab, Volvo Car Corporation
3 Department of Automatic Control, Lund University
{ (shuche, celle) @ @kth.se, sihao.ding @ volvocars, com, yiannis @ @control.Ithsse
ABSTRACT
Normalizing flows (NFs) have been shown to be advantageous in modeling com- plex distributions and improving sampling efficiency for unbiased sampling. In this work, we propose new class of continuous NFs, ascent continuous normalizing flows (ACNFs), that makes a base distribution converge faster to target distri- bution. As solving such flow is non-trivial and barely possible, we propose a practical implementation to learn flexibly parametric ACNFs via ascent regulariza- tion and apply it in two learning cases: maximum likelihood learning for density estimation and minimizing reverse KL divergence for unbiased sampling and vari- ational inference. The learned ACNFs demonstrate faster convergence towards the target distributions, therefore, achieving better density estimations, unbiased sampling and variational approximation at lower computational costs. Furthermore, the flows show to stabilize themselves to mitigate performance deterioration and are less sensitive the choice of training flow length T'.
INTRODUCTION
Normalizing flows (NFs) provide a flexible way to define an expressive but tractable distribution which only requires a base distribution and a chain of bijective transformations (Papamakarios
202T Neural ODE (Chen et al.) 2018) extends discrete normalizing flows (Dinh et al.
Paaamakarios et al.) 2017 7 Ho et al.] 2019) to new continuous-time analogue by defining the transformation via differential equation, substantially expanding model flexibility in comparison to the discrete alternatives. (Grathwohl et al., 2018 Chen and Duvenaud, 2019) propose a computationally cheaper way to estimate the trace of Jacobian to accelerate training, while other methods focus increasing flow expressiveness by e.g. augmenting with additional states (Dupont et al. 2019 Massaroli et al.) 2020), or adding stochastic layers between discrete NFs to alleviate the topological constraint (Wu et al. (020). Recent diffusion models like (Hodgkinson et al., 2020: Ho et al., 2020 Song et al. 2020 ZZhang and Chen 2021) extend the scope of continuous normalizing flows (CNFs) with stochastic differential equations (SDEs). Although these diffusion models significantly improve the quality of the generated images, the introduced diffusion comes with costs: some models no longer allow for tractable density estimation; or the practical implementations of these models rely on long chain of discretizations, thus needing relatively more computations than tractable CNF methods, which can be critical for some use cases such as online inference.
2014 2016
(Finlay et al. 2020: Onken et al. 2021; Yang and Karniadakis 2020) introduce several regularizations to learn simpler dynamics using optimal transport theory, which decrease the number of discretization steps in integration and thus reduce training time. (Kelly et al., 2020) extends the L2 transport cost to regularize any arbitrary order of dynamics. Although these regularizations are beneficial for decreasing the computational costs of simulating flows, they do not improve the slow convergence of density to the target distributions like trained vanilla CNF models shown in Figure I To accelerate the flow convergence, STEER (Ghosh et al.) 2020) and TO-FLOW (Du et al.) 2022) propose to optimize flow length T in two different approaches: STEER randomly samples the length during training while TO-FLOW establishes subproblem for T during training. To understand the effectiveness of

---

Published as a conference paper at ICLR 2023
CNF
4T { M M
0.5T
2T
ACNF
Figure I: Distribution transformations of two learned flows for Gaussian mixture from a Gaussian distribution at E [0, 4T Although the two flows reach similar densities at T, the density of ACNF converges faster to the target distribution before T and diverges slower after than that of CNF Color indicates the density of true Gaussian mixture.
Estimated log-likellhodds data samples log P+(x)
(x)
ONF
different confgguattins Optiall Ascent CNI
Ti72
73
72
74
The log-likelihood estimates of trained vanilla CNF models with various flow length T, and the steepest ACNF with dynamics defined in (6) at different 2-moon distribution. All vanilla CNF models reach their maximum around Tn and deteriorate rapidly afterwards while the log-likelihood estimate of ACNF elevates rapidly at initial and increases monotonically.
Figure 2:
these methods, we train multiple Neural ODE models with different flow length T, for 2-moon distribution and examine these flows by the estimated log- likelihoodds in Figure Although sampling or optimizing T dynamically performs a model selection during training and leads models to reach higher estimates at shorter flows, it cannot prevent the divergence after Tn. Furthermore, shorter flows are more limited in expressiveness for higher maximum likelihoods and sensitive to flow length.
In this work, we present new family of CNFs, ascent continuous normalizing flows (ACNFs), to address the aforementioned problems. ACNF concerns a flow that transforms base distribution monotonically to target distribution, and the dynamics imposed to follow the steepest ACNF However, solving such steepest flow is non-trivial and barely possible. We propose practical implementation to learn parametric ACNFs via ascent regularization. Learned ACNFs exhibit three main beneficial behaviors: 1) faster convergence target distribution with less computation; 2) self-stabilization to mitigate flow deterioration; and 3) insensitivity to flow training length T. We demonstrate these behaviors in three cases: modeling data distributions; learning annealed samplers for unbiased sampling; and learning a tractable but more flexible variational approximation.
CONTINUOUS NORMALIZING FLOWS
Considering a time-/ transformation z(I) = (x) on the initial value x, i.e. z(0) = x, the change of variable theorem reveals the relation between the transformed distribution (z()t)) and p(x):
Pr (z(()) = det (Jz,( lp(x),
where Ja. is the Jacobian matrix 4 As 4b, normalizes x towards some base distribution, pr (z(()) is referre to as the normalized distribution at time I, starting from the data distribution p(x).
Continuous normalizing flow the infinitesimal limit of the chain of discrete flows and the infinitesi- mal transformation is specified by an ordinary differential equation (ODE):
dz(t)
(x) f(z(t),t).
dt
dt
The instantaneous change variable theorem (Chen et al. 2018 theorem 1) shows the infinitesimal changes of log pa (x(()) is:
d log (i
f()
(3)
dt
2

---

Published as a conference paper at ICLR 2023
Normalization direction
Sampling/density estimation direction
eq) x(T) (y( x(() (x()))
z(0) = x
2(T) ~ p(()) Au(-) (4-))
2
x Reverse eq.(2)
(3.
(r(x(T)) pl))
Pr(2(()
pu(x) plx)
px(-)
pr(-)
(5.
(8) oo (x) Vlogp(-) (lg))(
Vlogp(x)
Figure 3: Upper: transformations on variables and densities on normalization and sampling directions. Lower left: data samples (orange) and the grid of states (blue) transformations along normalization direction. Lower right: density estimation Pl along sampling direction.
Thus, the log- negnoomml distribution log Pa (z(()) can be obtained by integrating eq. backwards with a common approximation to the base distribution μ. i.. pr 23 çu:
log Pe (z)()) = log pr(x(T)) J." V f(z(T() T) v log (ogu(((T)) - V f(z(()),)))
where z(1) x S0 f(z(T), r)dT. The accuracy of log px(),, obtained by the right hand side, depends the approximation enror of pr to μ and the enror can vary at different z(T)). To avoid the problems analysis and investigate how flow length affects on modeled distribution, we introduce (x()), estimating density of f-length flow o,, which is shown via the change of variable theorem:
Pe (x) det (Ja, (x)) | (x)x.
(4)
As indicated by eq. (4 and Figure #: initiates at the base distribution, i.e. (x)) p(().. Combining
eq. 1 and eq. (4). estimated density Pe relates normalized distribution Pe (z(2(() as:
(x(x)
n (x))
μ(z(t))
p(x)
Pe (x))
Pe (z(()))
It shows that p: - μ. (x) p(x). When there exists flow, of which the normalized density is
equal to the base distribution, i.e. pr = μu, then the estimated likelihood becomes exact to the data distribution, i.e. (x) = p(x). Like the instantaneous change of variable theorem in eq. 33. we derive the infinitesimal change of time-t estimated log- ligliielihood:
Proposition 1 (Instantaneous Change of Log-likelihood Estimate). Let z(t) be a finite continuous random variable at time t as the solution of differential equation da(t)) f(z(t),t) with initial value z(0) = x. Assuming that Po = μt at t = 0 and is uniformly Lipschitz continuous in and t, then the change in estimated log-likelihodd log fe (x) at follows a differential equation:
d log Pr (x) V f()(t).t) V log logg(z((() -(z(t),t)
dt
Proof. See Appendix A. for detailed derivation and its relation to eq. Unlike the integral for log Pe (z(()) that relies on the approximation and requires solve the whole trajectory z(r), T [0, T", log (x can be evaluated exactly simultaneously with z(t) for any or/and different t:
log (x log ((x) + / (V f(((T), (log((z(T)) f(((TT,T)) dr.
ASCENT CONTINUOUS NORMALIZING FLOWS
By using KL divergence as distance measure of distributions, we have the following duality:
(L(p())((.(x)) const - ff /((x) log (x))x( (L(p(((())))l(((((()))
that maximum likelihood learning of (x) for data samples from p(x) is equivalent to minimizing 1) the forward KL divergence between p(x) and pú (x) as the first equality: 2) the reverse KL divergence in normalization direction as the second equality. We can measure the rates of KL divergences or the expected log-likelihood by their time derivative, and define ascent continuous normalizing flows (ACNFs) that monotonically decrease KI. divergence or increase the expected log- ligllikelioood. i.e.
at /
à p(x) log (x)dxx 2 O; or KI. Ô
S.
It
By applying total variation, we can find the dynamics for the steepest descent of reverse KL divergence or the steepest ascent of the expected likelloooo::
3

---

Published as a conference paper at ICLR 2023
Theorem 1 (Dynamics for Steepest Ascent Continuous Normalizing Flows). Let z(t) be a finite
continuous random variable and the solution of a differential equation de value z(0) x. Its probability pi(z(t)) subjects to the continuity equation Oep: V (prf) = 0.
dx(t) ((z(t).t) 1) with initial
The dynamics of the steepest flow for decreasing (L(pi(z(t))ll(aa)
is
Vpo(z(t)) V Flog V logr:(z(t)). f(((()),)== V Vlog (lz(t( Pr(z(t))
(6)
Proof. See Appendix A.2 for detailed derivation. The steepest dynamics is the difference between two gradients: V log j and V loog Pt w.r.t state ztt). There are a few important implications of eq eq. 1) the dynamics time-variant as Pr evolves along the flow of ztt): 2) the dynamics at time I only depends on the current state z((), thus no history is needed; 3) the flow is initiated at the difference between V Vlogul (xx) and V logp( p(x), gradually slows down and eventually stops when pt converges to μ.. The convergence rate of the steepest flow can also be proven as the negative Fisher divergence, (KL ()) 0t = -F -F (pellm) = Ep, V log ((z) log pr [2 therefore this optimal deterministic CNF is related to (overdamped) Langevin diffusion, Appendix A.3 for the derivation of convergence rate and detailed discussion of their relation.
This optimal flow also can be considered special instance of Wasserstein gradient flow (Ambrosio eet al., (005) with KL divergence as the energy functional. Previous works (Finlay et al. [0200] Yang and Karniadakis 2020 Onken al. 202T) apply the optimal transport theory to regularize flow dynamics in Euclidean space, while Wasserstein gradient flow or eq.(6) instead regularizes flow in probability measure space. We refer readers to (Ambrosio al., (205) for accessible introduction. In some special cases, the flow can be solved by introducing an auxiliary potential, V(z, t) = (r())/((z), which has a partial differential equation (PDE):
OV (v.z,) f) AV 1) + 2V log VV(2, 1) + V (2, VV 1),
at
with the initial condition ((z(0))0) = (iz( أixi See Appendix A.4 for its derivation.
pa(x())
p(x)
Solving this PDE for p. (z())) is non-trivial as the closed form solution is typically Ily unknown. JKO integration practically used literature (Mokrov et al., 202T Fan et al., 202T) for the solution, which approximates the dynamics of density Pt by its time discretization. However, it requires to know the initial condition while p(x) is generally unknown and needs to be modeled for data. (Tabak) and Vanden-Eijnne Eijnden 2010) proposes to approximate p(x) by the spatial discretization of samples, which hardly can be scaled up even for intermediate dimensions.
To tackle these difficulties and accelerate unregulated flows for faster convergence, we propose ascent regularization to learn parametric ACNFs, as inspired by previous works (Yang and Karniadakis 2020. Onken al. 202 Finlay et al. 2020) Kelly et al. 2020 Ghosh et al. 2020) that enforce flows with certain behaviors via regularization in training. Ascent regularization penalizes the difference between the parametric dynamics and the steepest dynamics by |[ I1.. which needs to evaluate score function V log Pt (t)). Therefore, we propose the instantaneous change of the score function:
propose
Theorem 2 (Instantaneous Change of Score Function). Let z(t) be a finite continuous random variable with probability density Pe (z(()) at time t. Let dz(l) = f(z(t), t) be a differential equation describing iatiiuouu--i-tiig transformation of z(t). Assuming that f is uniformly Lipschitz continuous in z and t, the infinitesimal change in the gradient of log densiiyyt at t is
di
dV log (z(()) - -V Vlogp((((1)) of (f) dz(()
V (V f(z(1,)))
(8)
dt
Proof. See Appendix A.5 for detailed derivation. V Vog p(z(t),, follows linear matrix differential equation, where the linear coefficient is the Jacobian and the bias term is the gradient of divergence. To be noted, an alternative proof can be found in concurrent work (Lu et al., 2022 theorem D.1).
We discuss the training of ACNFs in two different learning cases: maximum likelihood learning for data modeling and density estimation in Section 4 minimizing reverse KL divergence for learning annealed samplers for unbiased sampling in Section 5
4

---

Published as a conference paper at ICLR 2023
Algorithm 1 Maximum likelihood learning of Algorithm 2 Training ACNF as annealed sampler for unbiased sampling with ascent regularization
ACNF with ascent regularization
Require:
Data samples X = {xx j=......MM parameteric dynamics of flow fg, length of flow T, ascent regularization coefficient X, mini-batch size N, base distribution μ Initialize 0
Require:
target distribution π = -7/Z, parameteric dynamics of flow fg. length of flow T, number of samples N, ascent regularization coefficient X, base distribution μ
Initialize 0
while 0 not converged do
while 0 is not converged do
Sample mini-batch of N data x X
Sample z' z0) ~ Po
Integrate augmented states
Evaluate log uz (0)) and V Vlog
z (t), log (x') forward with initial value
Integrate augmented states
x", log (u(x)) from O to T
(t), log (t),1og
(2 (t)), V Vog Pt (z' (t)) with initial
Integrate augmented states
value (z), log μu (1()), V log u (z)) from 0 to T
1z (1), V log Ps (I)) backwards with initial
Evaluate
value (z* (T), log p(z' (T)) from T to 0 Compute loss function L in (q. and
log wxx'
log (2'(()z)) log (T))
Compute loss function L in eq. eq.(T) and VeL by adjoint sensitivity method
Vel by adjoint sensitivity method
Update 0 by gradient descent algorithm
Update 0 by gradient descent algorithm end while
end while
4 MAXIMUM LIKELIHOOD LEARNING OF ACNF FOR DENSITY ESTIMATION VIA ASCENT REGULARIZATION
For maximizing likelihood learning of pr to fit data, the total objective with ascent regularization is:
N
- log pr + + f [1
min L = Σ
- V ((og((2((()) + ((2)
N
where À is the ascent regularization coefficient to control the trade-off between maximizing likelihood and regularization on the ascent behavior of the learned dynamics. When 0, ACNF degrades to vanilla CNF. The first term in eq. (9) is obtained by integrating eq. 5 over [0, T), simultane- ously with zt),, while the ascent regularization can be integrated backwards with augmented initial [z(T",, V log pT (z(T)),, with V log (z))) & V log (()( We summarize the pseudo-code for maximum likelihood learning of ACNFs Algorithm D We show the interpretation of ascent regularization score matching in Section A.6 in Appendix, thus Algorithm I can be implemented in more efficient ways like (Lu et al.
2022
Song et al. 2021) for some cases.
5 LEARNING ACNF AS ANNEALED SAMPLER FOR UNBIASED SAMPLING
Except modeling data samples and performing density estimation, NF as a sampler shows to be more sample efficient in Annealed Importance Sampling (AIS) (Neal) 2001) when comparing to classic MCMC methods (Arbel et al. 2021). A typical AIS and its extension use a sequence of annealed targets { k= that bridges an easy-to-sample and tractable distribution Tg μ to the target T = 2020) and AFT (Arbel al.) 202T) propose to fit K discrete NFs and each NF approximates the transport map between - and πд- However, the rate of sampling convergence is dependent on the pre-defined annealed targets. Besides, a larger K annealing step needed to decrease the variance of the estimator, which comes an additional computational cost (Doucet al. 2022).
7(-)/ that is known up to the normalization constant. SNF (Wu et al.
As ACNF can also define the flow from a base distribution to a target distribution, it can learn a continuous flow of the annealed target instead of the pre-defined discrete one, and later generate samples. Different to Grrsse et al. 2013), J, the annealed target by ACNF does not require a specific form of distribution. As ACNF enforces faster convergence to the target distribution, ACNF sampler potentially generates better samples than CNF or linear annealed scheduling especially at limited steps K, thus the estimate, e.g. on logarithm of normalization constant log Z, is more accurate.
5

---

Published as a conference paper at ICLR 2023
2T
0
Figure 4: Comparison on log potential field along the flow by trained vanilla CNF and ACNF with
A = 1 and the numerical PDE solutions of eq. 7 for 2-modal Gaussian mixture at E [0, 27" Color
indicates the value field: turquoise is O and the lighter the color is the larger the value is.
2
22.
2
2
2
2
2
2
2
a
a
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2T
Figure 5: Comparison density evaluation of trained vanilla CNF and ACNFs with À 0.0001,0.0000 0.001, 0.005 on 2-moon distribution along integral E [0, 2T].
Different to maximum likelihood in Section 4 training ACNF for annealed sampling is to minimize the reverse KI. divergence, (I((prz((T) (Z)))) It can be evaluated up constant by the logarithm of importance weights of samples and log w(z' (T)) = 1z With ascent regularization like previous section, the total objective becomes:
(z' (I)) log (T)).
log w(z' wzz'(((z)))))))) (T): + /-" (V (z (1); V Vlog (()))) + f(z (0,44)
min
Σ N
(10)
where f(z(t), t; 0) is the annealed generation dynamics. Unlike the previous section, as the sampler initiates by the base distribution, log (2 (1) and V logpe(z') are integrated simultaneously with
(t) = (0) + S f(z (T), r)dr with a sample (0) ~ μ.. We summarize pseudo-code for learning ACNF annealed sampler in Algorithm 2 Once ACNF sampler is learned, it can generate unbiased samples: generate one-shot samples from ACNF with flow length according to computation budget; correct samples by resampling according to importance weights like (Müller al., (2099) or by Markov Chain Monte Carlo methods with Metropolis-Hastee correction.
EXPERIMENTS
6.1
DENSITY ESTIMATION ON TOY 2D DISTRIBUTIONS
Before deploy ACNF for modeling complex distributions, first examine 2-modal Gaussian mixture in 2D and use standard Gaussian as the base distribution. Figure 4 shows that the potential field of the learned ACNF is very similar to the numerical PDE solutions eq. 7 while the potential of CNF converges much slower than that of ACNF and then diverges after T'. See Appendix A.7 for experiment details and comparison on the choices of X, and other regularization methods.
We then train vanilla CNF, RNODE (Finlay et al. 2020 and ACNFs to model for various 2D toy distributions and visualize the density estimation along flows. [iguree]show the densities at I E (0, 27], T= 10 by learned CNF and ACNFs with various regularization coefficients for 2-moon distribution. The densities that are close to the target distribution are highlighted inside the red border. We show that even slight regularization makes the learned flows to 1) converge much faster towards the target; 2) maintain the best estimations for long time after T. Seen from the left of Figure 6 the quantitative evaluation the log- likelihood estimates implies the same conclusion. More analysis on different and experiment setups are given in Appendix [A.8]
One may suspect that complex dynamics explain the faster ascent of likelihood estimates. To validate the actual improvements by ACNF, we report the number of function evaluations (NFEs) like (Finlay al. 2020) by counting the times when numerical solver calls evaluate dynamics

---

Published as a conference paper at ICLR 2023
AONF, 0
Estimann 010 Paaee
ACNF, 0.0301
0.0305
AONF,
80
ACNI. 0.095
83
ACNF,
40
RNODE
20
likethood
Ikeihhod
0.00 0.25
0.75
1.25 1.50
1.75
10
NFES
T
Figure 6: Left: comparison on estimated ligeliilsssodd of models trained under different regulariza-
tion À as Figure 5 Middle: log-likelihood vs NFE as the left figure. Right: comparison on NFEs of vanilla CNF, RNODE and ACNF trained with various flow length T and X.
evaluated at t/T
CNF
9
④
④
④
C
9
9
C
@
O
CNF
ACNF
CNF
+
66
88
299
88
88
38
ACNF
88
88
88
88
889
88
88
88
88
88
88
88
88
88
88
2T
Figure 7: Comparison on density estimations of trained ACNF and vanilla CNF models on various two-dimensional toy distributions along flows with increasing E [0, 27"]
function in integral, with and without log-likelihood estimate for all models. The marks in the left of F show NFEs along flows while the middle one plots log-likelihood estimates versus NFEs. ACNFs clearly demonstrate that they learn even less complex dynamics than CNF and RNODE, and log-likelihood gain per NFE of ACNFs are much higher than the two basclines especially at early stage. Regarding ascent regularization, larger coefficient leads to more rapid gain the log-likelihoods initially, however, too large regularization wer-constrains models to reach a good maximum. A moderate regularization benefits both maximum likelihood and faster convergence. Furthermore, we report NFEs at t/T = 1 for CNF, RNODE and ACNFs trained with various 0.0001, 0.005, 0.001, 0.005, 0.01, 0.05 for flow length T 0.5, 1,5, 10 on the right of Figure 6 ACNFs have generally lower NFEs than CNFs and RNODEs, and most models report the lowest indicates that optimizing T like TO-FLOW (Du et al.) (022) and STEER (Ghosh) et al., 2020 may decrease computational cost at T, however, neither strategy can accelerate the convergence of flow and prevent deterioration like ACNFs as shown by the density estimate of CNF in Figure 14 in Appendix A.8 Figure Tigure[[shows shows density evaluations on other multi-modal distributions.
NFEs T
Learned ACNFs show faster convergence than CNFs for all distributions and even give a better maximum density estimation on the challenging task, e.g. Olympics distribution.
6.2 DENSITY ESTIMATION ON REAL DATASETS
We demonstrate density estimations on real-world benchmarks datasets including POWER, GAS, HEPMASS, MINIBOONE from the UCI machine learning data repository and BSDS300 natural image patches. Like FFJORD, tabular datasets and BSDS300 are pre-processed as in (Papamakar ios et al. 2017). [T the averaged NLLs test data for FFJORD, RNODE and ACNFs trained with different X. The detailed description of experiments and models refers to Appendix A.9 Although FFJORD with multi-step flows increases the flexibility of flows, it tends have performance than the base distribution initially and then improves NLL mainly at the late stage of flows. A larger ascent regularization of ACNFs contributes more rapid initial increases NLL that these flows transform the base distribution faster towards the data distribution. When training on HEPMASS and BSDS300, too large regularization coefficient impedes model to converge.
6.3 ACNF AS FASTER ANNEALING SAMPLING PROPOSAL FOR UNBIASED SAMPLING
we train CNF and ACNFs with regularization coefficients À 0.0001, 0.001, 0.01 learn the flow of annealed targets. We evaluate the estimates of log Z on
Following Algorithm 2
7

---

Published as a conference paper at ICLR 2023
Table 1: Averaged negative log- ligllikelihoods (NI..s) on test data for density estimation.
Datasets POWER
Model
0.17
0.251
0.51
0.757
T
t 7'
FFJORD1S
7.47
5.97
4.63
2.55
-0.42
5.02
RNODE
7.80
6.11
4.80
2.64
-0.46
4.59
ACNE, 1e
7.03
5.02
2.63
0.81
-0.40
0.12
ACNE, 1e
6.26
4.15
1.99
0.48
-0.44
0.02
ACNE, 1e"
5.43
3.47
1.87
0.48
0.37
0..6
GAS
FFJORD1SS
11.76
241.13
145.75
132.12
8.60
279.97
RNODE
12.03
102.77
78.20
45.38
-8.79
90.23
ACNE, 1e
7.25
3.31
-1.28
5.5
8.56
6.88
ACNE, 1e
5.58
1.65
-173
5.61
-8.45
-7.63
ACNE, 1e"
3.57
-0.14
3.773
6..2
-8.57
-8.33
HEPMASS
FFJORD1S
35.51
40.19
82.08
106.81
14.90
12.09
RNODE
35.02
40.59
68.91
96.47
15.34
39.09
ACNE, le
27.75
24.88
20.63
17.17
14.95
15.36
ACNE 1e
25.24
22.78
18.81
15.94
14.88
15.52
MINIBOONE
FFJORDS
58.54
53.98
42.12
24.55
10.50
16.25
RNODE
57.98
53.65
41.79
24.12
10.77
15.86
ACNE, 1e
54.15
43.76
29.29
19.00
10.63
13.95
ACNE, 1e
52.64
42.87
28.92
18.72
10.40
12.57
ACNE, le
51.52
40.52
26.53
17.16
10.95
12.14
BSDS300
FFJORD1S
41.72
30.11
0.91
85.92
156.60
777.00
RNODE
12.38
33.29
1.74
84.03
-15.771
-9.35
ACNE le
35.24
24.10
-14.75
100.35
-55.0
125.67
ACNE le
37.45
21.53
-17.84
-105.11
155.55
-220.89
i FFJORD uses multi-step flow models for some datasets, so the total length of flow is no longer training configuration of T but T times the number of flow steps. T listed here refers the total length of flow. The flow length after T set slightly different among datasets due the multi-step FFJORD: 1.27 for POWER 1.25T MINIBOONE and BSDS300, but is always the same cross
GAS, 1.17 for HEPMASS, different models. 3FFJORDs are trained to match the performance as originally reported.
0
ACNF, D
logZ estimate
-10
ACNF, 0.001
MCMC Ibest tuned), steps
-30
MEMC, 25MC steps MCMC,
-40
True
0.0
0.2
0.4
0.6
0.8
1.0
10a
10
10²
VT
NFEs
Figure 8: Left: comparison estimated log Z by different methods along the flow over 5 different runs. Right: estimated log Z vs NFEs. See Figure 18 for generated sample comparisons.
Gaussian mixture target with 8 components whose means fixed evenly in space and standard deviations as 0.3 and the base distribution is a Gaussian, A (0 3²1), to give adequate support. Figure 8 compares the estimates evaluated along flows and reports estimates versus NFEs. We benchmark CNF and ACNFs with the linear annealed target log 7%. (-) .4 log 7-) + 33x) log where scheduling is 8x k K / and K 20, using 7170, 25, 10) 4t)-ttep Metropolis sampler between each intermediate target like (Arbel et al. 2021 Wu et al. 2020). As ACNFs converge faster towards the target than CNF, the estimates via ohe-shot samples from ACNFs are less biased than that from CNF especially at the initial of flows. Besides, ACNFs are more computationally efficient in terms of accuracy gain per NFEs. ACNFs with À 0.01, 0.001 show less biased estimates earlier than even the best tuned linear annealed target. Besides, the linear annealed target requires at least 1 order more computations than ACNFs for comparable accuracy due to slow mixing of Metropolis sampler, and its performance is very sensitive to the number of MC steps. Figure [8 in Appendix A0 shows the generated samples by all methods in Figure g Adding MC steps with learned ACNFs can further accelerate sample convergence and increase the expressiveness of flows.
8

---

Published as a conference paper at ICLR 2023
Table Averaged negative ELBO on MNIST datasets under different length of flows t.
Model
0.17
0.257
0.57
0.757
T
1.27
VAE-FFJORD
85.90
85.07
83.96
83.26
82.88(82.822)
(8)..82))
89.74
VAE-ACNE, le
85.62
84.60
83.45
83.06
82.74
85.67
VAE-ACNE, 1e
84.70
83.95
83.22
82.53
82.80
84.37
originally reported in FFJORD
5
8
9
5
4
9
9
4
6
S
2
/
3
4
1
6
8
8
9
8
2
3
9
5
8
9
8
4
4
8
6
6
4
6
5
8
8
/
8
9
8
9
6
8
3
3
0
7
2
5
8
9
9
5
4
4
6
4
8
6
S
é
/
3
9
8
9
6
8
5
9
8
2
2
3
3
9
2
5
9
Figure Reconstructions from VAE-ACNF and VAE and original data for some challenging samples.
6.4
VARIATIONAL. INFERENCE WITH ACNFS
In addition to density estimation and unbiased sampling, CNF provides more flexible variational approximation to improve variational inference (Rezende and Mohamed 2015). We follow the experiment setup as (Grathwohl et al., 2018), that uses an encoder/decoder with CNF architecture, of which the encoder gives the base latent posterior approximation for CNF and the decoder decodes latent inference at the end of the flow back to observation dimension. To train a VAE-ACNF model, the log weight in eq.410p is replaced by an ELBO estimate like (Kingma and Welling 2014).
We evaluate VAE-ACNF to VAE-FFJORD and vanilla VAE without flow on MNIST data. To make fair comparison, we fix the learned encoder-decoder when training all flows. A detailed description of model architecture and experimental setup be found in Appendix [I.II The averaged negative ELBO on test data of VAE is 86.50 and Table rreports that of VAE-FFJORD and VAE-ACNFs with = 1e 13 along the flows. Compared to VAE-FFJORD, VAE-ACNFs show faster descent on negative ELBO at the initial of the flows, and a larger coefficient shows faster convergence of the variational approximation. VAE-ACNFs also circumvent the flow deterioration by VAE-FFJORD, thanks to the self-stabilization behavior of ACNE. Figure D and Figure 19 in Appendix [. some reconstruction examples from VAE-ACNE. These reconstructions tend to correct some defects in original images, add details to strengthen identities while remaining sharp.
7 SCOPES AND LIMITATIONS
While we have demonstrated that ascent regularization is effective to learn flows that converge faster target distributions, there are still number of limitations, which we would like address in the future. First, more efficient implementations on score function evaluation e.g. by estimators or model design or learning via score matching (Song et 202I) can accelcrate training for high-dimensional problems. Second, Hypernet (Ha et al. (016) used in experiments is found suitable to illustrate faster convergence behavior of ACNF as time exerts a large impact on the dynamics, however, is slower to train than other simpler network architectures. A better architecture may improve the training speed while maintaining the desired characteristics of flows. Third, although the proposed ACNF and ascent regularization have been discussed under the framework of CNE, the concept can be easily extended and explored for score-based models and other stochastic flows. Finally, ACNF and ascent regularization be applied for sequence of distributions, c.g. inference of sequential data.
8 CONCLUSION
We introduce ACNFs. a new class of CNFs, that define flows with monotonic convergence toward a target distribution. We derive the dynamics for the steepest ACNF and propose a practical imple- mentation to learn parametric ACNFs via ascent regularization. We demonstrate ACNF in three use cases: modeling data and performing density estimation, learning an annealed sampler for unbiased sampling, and learning variational approximation for variational inference. The learned ACNF' illus- trate three beneficial behaviors: 1) faster convergence to the target distribution with less computation; 2) self- stfbitaabiiztiion mitigate performance deterioration; 3) insensitivity to flow training length T. Experiments on both toy distributions and real-world datasets demonstrate the effectiveness ascent regularization learning ACNFs for various purposes.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work is supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP). Y. K. is member of the ELLIIT Strategic Research Area at Lund University.
ETHICS STATEMENT
As this work mainly concerns to propose a flow-based model and practical implementation for learning, it does not involve human subjects, practices to data set releases, or security and privacy issue. At this stage of study, we do not foresee the effects of potential system failures due to weaknesses the proposed methods.
REPRODUCIBILITY STATEMENT
All proposition (proposition 1) and theorems (theorem and 2) proposed in this paper are proved with
details in Appendix A.1 A.2 and A.5 as well other minor derivations mentioned in main body of
the paper. The pseudo-code for both learning cases are provided in Algorithm| I and Algorithm 2 The datasets, models, experiment setups for each demonstration are described in details in Appendix AA.
A.11 Furthermore, we attach some source codes in supplementary material for further checkup.
REFERENCES
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57):: 64, 2021.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 6572-6583, 2018.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint 1rXiv:1410.851 2014.
Laurent Dinh, Jascha bohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 08803, 2016.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. In International Conference on Machine Learning, pages
PMLR, 2019.
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. International Confer- ence on Learning Representations, 2018.
Ricky TQ Chen and David K Duvenaud. Neural networks with cheap differential operators. Advances in Neural Information Processing Systems, 32, 2019.
Emilien Dupont, Amaud Doucet, and Yee Whye Teh. Augmented neural odes. Advances in Neural Information Processing Systems, pages 3140-3150, 2019.
Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dissecting neural odes. Advances in Neural Information Processing Systems, 33:3952-3963, 2020.
Hao Wu, Jonas Köhler, and Frank Noé. Stochastic normalizing flows. arXiv:2002.06707, 2020.
arXiv preprint
Liam Hodgkinson, Chris van der Heide, Fred Roosta, and Michael W Mahoney. Stochastic normaliz- ing flows. arXiv preprint 47X14:2002..95577, 2020.
10

---

Published as a conference paper at ICLR 2023
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-68535 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
Qinsheng Zhang and Yongxin Chen. Diffusion normalizing flow. Advances in Neural Information Processing Systems, 34, 2021.
Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam M Oberman. How to train your neural ode: the world of jacobian and kinetic regularization. In International Conference Machine Learning, 2020.
Derek Onken, S Wu Fung, Xingjian Li, and Lars Ruthotto. Ot-flow: normalizing flows via optimal transport. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1-18, 2021.
Fast and accurate continuous
Liu Yang and George Em Karniadakis. Potential flow generator with 12 optimal transport regularity for generative models. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Jacob Kelly, Jesse Bettencourt, Matthew Johnson, and David K Duvenaud. Learning differential equations that are easy solve. Advances in Neural Information Processing Systems, 33:4370- 4380, 2020.
Arnab Ghosh, Harkirat Behl, Emilien Dupont, Philip Torr, and Vinay Namboodiri. Steer: Simple temporal regularization for neural ode. Advances Neural Information Processing Systems, 33: [4831-14843) 2020.
Shian Du, Yihong Luo, Wei Chen, Jian Xu, and Delu Zeng. To-flow: Efficient continuous normalizing flows with temporal optimization adjoint with moving speed. arXiv preprinl arXiv:2203. 10335, 2022.
Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré. Gradient flows: in metric spaces and in the space of probubility measures. Springer Science & Business Media, 2005.
Petr Mokrov, Alexander Korotin, Lingxiao Li, Aude Genevay, Justin M Solomon, and Evgeny Burnaev. Large-scale wasserstein gradient flows. Advances in Neural Information Processing Systems, 34, 2021.
Jiaojiao Fan, Amirhossein Taghvaci, and Yongxin Chen. Variational wasserstein gradient flow. arXiv preprint arXiv:2 12.00444, 2021.
Esteban G Tabak and Eric Vanden-Eijndee Density estimation by dual ascent of the log- liceehooo.. Communications Mathematical Sciences, 8(1):217-233, 2010.
Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likelihood training for score-based diffusion odes by high order denoising score matching. In International Conference on Machine Learning, pages 14429-14460. PMLR, 2022.
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models, Advances in Neural Information Processing Systems, 34:1415-1428, 2021.
Radford M Neal. Annealed importance sampling. Statistics and computiing 11(2): ((12):125-133 2001.
Michael Arbel, Alex Matthews, and Arnaud Doucet. Annealed flow transport monte carlo. In International Conference on Machine Learning, pages 318-330, PMLR, 2021.
Arnaud Doucet, Will Grathwohl, Alexander GDG Matthews, and Heiko Strathmann. Score-based diffusion meets annealed importance sampling. arXiv preprint arXiv:2208.07690 2022.
Roger B Grosse, Chris J Maddison, and Russ Salakhutdinov. Annealing between distributions by averaging moments. Advances in Neural Information Processing Systems, 26, 2013.

---

Published as a conference paper at ICLR 2023
Thomas Mülllr, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. Neural importance sampling. ACM Transactions on Graphics (TOG), 38(5):1-19, 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, pages 1530-1538, 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. 2nd International
Conference on Learning Representations, ICLR 20/4, 2014.
David Ha, Andrew Dai, and Quoc V Le. Hypemetworks.. arXiv preprint 1et..9660900906, 2016.
Qiang Liu. Stein variational gradient descent gradient flow. Advances in neural information processing systems, 30, 2017.
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-33 1982.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 PROOF OF THE INSTANTANEOU'S CHANGE OF LOG-LIKELIHOOD ESTIMATE
Proposition (Instantaneous Change of Log-likelihood Estimate). Let z(t) be finite continuous
random vuriable al time 1 as the solution of a disferential equation dt value z()) Assuming that Po = μ at t 0 and f is uniformly Lipschitz continuous in and t, then the change in estimated log ooglikelihood log px(x) at follows a differential equation:
dn(t) f(z(t),) () with initial
d log (x) = f(z(())),) t) + V log u(z((t) f(z(t),
dt
Proof. To prove this theorem, we take the infinitesimal limit of finite changes log Pe (x) through time. As f is assumed be Lipschitz continuous in z(t) and (x(x)) represents the unique solution of the ODE (eq.(2) at time t on the initial value x:
fx(x) = x + / f( f(),(x), T)d7,
We also denote the transformation on (t) over an e change in time as:
z(1 + <) = $. (z()) +t+c(x).
Using the definition of estimated log density Po (xx in eq 4 the infinitesimal limit is:
d log Pi (x)
1 (log) det det (Jær.. (x)) log det (JT, (x)) log μu ((+e(())) - log μi (x)))
lim c+0+ e
dt
1
lim (log) det (Jart, (x)) log/det det (Ja, (x)) |) + lim -
(log (x)++(x)) - log (o (x))).
-
The derivation of first term is very similar (Chen et al. 2018 theorem 1) except the sign of the function:
1 (log det (Jo... (x)) log det (x))))))
lim c->0
1
lim
(log det (J+, (z()) + log det (Ja, (x)) log| det (J+, (x)) l)
c-+0+
1
lim
log/des (Jo, (z(())) ),
where summarize the main steps:
log det Js, (z(t))
log det Jo. (z(())
lim c-+0
lim
(L' (L'Hopital's rule)
+++
de dc
det J+. (z(l))
lim
de
c-+0
det Js. (z(t)) +
lim
det J+. (z(t))
0 à
=
lim
Tr adj @z(t)
& (z(()t))
de dz((t) @ (z(()))
(Jacobi's 's formula)
Tr
lim
$ (z(t))
(adjacent matrix + I as e + 0+)
c-+0+
de dz(()
Tr
lim
(z(t) + cf(z(t), t)
c-+01
de dz(()
I Ml(z(),))
dz(t) + o + +..))
Tr
lim
c-
de
(F(()()))))
+ o(c) +..)) dz(()
Tr
lim
c-+01
= V ff)())),
13

---

Published as a conference paper at ICLR 2023
Before deriving the second term, take the first-order Taylor expansion of lo ((( at z(t) = (x)):
log u Pete = log
=
hence,
log (ogu( - log (x,
lim c-0+
logu(z((t) V log ((zz(())) - (z(()(()) z(t)) + o(E2) +... log μ (z(t))
lim <-0+
lim (log)(z((t)) (e(z(t)) z(t)
+ + +...
--
= =V (olg(((((t) f(z(t),t).
Therefore, the differential of log pr(x) is:
d log pr (x) f(z(t)), log f()
dt
To show the relation between two differentials between log Pe (x) and log Pi (z(t)):
dlog (x) de and (x(t))
we first need the relation
de
log pr (x) log log u(z((z - log (zg))(z(t)).
Taking the total derivative on both l. and r.h.s. of last equation:
d logge( dllo ((((t))
dlog (z(())(z(t))
dl
di
dt
V (Vogu(z(tt) f(z(t), t) d d po (z(t))
di
=V =V Vog f(z(t), t) + V f(z(t),,
The total derivative (x(x)
de is defined on the fixed variable x, while the infinitesimal change on r.h.s. is evaluated on the variable zt). So solving log (x) requires to simulate z(t) simultaneously. Different solving log pa (z(t)) on the reversed direction solving z(t). log (x() only needs the trajectory of z(T), E [0, t), while Po (z(()) requires to know the whole trajectory of z(T), E [0, T). Therefore, using log (x) is more advantageous when evaluating models at any & other than T or at multiple I.
As for training, since pr is specified as μ at T, maximizing logp(x) in vanilla CNF essentially equivalent to maximizing log Pr (x) in ACNE.
we take the time partial derivative on the log-likelihood equation, then
log (x) Olog alog ((((t)) Ölog Pe (z((t) log (z(())
Ôt
Ōt
Ōt
Ot
so that the convergence rate of distribution estimate Py (x) towards p(x) is equivalent to the normalized distribution pr (z) towards pzz).
A.2 PROOF OF THE DYNAMICS FOR THE STEEPEST ASCENT CONTINUOUS NORMALIZING FLOWS
Theorem (Dynamics for Steepest Ascent Continuous Normalizing Flows). Let z(t) be finite
continuous random variable and the solution of differential equation dz(!) f(z(1), 1) with initial
14

---

Published as a conference paper at ICLR 2023
value z(0) x. Its probability pı(z(t)) subjects the continuity equation dep: V (pef) The dynamics of the steepest flow for decreasing KL(pp(z(t))ll((((tt) is
Vp((z(t)) V Vlogu(z(t)) V logp:(z(t)). pr(z(t)
(z((),, Vlogu(z(tt)
To keep this proof simple, derive this theorem in Euclidean space. If readers familiar with non- Enn-uucideen metric spaces, we refer more rigid of Wasserstein gradient flow proof in (Ambrosio)
al. 2005).
Proof. Assuming that N samples X {x: RNN are drawn from p(x), the averaged negative estimated likeliidl time is:
N , = N
N
1
Σ Po (x,,)) - log log,(( (x, = N
log lox)(x
Using the chain rule, the derivative of J ((+)) w.r.t. (x(x,) is: [V((F)),, = log $ (x,)) V log (x))(())))
where ()( is matrix that each row is for each sample = 1, ,,...,,.. .N and each column is for each dimension j 1,2, ......... d.
To numerically compute the solutions of Buler-Lagrange equation, i.c. ( = 0, we use gradient descent define the evolution of transformation &, for each x;:
dx, - [V), log u (x())) Vlog (x):)))
dt
which evolves + in the direction that decreases (( most rapidly, starting at initial Po(x,) X,
The next step is to extend the assumption of the finite number of data samples N to infinity, i.c. N + therefore, the objective J ((;,) at time t is updated as:
)( J. log (x))xx
[ (logre((,, - log (x)) - logp(x)) dx
J. L(x, (x(x)), (x)xxx)
where U C Rd and L(x, 2x(x), (x)( log pr (x))) log (x)x(())) logg For each j dimension of 4,, the functional derivative of J(P..) w.r.t. [ is:
OL
ØL
(x, (x). (x))
(x, (x(), V
,1
2 0.
aV
V ( V log μ( (x))),
01 σV ,
0. Therefore, the gradient descent that defines the evolution of transformation 4, is:
( dt
8J( = V log (x)()))) V log pa (x)),,
(11)
84;
therefore, the dynamics for the steepest ascent continuous normalizing flow is:
(z()))), ddb((x) = V log u(z((t) - V log (z())(z(t))
dl
15

---

Published as a conference paper at ICLR 2023
A.3 CONVERGENCE RATE OPTIMAL ASCENT CONTINUOUS NORMALIZING FLOWS AND ITS RELATION TO LANGEVIN DYNAMICS
The convergence rate of KL divergence w.r.t. can be derived (we start from general flow dynamics f):
a (L(px())llx(2)) KL (L(p(x))li
0t
/ p(x) ①
Pi (x))xx
Ot
/ p(xx (V- f(z(t), t) + Vlogu(z(tt) f(z(t), dx
f pi(a(t)) (V . f() v (log((z(()) f(z((), d2(1)
(12)
= / pi(z)) 1f1(2,1)
Glog
Si(z)
dz
Oz,
Oz:
Σ | -i((z,)) ()
(p)(z)
Olog
Ji(2))
0z:
Ozi
8 Ologp((z)
Olog
= Σ pr(2)(
Ji(zz 1)dz
O2i
0z:
= -p.. [(V log ju() - V po (z))
f(z, t)]
Liu 2017) 3.1] shows similar derivation from discrete transformation perspective and links to Stein variational gradient flows.
When dynamies f is equal to the fastest flow dynamics f* as eq 66 then the convergence rate becomes negative Fisher divergence (estimated w.r.t. p.):
i
(L(p(())ll(()) -pp Vog V log (z) Ot
This convergence rate can be easily proved the same to overdamped Langevin diffusion dynamics which is defined via stochastic differential equation at case of 1:
dz(l) V ((z(((())) + 223-IdW.:
(13)
where W, is Brownian motion. Under the Langevin dynamics, the transformed distribution has PDE:
Öpe(z)
(pr Vl) 3- 3-1 (z)
Øt
V (po( 3 . (Vpt
V (V 3 V log ))
The last line reveals the steepest gradient flow dynamics as cq when 3 1.
Therefore, the optimal ascent continuous normalizing flows and overdamped Langevin dynamics transform a distribution equivalently when / = 1. And this Fokker Plank equation is a linear (w.r.t Pz(z) and deterministic although Langevin dynamics is stochastic. The main difference between these two flows is that the dynamics of (optimal) ascent continuous normalizing flow is deterministic, so any particular sample trajectory; while Langevin dynamics defines stochastic process and sample trajectories are stochastic.
A.4 DERIVATION OF POTENTIAL FIELD PDE
The optimal dynamics defined in eq e can be rewritten in terms of the potential function V(z(t),
as Viz, t) = pi(z).
f = V Foog (log((z())) Vlogp(z(().)) log ((z(1),1).
(14)
16

---

Published as a conference paper at ICLR 2023
The continuity equation reveals the time derivative of the transformed density p(z(t), t) at t:
(pt(z(t) ()x()())) -p(x ((z(t),)) Vpr(z(()) f(x()),).
0t
Therefore, the time derivative of logp:(z) with dynamics defined in eq.
(1.41)) is:
a @logPe(((1)
@p.(z(()) V f(z(t), t) V logp((z(t)) f(z(t),t)
Pc(z(t) Øt
A log ((z(t),t) V logp:(z(t)) V log (()(t),
Using the last equation, the time derivative of log V(z, t) is derived as: a llog (z,tt)
Olog (z(())
a log u(z(t))
Ot
Ot
Ōt
f(z(1), v (lg))(((())) f(z((1,)) Vlogu(z(0) ((z(().)) A log (z((t),t) + (o (z(()) Vlogm(z(())) ((((t),)
log () + (2V log (z(()((()) + v log ((z()( -lg ((((t),
therefore, the time derivative of potential field is:
(v 1) Öt
=AV(2,t) + 2V (z)) (V(2, + + log (z,tt)
(V(2.t).
(15)
When 0, V(x,0)
p(x) p(x) when t V(z,t) 1, Vz.
A.5 INSTANTANEOUS CHANGE OF SCORE FUNCTION
Theorem (Instantaneous Change of Score Function). Let z(t) be a finite continuous random variable
with probability density pz(z(t)) time Let dæ(t) f(z(t), be a differential equation describing
a continuous in-time transformation of zt).. Assuming that is uniformly Lipschitz continuous in
and t. the infinitesimal change the gradient of log -eediistty is
dV log pr(2()) = po )f22()). V (V f(z(1),)). (z(()
dt
Proof. As f is assumed to be Lipschitz continuous in z(t) and (x)x) represents the unique solution map. We denote the transformation on z(t + e) reversed over an change in time as:
z(t + e) = 2. (z(t)), z(t) = t (z(t + +)),
and applying the change of variable theorem on log Pc+. (z(t + e)), defined on the variable z(t e):
log (z(t + c)) log P: (z(t)) logl det Jo. (z(t)) log + (z(( + c))) - log det Jo. (Ø +c)))
Taking the derivative of log P:+. E)) w.r.t. z(t + on both Lhs. and r.h.s. of the last equation:
0 (z(_ log +e) = - logp:(z(1)) log det Ja. (z(()))
dz(1 +e)
and the infinitesimal limit of finite changes of gradient of log density can be defined:
17

---

Published as a conference paper at ICLR 2023
dV lott)(z(t))
lim V
V (logp:(z(()))
dt
lim 1
((Vlogp(((()))) Vlog det Js. (z(1)))
04 ( (c)) Vlogpi(z(1))
c-0 6
dzit +c)
(22)(2(11)
-1
(Voog dell Jo. (z(1)) 0c. (0P,
)
lim =Vlogp(((()) -00
lim 0
8z(1)
Oz(t)
(t Oz(()
(logpi(z(t))
V (V f(z()).
(16)
where the two limits are derived in detail:
lim 6-0
1
00, (
1
Oz(()
6
1
(x((1) (z(() + +=((((1),))+ o(6)) + +...) -1
-1
-lim
<-10- 6
1
[I+ [I+ Üz(t)
Of(a(l),1) +(2)) +...)) - I
lim c-0
c
1
0f(x((),( +o(e2) +..))...)) )1
-1)
lim
(inverse by geometric power series expansion)
(z(()
c10
(f(z(t),tt
lim
o(c) +...
dz(t)
--
Of(z((),
dz(()
and
(o.((((L))
lim
Vogg del Jo. (z(1))
c00
Öz(()
I Of(z((),))
lii
Voog del Jo. (z(1))
+ ol(2) + +...)
c-00+
Ōz(())
lim
V log det Jo. (z(())
lim
log det Jo. (z(1))
Of (f(2(1),
(17)
c-001
Öz(()
V1-+0
log det Ja. (z(t))
lim
+
V f(z(t),t)).
Therefore, V log p(,, follows linear matrix differential equation, where the linear matrix is defined by the Jacobian of(2(0.)) and the bias term is the gradient of divergence of the differential function V V f(z((),1)).
82(()
A.6 INTERPRETING ASCENT REGULARIZATION AS SCORE MATCHING OBJECTIVE
To show the ascent regularization in eq. 9 and eq. 10) relates to the score matching objective, we first assume diffusion process defined via stochastic differential equation (SDE):
dz(()) h(x(1),)) + (() (()) x; p(x))
(18)
18

---

Published as a conference paper at ICLR 2023
where W, is Brownian motion and we denote Pr (z(()) the marginal distribution time and Pr as the path measure of the SDE up to time T.
(Anderson 1982) shows the reverse time process is also diffusion process which shares the same marginals as the forward process:
dz(() = (h(z((),)) (()V log (o di g(()d (T) ~ pT:
(19)
where (() is reverse-time Brownian motion. The reverse-time diffusion introduces the conditional path measure Pl-lz(Z)). As the score function, V log Pt(z)t)), generally unknown for an arbitrary
diffusion process, we approximate the reverse-time diffusion by secondary reverse-time diffusion process by parametric score function:
dz(t)= = (b() - g((t)so(z(()))) ()) dt (()) ~ pr:
(20)
which induces the conditional path measure (-()((T)) to approximate Pr(-Iz(T))
Under some regularity conditions that permit the definition of Radon-Nikodym derivative, ()Pr(-((T))/dPPPp) (T)), Girsanov theorem gives the expectation of KI. divergence between two path measures:
(-z(T)(T))
Epp (L(PT() (L(PF(((())I(((((())) EEp log dPr((Iz(T))
=Ep /' (()((())) (oool((),))-Vogpp(((())))))))) Vlogp(((())) + 2 L'' g (()])(((((())) ((0llsca(0.00-/lopppaa()1) (
1 Ep (t) /'" [[so(z(()))) (l)))((((t))
2
Using the chain rule of KI. divergence, we can show the KL. divergence between path measures:
KL(PxIIP7)
(L) + Epr (LPP)((((T)))(((((())))
gP(() [ [|so(z(t),))
(KL(pp((( + Ep
log))(z(t)))
(21)
2
=KL(p(x)(((x)) Ep
g (())ss((((())) (og))(a(()))at
Assume that the parametric dynamics f(z(t), t; = Vlogu(z(tt) - se(2(t), t) has the similar structure as the optimal dynamics in cq.(6) as so (z((), t) to approximate V log (z((t) and g(t) v2X, then recover the total learning objective with ascent regularization coefficient in eq. Therefore, the total objective is equivalent to minimize the KL divergence of two path measures on the joint (infinite) variable space. Similar analysis can also be applied to the objective eq. TO).
When À = 3 (t) and learned score se (z(t), t) matches to V log (z(t)) so that h(z(t), f) 2
V log (t)) then SDE in eq.(18) becomes the overdamped Langevin dynamics in e3. as well as optimal ACNF (eq. ()) with critical damping dynamics, i.e. A= -
As the ascent regularization can be interpreted as score matching objective, it possible to implement Algorithm I and Algorithm in more time efficient way for training like (Lu et al.) 2022, Song et al., 2021). However, note that the explicit score matching objective can hardly be used directly in the implementation as V log (z())) is intractable in general and requires to be evaluated e.g. via score function integral in ascent regularization. (Lu et al. 2022 Song et al. 2021 2020; Ho al. (2200) its surrogates, e.g. denoising score matching. To enable practical training, denoising score matching objective relies on the explicit form of conditional (noised) distributions
v log (2((2((()(() e.g. Gaussian. For image or data generation tasks, Gaussian assumption may
19

---

Published as a conference paper at ICLR 2023
2T
0
Figure 10: Comparison of log potential field, log (z((t),t) t), evaluated on trained vanilla CNF, RNODE with regularization coefficient 0.1 and ACNF models with regularization coefficient À as 0.1 and for 2-modal Gaussian mixture along flow at E [0, 27' and the numerical PDE solutions of 7. Color indicates the value of field: turquoise and the lighter the color the larger the value is, and vice versa.
2T
Figure 11: Comparison of log potential field, log V(z(t), t). evaluated trained vanilla CNF and RNODE (Finlay et al. 2020) models with T= for 2-modal Gaussian mixture along the flows at e [0, 27 Figure The kinetic energy regularization coefficients are 0, 0.01, 0.1, respectively. Color indicates the value of field: turquoise is O, and the lighter the color is the larger the value is, and vice versa.
not seem so limited as long as the chain of discrete transformation is adequately long for adequate expressivity of the marginal distribution at T. However, for inference tasks e.g. using flows as variational approximation or annealed sampler, constraining the distribution induced by flows with Gaussian assumption can hinder their approximate potential for true posterior.
A.7 ANALYSIS ON A TOY EXAMPLE: FROM A GAUSSIAN TO A MIXTURE OF GAUSSIAN
Before we deploy ACNF for complex distributions, we first demonstrate its validity on a simpler problem: to learn 2-modal Gaussian mixture with a standard Gaussian as the base distribution. Since the density of the target distribution is known in this case, we can numerically solve the potential field (z, for [0, I' in eq. 7 even though the exact solution still hard obtain for this simple case.
The PDE solution presented in Figure 4 and Figure 10
implemented using py-pde package.
fixed Cartesian grid is used which has the same center locations as the other potential fields evaluated by density estimations. The PDE solver py-pde uses the finite difference method, and we choose explicit solver to keep simulation simple.
To define the parametric dynamics function for training, we use hypernetworks (Ha et al. 2016) that smaller network generates the weights of layers. This architecture is suitable demonstrate ACNFs as the function of dynamics is supposed to evolve with time via changing the weights by the hypernetworks. We follow the same implementation of hypernetworks Neural ODE and use tor torchdiff ff for ODE solution and adjoint method.
The last row of Figure 10
Figure 4 shows the logarithm of the potential solutions, while the rest show the log potential field of learned flows evaluated by the ratio p(x) (xx))) when training 7 is set 10.
20

---

Published as a conference paper at ICLR 2023
0
t=2
Figure 12: Comparison on log potential field, log (z) of trained vanilla CNF and ACNF models with T = for 2-modal Gaussian mixture, evaluated along the flows at t E [0, as Figure The ascent regularization coefficients are 0, 0.01, 0.1, respectively.
Without ascent regularization, the potential field converges slower and only reaches close to uniform field at T. After T, some areas start to be under/ooer-reepresettee when the learned flow continues to move samples towards the center of the field. Nevertheless, the flows learned with ascent regularization transform densities faster to the target distribution. When the ascent regularization coefficient À is I, the evolution of the potential fields is very similar to that of PDE solutions which indicates the learned flow is close to the optimal ascent continuous normalizing flow.
Apart from vanilla CNE, we train RNODE models to demonstrate the effect of kinetic energy regularization the transformation of distributions. As known from (Finlay et al. 2020), the optimal flow that minimizes L2 transport cost induces straight sample trajectories and samples travel with constant speeds. Figure 11 shows the flows of RNODE models trained under the same configuration as Figure 4 and the kinetic energy regularization coefficients are O, 0.01, 0.1, 1 respectively.
Although RNODEs learn simpler ODE functions with lower NFEs compared to the flow without regularization, these flows do not induce the transformed distributions to converge faster. They are even slower at larger regularization coefficients. Like vanilla CNF, RNODE does not prevent the distribution to deteriorate after T. NFEs for each flow in Figure IT at the time that the transformed distribution gives the maximum estimated log- likeliiooood, are 38, 38, 36, 32, while the flows by ACNF are 26, 32, 36 under À = 0.01, 0.1, 1, nevertheless ascent regularization does not explicitly regularize for simpler ODE functions. We also tried Frobenius norm regularization on the Jacobian as suggested by (Finlay et al.) 2020). HJB regularization (Onken et al., 2021 Yang and Karniadakis] (020), second-order regularization (Kelly et al. 2020), however, the evolution of potential fields under these regularizations does not differ much to that of vanilla CNF and RNODEs as shown.
To demonstrate the effect of the length of flow in training configuration, we train vanilla CNF and ACNFs with other flow length, e.g. T = 5 and ascent regularization factors 0, 0.01, 0.1, 1, and evaluate the learned flows at t E [0, 2] as Figure 4 Under some suitable condition that there exists an optimal ACNF between the base and the target distributions, the flow is almost independent to the choice of flow length T. Comparing Figure 12 with T = 5 to Figure 4 with T == 1 but testing both E [0, 2]. the flow by vanilla CNF is idle at early stage for T = 5 and is very sensitive the choice of T. while the flows with ascent regularization are almost independent to the choice of T, which possibly makes tedious model selection on different T optimizing T' (Ghosh al. 2020 Du et al. 2022 no longer necessary.
A.8 DENSITY ESTIMATION ON 2D TOY DISTRIBUTIONS
Like Section A.7 we specify dynamics model by hypernetworks and all hypernetworks are defined by one hidden layer with 32 units and 64 for the width of hypernetworks learn all 2-dimensional distributions.
As shown in the last section, the flows learned with ascent regularization are almost insensitive to T for Gaussian mixture. To examine whether this conclusion still applies to more complex distributions, we retrain ACNF models with ascent regularization coefficients À = 0.0001, 0.0005, 0.001, 0.005 under different flow lengths T = 10, 5, 0.5. Figure 13 (T 5) and Figure [T (T
1) shows the
evolution of the density estimations for each model at t E [0, 2T] like Figure 5 (T = 10). When decreasing T from 10 to 5, the density estimations are almost identical under the same regularization coefficients. When T' decreases from to 1, the highlighted area shrinks slightly at low regularization
coefficients, e.g. 0.0001, 0.0005. Model trained with smaller T may require larger À to have
21

---

Published as a conference paper at ICLR 2023
2
e
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
3
2
2
2
2
2
2
2
2
2
2
2
2
2T
Figure 13: Comparison on density estimations of trained vanilla CNF and ACNFs with regularization
coefficients À = 0.0001, 0.0005, 0.001, 0.005 and ' = 5 on 2-moon distribution at E [0, 27'.
R
2
2
2
2
2
2
2
2
2
2
2
2
2
2
e
2T
Figure 14: Comparison on density estimations of trained vanilla CNF and ACNF's with regularization
coefficients À 0.0001, 0.0005, 0.001, 0.005 and T 1, 2-moon distribution / (7)
@
0
@
8
8
0
O
o
00000000000000000000000000000000
O
O
0
01010000001000000000000000000000
0
0
9
0
2T
Figure 15: Comparison on density estimations of trained vanilla CNF and ACNFs with regularization
coefficients À = 0.0001, 0.0005, 0.001, 0.005 and T 10 on 2-cirele distribution at t [0, 277.
+
+
29
000
0O0
328
900
205
899
209
909
209
909
829
929
829
98
85
35
000
000
000
000
000
300
2T
Figure 16: Comparison on density estimations of trained vanilla CNF and ACNFs with regularization coefficients À = 0.0001, 0.0005, 0.001, 0.005 and 10 on Olympies distribution at E (0, 27).
similar regularization that with a larger T as the flow length serves as an implicit regularization factor. Although the effect of regularization depends slightly more the choice of T for complicated distributions, the flows by ACNF are still much less sensitive to T, compared to that by CNE.
Apart from the 2-moon distribution, we show the density estimations of learned vanilla CNF and ACNF with different regularization coefficients for modeling 2-circle, Olympics and checkerboard distributions in Figure [5] Figure [6]and and Figure 17 They show that ascent regularization is effective in learning different distributions that a larger coefficient induces densities to converge faster to the target distributions and prevents them from deterioration. Comparing across different distributions,
22

---

Published as a conference paper at ICLR 2023
2T
Figure 17: Comparison on density estimations of trained vanilla CNF and ACNFs with regularization coefficients À= 0.0001, 0.0005, 0.001 and 10 on checkerboard distribution at E [0, 27).
Dataset
hypernetworks layers
encoding dim
T
# flow steps
batch size
POWER
2
5
10000
GAS HEPMASS MINIBOONE
5
5
1000
3
10
1
10
10000
4
10
1
1000
BSDS300
4
10
5
2
10000
Table 3: Model architectures of ACNFs for density estimations on tabular data reported in Table
the highlighted areas larger for 2-moon, 2-circle and checkerboard distribution than Olympics distributions, since the Olympics distribution more challenging and requires a relatively large regularization coefficient.
A.9 DENSITY ESTIMATION ON TABULAR DATASETS
For tabular datasets, follow the experiment setup and model configurations recommended by FFJORD (Grathwohl et al. 2018) and all data are pre- pre-processed according to (Papamakarios et al. 2017p. We found that the concatenate layer used in FFJORD, that concatenates time t and
states z(t) as a flat input vector for differential function, dilutes the ascent regularization on the parameters, especially when data dimensions are high, e.g. for MINOBOONE and BSDS300 datasets. Nevertheless, the hypernetwork architecture used in previous sections, even a deeper one, turns out to be inadequate reach similar log- likelihoooo evaluation as FFJORD and slow train. To tackle this issuc, we use an encoder to encode states ztt) to a lower dimension and apply the weights by the hypernetworks the encodings and later decoder maps the transformed encodings back the data dimension. We summary model architectures and training configurations for each dataset in Table 3
A. ACNFS AS ANNEALED SAMPLER FOR UNBIASED SAMPLING AND ESTIMATE OF NORMALIZATION CONSTANT
To extend ACNF annealed sampler with stochasticity, we replace the discrete NF blocks in SNF by the discrete realization of each adaptive step of ACNF and each is followed with a stochastic block by e.g. discrete Langevin flow or MCMC flow as SNF. The original importance weight update for discrete flows also needs to be replaced by the integral of negative divergence of dynamics and resampling steps are added as AFT (Arbel al. 2021) The complete algorithm is summarized in Algorithm
Figure shows the generated samples all different methods as reported Figure & plus adding MC steps on top of trained ACNF to form SNF models by Algorithm 3 Like quantitative evaluation shown in Figure g learned ACNFs with regularization coefficient À 0.01 has distinctly faster convergence than CNF, best tuned linear annealed target and less regularized ACNFs, but uses less computation. The add-on MC steps on trained ACNF boasts the convergence slightly as shown by the last two rows. Although diffcomorphism constraint does not show much effect on limiting the expressiveness of CNF/ACNF this experiment, adding stochastic blocks is still very beneficial especially at the beginning stage of the flows.
23

---

Published as a conference paper at ICLR 2023
Algorithm 3 Asymptotically unbiased sampler with learned ACNF To
Require:
parameteric dynamics of ACNF generation flow fa, base distribution target distribution up to the normalization constant (-)) = r(-) length of flow T, number of samples N, MC step size number of MC steps J
sample samples from base distribution {zi)i=1:N ~ μ 40
log w/- log ((zz), to =
while & < T do
ODE solver chooses step size Atg, if lR 1--11 + Ar, T else I, T
Integrate augmented states (t), log qe(2 (t)) using generation dynamics fo until b from the
initial log Qr (z at 14-1
ASk f = log (44-1 (z/-1) log tt, (1+))
z z (1x)
MCMC update with π invariant kernel via Metropolis-Hastinne for j = I.....JJ J do
propose x² z + en's ((O,I)) at = 7(2 (/7(())) if < aa, (0, then
update zi.- z'k end if end for
AS log 7z (tx)) - log (zz)),
Update weights log ij. log - 1 + AS +AS
Resample z according to normalized weights viद = w w/)
Update weights w 1/N
end while
Target
0
03
33
53
T
Figure 18: Comparisons of generated samples from different methods 2D Gaussian mixture
distribution with 8 components as Figure g From top to bottom, samples are from: (1-3) linear
annealing importance sampler with (170, 25, 10) MC steps between each annealing target; (4) CNF:
(5-7) ACNF with ascent regularization factor X = 0.0001, 0.001, 0.01, (8-9) SNF with trained ACNF 0.01 (as 7th row) and {1, 5) MC step as the stochastic block Algorithm
A.11 VARIATIONAL INFERENCE WITH ACNES
Our experiment setup mimics (Grathwohl et al. 2018), and the encoder and decoder are defined by 7-layer neural networks with specified latent dimension as 64. The first layers of the encoder are implemented gated convolutional networks and the last one is a linear layer output mean and diagonal covariance. For the decoder, the first layers are also gated convolutional networks while the last layer a vanilla convolutional network. We define the length of flow for both VAE-FFJORD and VAE-ACNF as T and the number steps as 2. The networks for modeling differential function of flows the modified hypernetworks as for the tabular datasets, with 4 layers, and the
24

---

Published as a conference paper at ICLR 2023
7
2
1
0
4
4
9
5
9
0
6
9
0
5
9
7
8
7
2
1
0
4
4
9
4
9
0
6
9
0
8
7
2
1
0
4
1114
9
6
9
0
6
09039973
8
7
6
6
5
9
6
3
3
9
09407401311477211
5
12445612466356601111
7
|
134435144663550001
7
7 8 99374443070911
7 2 7
8 9377044070271
7
2
7 8 0
7
3
2
7 6 6
9
7 6 3
3
1
4
7
6 3
9
3
1
4
7
3
9
3
1
4
7
6 9
6 6054992(948)
7
3
9
7
4
4
4
9
6054992(94
7
3
9
7
4
44494
6
3
9
7
44444
5 6
8
10/664
5 1
000004 64
5 04767900
8
5
6
1
/ 6
673177
(
8
2
0
2
5
06004
6
3
3
1
7
8
2
0
2
9
5
5
4
6
7
3
|
7
8
5
5
5
5
6
0
3 4
Figure 19: More reconstructed samples from VAE-ACNE, vanilla VAE and original data. The first row three is the reconstruction from VAE-ACNE, the second one the reconstruction from vanilla VAE while the last is the original data samples.
activation function is tanh. All models reported in Table are trained under the same learning rate as 0.001, Adam optimizer and batch size as 100.
Figure 19 shows more reconstructed samples from VAE-ACNF and vanilla VAE, with comparison of original data. general, the reconstructions from VAE-ACNE are smoother than ones from vanilla VAE and original data samples. Figure 9 shows some challenging examples for VAE to reconstruct. VAE-ACNF tends to reconstruct images by adding more details, not only to make smoother, but also to possibly strengthen their identity of classes. Furthermore, due to the coarse variational approximation, some reconstructions of VAE fail to retain their features in original data and change the identity of classes.
25


Reviews:
[{"review_id": "k6p6xmCXrnn", "reviewer": "Reviewer_XQ2y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper discusses ascent regularization for training continuous normalizing flows (CNFs). This is motivated from Wasserstein gradient flows and results in an interesting regularization that encourages the learned model to be similar to the target distribution around a large interval of time values. I find this to be an interesting alternative to optimal transport-based approach for reducing the compute cost to transport samples between a base distribution and a target distribution. Experiments include maximum likelihood on tabular data and as a variational inference model within a VAE.", "summary_of_the_review": "I think this paper is interesting and has potential. However, I feel hesitant to recommend acceptance in its current form, particularly since multiple concepts and equations are introduced but I did not understand how these connect with the final training algorithm. My understanding is that the main contribution seems to be using the time evolution of the score function for regularizing the ODE to model the steepest descent direction on the KL, but the connection to \\tilde{p}_t is unclear to me. If the authors can clarify this, I would be willing to change my rating.", "strength_and_weaknesses": "Strengths:\n  - The proposed approach is interesting, as a way to mimic Wasserstein gradient flow paths. Using the time evolution of the score function is interesting as well.\n  - The algorithm works well at regularizing the path and seems to help accelerate convergence towards the target distribution. This is shown across both MLE and VI settings, as well as in a toy setting where it is shown to improve upon MCMC in terms of the number of function evaluations.\n\nWeaknesses:\n  - I found the writing poor and can be significantly improved. Specifically, there seems to be multiple concepts being introduced \\tilde{p}_t (Eq 4) and V (Eq 7) that do not appear in the experiments. See clarity section below for more comments. If I am mistaken, please correct me.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SUMNud61t0G", "reviewer": "Reviewer_niTY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors  propose a new class of continuous Normalizing flows (NFs), ascent continuous normalizing flows (ACNFs), that makes a base distribution converge faster to a target distribution.  The learned ACNFs demonstrate faster convergence towards the target distributions, therefore, achieving better density estimations, unbiased sampling and variational approximation at lower computational cost. ", "summary_of_the_review": "This paper is above the average. I would suggest to accept this paper. ", "strength_and_weaknesses": "Pros:The paper proposed ACNFs, a new class of CNFs, that define flows with monotonic convergence toward a target distribution. The authors derive the dynamics for the steepest ACNF and propose a practical implementation to learn parameteric ACNFs via ascent regularization. The learned ACNFs illustrate three beneficial behaviors: 1) faster convergence to the target distribution with less computation; 2) self-stabilization to mitigate performance deterioration; 3) insensitivity to flow training length T . Experiments on both toy distributions and real-world datasets demonstrate the effectiveness of ascent regularization on learning ACNFs for various purposes.\n\nCons: The definition of \\tilde p_t, at least in the main body of the paper, say in page 3, is not very clear. I would suggest the authors to make the derivation part before more clear, say those around prop. 1. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HyUPhqwO26Z", "reviewer": "Reviewer_XKEH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a way to regularize continuous normalizing flows by regularizing them to have monotonically decreasing KL divergence wrt to the base distribution along the integration path. In doing so the paper builds on previous work of NODE, RNODE and CNFs and adds to the collection of papers around stabilizing continuous normalizing flows. Through a bunch of experiments on tabular datasets and some experiments on MNIST images the paper shows that their method can indeed lead to flows that converge faster (less NFEs) and are more stable.\n", "summary_of_the_review": "I think the paper makes a good contribution to the bijective normalizing flows literature. It supports the claimed contribution with theory and appropriate empirical analysis.", "strength_and_weaknesses": "Strengths:\n\n- The authors have formulated a novel way of regularizing continuous flows that not only makes the flow stable but also improves efficiency (in terms of NFEs) and final performance (in terms of NLL)\n- The authors have backed their regularization functional with ample explanations and theoretical justification and followed it up with experiments that reflect the contributions they claim.\n\nWeaknesses:\n- While not specific to this work but continuous flows papers in general should discuss an important limitation: scaling to high-dimensional datasets like ImageNet or FFHQ. The authors show some results on MNIST but I think the stronger case should be made on high-dimensional images. If not, commenting on a possible roadmap to get there might also be useful.\n- I am unsure how the computational cost compares wrt to previous work. From Algorithms 1 and 2 it seems like integrating augmented states for z_t and then for log(p_t(z_t)) should incur a higher computational cost. It would be better if the authors could discuss the overall compute cost as well and not just NFE.\n- Another thing to discuss is more theoretical: why should there exist a map from the data distribution to a base distribution that has the ascent continuous property? The topology of the support of the data distribution may not be homeomorphic to a ball in high-dimensional space (in case of Gaussian) ? Moreover even if it is homeomorphic, it is not obvious to me why should an ascent-continuous map always exist.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster