Inflationary Flows: Calibrated Bayesian Inference with Diffusion-Based Models
Authors: Daniela F De Albuquerque
John Pearson
Venue: NeurIPS 2024 poster (2024)
Keywords: Deep Generative Models
Diffusion-Based Models
Probability Flow ODE
Inference
Bayesian Inference
Calibrated Inference
Compression
Dimension Reduction
Primary area: probabilistic_methods
TL;DR: In this work we introduce Inflationary Flows, a class of highly expressive generative models that allows us to map complex data distributions to uniquely defined and lower-dimensional latent spaces while also affording principled Bayesian inference.

Abstract: Beyond estimating parameters of interest from data, one of the key goals of statistical inference is to properly quantify uncertainty in these estimates. In Bayesian inference, this uncertainty is provided by the posterior distribution, the computation of which typically involves an intractable high-dimensional integral. Among available approximation methods, sampling-based approaches come with strong theoretical guarantees but scale poorly to large problems, while variational approaches scale well but offer few theoretical guarantees. In particular, variational methods are known to produce overconfident estimates of posterior uncertainty and are typically non-identifiable, with many latent variable configurations generating equivalent predictions. Here, we address these challenges by showing how diffusion-based models (DBMs), which have recently produced state-of-the-art performance in generative modeling tasks, can be repurposed for performing calibrated, identifiable Bayesian inference. By exploiting a previously established connection between the stochastic and probability flow ordinary differential equations (pfODEs) underlying DBMs, we derive a class of models, \emph{inflationary flows,} that uniquely and deterministically map high-dimensional data to a lower-dimensional Gaussian distribution via ODE integration. This map is both invertible and neighborhood-preserving, with controllable numerical error, with the result that uncertainties in the data are correctly propagated to the latent space. We demonstrate how such maps can be learned via standard DBM training using a novel noise schedule and are effective at both preserving and reducing intrinsic data dimensionality. The result is a class of highly expressive generative models, uniquely defined on a low-dimensional latent space, that afford principled Bayesian inference.

Inflationary Flows: Calibrated Bayesian Inference with Diffusion-Based Models
Daniela Albuquerque
John Pearson
Department of Electrical & Computer Engineering School of Medicine Duke University Durham, NC 27708
Department of Neurobiology
Department of Electrical &
Computer Engineering
Center for Cognitive Neuroscience
Duke University
daniela. de. htt.......buuuuuuuuuuueeee.eeee. edu
Durham, NC 27708
john pearsondduke. edu
Abstract
Beyond estimating parameters of interest from data, of the key goals of statisti- cal inference is to properly quantify uncertainty in these estimates. In Bayesian inference, this uncertainty is provided by the posterior distribution, the computation which typically involves an intractable high-dimenssonal integral. Among avail- able approximation methods, sampling-ben baaeed approaches come with strong theo- retical guarantees but scale poorly to large problems, while variational approaches scale well but offer few theoretical guarantees. In particular, variational methods are known to produce overconfident estimates of posterior uncertainty and are typically non-identifiable, with many latent variable configurations generating equivalent predictions. Here, we address these challenges by showing how diffusion-basee models (DBMs), which have recently produced state-of-the-arr performance in generative modeling tasks, can be repurposed for performing calibrated, identifiable Bayesian inference. By exploiting a previously established connection between the stochastic and probability flow ordinary differential equations (pfODEs) underlying DBMs, we derive a class of models, inflationary flows, that uniquely and determin- istically map high-dimensional data to a lower-dimensional Gaussian distribution via ODE integration. This map both invertible and neighborhood- pewooooddppeeevingg.. with controllable numerical error, with the result that uncertainties in the data are correctly propagated to the latent space. We demonstrate how such maps can be learned via standard DBM training using novel noise schedule and are effective at both preserving and reducing intrinsic data dimensionality. The result a class of highly expressive generative models, uniquely defined low-dimensional latent space, that afford principled Bayesian inference.
Introduction
In many fields science, the aim of statistical inference is not only to estimate model parameters of interest from data but quantify the uncertainty in these estimates. In Bayesian inference, for data x generated from latent parameters z via a model p(x/z), this information is encapsulated in the posterior distribution p(z]x0 x), computation of which requires evaluation of the often intractable normalizing integral p(x) = fp(x, z) dz. Where accurate uncertainty estimation required, the gold standard remains sampling-be based Markov Chain Monte Carlo (MCMC) methods, which guaranteed (asymptotically) to produce exact samples from the posterior distribution [I] However, MCMC methods can be computationally costly and do readily scale either large high- dimensional data sets.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Alternatively, methods based on variational inference (VI) attempt to approximate posterior distri- butions by optimization, minimizing some measure of divergence between the true posterior and a parameterized set of distributions (x(2x)) (2]. For example, methods like the variational autoencoder (VAE) 3 4 minimize the Kullback-Leibler (KI.) divergence between true and approximate posteri- ors, producing bidirectional mappings between data and latent spaces. In vanilla VAEs, posterior uncertainty estimates are typically overconfident due to minimization of the reverse (mode-secking) KL divergence 6 While some lines of work have sought to mitigate this posterior mismatch prob- lem by utilizing different divergences 77100 VAEs still tend to produce blurry data reconstructions and non-unique latent spaces without additional assumptions [1..].
By contrast, normalizing flow (NF) models [4] work by applying a series of bijective transforma- tions to simple base distribution (usually uniform or Gaussian) to deterministically convert samples to desired target distribution. While NEs have been successfully used for posterior approximation [16-20] and produce higher-quality samples, the requirement that the Jacobian of each transforma- tion be simple to compute often requires high number of transformations and, traditionally, these transformations do not alter the the dimensionality of their inputs, resulting in latent spaces with thousands of dimensions. More recent lines of work on injective flow models 21425 address this limitation by allowing practitioners to use flows to learn lower dimensional manifolds from data, but most compression-capable flow models still fail to reach high generative performance on key benchmark image datasets (ef. (23)).
More recently, cestussion-bbbede models (DBMs) 26 33 have been shown to achieve state-of-the-art results in several generative tasks, including image, sound, and text-to-image generation. These models work by stipulating a fixed forward noising process (e.g.. a forward stochastic differential equation (SDE)), wherein Gaussian noise is incrementally added to samples of the target data distribution until all information in the original data is degraded. To generate samples from the target distribution, one then needs to simulate the reverse de-noising process (reverse SDE requires knowledge of the score of the intermediate "noised" transitional densities. Estimation of this score function across multiple noise levels is the key component of DBM model training, typically using a de- do-noising score matching objective [35 28 30). Yet. despite their excellent performance as generative models, DBMs, unlike VAEs or flows, do not readily lend themselves to inference. In particular, because DBMs use a diffusion process to transform the data distribution, they fail to preserve local structure in the data (Figure [T. and uncertainty under this mapping is high at its endpoint because of continuous noise injection and resultant mixing. Moreover, because the final SoaobbuttonnGuussan white noise of the same dimension must have higher entropy than the original data, there is no data compression.
(34) which
Finally, emerging work on flow matching models 46-42] has achieved impressive generative perfor- mance on several benchmark image datasets. Such models utilize simple conditional distribution families to learn vector field capable of transporting points between two pre-specified densities. These are closely related to the probability flow ODE (pfODE) view of DBMs, and, in fact, have been shown to be equivalent such models for specific choices of "interpolant" functions and conditional distributions. Despite their exceptional generative performance and deterministic nature, existing flow matching approaches do not allow for compression and, therefore, do not allow practitioners to infer a lower dimensional latent space from data.
Thus, despite tremendous improvements in sample quality, modern generative models do not lend themselves to one of the key modeling goals in scientific applications: calibrated Bayesian inference. Note that while many works focus predictive calibration, how well the inferred marginal p(x) matches real data (43 47), our focus here is on posterior calibration, how well q( matches the true posterior p( p(z/x) We address this challenge by demonstrating how novel DBM variant that call inflationary flows can, in fact, produce calibrated Bayesian inference in this sense.
Specifically, our contributions are: First, focusing on the case of unconditional generative models, we show how previously established link between the SDE defining diffusion models and the probability flow ODE (pfODE) that gives rise to the same Bockkrr-Planck equation (30) can be used to define a unique, deterministic map between the original data and an asymptotically Gaussian distribution. This map is bidirectional, preserves local neighborhoods, and has controllable numerical error, making it suitable for rigorous uncertainty quantification. Second, we define two classes of flows that correspond to novel noise injection schedules in the forward SDE of the diffusion model. The first of these preserves a measure of dimensionality, the participation ratio (PR) [48), based on

---

Diffusion Model
Fokker-Planck Equation
Probability Flow ODE
"Inflation"
To Gaussian
NNossigg
"Generation"
Denoising"
Original Data
f(x, G(x.0)
Σ [a1
dx ((x)))) 1 - (Gxx))
Σ (ul
f00.0000000 "V..eeGGGovvvveeeeeoo Valoents)
Figure 1: SDE-ODE Duality of diffusion-based models. The forward (noising) SDE defining the DBM (left) gives rise sequence of marginal probability densities whose temporal evolution is described by Fokker-Planck equation (FPE, middle). But this correspondence is not unique: the probability flow ODE (pfODE, right) gives rise the same FPE. That while both the SDE and the pIODE possess the same marginals, the former is noisy and mixing while the latter is deterministic and neighborhood-pee preeoediie...igg...... Both models require knowledge of the score function Vx log pe (x), which can learned by training either model.
second-order data statistics, preventing an effective increase in data dimensionality with added noise, while the second flow reduces PR, providing data compression. We demonstrate experimentally that inflationary flows indeed preserve local neighborhood structure, allowing for sampling- based uncertainty estimation, and that these models continue provide high-quality generation under compression, even from latent spaces reduced to as little as 0.03% of the nominal data dimensionality. As result, inflationary flows offer excellent generative performance while affording data compression and accurate uncertainty estimation for scientific applications.
2 Three views of diffusion-based models
As with standard DBMs, assume a data distribution (x Pdaca (x) at time t = 0. transformed via a forward noising process defined by the stochastic differential equation [c.g.. [c.g..26.77 28):
dx = f(x, t)dt + G t) . .dW,
(1)
with most DBMs assuming linear drift (f = (f)x) and isotropic noise (G (t) that mono- tonically increases over time [49]. As result, for 1f (T))d >> datan pr(x) becomes essentially
indistinguishable from an isotropic Gaussian (Figure I left). DBMs work by learning an approxima- tion the reverse SDE
34
28
30,500 50),
dx (f(x,t) V [G tx G(x,t) Vx (x))) G(x,t) dW, (2)
where W is time-reversed Brownian motion. In practice, this requires approximating the score function Vx log (x) by incrementally adding noise according to the schedule (t)) of the forward process and then requiring that denoising by (2) match the original sample. The fully trained model then generates samples from the target distribution by starting with xy ~ A(00, (T) and integrating (2 in reversed time.
As previously shown, this diffusive process gives rise to series of marginal distributions px(x) satisfying Fokker-Planck equation (Figure I middle) [30 [3]49].
(x):(x)= Σ ()px(())))xxx)) +
Σ a,0,
Σ k
Galx, t) (x
(3)
3

---

where 0 0_ In the "variance preserving" noise schedule of [3] (3) has as its stationary solution ōg;" an isotropic Gaussian distribution. This "distributional" perspective views the forward process as means of transforming the data into an easy-to-sample form (as with normalizing flows) and the reverse process as a means of data generation.
However, in addition to the SDE and FPE perspectives, Song et al. 30 also showed that (3) is satisfied by the marginals of a different process with no noise term, the so-called probability flow ODE (pfODE):
dx - - (((x)))((x - G(x, (x)
Vx 1x }
}ar. dt.
(4)
Unlike (T), this process is deterministic,, and data points evolve smoothly (Figure I. right), resulting in a flow that preserves local neighborhoods. Moreover, the přODE is uniquely defined by f(x, t). t(,,)), and the score function. This connection between the marginals satisfying the SDEs of diffusion processes and deterministic flows described by an equivalent ODE has also been recently explored in the context of flow matching models 36 42 connection on which we elaborate in Section
In the following sections. show how this pfODE, constructed using a score function estimated by training the corresponding DBM, can be used to map points from (x)ta(x) to a compressed latent space in manner that affords accurate uncertainty quantification.
3 Inflationary flows
As argued above, the probability flow ODE offers means of deterministically transforming an arbitrary data distribution into a simpler form via score function learnable through DBM training. Here, we introduce a specialized class of pfODEs, inflationary flows, that follow from an intuitive picture of local dynamics and asymptotically give rise to stationary Gaussian solutions of (3).
We begin by considering sequence of marginal transformations in which points in the original data distribution are convolved with Gaussians of increasingly larger covariance C(t):
pt (x) (x(x) Nx(x; 0, C(t)).
It is straightforward to show (Appendix A.1) that this class of time-varying densities satisfies (3) when f = 0 and GG è. This can be viewed as process of deterministically "inflating" each point in the data set, or equivalently as smoothing the underlying data distribution on ever coarser scales, similar to denoising approaches DBMs [51 52). Eventually, if the smoothing kernel grows much larger than Eg. the covariance in the original data, total covariance E(t = Eo C(l) C(L), p.(x) R NO0 C())), and all information has been removed from the original distribution. However, because it is numerically inconvenient for the variance of the asymptotic distribution Poo (x) grow much larger than that of the data, we follow previous work in adding a time-dependent coordinate rescaling x(t) = Att) x(t) 30 49). which results in an asymptotic solution Poc (x) = -O, AEAT) T) of the corresponding Fokker-Planck equation when Σ = and
λEAT + AΣÄT 0 (Appendix A.2). Together, these assumptions give rise to the pOODE (Appendix A.3
dx
dt A(t)
(100)(116 (-20(t)
Vx (=x + (A(t) (()))A-'(1)) x
(6)
where the score function is evaluated x -11 x. Notably, (6) is equivalent to the general pfODE form given [4 in the case both C(t) and (1) are isotropic (Appendix(A.4). with C(t) playing the role of injected noise and A (t)) the role of the scale schedule. In the following sections, we will show how to choose both of these in ways that either preserve or reduce intrinsic data dimensionality.
3.1 Dimension- flows
In standard DBMs, the final form of the distribution pr(x) approximates an isotropic Gaussian distribution, typically with unit variance. As a result, these models increase the effective dimension- ality of the data, which may begin low-dimensional manifold embedded within Bd Thus, even

---

Dimension Preserving Toy Simulations
Sine Moons Circles
Curve
Swirl
Inflation
Generation
Infiation
Generation
Figure Dimension-preserving flows for toy datasets. Numerical simulations of dimension- preserving flows for five sample toy datasets. Left sequences of sub-panels show results for inte- grating the pfODE forward time (inflation); right sub-panels show results of integrating the same system backwards in time (generation) (Appendix B.3). Simulations were conducted with score approximations obtained from neural networks trained on each respective toy dataset (Appendix
B.4.1
maintaining intrinsic data dimensionality requires both definition of dimensionality and a choice of flow that preserves this dimension. In this work, consider particularly simple measure of dimensionality, the participation ratio (PR), first introduced by Gao et al. [48]:
tr(E)² E2 Σ.
PR(E)
where is the covariance of the data with eigenvalues {o?)- PR is invariant to linear transforms of the data, depends only second-order statistics, is when Σ is rank-1, and is equal to the nominal dimensionality d when x ldxd. In Appendix C.1 we report this value for several benchmark image datasets, confirming that in all cases, PR is substantially lower than the nominal data dimensionality.
To construct flows that preserve this measure of dimension, following 5), we write total variance as E(t) = diag(o²(t)) C(t) E., where Eg is the original data covariance and C(t) is our dime-dependent smoothing kernel. Moreover, will choose C(t) be diagonal in the eigenbasis of Eo and work in that basis, in which case E(t) = ( and we have (AppendixAA.6)
(1 11-PR (1-R)(0)) Σ
dPR 0
do 0.
(8)
The simplest solution to this constraint is proportional inflation, rescaling along each principal axis:
d (a = pp?? along with
di
Aoj
Aoj -pt/2
0āg (t) - -
obglett
Aty(t)
(t
As with other flow models based on physical processes like diffusion (66 or electrostatics
54 our
use of the term inflationary flows for these choices is inspired by cosmology, where similar process of rapid expansion exponentially suppresses local fluctuations in background radiation density [55]. However, as result of our coordinate rescaling, the effective covariance Σ AEA" = diag(A)) (Ag)) remains constant (so 2ō2 0 trivially), and the additional conditions of Appendix A.2 are satisfied, such that (0, Σ) is a stationary solution of the relevant rescaled Fokker-Planck equation. As Figure 2 shows, these choices result version of (6) that smoothly maps nonlinear manifolds Gaussians and can be integrated in reverse to generate samples of the original data.
3.2 Dimennion--rdducing flows
In the previous section, we saw that isotropic inflation preserves intrinsic data dimensionality as measured by PR. Here, generalize and consider anisotropic inflation at different rates along each of the cigenvectors of E: di ( pg o² In addition, denote g. = max(g), so that the

---

Dimension Reducing Toy Simulations
Sine Moons Circles
Curve
Swirl
Inflation
Generation
Inflation
Generation
Figure Dimension-reducing flows for toy datasets. Numerical simulations of dimension dimension-reducing flows for the same five datasets in Figure 2 For 2D datasets, we showcase reduction from two to one dimension, while 3D datasets are reduced to two dimensions. Colors and layouts are the same in Figure 2 with scores again estimated using neural networks trained each example. Additional results showcasing (1) similar flows further compressing two-dimensional manifolds embedded in D 3 space, and (2) effects of adopting different scaling schemes for target data given Appendices C.2.2 and C.2.3 respectively.
fastest inflation rate is pg+. Then, if we take 9i g+ for f21, .ik and gi g- for the other dimensions,
PR(E()) C, E, s.Jpptz
C k= The )² Σj=1 oo,
(10)
which is the dimension that would be achieved by simply truncating the original covariance matrix in manner set by choice of Here, unlike in (9), we do not aim for rescaling to compensate for expansion along each dimension, since that would undo the effect of differential inflation rates.
Instead, we choose single global rescaling factor (tt) Ag exp(-pg ptlttpgt//// 22), leading Gaussian asymptotic solution with the original data covariance in dimensions E {i1 12..2...... ik).
Two additional features of this class of flows are worth noting: First, the final scale ratio of preserved to shrunken dimensions for finite integration times T is governed by the quantity e 99)) in (IO). For good compression, we want this number to be very large, but we show in Appendix A.4 this corresponds to maximum injected noise of order e )T/2 in the equivalent DBM. That is, the compression one can achieve with inflationary flows is constrained by the range of noise levels which the score function be accurately estimated, and this is quite limited in typical models. Second, despite the appearance given by (TO). the corresponding flow is not simply a linear projection to the top K principal components: though higher PCs are selectively removed by flows via exponential shrinkage, individual particles are repelled by local density as captured by the score function (6). and this term couples different dimensions even when C and A are diagonal. Thus, the final positions of particles in the retained dimensions depend on their initial positions in the full space, producing nonlinear map (Figure)
dimension-redee
4 Score function approximation from DBMs
Having chosen inflation and rescaling schedules, the last component needed for the pfODE (6) the score function s(x, f) Vx log px(x), Our strategy will be exploit the correspondence described above between diffusion models (I) and pfODEs { that give rise to the same marginals B That is, will learn approximation s(x, k, by fitting the DBM corresponding to our desired pfODE, since both make use of the same score function.
Briefly, in line with previous work DBMs [49], train neural networks to estimate de-noised version, Dxx, C(t)). for the correspondence between C(t) and noise). That is, we model D C(t)) using neural network and train it by minimizing a standard L2 de-noising crror:
noise-corrupted data sample x given noise level C(t) (cf. Appendix A.4
D(y + n: C(t)) y
(11)
De-noised outputs can then be used to compute the desired score term using V log C(I)
C-1(t) (x(x; C(t)) x) [30 49 Moreover, as in [49], we also adopt series of preconditioning

---

(A)
HMC Experiment Schematic
(B)
Dimension Preserving
(C)
Dimension Reducing
0.21
0254
1 Component Weight
1s Component Weight
Generation
0.250
0251
Inflation
0.28
650
0:319
0:301
Component Gaussian Mixture
Data Space
0th Component Weight
Component Weight
Figure 4: Calibration experiments. To assess error our posterior model estimates, we used Hamiltonian Monte Carlo (HMC) to perform inference in one of our toy datasets (2D circles). Drawing samples from a 3-component Gaussian Mixture Model (GMM) prior, we integrated the generative process backward in time obtain corresponding data space samples (A, components shown in orange, blue, and purple). We then used HMC to obtain posterior samples from the posterior distribution over the weights of the GMM components. (B, C) Kernel density estimates from the joint posterior samples over the mixture distribution weights in the dimension-preserving and dimension- reducing cases. Dashed vertical and horizontal lines indicate posterior means for each component. Reference ground-truth weights were w = [0.5, 0.25, 0.25).
factors aimed at making training with the above L2 loss and our noising scheme amenable to gradient descent techniques (Appendix(B..
Calibrated uncertainty estimates from inflationary flows
Several previous lines of work 43 have focused on assessing how well model-predicee marginals p(x) match real data (i.e., the predictive calibration case). Though we do compare our models' predictive calibration performance against existing injective flow models (Table 3). here we are primarily focused on quantifying error in unconditional posterior inference. That is, we are interested in quantifying the mismatch between inferred posteriors (zz/x) and true posteriors p(z p(z[x), especially contexts where the true generative model unknown and must be learned from data. This is by far the most common scenario in modern generative models like VAEs, flows, and GANs.
As with other implicit models, our inflationary flows provide deterministic link between complex data and simplified distributions with tractable sampling properties. This mapping requires integrating the PIODE (6) for given choice of C(t) and A(t) and an estimate of the score function of the original data. As a result, sampling- based estimates of uncertainty are trivial to compute: given a prior (x) over the data (e.g.. a Gaussian ball centered on particular example Xg). this can be transformed into an uncertainty on the dimension -edduude space by sampling (x, } ~ x(x) and integrating (6) forward to generate samples from p(xx (x,) J (Xx) dxy. As with MCMC. these samples can be used to construct either estimates of the posterior or credible intervals. Moreover, because the pfODE is unique given C, A, and the score, the model is identifubbee when conditioned on these choices.
The only potential source of error, apart from Monte Carlo error, in the above procedure arises from the fact that the score function used in (6) is only an estimate of the true score. To assess whether integrating noisy estimates of the score could produce errant posterior samples, we conducted the experiment showcased in Figure 4/A (Appendix [B.6). Briefly, we constructed Gaussian Mixture Model (GMM) prior with three pre-specdfied components (Appendix B.6) from which we drew samples of z, integrating backwards time using our trained pFODE networks to construct corresponding observed data points x. We then utilized Hamiltonian Monte Carlo (HMC) 56 to obtain posterior samples for the GMM component weights. As shown Figure 4B, C. the resulting posterior correctly covers the original ground-truth values, suggesting that numerical errors in score estimates, at least in this simplified scenario, do not appreciably accumulate. This is likely because, empirically, score estimates do not appear to be strongly auto-correlate in time (Appendix C.3
7

---

suggesting that $(x, is well approximated as scaled colored noise process and the corresponding pIODE as an SDE. In such a case, standard theorems for SDE integration show that while errors due to noise do accumulate, these can be mitigated by a careful choice of integrator and ultimately controlled by reducing step size [59] 60]. In addition, we verified this empirically in both low- dimensional examples (Figure 4 Appendices B.6 C.2.1) and with round-trip integration of the pIODE in high-dimensional datasets (Tables Appendix B.5.1)
Experiments
For the PR-Reducing flows, the final scale ratio between preserved vs. shrunken dimensions for finite integration times is dependent on the quantity eo(a. >,) Therefore, for fixed end integration time T and rate p. this scaling is dictated by 9. It. which we call the "inflation gap" (IG), Appendix B.2] As this inflation gap increases, compressed dimensions are shrunken to greater extent, and the denoising networks are required amortize score estimation over wider noise scales, harder learning problem. Therefore, for our proposed model, compression should be understood both in terms of the number of dimensions being preserved and the size of this inflation gap.
To assess how these two factors affect model performance, we performed two sets of experiments on two benchmark image datasets (CIFAR-10 10 [6]] and AFHQv2 [62]: Appendix B.4.2 code:
63 :: project website: (64)). In the first set of experiments, we fixed T, p. and the inflation gap (IG 1.02) while varying only the number of preserved dimensions d between d (compression to z 0.03%) and d = 3072 (no compression) for both datasets. For the second set of experiments, we worked with the AFHQv2 dataset and fixed T, p. and 2, while varying the inflation gap (IG = 1.10, 0, 1.25, 1.35, 1.50). In Tables T and 2 we showcase Frechet Inception Distance (FID) scores 65 (mean 4:20 over 3 independently generated sets of images, each with 50,000 samples) and round-trip integration mean squared errors (mean MSE 4:2o over 3 randomly sampled sets of images, each with 10,000 samples) for each (d, IG) combination explored (Appendices [B5.1 B.5.2 B.5.4). Figures 5 6 and 7 showcase 24 randomly generated images (top rows) along with round-trip integration results for randomly sampled images (bottom rows), across select (d. IG) combinations.
Finally, we also compared our inflationary flows (IFs) model generative performance on CIFAR-10 against three existing injective flow model baselines (Appendix B.5.2) M--lows (21. Rectangular Flows (RFs) [22]. and Canonical Manifold Flows (CMF) (23 for different numbers of preserved dimensions (d 30, 40, 62). Table 3 showcases best FID scores (out of 3 independently gener- ated sets of images, each with 10,000 samples) for each such experiment. For these comparison experiments, we fixed 1G=1.02 when training our networks for the different values.
As general trend, increasing the number of preserved dimensions at a constant inflation gap led to improvements in generative quality (lower FID scores) and reduced MSE (Table D.. However, some schedules we assessed are not entirely consistent with this trend. We hypothesize this is at least partially due to variance arising from different network initializations for each schedule (Appendix ).5.3). as well as differences between the two datasets explored here. As expected, increasing inflation gap while maintaining the number of preserved dimensions leads to worsened generative performance (higher FID scores, Table 2). Finally, in terms of predictive calibration, our model provides substantial gains when compared to existing injective flow model baselines (Table 3)
Table 1: FID and round-trip MSE (mean =20) at 1.02 Inflation Gap (IG)
AFHQv2 FID
CIFAR-10 FID
Dimensions
MSE
Dimensions
MSE
12.65 + 0.07
1.47 + 0.07
20.76 + 0.09
1.07 + 0.10
2
11.95 + 0.06
+
2
21.29 + 0.04
0.11
30
13.64 1 0.02
0.13
30
23.36
0.14
2.21
62
14.05 : 0.18
4.12
62
23.30
+ 0.19
2.27
307
15.64
0.10
0.13
307
28.07
0.13
0.02
615
14.63
0.07
2.12
615
#4.49
0.09
1536
13.36 0.12
0.04
1536
17.44
0.06
3041
13.97 - 0.13
0.28
3041
16.60
0.00
3072
11.90 0.08
0.38 0.04
3072
17.01 0.10
0.22 +.00
8

---

PRP
to
PRR 30D
PRR 307D
Generation
Round-Trip
Figure 5: Generation and round-trip experiments for AFHQv2 at IG=1.02 and varying number of preserved dimensions. Top row: Generated samples for select flow schedules PrR-Preserving (PRP), IR-Reducing to 2D (~2 0.07%), 30D(= 1%), and 307D(> 10%), 1.02 IG. Bottom row: Results for round-trip experiments under same schedules. Leftmost columns are original samples, middle columns are samples mapped to Gaussian latent spaces, and rightmost columns are recovered samples.
Table 2: FID and round-trip MSE (mean +:20) for AFHQv2 varying Inflation Gaps (IG)
Dimensions
IG
FID
MSE
2
1.02
11.95
0.06
1.55
0.21
2
1.10
13.98
0.13
1.35 + 0.08
2
1.25
1.15
1.65 0.09
2
1.35
34.6840.33
0.18
2
1.50
107.64 0.43
0.11 0.02
7 Discussion
Here, we have proposed a new type of implicit probabilistic model based on the probability flow ODE (pfODE) in which is possible to perform calibrated, identifiable Bayesian inference on a reduced- dimension latent space via sampling and integration. To do so, we have leveraged a correspondence between pfODEs and diffusion-based models by means of their associated Fokker-Plance equations, and we have demonstrated that such models continue to produce high-quality generated samples even when latent spaces are as little as 0.03% of the nominal data dimension. More importantly, the uniqueness and controllable error of the generative process make these models an attractive approach cases where accurate uncertainty estimates are required.
One limitation of our model is its reliance on the participation ratio (7) as a measure of dimensionality. Because PR relies only second-order statistics and proposals (9) formulated in the data eigenbasis, our method tends to favor the top principal components of the data when reducing dimension. However, as noted above, this is not simply truncation to the lowest principal components, since dimensions still mix via coupling to the score function in (6).
Limitations:
Nonetheless, solutions to the condition (8) that preserve (or reduce) more complex dimensionality measures might lead to even stronger compressions for curved manifolds (Appendix C.2.2). and more sophisticated choices for noise and rescaling schedules 6 might lead to compressions that do not simply remove information along fixed axes, more similar to [66]. That is, we believe much more interesting classes flows are possible. A second limitation is that mentioned Section 3.2 and in our experiments: our schedule requires training DBMs over much larger ranges of noise than are typically used, and this results in noticeable tradeoff' in compression performance as the inflation gap and number of preserved dimensions are varied.

---

Table 3: FID score comparison with injective flows for CIFAR-10
Dimensions Preserved
IF's (IG=1.022
M-Flow
RFs
CMFs
30
23.3
541.2
544.0
532.6
40
24.3
535.7
481.3
444.6
62
23.2
280.9
280.8
287.9
This work draws on several related lines of research, including work on using DBMs as likelihood estimation machines [50, 67 31. relations with normalizing flows and hierarchical VAEs [67 33 68]. injective flow models 25] and generative flow networks [69]. By contrast, focus is on the use of DBMs to learn score functions estimates for implicit probabilistic models, with the ultimate goal of performing accurate posterior inference. In this way, it is also closely related to work
Related work:
that cast that process in terms of statistical inference and to models that use DBMs for de-blurring and in-painting [71 (2). However, this work distinct from several models that use reversal of deterministic transforms to train generative models (73776). Whereas those models work by removing information from each sample x. our proposal relies critically on adjusting the local density of samples with respect to one another, moving the marginal distribution toward Gaussian.
denoising models [51 52
66 70
Our work is also similar to methods that use DBMs to construct samplers for unnormalized distribu- tions 87) Whereas we begin with samples from the target distribution and aim to learn latent representations, those studies start with a pre-specified form for the target distribution and aim to generate samples. Other groups have also leveraged sequential Monte Carlo (SMC) techniques to construct types of denoising diffusion samplers for, e.g., conditional generation [82 228 While our goals are distinct, we believe that the highly simplified Gaussian distribution of our latent spaces may potentially render joint and conditional generation more tractable in future models. Finally, while many prior studies have considered compressed representations for diffusion models [85)88) typically in an encoder-decoder framework, the focus there has been on generative quality, not inference. Along these lines, the most closely related to our work here is [89]. which considered diffusion along linear subspaces as means of improving sample quality in DBMs. though there again, the focus was on improving generation and computational efficiency, not statistical inference.
Yet another line of work closely related ours is the emerging literature on flow matching 36-38 90) models, which utilize a simple, difeeeffferentiblee, "interpolant" function to specify conditional families of distributions that continuously map between specified initial and final densities. That is, the interpolant functions define flows that map samples from a base distribution (x(x) to samples from a target distribution PI (x). Typically, these approaches rely on a simple quadratic objective that attempts to match the conditional flow field, which can be computed in closed form without needing to integrate the corresponding ODE. As shown in Appendix A.5 the pfODEs obtained using our proposed scaling and noising schedules are equivalent to the ODEs obtained by using the "Gaussian paths formulation" from [36] when the latter are generalized to full covariance matrices. As result, our models are amenable to training using flow-matching techniques, suggesting that faster training and inference schemes may be possible through leveraging connections between flow matching and
optimal transport 40 42
38
Broader impacts: Works like this one that focus on improving generative models risk contributing to an increasingly dangerous set of tools capable of creating misleading, exploitative, or plagiarized content. While this work does not seek to improve the quality of data generation, does propose a set of models that feature more informative latent representations of data, which could potentially be leveraged to those ends. However, this latent data organization may also help to mitigate certain types of content generation by selectively removing, prohibiting, or flagging regions of the compressed space corresponding to harmful or dangerous content. We believe this is a promising line of research that, if developed further, might help address privacy and security concerns raised by generative models.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by NIH grants F30MH129086 (DdA) and IRFIDA056376 (JMP).
We also thank Eero Simoncelli for comments and discussion on an early version of this work.
References
[1] Christian Robert and George Casella. Monte Carlo Statistical Methods. Springer Texts in Statistics. Springer New York, New York, NY, 2004. ISBN 978-1-4419-1939-7 doi: 1001007/ 978-1-4757-4145-2 /ttp:///://iin aarrtigg/iinkkspringger.... 1ttp:/10..077/77--11111 httpp//r....///../0000011111111-
[2] David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational infereenc: review for statisticians. Journal of the American Statistical Association, 1 112(518):559-877 April 2017. arXiv: 1601.00670.
ISSN 0162- 1459, 1537-274X. doi:
2017
2017.
[3] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banjf, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL
http:// www.i.avvo............ .ww. /0ob/1312.. 6114
[4] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings the 31th International Conference on Machine Learning, ICMI. 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pages 1278-1286. JMLR.org, 2014. URL http://prroceddings mmlr. press//32/rezeeddee html htt.a//a......eeeee//e///eaaaaae
[5] Christopher Bishop. Pattern Recognition and Machine Learning. Springer, New York, 2006.
[6] Ryan Giordano, Tamara Broderick, and Michael Jordan. Covariances, robustness, and variational bayes. Journal of machine learning research, 19(51):1-49, 2018.
2018
[7] Simón Rodríguez Santana and Daniel Hernández-Lobate Adversarial a r-divergence minimiza- tion for bayesian approximate inference, Neuroccomputingg 471:260- 2022. doi: 10.101 NEUCOM.2000.00.077 URL BtL/https://dao org/10. htrg/10.1016// neucom. 2020. 0920..9.000 09.076
[8] Jacob Deasy, Nikola Simidjievski, and Pietro Li6. Constraining variational inference with geometric jensen-shannon divergence. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tice Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Pro Processing Systems 2020, NeurlPS 2020, December 6- 2020, virtual, 2020. URL /tLp cc httppp//000o/aaaa///111111112111
neurips.
Abstract.
[9] Tom Minka et al. Divergence measures and message passing. Technical report, Technical report, Microsoft Research, 2005.
[10] Yingzhen and Richard E. Turner. Rényi divergence variational inference. Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Gareet, ed- itors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain. heweiiinnnnnnnnuisseeee.e
pages 1073-1081, 2016. URL URtptttp:://rrcedddng neurips ctcpiiiss.ccppepee httpp/appeee/////e/eee/e TT55aa355ee1aa11222233333ff11
[11] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational au- toencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217. PMLR, 2020.
[12] Miles Martinez. and John Pearson. Reproducible, incremental representation learning with Rosetta VAE. In NeurIPS Bayesian Deep Learning Workshop, 2021. URI. http://
bayessiandeepLearming.ooo

---

[13] Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. In The Eleventh International Conference on Learning Representations, 2023. URL https
het.://eeneeetttommiai---------- nwie]]
[14] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57): 2021.
[15] Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker. Normalizing flows: An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence, 43(1 1):3644-9979, 2020.
[16] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1530-1538, Lille, France, 07-09 Jul 2015. PMLR.
mlr wwtpp////ee/eeeeneeeeee
html
[17] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
[18] Rianne van den Berg, Leonard Hasenclever, Jakub M. Tomezak, and Max Welling. Sylvester normalizing flows for variational inference. In Amir Globerson and Ricardo Silva, editors, Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pages 393-402. AUAI Press, 2018. URL ppwoceedings/pppppeeeeeeeeeeeeee /1ceee///16...0 1
http://auai... www./oor/g///i0. httpaaaa......aaa//////////ag/gp
[19] Christos Louizos and Max Welling. Multiplicative normalizing flows for variational bayesian neural networks. In International Conference on Machine Learning, pages 2218-227. PMLR, 2017.
[20] Jakub M. Tomezak and Max Welling. Improving variational auto-encoders using householder flow. CoRR, abs/161 I. 1.09630, 2016. URL
09630
[21] Johann Brehmer and Kyle Cranmer. Flows for simultaneous manifold learning and density estimation. Advances in neural information processing systems, 33:442-453, 2020.
[22] Anthony L Caterini, Gabriel Gonez,, Geoff Pleiss, and John P Cunningham. Rectangular flows for manifold learning. Advances in Neural Information Processing Systems, 34:30228- 30241, 2021.
[23] Kyriakos Flouris and Ender Konukoglu. Canonical normalizing flows for manifold learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http::/paperr..a.t.. www.eerrr..iii...mme..ee..ee nips cctccc/cpppec wttpssscccppppppppeee http:papprr__fer fileesffieee_eeeeeeeeeeeeeeeeeer wwllss/pppprr////e papp///ppp//// Seaaper/222//// 22022/22222//2// www.ecbbbb--bbttacacc Abstract- Conference, html
[24] Rob Cornish, Anthony Caterini, George Deligiannidis, and Arnaud Doucet.
Relaxing bijectivity constraints with continuously indexed normalising flows. In International conference on machine learning, pages 2133-2143. PMLR, 2020.
[25] Edmond Cunningham, Adam D Cobb, and Susmit Jha. Principal component flows. In Interna- tional Conference on Machine Learning, pages 4492-4519, PMLR, 2022.
[26] Jascha
Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep un- supervised learning using nonequilibrium thermodynamics.. In Francis R. Bach and David M. Blei, editors, Pruceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, Frunce, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceed- URL. sttp://proceedingg mlr. prwss/ vv7 wtt.gggs.w.........eee.mmmmese
ings, pages 2256 2265-2265. R.rg, 2015.
html
12

---

[27] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020. URL httpps://pprcceediggss wwtgggs..euriipe nwuripu cc/ wwtpppppp///////eeee/eeepee Abt.rebbb-bbbtttt...h.h..
[28] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence d' A'Aché--uue, Emily B. Fox, and Roman Garett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8-14, 2019. Vancouver, BC, Canada. pages 119999--1901, 2019. URL ptttps://proceedingg hew.iiiinnnnn..isnnseee.e... neurips..... cttpiiisscccpppppe hccppappr/221///aaa 10001e22577400a53377996666a228
[29] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12. 2020, virtual, 2020. URL httpp:://pooceeeiio hwwpooeeednnngnnnnriip.p...... neurips cc/pappr/2020/ /aahh// 92c3b91631 $22Db118111455176229766633333333 Abstraattddstttttt.t... .ttll
[30] Yang Song, Jascha
Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th In- ternational Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3- PXIIG112RRSS
7, 2020. OpenReview.net, 2021. URRLhtttps://pperreeee
[31] Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In Marc' Aurelio Ranzato, Alina Beygelz- imer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Ad- vances Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 1415-1428, 2021. URL bastlbbb.feocccooe/4066666622222
hewpeeeediinnn.euiiiisss.p neurips cttpiiss.ccppppaee wwtpcppppppppppteeetccccccceeaet panperr/222/hhaa 2021/hash/
[32] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image syn- thesis. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December URL httpo//000/.eseeacaaff.f.faaaaa.
6-14, 2021, virtual, pages 3780-8794, 2021.
neur sourips.com cc/
[33] Calvin Luo. Understanding diffussin models: A unified perspecttve.. CoRR, abs/2208. 11970, 11970. 2022. doi: 10448550/ARXIV.2000. 1970. http::ttt////:ooorrgg/ wtt.://///.....///r///..../ .208
11970
[34] Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic ProsicerrPeeeeee Processes and their Applications, 12(3):313-326, May 1982. ISSN 03044149. doi: 10.101603304441100888900011
[35] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation,
2011. doi: 10.1162/NECO, a_00122.
[36] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL htttps://openreviex.. net: http:/ppenrvieee.neee/frrrrore Piw.:mmmDdddPPPPD...SS...
[37] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL.
https://openreviees http://wwwwweeeeeeoooooo///dd/d
13

---

[38] Michael S. Albergo and Eric Sunden-Eijnden Building normalizing flows with stochastic interpolants. The Eleventh International Conference on Learning Representations. ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. https:///://opem
het/fooru21d=-11qqeBB0
[39] Nicholas M. Boffi, Michael Albergo, and Eric Vanden-Eijnden. Flow map matching. CoRR. abs/2406. 07507, 2024. doi: 1044550/AAXX1.224600777777 URL https://doi.org:: 48550 18650/arXiv 2406 07507
[40] Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector- Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. Trans. Mach. Learn. Res., 2024, 2024. URL
https://opeareiie..nes
[41] Alexander Tong, Nikolay Malkin, Kilian Fatras, Lazar Atanackovic, Yanlei Zhang, Guillaume Huguet, Guy Wolf, and Yoshua Bengio. Simulation-fiee schrödinger bridges via score and flow matching. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, International Conference on Artificial Intelligence and Statistics, 2-4 May 2024, Palau de Congressos, Valencia, Spain, volume 238 of Proceedings of Machine Learning Research, pages 1279-1287. PMLR, 2024. URL
mlr prwss/
pwwps/v238/ v238/y y tong24a......... 24t.htmllllmmm.mmmmmmmmmmmm..... html
[42] Aram-Alexande Pooladian, Heli Ben-Hamu, Carles Domingo-Enrinn Enrrch, Brandon Amos, Yaron Lipman, and Ricky T. Q. Chen. Multisample flow matching: Straightening flows with minibatch couplings. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 28 1000288127. PMLR, 2023. URL https:// .mlr vress/v202/pooladdi an23a html
[43] Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359 2007. doi: 10.1198/
016214506000001433
[44] Francis X Diebold and Robert Mariano. Comparing predictive accuracy. Journal of Business Economic Statistics,
2002. doi: 10. 10.11940073501000753344444 URL https://doi. org/ 10. 1198 1Tt.001180//70000077575110000000
[45] Andrey Malinin and Mark J. F. Gales. Predictive uncertainty estimation via prior networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, neurips
2018, Montréal, Canada, pages 7047-7058, 2018. http::t:s///:ooetto
[46] Jiayu Yao, Weiwei Pan, Soumya Ghosh, and Finale Doshi-Velez, Quality of uncertainty quantification for bayesian neural network inference. CoRR, Abs/1906.00966, 2019. URL
a9900 09686
[47] Iñigo Urteaga, Kathy Li, Amanda Shea, Virginia Vitzthum, Chris Wiggins, and Noemie Elhadad. A generative modeling approach to calibrated predictions: A use case on menstrual cycle length prediction. In Ken Jung, Serena Yeung, Mark P. Sendak, Michael W. Sjoding, and Rajesh Ranganath, editors, Proceedings of the Machine Learning for Healthcare Conference, MLHC 2021, 6-7 August 2021, Virtual Event, volume 149 of Proceedings of Machine Learning Research, pages 535-566 PMLR, 2021. CeLLhtts::// https::/pproceediinggeee//// mlr wttp://gw..mml..eess//sssas press/v149/ urteaga21a, html
[48] Peiran Gao, Eric Trautmann, Byron Yu, Gopal Santhanam, Stephen Ryu, Krishna Shenoy, and Surya Ganguli. A theory of multineuronal dimensionality, dynamics and measurement. bioRxiv, 2017. doi: 10.1 10.110./2.4 URL TRR.httpss:/// //wwps::/www.... biorxiv. www.://wwww.......tttatttat///// 12/21422
14

---

[49] Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-basel generative models. Advances in Neural Information Processing Systems, 35: 26565-26577 2022.
[50] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems,
2021.
[51] M Raphan and E P Simoncelli. Least squares estimation without priors or supervision. Neural Computation, 23(2):374 420, Feb 2011. doi: 10.1162/NECO a 00076. Published online, Nov 2010.
[52] Zahra Kadkhodaie and Eero P. Simoncelli. Stochastic solutions for linear inverse prob- lems using the prior implicit in a denoiser. In Marc' Aurelio Ranzato, Alina Beygelz- imer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Ad- vances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 13242-13254, 2021. URL http:ttss////cceeee//ii htw.oeeeedinnnnneiiip.p..p. cc/ http:/appppe////e//eeee 072203203aaaab00fo00000002220002 Abstratttaaaatttttttt.... html
[53] Yilun Xu, Ziming Liu, Max Tegmark, and Tommi Jaakkola. Poisson flow generative models. Advances in Neural Information Processing Systems, 35:16782-16795, 2022.
[54] Yilun Xu, Ziming Liu, Yonglong Tian, Shangyuan Tong, Max Tegmark, and Tommi Jaakkola. Pfgm++: unlocking the potential of physics- inspired generative models. In Proceedings of the 40th International Conference on Machine Learning. ICML'23, JMLR.org. 2023.
[55] Alan H Guth. Inflationary universe: A possible solution to the horizon and flatness problems. Physical Review D, 23(2):347, 1981.
[56] Adam D. Cobb and Brian Jalaian. Scaling hamiltonian monte carlo inference for bayesian neural networks with symmetric splitting. In Cassio P. de Campos, Marloes H. Maathuis. and Erik Quaeghebeur, editors, Proceedings of the Thirty-Sevente Seventh Conference on Uncertainty in Artificial Intelligence, UAI 2021, Virtual Event, 27-30 July 2021, volume 161 of Proceedings of Machine Learning Research, pages 675-685. AUAI Press, 2021. URL https: htttps://pproceedings. mll cress/v161/com cobbb1aa hhmml
[57] Tianqi Chen, Emily B. Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pages 1683-1691.JMM Jor.R.rrg 2014. URL htt.:tttp//prroeddnngs mlr. htw.p//ssmm.m....e.eee..e...eeee v32/ cheni14
html
[58] Matthew Hoffman and Andrew Gelman. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo. J. Mach. Learn. Res., 159311159,,, 2014. doi: 7..555/2627435.2638888
httpstss::// acm. org/doi//0 3555//627435..... 2638586
[59] Peter E Kloeden, Eckhard Platen, Peter E Kloeden, and Eckhard Platen. Stochastic differential equations. Springer, 1992.
[60] Wenlong Mou, Nicolas Flammarion, Martin Wainwright, and Peter L Bartlett. Improved bounds for discretization of langevin diffusions: Near-optimal rates without convexity. Bernoulli, 28(3) 2022.
[61] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[62] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[63] httee///gghhooccocommmm///////rp
2024.
[64] https://aamffa.iithun htt.:a/ny....iiiii..bi///ee.ee.e 2024.
15

---

[65] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler. and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 6629 6640, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
[66] Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from scometry-adaptive harmonic representations. In The Twelfth International Conference on Learning Representations, 2024. URL perttps://opereeee htt.//orrummiidd/iv////222222
[67] Chin-Wei Huang, Jae Hyun Lim, and Aaron C Courville. variational perspective on diffusion- based generative models and score matching. Advances in Neural Information Processing Systems, 34:22863- 22876,, 2021.
[68] Diederik P Kingma and Ruiqi Gao. Understanding diffusion objectives the ELBO with simple data augmentation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL http::::/tppereeaaaeeae/ http://ppaaaeeeeee...eeo/or///ap
[69] Nikolay Malkin, Salem Lahlou, Tristan Deleu, Xu Ji, Edward J Hu, Katie E Everett, Dinghuai Zhang, and Yoshua Bengio. GFlownets and variational inference. In The Eleventh International Conference on Learning Representations, 2023. URL htttps://operrett www.urrriiiee.eeeorr.... net/forum?
id=uKiEOVIluk
[70] 7ahra Kadkhodaie, Florentin Guth, Fero P Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representations. In The Twelfth International Conference on Learning Representations, 2024. URL htRts:tts::/// httph::ttp/:seeeee //tperevv//eeenevie http/forrummi-ANNww//..s
[71] Berthy T. Feng, Jamie Smith, Michael Rubinstein, Huiwen Chang, Katherine L. Bouman, and William T. Freeman. Score-based diffusion models as principled priors for inverse imaging. In 2023 IEEE/CVF International Conference Computer Vision (ICCV), page 14486-104977 Paris. France, October 2023. IEEE. ISBN 9798350307 184, doi: 10.1 10.100//CCV5100002223 00965. URL httphttt///eeeeepllrrs ieee. org/ htt////ottttt///////
[72] Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guine guided diffusion models for inverse problems. In International Conference on Learning Representations, 2023.
URL httppss///ppeereeiiiee http://wwwwweeefeeeeeeeeeeeeee wwwneeeet/emiiiee 9iteummmmiiiiie _g____MM
[73] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. URL https://ppeerreiie
[74] Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. Advances in Neural Information Processing Systems, 36, 2024.
[75] Severi Rissanen, Markus Heinonen, and Arno Solin. Generative modelling with inverse heat dissipation. In The Eleventh International Conference on Learning Representations, 2023. URL https:///penreviiet www.:/eeetwndddeeeeooe.... 4UBBT99F00
[76] Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. In The Eleventh International Conference on Learning Representations, 2023. URL http::///opprreeeeeeeeeeforrreee
id Dx0077522
[77] Julius Berner, Lorenz Richter, and Karen Ullrich. An optimal control perspective on diffusion- based generative modeling, Transactions on Machine Learning Research, 2024. ISSN 2835- 8856. URL /teL/atttt://peerevvier
[78] Lorenz Richter and Julius Berner. Improved sampling via learned diffusions. In The Twelfth International Conference on Learning Representutions, 2024. URL https: //ppnrvvveeeee het//forum?.cc-h4pllbbbbo0
16

---

[79] Francisco Vargas, Andrius Ovsianas, David Lopes Fernandes, Mark Girolami, Neil D Lawrence, and Nikolas Nüsken. Bayesian learning via neural Shhrddinge-fölllmen flows. In Fourth Sym- posium on Advances in Approximate Bayesian Inference, 2022. httph:ttss:/:oeeeeeeeeeee Ftt
[80] Xumpeng Huang. Hanze Dong, Yifan HAO, Yian Ma, and Tong Zhang. Reverse diffusion monte carlo. In The Tweifth International Conference on Learning Representations, 2024. URL https::/openreevles. wwwe/fooummmiee KIPEMMSPFV
[81] Curtis McDonald and Andrew Barron. Proposal of a score based approach to sampling using monte carlo estimation of score and oracle access target density. CoRR, abs/2212.03325. 2022. doi: (044550//RRXI.2222..5555 03325
htLps:tts::///o org/ 10. 48550/ 2212
[82] Angus Phillips, Hai- Dau, Michael John Hutchinson, Valentin De Bortoli, George Deli- giannidis, and Arnaud Doucet. Particle denoising diffusion sampler. CoRR, abs/2402.063200 2024. doi: (0448500ARXXIV2400.0666
URL //doi////oio org/ 11. 48550/855iii .402
06320
[83] Gabriel Cardoso, Yazid Janati el idrissi, Sylvain Le Corff, and Eric Moulines. Monte carlo guided denoising diffusion models for bayesian linear inverse problems. In The Twelfth Inter- national Conference on Learning Representations. 2024. URL htttt:://ppenrrett -idiumdddd-EESSwx.....
net/
[84] Brian L. Trippe, Luhuan Wu, Christian A. Naesseth, David Blei, and John Patrick Cunningham. Practical and asymptotically exact conditional sampling in difffusion models. In ICML 2023 Workshop on Structured Probabilistic Inference & Generative Modeling. 2023. URL https
//openreview. http:/eewweeeeeeeeeeeeee net/forummiii re33ddd22
[85] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in neural information processing systems, 34: 1287-11302, 2021.
[86] Andreas Blattmann, Robin Rombach, Huan Ling. Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22563-22575, 2023.
[87] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasom Suwajanakom. Diffusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10619-10629, 2022.
[88] Drew Hudson, Daniel Zoran, Mateusz Malinowski, Andrew Lampinen, Andrew Jaegle, James I. McClelland, Loie Matthey, Felix Hill, and Alexander Lerchner. Soda: Bottleneck diffusion models for representation learning. arXiv preprint arXiv:2311.17901, 2023.
[89] Bowen Jing, Gabriele Corso, Renato Berlinghieri, and Tommi Jaakkola. Subspace diffusion generative models. In European Conference on Computer Vision, pages 279-289. Springer, 2022.
[90] Michael S. Albergo, Nicholas M. Boffi, and Eric -ajndenEijnnen. Stochastic interpolants: A unifying framework for flows and diffusions. CoRR, abs/2303.08799 2023. doi: 10.48550/ ARXIV.2303.08797 URL https://////o org/10 10. 480.48850//4 48550/arXiv 2303 08797
[91] Nataraj Akkiraju, Herbert Edelsbrunner, Michael Facello, Ping Fu, EP Mucke, and Carlos Varela. Alpha shapes: definition and software. In Proceedings of the 1st international computational geometry software workshop, page 63-66, 1995.
[92] Herbert Edelsbrunner and Ernst Mücke. Three-dimensional alpha shapes. ACM Transactions On Graphics (TOG), 13(1):43-72, 1994.
[93] Fabian Pedregosa, Gael Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning in python. Journal of machine learning research, 12(0ct):2825-288)
2011.
17

---

A Appendix: Additional Details on Model and Preliminaries
A.1 Derivation of the inflationary Fokker-Planck Equation
We start with derivatives of the smoothing kernel <<x, () = 14,
1 tr(CC= + 2t (x- 1) H) T ---1-)) x(x,t)
0n
(12)
2
VK C ---1x--- (x- u)s
(13)
0,0,n |[ (x (x- (c-));y)+
(14)
and combine this with 5 calculate terms in
Oep po(x) 0,k(xx,) t)
(15)
---xxxx-x+----- - Po 1 jur (c- - (x u) -) -)
(16)
Σ a.fipll = -po *
- f 1))),a
(17)
Σ 0,0, 2
Σ
GuGGjp
= 2PO * Σ
0,0,
Σ GikGjh K
(18)
k
1j
20,
(x k))a
Σ GinGith 1
(x))(= (1)) -CC-
+
Assuming f = 0 and 0,G,a (x, 1) then gives the condition
1 1 20 (c (x- - 2 - +
(x- (x- (C) =
-C)==
-G
1 2 (x- m)(x (1)x-N)TCC GG" (19)
which is satisfied when GG
A.2 Stationary solutions of the inflationary Fokker-Plande Equation
Starting from the unscaled Bokker-Plancke Equation corresponding to the process of Appendix A.I
@xp((x) Σ (,,(t)),,,,((xx)).
(20)
introduce coordinates ă A(t) x, i leading to the change of derivatives
a aā; o, + oi āe Öt Ot =.lAA, (t)xz 5
(21)
(22)
[(0,A)A 1x),0 Ō.
(23)
Fāx 05. ōr. Cij,,0, Ois Dx, Ox,
(24)
Ai = 3r00
(25)
(26)
18

---

and the Bokker-Plancc's Equation
[[0,A)A-*]]]] + 0. p:(8) 2
/ (AČA (x
(27)
where ?;(x) = Pe (x(x) is simply written in rescaled coordinates. However, this not a properly
normalized probability distribution in the rescaled coordinates, so we define q(x, 8) J (x), which in turn satisfies
[(o,A)A-')]] + a. ã. log J ((x, _ (AĆA (ACAT)a&r 01020)1((R 8).
(28)
Now consider the time-dependent Gaussian density
(- (- Ap) (AEAT) Awl
q(x, E)
exp
(29)
(2 E A ATAA
with rescaling factor ((6) A (t)l We then calculate the pieces of (28 as follows:
Vg (AEA )) - Ap)y
0,0,g [AEA] ) (x (4) [(EAA)- (x-
AW),, 9- [(EAA] "
0, log J &, log AA T = ur(ā, log AAT) = ((AAT)-1 ( (8.A)AA++ (
1 Q.g
2 - (A ))q
+ qu (AT(AAAT) AH)
9 tr (X - AH)(E Ap) (AEAA T)-2
2
δ, log J
(AEAT) (AAAA 1) - ((AAAA)(AAAT))
=AAAAA (()A)AA))
(AEAA T)-1
A E- EE-'''--'''!
With these results, the left and right sides of (28) become
x 8, log A T. V â å, log f1g
- = "TEA Ap)q
2
+ A.A TA (AEAT) - Aply
2 tr ((& ( AR) (OAAAATT)))
log IAAT lg
+tr ((AAT)- GRAJAT+A(OAT)
9tr
2 ([(AAA)(AEAT))
+ !ir (IAEAT) ) 1( A - AR)T (ACAAAT(AZAT)
2
(AČAT Jw (AAAAAT(AAAA)
+ (x- T -ACAAT) (AEAT) -AU)
and qxx,7) is a solution when
AČAT(AEA AČAT (AEA 1)-1 1 -.
AČAT- = -O(AAA))
(30)
19

---

Thus, for to be a solution in the absence of rescaling (A = 1) requires Σ = C, and combining this with (30) gives the additional constraint
ÅEA T + AEAT- 0.
(31)
Finally, note that, under the assumed form of pt (x) given in (5). when C(t) increases without bound, q t) - (0, ACA T (t)) asymptotically (under rescaling), and this distribution is stationary when [(1) = AEAT - ACAT is time-independent and a solution to
A.3 Derivation of the inflationary pOODE
Here, we derive the form of the pEODE (6) in rescaled coordinates. Starting from the unscaled inflationary process (Appendix A.1 with f = 0 and GG (x, = C(t), substituting into gives the PFODE
dx dt
c(t) Vx (x)
(32)
As in Appendix A.2 the derivation, we start by parameterizing the particle trajectory using a worldline time r such that dt dr while A remains function of t. With this convention, the pFODE becomes
we again consider the rescaling transformation ğ A(t x. ==i=t t. To simplify
dē:
0e dt ai; dæy
+
(33)
dr
Oz, dr
Ot dr (Ax);
dæj + ØL
Aij dr
(34)
drj +
Aij dr
Σ
(ig A
Agiat
(35)
ik
dx
A dx
+ [(0,A) 8
(36)
dr
dr
A (-2 C (- 2
[(OAA) valopp()))))) + A- -
(37)
Two important things to note about this form: First, the score function Vx log (x() is caleulated in the unscaled coordinates. In practice, this is the form we use when integrating the pIODE, though the transformation to the scaled coordinates is straightforward. Second, the rescaling has induced second force due to the change of measure factor, and this force points inward toward the origin when A is contraction. This overall attraction thus balances the repulsion from areas of high local density due to the negative score function, with the result that the asymptotic distribution is stabilized.
More formally, recalling the comments the conclusion of Appendix A.2 when C(t) grows without bound in (5) Pt (x), the unsealed density, is asymptotically Gaussian with covariance C(I), and its rescaled form (xx, t) is a stationary solution of the corresponding rescaled Fokker-Plant Planck Equation. In this case we also have
dx
(EACO-I+A ( x x=0, 0,
(38)
dr 1-
where we have made use of (31) with Σ - That is, when the rescaling and flow are chosen such that the (rescaled) diffusion PDE has stationary Gaussian solution, points the (rescaled) flow ODE eventually stop moving.
A.4 Equivalence of inflationary flows and standard pIODEs
Here, we show that our PfODE in 66) is equivalent to the form proposed by 49 for isotropic C(t) and (t). We begin by taking equation (6) and rewriting it such that our score term is computed with respect to the rescaled variable x:
d& df
1 (-1 2
(
C. sx (C-.T.s((A-xX)) (A-))))))) + ((0AA))
A
(39)
20

---

where we have made use of the transformation properties of the score function under the rescaling, If we then choose C(1) (1)1 and (1) a(() (i.e., isotropic noising and scaling schedules), this becomes
dx
dt = v(0))6())(()) logp (a()):))
x
(40)
(t)
where we have dropped tildes x and t. This is exactly the same the form given in Equation 4 of if we substitute (xl)) + s(), c(t) + (().
49
A.5
Equivalence of inflationary flows and flow matching
Here, we show the equivalence of our proposed un-scaled 32 and scaled (37 pOODEs to the un- scaled and scaled ODEs obtained using the "Gaussian paths flow matching formulation from [36]. Here, we will use the convention of the flow-matching literature in which - 0 corresponds to the easily sampled distribution (e.g., Gaussian), while / corresponds to the target (data) distribution. In this setup, the flow (xo) is likewise specified by an ODE:
d 2b (xo)) = (1b
(41)
dt
where again, X1 is a point in the data distribution and xo ~ N/, 1). In [36,. the authors show that choosing
vx(x xx(x)) (x (x(x)) (x1) xx(x1)
(42)
with "dots" denoting time derivatives leads flow
vb = of (x1)xx + 14 (x1):
(43)
that is, conditionally linear transformation of the Gaussian sample Xg-
For our purposes, we can re-derive (42 for the general case where (x1x)) is no longer a scalar but matrix-valued function of x1 and time. That is, we rewrite (43) (equation 11 in (36) with full covariance matrix Ex
x, (x0) E (x)) xo + (x))..).
(44)
Similarly, we write
(x
ンラ (x1) E (x1) (x-4((x)) + is (x1),
(45)
from which it is straightforward to show that (41 is again satisfied.
This can be related to our pOODE (6) as follows: First, recall that, under the inflationary assumption (5) plus rescaling, our time-dependent conditional marginals are
p(x; x1) = (A xx, ACCA,)
(46)
which is equivalent to 44 with μ4 (x1) = A. x1. (x,) = C,A,A. Note that, here again, we have reversed our time conventions from the main paper follow the flow-matching literature: t 0 is our inflated Gaussian and = 1 is the data distribution. From these results, along with the constraint
(31) required for inflationary flows produce stationary Gaussian solution asymptotically, then have, substituting into (45
-
2
(47)
(48)
X = vx(xxxx A,A.AT, (x, - A + A.
(49)
A.C.A.T Vx, log logp(x/ + Á.A -1 Xt,
(50)
which is the pFODE 66) written in the rescaled form Gaussian paths flow matching approach for a particular choice of (matrix-valued) noise schedule and mean.
39
Thus, our inflationary flows are equivalent
21

---

A.6 Derivation of cemnension-preservvinn criterion
Here, for simplicity of notation, denote the participation ratio 7 by R(E) and let Σ diag(r) in its eigenbasis, so that
R(y) %) ,n
(51)
and the change in PR under change in covariance is given by
E,
(E, <))
dR(r))
dy
Σ
7kdry
(52)
Σj 7
(Σ₂
k
E. 1 Σ, - 7/
R(7)
Σ dr. 9/8
(53)
Requiring that PR be preserved (dR then gives (
Now, we would like to consider conditions under which PR not preserved (i.e., 8
does not hold).
Assume are given (t) (along with initial conditions ((0) and define
E (), (g)
R(t)=
(54)
so that
1 11-R(1) R(1)
7
.9=1 =
(55)
Σ, 9/.
by definition. Then we can rewrite
dR(y)
-.7%
1-R(t)
1 R(t)
7
. ((R()) R(t)) -R())
Σ, -
dt Σ.77
Σ, 9i
d
0 (R(7) R((t) (log) Σtの
dt
(R(()) - R())) d
dt (log Tr(C²))
(56)
In the cases we consider, flows are expansive (d(log TT(C²)) > 0), with the result that (56) drives R(y) toward Rt).. Thus, in cases where R(t) has an asymptotic value, the R(y) should approach this value as well. In particular, for our dimension-reducing flows, we have y = Pg @ 7 giving
R(t) (ET Ep
7o:)2
(57)
PΣ 9NTR
Σ
2
k= Yok
where i = 1...A K are the dimensions with 9i = g. and 7h (0) value of R(t) (and thus the asymptotic value of PR) is that of the covariance in which only the eigendimensions with g g. have been retained, as in (10).
Tok That is, the asymptotic
22

---

B Appendix: Additional Details on Model Training and Experiments
B.1 Derivation of Training preconditioning Terms
Following an extensive set of experiments, the authors of 49 propose a set of preconditioning factors for improving the efficiency of denoiser training (II) that forms the core of score estimation. More specifically, they parameterize the denoiser network (x as
Da(x, a) = (o) Cour(a) F (o)x; Cnaise(a)).
(58)
where Fo is the actual neural network being trained and Cins Couts Cskip, and Cnoise are preconditioning factors. Using this parameterization of (x o), they then re-write the original L2 de-noising loss as
1
(D)) = E w(a) (y + n); Croise (r))
(y - Csksplo) (y + n)) (59)
Cout
where w(o) is also preconditioning factor, y is the original data sample, n is a noise sample and y n. As detailed in 49). these "factors" stabilize DBM training by:
1. Cin: Scaling inputs to unit variance across all dimensions, and for all noise/perturbation levels. This is essential for stable neural net training via gradient descent.
2. Cout: Scaling the effective network output to unit variance across dimensions.
3. Cakip:
Compensating for Cout, thus ensuring network errors are minimally amplified. The authors of [49] point out that this factor allows the network to choose whether to predict the target, its residual, or some value between the two.
4. w()):: Uniformizing the weight given to different noise levels in the total loss.
5. Cnoise: Determining how noise levels should be sampled during training so that the trained network efficiently covers different noise levels. This is the conditioning noise input fed to the network along with the perturbed data. This quantity determined empirically.
In [49] the authors propose optimal forms for all of these quantities based on these plausible first principles (cf. Table and Appendix B.6 of that work). However, the forms proposed there rely strongly the assumption that the noise schedule is isotropic. which does not hold for our inflationary schedules, which are diagonal but not proportional to the identity. Here, we derive analogous expressions for our setting.
As in the text, assume we work the eigenbasis of the initial data distribution Eo and let C(t) di))(r(t)) be the noising schedule, such that the data covariance at time t is Σ(t) = Eo + C(t). Assuming noise-dependent weighting factor A(t) analagous to /w(o) above, then rewrite
as
C(D,) Eyy. (I) + y()( - =)
(60)
Ex,y. [I))(() (Cout (C(Cin (y + n); Cnoise) - (y Cskip + n)))
(61)
= Eryy [l)
+ n); Cnoise) C- - Cskip + n))) [?]
) -
(62)
This clearly generalizes 59 by promoting all preconditioning factors either to matrices (Cin: Cout: Cskip, A) or vectors (Cnoise). We now derive forms for each of these precondition- ing factors.
B.1.1 Cin
The goal is to choose Cjn such that its application to the noised input + has unit covariance:
Vary,n in (y + n)]
(63)
Cin Vary. n [[y + n)] C.n.
(64)
Cin (Eo + C(t)) C.
(65)
Et) Cj (
(66)
(67)
23

---

More explicitly, if W is the matrix whose columns are the eigenvectors of Eo. then
Cin = Wdiag (1V08 + W
(68)
where the square root is taken elementwise.
B.1.2 Cout, Cakip
We begin by imposing the requirement that the target for the neural network should have identity covariance:
1 = Vary, Cout (I Cykip + n))]
(69)
Cout Cout Vary, [y Cskip (y n)]
Vary,n [[1 Cskiply Cskip
(1 Cskip) Hoo(1 - Cskip) + Cskip C(t)C skip
(70)
This generalizes Equation 123 Appendix 6.6 of 449.
Again by analogy with |49 we choose Cskip to minimize the left-hand side of 77
0 (1 Cskip) + Cskip C(t)
(71)
Eo = Cskip (()
(72)
C skip = Σ,Σ = Wding (o// (o) +())) W
(73)
which corresponds to Equation 131 in Appendix B.6
49
Using
73
in
70
then allows us to solve for Cout:
Cout C.ut = (1 EoΣ EoΣ Eo - E,E T + Eo
(74)
Eo -20 + EoΣ Eo + C)
(75)
Eo ΣoΣ-EE
(76)
= (Io' + (t)
(77)
Cout = Wdiag
( /1)/ ( +x(1) W
(78)
B.1.3 A(t)
Our goal in choosing Att) is to equalize the loss across different noise levels (which correspond, via the noise schedule, to different times). Looking at the form of (62) we can that this will be satisfied when A(t) is chosen to cancel the outermost factor of Cout
(t) C. E + Ct)(t) Wdiig (Voo²+ + +
(t) W
(79)
B.1.4 Re-writing loss with optimal preconditioning factors
Using these results, we now rewrite (2 using the preconditioning factors derived above:
C(D,) = Et,y n [|])(t)Cout (Fe(Cin (y + n); Cnoise) -C (y Cakip + +]]
|F & (t) (y + n); Cnoise) (2o' +c) (y-Eph (x= (y + m))
Et,y.
In practice, we precompute W and of via SVD and compute all relevant precoditioners in eigenspace using the forms given above. For Cnoise. we follow the same noise conditionining scheme used in the DDPM model (27). sampling uniformly from some interval t Li[t tmaxl and then setting Cnoise = (M 1)t, for some scalar hyperparameter M. We choose M = 1000, in agreement with 49 27). After this, indicated above, our noise is sampled via n A/( C(t)) with
C(t) (d))
24

---

B.2 Construction of and its impact on compression and generative performance of PR-Reducing PSODEs
As highlighted in main text, for constant end integration time T and p, the final scale ratio between preserved and compressed dimensions is dictated by the quantity 9- - 9i. which we called the inflation gup (IG). Higher inflation gaps (IGs) lead to more stringent exponential shrinkage towards zero in
compressed dimensions (Tables 6 1T and worse off generative performance (Table 2
In PR-Reducing experiments, we set p and constructed by making all elements of g correspond- ing preserved dimensions equal to (i.c., 9preaerved = 9moa = 2) and all elements corresponding to compressed dimensions equal to Icompressed = 9min Ipreserved - IG (Tables 10p. Of note, for PR-Preserving experiments, all elements of g are set to (i.e., g IG 0) and we chose
2, such that all dimensions are inflated to the same extent and we match exponential constant used for preserved dimensions in RR-Reducing experiments.
B.3 Details of pFODE integration
B.3.1 pOODE in terms of network outputs
Here we rewrite the pOODE (6) in terms of the network outputs D x, diag(r (1))) learned during training and queried in our experiments. As described in Appendix B.1.4 and in line with previous DBM training approaches, we opt to use time directly as our network conditioning input. That is, our networks are parameterized as (xx, t). Then, using the fact that the score can be written in terms of
the network as 30 49
Vx
(D)((x))))
(80)
we rewrite 6 as
dx
AÈ [C (D(x xi] + [()AA) A "1 x
(81)
df
2
1
- AC (C - xxt))
A- x) + [0.AA A A- xx
(82)
2
1
i(t) 7(t)
(D
x 1)
x (t))
2 a()
@
Ox
(83)
where in the last line we have expressed (t) and ČC- in their respective eigenspace (diagonal) representations, where the divisions are to be understood element-wise. For PR- PR-Reducing schedules, this expression simplifies even further, since our scaling schedule becomes isotropic i.e., A() = (f)
B.3.2 Solvers and Discretization Schedules
To integrate (83). we utilize either Euler's method for toy datasets and Heun's 's method (see Algorithm 1) for high-dimensional image datasets. The latter has been shown to provide better tradeoffs between number of neural function evaluations (NFEs) and image quality as assessed through FID scores in larger data sets [49.
In toy data examples, we chose simple, linearly spaced (step size h 10-2) discretization scheme, integrating from = 0 to / Amar when inflating and reversing these endpoints when generating data from the latent space. For higher-dimensional image datasets (CIFAR-10, AFHQv2), we instead discretized using & = x - e,) + E, when inflating, where tmaz is again the maximum time at which networks were trained to denoise and c, 10-2, similar to the standard discretization scheme for VP-ODEs [49] (though we do not necessarily enforce tmaz = 1). When generating from latent space, this discretization is preserved but integration is performed in reverse.
B.4
Training Details
B.4.1
Toy DataSets
Toy models were trained using a smaller convolutional UNet architecture (ToyConv and our proposed preconditioning factors (Appendix B.1). For all toy datasets, we trained networks both
25

---

Algorithm 1 EEign--Basie přODE Simulation using Heun's 2nd order method
1: procedure HEUNSAMPLER(D, t), 7(t), a(t), WT, (o......,
2: if running "generation" then
Generate initial sample at to
3:
Xo ~ (0,, diag((a(to) ((to)))
Sample from Gaussian latent space
4: else
i.e., if running "inflation" Sample from target distribution
5:
xo ~ Pdota (x)
6:
xo -a( xp)
Transform to eigenbasis, scale
7:
end if
8:
for i .0.,, N do:
> Solve equation 83
N times
9:
d 6 -1(44)
C
7(+)
tit.
D
artt)s 1s)
x (x)
10:
+ axti))
a(t))
@ x
b Evaluate at at t,
11:
x < x + (ti+1 - t.) t.)
Xi+1 = a(f+1))
S++1
> Take Euler step from t; to f
12:
àz <- 1 (1i+1) c +((++1
oc(li11)> (1n1)++1))))
R1-1 a(fi
7(1111)
13:
+ alf+1)
d(f++))
@ X
di b Evaluate
2 at 4:41
14:
8141 < x; + (1++1 - 4i) (1d. + =
Apply trapezoidal rule at 4++1
15:
return öN
Return Sample
16: end procedure
by using original images as inputs (i.e., "image space basis") or by first transforming images to their PCA representation (i.e., "eigenbasis"). Networks trained using either base choice were able to produce qualitatively good generated samples, across all datasets. For all cases, we used a learning rate of 10-5, batch size of 8192, and exponential moving average half-life of 50 x 10a For PR-
Reducing schedules, we set p = and constructed g as described Appendix B.2 (Table 5). The only exceptions were networks used on mesh and HMC toy experiments (Appendices C.2. B.6). where we used instead Spreserved = 1.15 across all preserved dimensions (circles, S-curve) and Scompressed = 0.85 (circles), or 9compressed = 0.70 (S-curve) Table 2nd and 6th rows. This yields softer effective compression (i.e., smaller IGs) and is needed to avoid numerical instability in these experiments.
As explained in Appendix B.1 to construct our CRoise preconditioning factor, we sampled t ~ (1mminne/mas with byin = 10 across all simulations and maaz equal the values shown in in (49).
Table 4 In the same table, we also show training duration (in 106 images (Mimgs), along with both the total number of dimensions (in the original data) and the number of dimensions preserved (in latent space) for each dataset and schedule combination. In Table 6 we showcase latent space (i.e., end of "inflation") compressed dimension variances achieved for the different toy PR-Reducing experiments function of inflation gap (IG). As expected, higher IGs lead to more stringent shrinkage of compressed dimensions in latent space.
B.4.2 CIFAR-10 and AFHQv2 Datasets
For our image datasets (i.e., CIFAR-I0 and AFHQv2), we utilized similar training hyperparameters to the ones proposed by [49 for the CIFAR-10 dataset, across all schedules explored (Table (7).
26

---

Table 4: Toy Data Training Hyperparameters
Dataset
Schedule
Total Dimensions
Dimensions Kept
tmax (s)
Duration (Mimg)
Circles
PRP
2
2
7.01
6975
Circles
PRR
11.01
8601
Sine
PRP
2
7.01
12288
Sine
PRR
2
1
11.01
12288
Moons
PRP
8.01
6400
Moons
PRR
2
1
11.01
8704
S
Curve
PRP
3
3
9.01
6144
S Curve
PRR
2
15. 01
5160
Swirl
PRP
3
3
11.01
8704
Swirl
PRR
2
15.01
12042
Table 5: 9: Values for Preserved vs. Compressed Dimensions for Toy Experiments.
Dataset
Schedule
Dimensions Kept
IG
Spreserved
Icompressed
Circles
PRR
2.0
2.0
0.0
Cireles
PRR
0.3
1.15
0.85
Sine
PRR
I
2.0
2.0
0.0
Moons
PRR
I
2.0
2.0
0.0
Curve
PRR
2
3,0
2.0
-1
Curve
PRR
2
0.45
1.15
0.70
Swirl
PRR
2
3.0
2.0
-1
Shown in Tables 8 are our specific choices for the exponential inflation constant (p) and training duration (in 10 images Mimgs) for the main sets of experiments performed image datasets, namely (1) experiments with constant inflation gap (IG=1.02) and varying the number of preserved dimensions d on both datasets (Table 8). and (2) experiments with fixed d (d 2) and varying inflation gaps for the AFHQV2 dataset (Table 9). Here, training duration was determined for each schedule based when computed Frechet Inception Distance (FID) scores (65 stopped improving. We also showcase Table To the specific values used for elements of g corresponding to preserved vs. compressed dimensions different inflation gaps.
All networks were trained on the same DDPM++ architecture, as implemented in 149 and using proposed preconditioning scheme and factors in the standard (e.g., image space) basis. No gradient clipping or mixed-precision training were used, and all networks were trained to perform unconditional generation. We run training the image space basis (as opposed to in eigenbasis) because this option proved to be more stable in practice for non-toy datasets. Additionally, we estimate the eigendecompositionn of the target datasets before training begins using 50K samples for CIFAR-10 10 and 15K samples for AFHQv2. Based on our experiments, any sample size above total number of dimensions works well for estimating the desired eigenbasis.
Table 6: Toy Experiments Compressed Dimension Variance by Inflation Gap (IG)
Dataset
Schedule
Dimensions Kept
IG
Compressed Dimension Variance
Circles
PRR
2.0
4 10-7
Circles
PRR
0.3
1 x 10-2
Sine
PRR
2.0
4 10-7
Moons
PRR
I
2.0
4 -0
S Curve
PRR
2
3.0
2 10-12
S Curve
PRR
0.45
2.5 x 10-3
Swirl
PRR
2
3.0
2 10
27

---

Table 7: CIFAR-I0 & AFHQv2 Common Training Hyperparameters (Across All Schedules)
Hyperparameter Name
Hyperparameter Value
Channel multiplier
128
Channels per resolution
2-2-2
Dataset x-flips
No
Augment Probability
12%
Dropout Probability
13%
Learning rate
10
LR Ramp-Up (Mimg)
10
EMA Half-Life (Mimg) Batch-Size
0.5
512
Table 8: Training Duration (in Mimgs) and Exponential Inflation Constant (p) for Dimension
Reducing Experiments Using 1.02 Inflation Gap (IG) and Dimension Preserving Experiments (IG 0.0)
Dataset
Total Dimensions
Dimensions Kept
IG
Training Duration p
CIFAR-10
3072
1.02
300
AFHQV2
3072
1.02
250
CIFAR-10
3072
1.02
300
I
AFHQV2
3072
2
1.02
250
1
CIFAR-10
3072
30
1.02
300
AFHQV2
3072
30
1.02
450
CIFAR-10
3072
40
1.02
300
1
CIFAR-10
3072
62
1.02
250
AFHQV2
3072
62
1.02
450
CIFAR-10
3072
307
1.02
300
AFHQV2
3072
307
1.02
300
CIFAR-10
3072
615
1.02
450
AFHQV2
3072
615
1.02
450
CIFAR-10 10
3072
1536
1.02
300
AFHQV2
3072
1536
1.02
250
CIFAR-10
3072
3041
1.02
300
AFHQV2
3072
3041
1.02
200
CIFAR-10
3072
3072
0.00
275
2
AFHQV2
3072
3072
0.00
275
2
Times utilized to construct conditioning noise inputs to networks (Cnoise(()) were uniformly sampled (L ~ (Imiin (mma))) with Lmin = 10-7 and Imaz 15.01, across all experiments. For the AFHQv2 dataset, we chose to adopt a 32x32 resolution (instead of 64x64 as in (49) due to constraints on training time and GPU availability. Therefore, for our experiments, both datasets have total of 3072 (i.e., 3x32x32) dimensions.
Finally, training was performed in a distributed fashion using either 8 or 4 GPUs per each experiment (NVIDIA GeForee GTX TITAN X, RTX 2080) in a compute cluster setting. Generation (FID) and round-trip (MSE) experiments were performed on single GPU (NVIDIA RTX 3090, 4090, A5000, A6000). We report training duration in Mimgs and note that time needed to achieve 200Mimgs approximately days on 8GPUs (4 days GPUs) using hyperparameters shown in Tables 7 g This is in agreement with previous train times reported in 49 using an GPU distributed training set up.
28

---

Table 9:
Training Duration (in Mimgs) and Exponential Inflation Constant (p) for AFHQv2 Experi- ments Using Variable Inflation Gaps (IGs)
Total Dimensions
Dimensions Kept
IG
Training Duration p
3072
2
1.10
200
3072
2
1.25
250
I
3072
2
1.35
250
3072
2
1.50
200
1
Table 10: 9i Values for Preserved vs. Compressed Dimensions at Different Inflation Gaps (IGs)
Inflation Gap (IG)
Spreserved
Jcompressed
1.02
2.0
0.98
1.10
2.0
0.90
1.25
2.0
0.75
1.35
2.0
0.65
1.50
2.0
0.50
B.5
Details of Roundtrip MSE and FID calculation Experiments
B.5.1 Roundtrip Experiments
For image datasets (CIFAR-I0 10 and AFHQv2), we simulated full round-trips: integrating the pIODEs (6) forward in time map original images into latent space and then backwards in time reconstruct original samples. We run these round-trips for a set of 10K randomly sampled images, three times per each schedule investigated and compute pixel mean squared error between original and reconstructed images, averaged across the 10K samples. Values reported in Tables represent mean #2 standard deviations of pixel MSE between these three different random seeds per each condition. For pfODE integration, we used the discretization schedule and Heun solver detailed above (Appendix B.3.2). with bmax 15.01, 6, 10-2, and N 118 for all conditions.
B.5.2 FID Experiments
For image datasets, also computed Frechet Inception Distance (FID) scores 65 across inde- pendent sets of 50K random samples, per each schedule investigated. Values reported in Tables of random samples per each condi-
2 represent mean 22 standard deviations across these ton. Here again, we used the discretization scheme and solver described in Appendix [B.3.2 with Iaaz = 15.01, 6, = 10 and N 256 across all conditions. We chose N 256 here (instead of 118) because this provided some reasonable trade-off between improving FID scores and reducing total compute time.
To obtain our latent space random samples x(T" at time to T (i.e, at the start of generation) sample from diagonal multivariate normal with either (1) all diagonal elements being 1 (for PR-Preserving schedule) or (2) all elements corresponding to preserved dimensions being and all elements corresponding compressed dimensions being equal the same small value for given inflation gap (see Table
Table 11: Latent Space Compressed Dimensions Variance per Inflation Gap (IG), Both Datasets
Inflation Gap (IG)
Latent Space Compressed Dimensions Variance
1.02
2.15 10-7
1.10
6.00 10
1.25
6.80 x 10
1.35
1.50 10-9
1.50
1.76 x 10-10
29

---

For our CIFAR-I6 10 comparison experiments against existing injective flow models, we used the same implementations for M-Flow (2T] Rectangular Flows [22], and Canonical Manifold Flows 23 as in proposed Appendix G.1 of 23 for the CIFAR-10 dataset. The only difference here that we trained models with latent dimensions equal to d = [30, 40, 62). Finally, comparison FID scores reported in represen best score out of independently generated sets, each with 10K samples. For our comparison models, inflation gap was fixed to IG 1.02 while d was varied between 30, 40, and and we utilized the same training computed using the same discretization, solver, and general set up described above.
23
When training the comparison injective flows, we used the same
reported in Tables 7 FID scores were
B.5.3 FID and Roundtrip Integration Experiments with Additional Initialization Seeds
In the FID and roundtrip pfODE integration experiments in Tables 1 2 we showcased variation arising from using different seeds when constructing initial generation roundtrip samples. Another relevant source of variability in FID and MSE scores reported arises from different parameter initializations when training our denoiser networks. To assess this source of variability, trained networks using three additional initialization seeds for our worse performing schedule at constant inflation gap (IG = 1.02, PR-Reducing 307 dimensions), for both datasets (AFHQv2, CIFAR-10). For each such initialization seed, we conducted similar FID and roundtrip integration MSE experiments as detailed in Sections B.5.1 B.5.2
Tables 12 13 showcase results for these experiments on each individual seed tested (top 4 rows) and aggregated across all seeds (bottom row). For top 4 rows, values reported represent mean 2or of scores computed across three different sets of generation/roumddrpp seeds (as Tables I 2) For bottom rows, values reported represent mean #2 of scores computed across all initialization and generation/roundtrip seeds (i.e., these represent mean =o over all aggregated experiments conducted for the given schedule and dataset).
Table 12: AFHQv2, FID and Roundtrip Experiments with Additional Initialization Seeds
Seed
FID MSE
1
16.41 + 0.10
3.24 0.16
2
15.63 0.17
3.20 0.15
3
16.53 0.11
3.06
4
15.64 0.10
3.33 + 0.13
all (aggregated)
16.05 + 0.85
3.21 = 0.26
Table 13: CIFAR-10, FID and Roundtrip Experiments with Additional Initialization Seeds
Seed
FID
MSE
1
26.32 0.07
1.04 0.09
2
27.43 + 0.15
0
3
27.80 0.19
0.84 0.04
4
28.07 + 0.13
0.02
all (aggregated)
27.41 1.34
0.89 = 0.26
B.5.4 Additional Figures for FID and Round-Trip MSE Experiments Image Benchmark Datasets.
Below we showcase additional examples of generated images and results of roundtrip pIODE integra- tion for different schedules explored in experiments. More specifically, of FID and roundtrip integration experiments on CIFAR-10 at constant inflation gap (IG = 1.02) and varying number of preserved dimensions. Figure ssoweessss results of FID and roundtrip integration experiments done on the AFHQv2 dataset, with varying inflation gaps IG = [1.10, 1.25, 1.35, 1.50] and constant number of preserved dimesions (d 2).
shgura/sshwwassee results
30

---

PRP
PRR to 2D
PRR to 30D
PRR to 307D
Generation
Round- -rip
Figure 6: Generation and Round-Trip Experiments for CIFAR-10 at IG=1.02 and varying number of preserved dimensions. Layout and setup same for Figure 5 see Appendices B.5.2 B.5.1 for details.
1.10
1.25 IG
1.35IG
1.50IG
Generation
Round-Trip
Figure 7: Generation and Round-Trip Experiments for AFHQv2 dataset with dimension reduction to 2D (PRR to 2D) at different inflation gaps (IGs). Top Generated samples for each inflation gap (IG) flow schedule (1.10, 1.25, 1.35, and 1.50), all with = 2. Bottom row: Results of round-trip experiments for same schedules. Lefumost columns are original samples, middle columns are samples mapped to Gaussian latent spaces, and rightmost columns are recovered samples.
31

---

B.6 Details of Toy HMC Experiments
As highlighted in Section 5 we utilized Hamiltonian Monte Carlo (HMC) I 56 58 to assess if errors in our network score estimates could result in mis-calibrated posterior distributions. In these experiments, we worked with the toy 2D circles dataset (using both PR- PR-Preseg and PR- Reducing schedules) and began by constructing our observed data samples Xobs follows: First, we sampled a set of latent variables z from 3-component Gaussian Mixture Model (GMM) p(z)
Eizo (H, Σ,) with known means (11), diagonal covariances (E), and weights (w) (Table 14). Second, we integrated the sampled z points backwards in time ("generation") using proposed pfODES with score estimates taken from trained networks to obtain "noise-free" observed data samples XnI- Finally, we added small amount of isotropic Gaussian noise these samples (n NV, o²), x² = 10-2), to obtain our final observed data, Xobs-
Table 14: Ground-Trun Mcans, Covariance Diagonals, and Weights for Gaussian Mixture Model (GMM) Components Used in Toy HMC Experiments
GMM Component
Schedule
Mean
Covariance Diagonal
Weight
0th
PR-Preserving
[0.0, 0.0)
5.625 10-1 5.625 x 10-1
0.50
0th
PR-Reducing
10.0,0.0)
[5.625 10 10011..600 1.6.25 x 10-3
0.50
1st
PR-Preserving
[-5 x 10-2 0.0)
110 1.0)
0.25
1st
PR-Reducing
[-5 10 2.00
[10 10
0.25
2nd
PR-Preserving
5 x 10-2,0.0
[100, 10
0.25
2nd
PR- Reduucinn
5 x 10 2,0.0]
[1.0, 10
0.25
We then used these observations, Xobe. along with the HMC implementation provided in the hamiltorch library (56). to jointly sample from the posterior over ((z,), ), w). assuming {M,, E known.
For both PR-Preserving and PR-Reducing experiments, we generated 2000 samples (Xobs). For sampling, we used L = 15 steps per sampling trajectory, discarding the first 500 samples as "burn-in." Step sizes were 10-2 2 for PR-Preserving and 10-3 for PR- PR-Reducing schedules. Because sampling
required integration over the full generative trajectory and was slow to mix, requiring roughly 40 minutes per sample, we initialized our w and 2, estimates to ground truth values. other experiments, we verified that other initializations quickly converged to these values, but this procedure avoided numerical instabilities associated with integration of the generative pfODE during the burn-in in phase. Finally, to reduce sample autocorrelation, thinned the resulting chains by factor of 5.
As mentioned above, this procedure required multiple neural function evaluations (NFEs) for PIODE integration per HMC integration step. producing very long sampling times. For instance, using the single-GPU setup of hami ltorch required weeks to pass burn-in for our PR-preserving schedule and = 4 weeks for our PR-preserving schedule. As result, sample numbers were somewhat small (N = 1872, PR-preserving; N 1635, PR-reducing) and thinned traceplots still exhibited some consideabbe correlation (Figure underscoring the impracticality of using sampling-base based inference these models.
Appendix: Additional Experiments and Supplemental Information
C.1 Spectra and PR-Dimensionality for few common image datasets
Shown in Table 15 are participation ratio (PR) values for some benchmark image datasets. Figure showcases spectra (zoomed in to first 25PCs) for same image benchmarks.
C.2
Additional Toy Experiments
C.2.1
Toy Alpha-Shape/Mesh Coverage Experiments
To assess numerical error incurred when integrating our proposed pfODEs, we performed additional coverage experiments using 3D meshes and 2D alpha-shapes [9] 9 in select toy datasets (i.c., 2D
32

---

(A)
PR - Preserving
Chain
M m
0.595
0.5027
0.508G
MM
0.4907
AM
0.5085
6.0000
0.4009
25
30 75
180
325
150
175
25
50
75
300
125
150 175
25
130 175 56 75
200
125
Sample Number
Sample Number
Sample Number
(B)
PR - ReduRuu
Chain 1
Chain
Chain
0.69953
0.4020
mrr M
My WolnaHin
0.60915
0.4092
6.60875
0,4588
150
Sampla Number 375
Sample Numbar
Sample Number
Figure 8: Traceplots (post-thinning) for 3 random chains for PR-Preserving and PR-Reducing schedules. A: Traceplots for random PR-Preserving chains, after thinning by a factor of 5, "X axis" represents sample number and "Y axis" represents value of zeroth dimension of sample (wo).B: Same set up, only for 3 random PR-Reducing chains. Note that there is still some considerable correlation the samples, even after thinning. Additionally, mixing is not particularly good.
Table 15: Participation ratio (PR) for some commonly used image datasets.
Dataset
PR
MNIST
30.69
Fashion MNIST
7.90
SVHN
2.90
CIFAR-10
9.24
Here, we began by sampling 20K test points from a Gaussian latent space with appropriate diagonal covariance. For PR-Preserving schedules, this is simply a standard multivariate normal with either 2 or 3 dimensions. For PR- Pe-Recuging experiments, this diagonal covariance matrix contains I's for dimensions being preserved and a smaller value (10-2 for Circles, 2.5 x 10-3 for S-curve) for dimensions being compressed.
circles and 3D S-curve), Figure 10
Next, we sampled uniformly from the surfaces of balls centered at zero and with linearly spaced Mahalanobis radii ranging from 0.5 to 3.5 (200 pts per ball). We then fit either a 2D alpha-shape (2D Circles) or mesh (3D SCurve) to each one of these sets points. These points thus represent "boundaries" that we use to assess coverage prior to and after integrating our pfODEs. We define the initial coverage of the boundary to the set of points (out of the original 20K test points) that lie inside the boundary. We then integrate the pEODE backward in time (the "generation" direction)
33

---

Zoomed-In Spectra for Common Datasets
Log Explained Variance (Normatized)
MNIST
Fashion MNIST
SVHN
CIFAR-10
5
10 15
20
25
Principal Component
Figure 9: Zoomed-in spectra for some standard image datasets. Log of explained variance versus number of principal components (PCs) for 4 common image datasets (MNIST, Fashion MNIST, CIFAR- 10, and SVHN). We plot only the first 25 PCs across all datasets to facilitate comparison.
(A)
Experiment Schematic
(B)
Coverage
(C)
SCurve Coverage Probability Difference
Probability Difference
Dimension Preserving
1m-3
1.5 1e-2
Generation
Dimension Preserving
Initial Final Coverage
1.0
Dimension Reducing
Initial Final Coverage
1.0
Inflation
0.8
Probability
0.6
Probability
0.5
Dimension Reducing
Boundaries
Generation
-1.0
Inflation
Preserving Dimension Reducing
0.3
1.0
Latent Space
Data Space
Initial Coverage Probability
Initial Coverage Probability
(D)
Dimension Preserving
Dimension Reducing
Data Space
Latent Space
Data Space
Latent Space
5
Inflation
Inflation
-22
-22
Figure 10: Mesh/Alpha-Shape Calibration experiments. For select toy datasets, we numerically assessed coverage during the inflation and generation procedures using (3D) meshes and (2D) alpha- shapes. (A) We constructed fixed coverage sets by sampling data points at fixed Mahalanobis radii from the centers of each distribution and creating alpha shapes (2D) or meshes (3D). (B-C) We then quantified the change in coverage fraction for each of these sets at the end of either "inflation" or "generation" procedures. Lines represent means and shaded regions +2 standard deviations across three sets of random seeds. (D) Illustration of the effect of flows on set geometry. While both types of flows distort the shapes of initial sets, they do preserve local neighborhoods, even when one dimension is compressed by five orders of magnitude.
for each sample and boundary point. At the end of integration, we again calculate the mesh or 2D alpha-shape and assess the number of samples inside, yiclding final coverage numbers.
34

---

Front View
Side View
Front View
Side View
No Noise
Flat Manifold
Noised
No Noise
Curved Manifold
Noised
Start
End
Inflation
Figure 11: Additional PR-Preserving experiments for 2D data embedded in 3D space. Here we integrate our PR-Preserving pfODEs forwards in time (i.e., inflation) for different toy datasets, constructed by embedding the 2D Circles data in 3 dimensional space as either a flat (top rows) or curved (bottom rows) manifold. We present results for such simulations both without any added noise (1*t and 3rd rows) and with some small added noise (0.2 and 0.5 a for flat and curved cases, respectively 2nd and 4t rows).
Similarly, take our samples and boundary points the end of generation, simulate our pfODEs forwards (i.e., the "inflation" direction), and once again, use 2D alpha-shapes and meshes to assess coverages the end of this round-trip procedure. If our numerical integration were perfect, points initially inside these sets should remain inside at the end of integration; failure to do so indicates mis-calibration of the set's coverage. As shown in Figure 10 B-C), we able preserve coverage up to some small, controllable amount of error for both schedules and datasets using this process.
C.2.2 Toy Experiments on Datasets with Lower Intrinsic Dimensionality
The pfODEs proposed here allow to infer latent representations of data that either preserve reduce intrinsic dimensionaltiy as measured by the participation ratio. In this context, it is important to characterize PR-Preserving pSODEs' behavior in cases where data are embedded in higher- dimensional space but are truly power-dimensiee cases, would expect inflationary PfODEs map data into low-rank Gaussian that preserves the true intrinsic PR- dR-dimensionaltty of the original data.
(e.g., 2D data embedded in 3D space). In such
To confirm this intuition, constructed 3D-embedded (2D) circles datasets using two different approaches: (1) by applying an orthonormal matrix M to the original data points, embedding them into 3D as tilted plane (Figure I1. top 2 rows) (2) constructing third coordinate using
35

---

Additional Toy 3D - 2D Dimension-Reducing
Simulations
Scaling IIsotropical
S Curve
Alternative Scaling
Inflation
Generation
Scaling to (Isotropical)
Swirl
Alternative
Scaling
Inflation
Generation
Figure 12: Toy 3D - 2D dimension-redueing experiments with alternative scalings. Shown here are simulations of our 3D 2D PR-Reducing pfODEs for 3D toy datasets (S-curve, Swirl) scaled either to unit variance across all dimensions (first and third rows) or scaling the thickness dimension 0.5, while leaving other dimensions scaled (second and fourth rows). Note that scaling all dimensions to leads to some loss in original shape content when running generation (first and third rows, rightmost column). This is not the case when we make total variance contribution of the "thickness" dimension smaller (i.e., under the alternative scaling; second and fourth rows, rightmost column).
= =ign(w)))² which creates curved (chair-like) shape in 3D (Figure 11 bottom 2 rows). We then simulated our PR- TR-Preserving pfODE both embedding procedures and considering both the case in which noise added to the data alternatively, where some Gaussian noise is added to the initial distribution, giving it a small thickness. We used zero-mean Gaussian noise with a of 0.2 and 0.5 for embedding types (I) and (2), respectively.
As shown in Figure II when noise added, our PR-Preserving pfODEs Gaussianize the original data points along the manifold plane (rows and 3, rightmost columns). Alternatively, when noise is added and the manifold plane has some "thickness" the inflationary flows map original data into lower-rank Gaussian (rows 3 and 4, rightmost columns). In both cases, the original PR is preserved (up to some small numerical error), as expected.
C.2.3 3D Toy PR-Reducing Experiments with Different Dimension Scaling
For our 3D toy data PR-Reducing experiments, we tested how changing the relative scaling of different dimensions in the original datasets qualitatively changes generative performance.
36

---

2D Circles
CIFAR-10
1.0
PR-Preserving
PR-Preserving
PR-Reducing
PR-Reducing to 2%
0.8
0.8
PR-Reducing to 10%
PR-Reducing to 20%
Rtt,,22 tt,tt
0.6
22
0.6
R(t
0.4
0.2
0.2
0.0
0.0
0
2
3
7
2
4
6
8
10
12
14
Time (s)
Time (s)
Figure 13: Autocorrelation of denoiser network residuals. Scaled autocorrelations of denoising network residuals c(x(t)) for two sample toy networks (left, 2D eircles PR-Preserving (green) and PR-Red PR-Reducing to dimension (pink)) and for networks trained on CIFAR-10 (right) for both PR- Preserving (green) and select PR-Reducing schedules (62D, 2%, (pink); 307D, 10%, (violet); 615D, x 20%, (blue), all at IG=1.02). Toy data exhibit minimial autocorrelation along integration trajectories, while the CIFAR score estimates have some autocorrelation along one third to one half of the integration trajectory.
For the first experiment, we scaled all dimensions to variance (Figure 12 first and third rows). In this case, all dimensions contribute equally to total variance in the data. In contrast, for the second experiment (Figure 12 second and fourth rows), we scaled the thickness dimension to variance 0.5 and all others to In this case, the non-thickness dimensions together account for most of the total variance.
We then trained neural networks on 3D S-curve and Swirl data constructed using these two different scaling choices and used these networks to simulate our PR-R PR-Reducing pfODEs (reduction from 3D > 2D) both forwards (Figure left panels) and backwards (Figure 12 right panels) in time. Of note, the first scaling choice leads to generated samples that seem to loose some of the original shape content of the target dataset (first and third rows, rightmost columns). choice 2 is able to almost perfectly recover the original shapes (second and fourth rows, rightmost columns). This because scaling the thickness dimension to 0.5 reduces the percent of total variance explained along that axis, and our PR reduction preferentially compresses in that direction, preserving most information orthogonal to it. By contrast, the first scaling choice spreads variance equally across all dimensions and, therefore, shape and thickness content of target distribution are evenly mixed among different cigendimensions. As a result, compressing the last dimension in this case inevitably leads to loss of both shape and thickness content, observed here.
contrast, scaling
C.3 Autocorrelation of Network Residuals
In Section 5 above, considered the possibility that numerical errors in approximating the score function might result in errors in PIODE integration and thus miscalibration our proposed inference procedure. There, argued that if these score estimation errors can be modeled as white noise, inte- gration using sufficiently small integration step sizes will maintain accuracy, as dictated by theorems on numerical integration of SDEs [59]. Here, investigate the validity of this approximation for our trained score functions.
As detailed in Appendices B.1 and 1 we did not directly estimate scores but trained networks
to estimate denoiser ý (x C()), where y are samples from the data and x = y + are the noised samples with N( C(t).. In this case, can then compute scores for the noised
37

---

distributions using:
V log p(x, C(t)) - (-)()) (ttt(tt(xx,,....))e C(t)) -x)
(84)
In practice, however, this de-noised estimate contains error in our network estimates. Therefore, we rewrite our score expression as:
= -y., which is the true residual
Vx log, C(t) = (t) (() x) +
(85)
where (5 x) can be understood the magnitude of the correction made by the denoiser at x [5]]. Note that e = O for the ideal denoiser (based on the true score function), but nonzero will result in errors in our pfODE integration.
As argued above, these errors can be mitigated if they are uncorrelated across the data set, but this need not be true. To assess this in practice, we extracted estimation errors c(x) across large number of data samples (10K for 2D circles toys, 50K for CIFAR-10 10) and for networks trained on both PR-Preserving and select PR-Reducing schedules (PR-Reducing ID for circles IG=2.0, and to 62D, 307D, and 615D for CIFAR- 10, all at IG=1.02) and then computed cross-correlations for these emrors along integration trajectories x()):
Rt, (2) = Ex((<(x(tt) E) (e(x(t2)) - =2)"
(86)
where is the mean residual across the entire data set. In practice, we use scaled correlations in which entry Ra is normalized by o:a, the (zero-lag) variance of the residuals along the corresponding dimensions.
Results of these calculations are plotted in Figure 13 for the mean across diagonal elements of R. As the left panel of Ih residuals display negligible autocorrelation for networks trained to denoise toy data sets, while for CIFAR-10 (right panel), there is some cross-correlation at small time lags. This likely due to the increased complexity of the denoising problem posed by larger data set of natural images, in addition to the limited approximation capacity of the trained network. As result, points nearby in data space make correlated denoising errors. Nevertheless, this small amount of autocorrelation does not seem to impact the accuracy of our round-trip experiments nor our ability to produce good-quality generated samples (Figures Table T
C.4 Dataset Pre- Pre-Processing
Toy datasets were obtained from scikit-learn 993 and were de-meaned and standardized to unit variance prior training models and running simulations. The only exceptions to this are the alternative 3D toy datasets detailed in Appendix C.2.3 where the third dimension was scaled to slightly smaller variance.
For CIFAR-10 and AFHQV2 datasets, we apply the same preprocessing steps and use the same augmentation settings as those proposed for CIFAR-10 in 49 (cf. Appendix F.2), with the only change that we downsample the original AFHQv2 data to 32 32 instead of 64 x 64.
C.5
Licenses
Datasets:
CIFAR-10 61
MIT license
AFHQv2
62
Creative Commons BY-NC-SA 4.0 license
Toys
g
BSD License
38

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose new set of pfODEs (Inflationary Flows) that allows practi- tioners to deterministically map data into a (potentially) lower-dimensional, unique, and neighborhood- wewerrhooodpppeeviinnee preserving latent space, while also controlling for numerical error. Addition- ally, we perform multiple experiments using our proposed model in both toy and benchmark image datasets to support claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: As highlighted in Section 7 one of the main limitations of the proposed method lies our choice of Participation Ratio (PR) as our dimensionality measure. This measure favors top principal components of the data when doing compression. Utilizing different (more complex) dimensionality metrics and noise and scaling schedules might yield pIODEs with more interesting compressive behavior and properties. We also note the need to train DBMs over much larger noise ranges than at present as a key limitation.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
39

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide most important set of assumptions and equations needed to understand the work (in main text) and provide full assumptions, proofs, and theoretical
detail in Appendices
B
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed information about all of our experiments (including additional experiments, not included in main text) in Appendices B C Additionally, provide entire code needed to reproduce results of paper in this repository [63]. All datasets
utilized are publicly available and we provide details how download and pre-process these data in our repository and in the appendices. Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
40

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide entire code needed to reproduce results of paper in this repository [63]. All datasets utilized are publicly available and provide details on how to download and pre-process these data in our repository and the appendices.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssionooiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide in Appedix details on model hyperparameter choices, training, POODE discretization and integration, as well as how these were used to perform experiments showcased in paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
41

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Experiments, FID and MSE Experiments), we report mean +2 standard deviations results run across at least sets of independent random seeds/samples to provide readers with an estimate of uncertainty in our experiments. Additionally, we explain detail how such means and standard deviations are computed in Appendix B
Justification: For all quantitative experiments
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
In Appedix B we provide training time utilized for each model/schedule and dataset in millions of images (Mimgs) and also provide estimate of what these values mean (in terms of clock time) using our computing resources. We also specify hardware (GPU cards) used to run these experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////eeerrppppc///a////cchc
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and believe that the research conducted in this paper conforms to it (in every respect), to the best of our knowledge.
42

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We include discussion of potential societal impacts of the work presented herein as part of section 7
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Although we proposed new class generative models, work presented here does not constitute high risk for misuse (we do not release our pre-trained image generation models). We do not use scraped datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
43

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite and provide licenses for all assets (datasets, code, models) utilized in this paper. We respect all such license agreements.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode...t.....oo..o com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The main asset introduced in this paper is our code for training the proposed models and running the experiments presented herein. We provide this code under this repository [63] and also provide detailed documentation (under same repository link) on how utilize this code to reproduce results shown.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Paper does NOT involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
44

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Paper does NOT involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
45


Reviews:
[{"review_id": "NW88lVa6nT", "reviewer": "Reviewer_y6tA", "summary": "This paper proposes a new version of a probability flow-based diffusion model that allows for the effective reduction in dimensionality of the data after it has been diffused. Furthermore, the proposed approach allows for proper uncertainty measures of the data by preserving local neighborhoods from latent space to data space---in both dimension preserving and dimension reducing setups. Extensive empirical results are generated demonstrating the effects of the proposed scheme.", "questions": "No additional questions, please address the main weakness I brought up. Please let me know if I misunderstood something as well. I am more than happy to be convinced otherwise on my points.", "limitations": "The authors adequately discussed the limitations.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This is a unique and interesting problem to try and solve with diffusion models. The approach is well justified and presented clearly. The experimental setups are well motivated and the corresponding results are largely convincing.", "weaknesses": "For the experiments with high-dimensional data, I found the trend of having a higher percentage of dimension preservation resulting in worse performance to be concerning and the lack of investigation on this to be unsatisfactory. Only values of 2%, 10%, and 20% were chosen with the explanation of the results being \"retaining more dimensions in our PR-Reducing schedules leads to larger scale gaps between our preserved and compressed dimensions (i.e., larger $e^{ρ(g_{*}−g_i)T}$), thus increasing the required noise range over which networks must optimally estimate scores;\"  however, I do not believe that this tells the full story. The dimension-reducing flow approaches the original dimension-preserving flow as the number of $K$ reduced dimensions approaches the original data dimensionality $N$. As such, one would expect the performance to at some point start improving again. Should it not, and experience a discontinuity in performance from $K=N-1$ to $K=N$ then that would be interesting to note. Additionally, clearly it stands to reason that we cannot keep reducing the dimensionality further and further without eventually experiencing a decrease in performance, i.e., $K=1$ is most likely going to perform poorly for images. Finding out where this inflection point would be interesting to note as that is naturally going to be the optimal choice.\n\nWith the lack of comparison to other methods, which is understandable in and of itself given the focus of the paper, it is even more important then to fully explore the failure modes and general behavior of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's positive assessment of our work. We hope that our replies to other reviewers, as well as the General Rebuttal, reinforce this assessment.\n\nThe reviewer also raises an important issue in our experiments with high-dimensional image data: the paradoxical finding that performance appears to be _worse_ as _more_ dimensions are retained the PR-reducing flows. As we attempted to explain in Appendix B.2, this is a direct consequence of our choice of $\\mathbf{g}$, the variable that determines our inflation schedule. In the experiments underlying Table 1, we chose $g_i = 2$ for preserved dimensions and $g_i = 1 - \\frac{d}{D - d}$ for compressed dimensions (with $d$ the preserved and $D$ the number of data dimensions), so that $\\sum_i g_i = D$. We also fixed the total inflation time $T$ and inflation rate $\\rho$ across experiments. Given these choices, the quantity we call the inflation gap, the ratio between the variances of the preserved and compressed dimensions, is given by $e^{\\rho(g_{\\mathrm{max}} - g_{\\mathrm{min}})T} = e^{\\rho T D/(D - d)}$, and this is \\emph{larger} when $d$ is higher (cf. Table 5 on p26, last 2 rows). That is, when we elect to preserve more dimensions, we perform more compression  (cf. Table 7 on p27), which, as explained in the main text, results in a much more difficult learning problem, since scores must be estimated over this larger range of scales.\n\nWe hypothesize that, were we to choose $\\mathbf{g}$, $T$, and $\\rho$ to hold the inflation gap constant as $d$ is varied, preserving more dimensions should indeed result in better FID scores, and we plan to include these additional experiments (which the rebuttal period does not allow time for performing) in a revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xKL58s8p53", "reviewer": "Reviewer_v9FZ", "summary": "This paper presents a novel approach to Bayesian inference using diffusion-based models (DBMs). Traditional Bayesian inference methods struggle with high-dimensional integrals, and existing approximation methods either scale poorly (sampling-based) or offer limited theoretical guarantees (variational methods). This work leverages DBMs to develop \"inflationary flows\": these flows utilize a deterministic map from high-dimensional data to a lower-dimensional Gaussian distribution, achieved through ODE integration. The proposed method ensures local neighborhood preservation and controllable numerical error, enabling accurate uncertainty quantification. The approach also includes novel noise schedules and demonstrates high generative performance, even under significant data compression.", "questions": "* Could you please discuss the potential impact of errors in the score function estimation on the accuracy and stability of the pfODE integration? How do you address the sensitivity of your method to these errors, and what strategies can be employed to mitigate their effects? This would be particularly interesting for complex datasets.\n* When is it the PR measure for dimensionality expected to perform best? Are there any cases where this measure can lead to a bad performance due to the reliance only on second-order statistics?", "limitations": "* The article relies on the participation ratio (PR) as a measure of dimensionality, which might not capture more complex data structures effectively.\n* Accuracy of the method depends heavily on the quality of the estimated score function, which can introduce errors if not precisely learned.\n* The need for training diffusion-based models over larger ranges of noise can lead to trade-offs in compression performance and score estimation accuracy.\n* The experimental phase can be bolstered with the addition of a few extra comparisons with other VI-based models, as well as regular DBMs and MCMC-based models in order to provide a fuller comparative picture of the performance of the proposal.\n\n## References:\n[1] Rodríguez Santana, S., & Hernández-Lobato, D. (2022). Adversarial $\\alpha$-divergence minimization for Bayesian approximate inference. Neurocomputing, 471, 260-274.\n\n[2] Minka, T. (2005). Divergence measures and message passing (p. 17). Technical report, Microsoft Research.\n\n[3] Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477), 359-378.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The article is very well written and structured. Moreover, the authors provide good explanations that aid in developing intuition around the model, s.a. the diagram presented in Figure 1.\n* The idea behind the inflationary flows is a very interesting one from the point of view of the properties of the method. It does seem to provide new key features that are missing in the literature and that may be helpful in many other problems.\n* The theoretical contributions made by the authors are strong, with enough support behind the main points. The discussion makes it so that the inflationary flows model seems a natural consequence of previous works, filling previous gaps in the literature.\n* The method produces calibrated, identifiable Bayesian inference by leveraging a deterministic map between the original data and a Gaussian distribution. This ensures accurate propagation of uncertainties from the data to the latent space, leading to reliable predictive distributions.", "weaknesses": "* While it is true that the KL divergence used in VI induces a mode-seeking behavior, this can be easily fixed alongside the properties of the posterior distribution obtained via VI. This is explored in articles s.a. [1], although it has become a widely-known fact for some time [2] and its properties are usually exploited in the context of VI-based methods. $\\alpha$-divergences improve the performance of VI-based methods with very simple changes in the formulation. I think this should be discussed in the context of the introduction and the related work of the article.\n* The accuracy of the method heavily depends on the quality of the estimated score function. Errors in this estimation can propagate through the model and impact the final results.\n* The training process for diffusion-based models requires significant computational resources, particularly due to the need for training over larger noise ranges. This can limit the practical applicability of the method to very large datasets.\n* I would appreciate it if the authors provided further metrics to fully ensure the quality of the predictive distributions. These can range from the usual log-likelihood estimates to the CRPS or other proper scoring rules [3], alonside the ECE compared to other baseline models (e.g. some based on VI, regular DBMs and MCMC-based approaches). This would help in comparatively assessing the quality of the predictions obtained in a broader sense.\n\n## Minor: \n* Check again for typos, e.g. \"asympototically\" (first paragraph of the introduction). I only found this one, but it is worth checking again.\n* On the first equation, add the definition of $\\mathbf{W}$ for completeness (as is done later for the time-reversed $\\bar{\\mathbf{W}}$).\n* Although I do like the representation, I would add some short descriptions on the arrows of Figure 1 connecting the three paradigms (diffusion model $\\rightarrow$ Fokker-Plank $\\leftarrow$ pfODE), making it more explicit. \n\n_(please see the 'Limitations' section for the references)_", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we want to thank the reviewer for their careful assessment. We believe the reviewer has correctly identified the aims and strengths of our work.\n\nAs to the weaknesses:\n\n- The reviewer is correct that several alternative variational approaches have been able to mitigate some of the shortcomings of vanilla VAEs for predictive accuracy and uncertainty estimation. We are happy to include a discussion of these efforts in our introduction.\n    \n However, we will note that works such as [1] focus on calibration of _predictions_ rather than of _inferences_. While both are important, our focus here is on the latter (see our General Rebuttal). \n\n Thus, in the attached pdf, we include results from additional predictive calibration experiments on CIFAR-10 using 3 different types of VAE models as comparisons: a vanilla VAE (i.e., reverse-KL), maximum mean discrepancy (MMD) VAE, and an $\\alpha$-VAE [1], which uses the dual skew-geometric Jensen-Shannon (gJS) divergence with a learnable skew parameter. For these experiments, we utilized a similar implementation to the one used in [2] and trained models with latent spaces equal (in size) to the ones for our 2, 10, and 20% PR-Reducing experiments. All models were trained for 1K epochs, with $1e^{-4}$ learning rate, 256 batch-size, and the same architecture as detailed on Table 3 of [2] (see architecture for MNIST; we varied only latent space sizes to match compression levels in our experiments). As the table shows, these models all produce substantially worse FID scores than our approach, even when given the easier task of simply reconstructing data, no _de novo_ generation.\n\n- We agree with the reviewer that because our model is deterministic, it preserves neighborhoods by construction, with the result that accuracy in score function estimation is the central difficulty in practice. In response, we note make three observations in our work that speak to this concern:\n    - As argued in the main text (ll. 206-210), when score estimates are accurate in the mean, high variance in score estimates can be mitigated by using smaller integration step sizes.\n    - As we demonstrate in Appendix C.3 (cf. Figure 10), score function estimates do exhibit autocorrelation across flow time in the case of our larger data sets. This is less than ideal, since it means that fluctuations in the estimates may accumulate during integration, even with smaller step sizes. This is inherited from the methods we use to train the corresponding diffusion models and might be mitigated with better DBM training methods.\n    - Nonetheless, even when the score function is estimated inaccurately, if the final marginal distribution is a multivariate Gaussian, the model has learned a well-behaved latent space. That is, local inaccuracies in score approximation may be benign from the standpoint of learned latent spaces. In the rebuttal pdf, we have included q-q plots for both the first and last three principal component dimensions for some of our AFHQv2 experiments (both dimension-preserving and dimension-reducing to 2%), demonstrating that, regardless of possible score errors and autocorrelation, the final learned latent representation is quite close to Gaussian.\n    - Agreed. Diffusion models indeed require significant training resources, as our experiments show. However, as detailed in our General Rebuttal, our recent experiments with flow-matching training methods have significantly reduced these requirements, down to minutes rather than hours on toy data, increasing our confidence that methods like ours may be trained efficiently enough to compete with less intensive methods like VAEs.\n    - While, as we noted above, our primary focus in this paper is on the calibration of inferences (correct $p(\\mathbf{z}|\\mathbf{x})$), we agree that it is also important that the generation process be well calibrated (correct $p(\\mathbf{x}) = \\int p(\\mathbf{x}|\\mathbf{z}) \\pi(\\mathbf{z}) \\, d\\mathbf{z}$). To this end, we did additional predictive calibration experiments using VAEs (as detailed above) and plan to also include similar experiments using injective normalizing flows in a revised version.\n    - We expect the participation ratio to perform well when the data we wish to preserve are well aligned with the dimensions of highest variability, which will usually be cases where manifold curvature is mild. In Appendix C.2.1 (cf. Figure 8), for instance, we found that PR-reducing flows did not \"stretch\" the curved 2d manifold so much as \"squish\" it. We believe that the problem of identifying other intrinsic dimensionality measures that can be used to derive improved flows is a valuable direction for future research.\n\n\n[1] Rodriguez-Santana and Hernandez-Lobato. Adversarial α-divergence minimization for Bayesian approximate inference. Neurocomputing, 471:260–274, 2022. \n\n[2] Deasy et al. Constraining variational inference with geometric Jensen-Shannon Divergence. NeurIPS 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UwrlYDMLUs", "reviewer": "Reviewer_pn8r", "summary": "The paper proposes inflationary flows (IFs) for sampling-based density estimation. IFs belong to the growing family of diffusion-based models (DBMs) that are trained to transform distributions via a continuous process. However, the current paper notes that typical DBMs actually increase the intrinsic dimensionality of the data in the process, which may be disadvantageous in certain cases. Thus the authors propose to constrain the time-varying covariance of the process in order to approximately preserve the intrinsic data dimensionality as indexed by the participation ratio metric. In addition, the constraint can be modified to encourage dimensionality reduction, with interesting implications for modeling nominally high-dimensional data. The properties and performance of IFs are evaluated across low-dimensional toy data sets and image benchmarks.", "questions": "- Does the model admit a procedure for density estimation in addition to sampling? Is there a danger of this density ending up degenerate due to errors in compression?", "limitations": "The authors openly discuss limitations of their method and future work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and the flow of argument can be easily followed. I particularly appreciate Section 2.\n\n- The method is an interesting addition to the growing number of expressive sampling methods for density estimation, while allowing for preservation or reduction of the intrinsic data dimensionality.\n\n- The method can be useful way beyond the empirical settings considered in the paper. \n\n- The appendix is rich with useful details and additional results.", "weaknesses": "- Generic claims that NFs cannot operate on compressed representations and invalidated by recent work on relaxing architectural constraints [1-3]. These architectures need to be at least considered in the related work and possibly make it into the benchmarks as contenders.\n\n- The abstract and introduction begin as if the paper is situated in the context of sampling methods for Bayesian estimation (i.e., conditional estimation), but doesn’t feature any examples relevant to the field (i.e., only unconditional density estimation) and only references generic generative models. Moreover, with this framing, it is written as if the field of simulation-based inference (SBI, [4]), or amortized Bayesian inference (ABI) in particular, does not exist. However, SBI has long been concerned with calibrated Bayesian inference (especially for non-linear, rather low-dimensional problems) and several recent papers have used continuous probability models, such as diffusion models [5] or consistency models [6], to name just a few.\n\n- Section 3 can profit from a bit more consistency overall, for example some symbols change meaning, such as $\\boldsymbol{\\Sigma}_0$ and $\\boldsymbol{\\Sigma}$, P4L141 seems to imply that $\\boldsymbol{\\Sigma}(t)$ is diagonal for any $t$, but in my understanding this will not hold for $t \\approx 0$, PR is defined as a function over matrices (7) and vectors (8), etc.\n\n- An empirical example where the dimensionality-preserving property of IFs actually brings about a palpable advantage over a standard DBM would further strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for their positive assessment of our work. We believe they have done a good job of articulating our contributions.\n\nIn responses the weaknesses noted by the reviewer:\n\n- We regret our omission of injective normalizing flows (e.g., [1-4]) from the introduction and related work sections and thank the reviewer for bringing them to our attention. These models have very similar goals to our own, and of course, our approach can be viewed as a species of continuous normalizing flow (see discussion of Flow Matching in the General Rebuttal). Like ours, these models should have well-calibrated posteriors to the degree that they are deterministic, though we note that the connection with Bayesian inference is _not_ often discussed in the flow matching literature. Although we could not perform comparison experiments against these models during the rebuttal period, we believe, based on previously reported FID scores for benchmark image data (see Table 1 of [4], CIFAR-10), that our model would have better predictive calibration (i.e., better FID scores). If accepted, we also plan to include comparisons with several of these normalizing flow models in the simplified inferential setup described in the General Rebuttal, where we can compare model inferences with MCMC ground truth.\n\n- We apologize for the confusion and have attempted to clarify these points in our General Rebuttal and our reply to Reviewer 13gp, but to reiterate briefly:\n    - We are focused in this work on the problem of unconditional generative modeling in the absence of an assumed likelihood model. That is, as with VAEs, we are interested in both learning a generative model and a latent space at the same time. We are big fans of simulation-based inference, but our understanding is that these approaches typically begin with a well-defined generative model and then attempt to perform approximate inference on the parameters of this model. By contrast, we do not assume a fixed parameterization or generative model, which creates issues when assessing calibration of inferences.\n        \n     However, we agree that it is important to explain this distinction, which we would plan to add to the introduction of a revised version if accepted.\n        \n    - As also discussed in the General Rebuttal, our primary focus in this work is on calibration of inferences, not predictions. While both are important, we have focused on the former, though as detailed in our rebuttal to Reviewer v9FZ and in the attached pdf, we have performed additional predictive calibration experiments.\n\n- We have checked, and we believe the notation in Section 3 is consistent. $\\boldsymbol{\\Sigma} = \\boldsymbol{\\Sigma}(t) = \\boldsymbol{\\Sigma}_0 + \\mathbf{C}(t)$ is the time-dependent covariance of the smoothed data, where $\\boldsymbol{\\Sigma}(0) = \\boldsymbol{\\Sigma}_0$ is the covariance of the data. Moreover (ll.142-143), we choose $\\mathbf{C}(t)$ to be diagonal in the eigenbasis of the data, $\\boldsymbol{\\Sigma}_0$, so that $\\boldsymbol{\\Sigma}(t)$ does indeed remain diagonal for all $t$. It's for this reason, along with the fact that PR is invariant to linear transformations (ll. 136-137) that the definition in (8) is the same as the one in (7). That is, in the eigenbasis of $\\boldsymbol{\\Sigma}_0$, $\\boldsymbol{\\Sigma}(t) = \\mathrm{diag}(\\boldsymbol{\\sigma^2}(t))$.\n- We view our method as distinct in aim from DBMs. While our flows can be used as generative models, data compression does result in FID scores slightly worse than those for the best DBMs. What inflationary flows provide that DBMs do not are neighborhood-preserving, low-dimensional latent spaces.\n- This is an interesting question. One might think of integrating the Fokker-Planck Equation (3) backward from the smoothed low-rank Gaussian distribution, but because _both_ the compression and smoothing eliminate information, there is an exponential loss of initial conditions, and integrating backward from a Gaussian (i.e., the asymptotic solution) cannot recover the data. We think the problem of density estimation would be interesting to pursue here but don't currently have a solution to it. This is an advantage offered by some manifold learning normalizing flows over our approach.\n\n[1] Brehmer and Cranmer. Flows for simultaneous manifold learning and density estimation. NeurIPS 2020.\n\n[2] Caterini et al. Rectangular flows for manifold learning. NeurIPS 2021.\n\n[3] Cornish et al. Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows. ICML 2020.\n\n[4] Flouris and Konukoglu. Canonical normalizing flows for manifold learning. NeurIPS 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SCvJoYmVe2", "reviewer": "Reviewer_13gp", "summary": "This paper introduces inflationary flows, a diffusion model with deterministic denoising trajectories that allow mapping high-dimensional data to low-dimensional Normal distributions. The dimensionality reduction is controllable, and it is claimed that, as long as the score is correctly estimated, the method provides calibrated uncertainty estimates.", "questions": "* Do you have evidence that strong miscalibration can be observed when using other methods with perfectly learned scores that would strengthen the significance of your method?\n\n* Is there any comparison available between your method and other methods both on the quality of the generated objects and on the calibration of the uncertainty estimates? \n\n* Could you clarify exactly what properties this method has that no other existing methods have and how those properties lead to calibrated uncertainty estimates?", "limitations": "Limitations are properly addressed.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "### Originality\n* The method is new to me.\n\n### Quality\n* I did not identify flaws in the mathematical derivations but did not check derivations in the appendices carefully.\n\n### Clarity\n* The flow is good.\n* Most of the implementation details are provided.\n* The authors made an effort to be pedagogical and introduce all the necessary background.  \n\n### Significance\n* Having calibrated uncertainty estimates is of high significance in risk-sensitive tasks.", "weaknesses": "### Originality\n* No concerns about originality.\n\n### Quality\n* I have some concerns about the empirical evaluation of the method. My main concern is that there are no comparisons to existing methods while there exist plenty of generative models, such as other diffusion models, and normalizing flows,... Additionally, the experiments did not convince me that this method indeed provides calibrated uncertainty estimates, probably because I did not understand what Figure 4 shows (see clarity concerns).\n\n### Clarity\n* To me, the central part of the submission, the calibrated uncertainty estimates, is not detailed enough. Only 7 lines at the beginning of section 5 mention how to obtain a calibrated uncertainty estimate, and after reading the manuscript, it is still not clear to me as to why this method provides calibrated uncertainty estimates while others don't. \n* I read Appendix B6 three times and still don't understand what is done, how Figure 4 is obtained, and what it is supposed to show. \n\n### Significance\n* This paper presents a method that produces calibrated estimates as long as the score is well estimated. My intuition would be that estimating the score correctly is the root of the problem. other methods, such as normalizing flows, also provide calibrated estimates if the neural network is perfectly trained. The miscalibration of other methods, given perfect score estimates, has not been quantified in order to showcase the significance of their method. \n* It is not clear to me what this method brings in comparison to other diffusion models. The authors seem to claim that their method provides calibrated uncertainty estimates because it is based on a deterministic process, but as mentioned by the authors, equation (4) is also a deterministic process. What does this method bring that equation (4) doesn't have? Additionally, Normalizing flows are also deterministic transformations of data to a base (usually Normal) distribution and satisfy this property.  It is also not clear to me why a deterministic process is required for calibrated sampling-based uncertainty estimates (probably linked to clarity issues in section 5).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the author's time and effort in reviewing our paper and apologize for some confusion.\n\nPlease see our General Rebuttal, in which we attempt to clarify several points raised by multiple reviewers, particularly \n- the particular calibration and modeling task we set out to solve\n- the difficulties of assessing posterior calibration in this task\n\n**Relation to other models:** Several of the reviewer's concerns related to the relationship of our approach to other models. Given our aim of providing calibrated posteriors over structured latent representations, we make the following comparisons:\n\n- **VAEs:** Variational autoencoders and related models offer structured latent spaces, but posterior uncertainties are approximated by Gaussians and often overconfident. Several approaches, including [1], attempt to mitigate this posterior mismatch. VAEs typically offer poor generative performance on challenging benchmark data sets.\n- **Simulation-based inference:** We are not experts on SBI, but our understanding is that these approaches usually assume a parameterized generative model (often likelihood-free and based on substantial domain knowledge) and aim for both good generative performance and well-calibrated inference over model parameters. By contrast, we do not assume a particular form for our generative model (more similar to VAEs and normalizing flows) and are focused on posterior uncertainty in latent variables, not generative model parameters.\n- **Diffusion models:** Diffusion models offer state-of-the art generative performance, and our model is trained as a diffusion model using a particular noise schedule. But standard diffusion models _do not_ produce well-organized latent representations (Figure 1, left). Rather, they are constrained to \"latent\" representations that are the same size as the data dimension, inflate the intrinsic data dimension, and mix widely disparate data together.\n- **Normalizing flows:** One reviewer pointed out that our introduction ignored a prominent line of work on injective normalizing flows, which map low-dimensional distributions to high-dimensional data [e.g., 2, 3, 4]. This is a regrettable omission on our part. Indeed, the goal in those works is very similar to our own. As with our model, a deterministic normalizing flow that changes dimensionality will, by definition, produce well-calibrated posteriors and low-dimensional latent representations. In fact, our model can be viewed as a species of continuous normalizing flow that works by learning an ODE rather than composing a sequence of functions. Our main advantage over these models is primarily in predictive calibration (i.e., generation quality), where our model inherits the performance of diffusion models in producing better coverage of the data distribution and much lower FID scores (cf. [5], FID scores for CIFAR-10 at $d=40$). \n\nThus, VAE-like models and injective normalizing flows are the most suitable comparison model classes for our purposes.\n\n**Calculation of calibration:** We apologize for the confusion on this point, particularly the analysis underlying Figure 4. For reasons detailed in the General Rebuttal, it is extremely difficult to assess calibration for high-dimensional models, so our goal in this analysis was to assess the following question in the context of low-dimensional synthetic examples: **Given a set in the data space with a particular probability measure, can we identify a set in the latent space with the same probability measure that maps to it?**\n\nIt is fairly obvious that if the score function is perfectly known and there are no numerical integration errors, this _must_ be the case: we just integrate the pfODE (6). However, if there are errors in estimating the score function or integrating the ODE, a set that contains 20% of points sampled from a probability distribution in $\\mathbf{x}$ (data) space may only contain 10% of these points when transported to $\\mathbf{z}$ (latent space). \n\nFigure 4 is our attempt to assess whether this is the case. We first trained a score function network to approximate the score for the datasets shown in (A). We then defined a collection of sets (dashed lines) at various Mahalanobis distances from the mean of the data distribution and calculated how much of the data was contained within each set. This is the \"Initial Coverage Probability\" used in (B) and (C). The next question is how to \"transport\" these sets to the latent space --- tricky, because they are defined as contours. To approximate this, we sampled boundary points on these contours, constructed meshes or alpha shapes based on them, and transported the sampled boundary points (and thus meshes) to the latent space. Finally, we counted the percentage of latent data points inside these sets. This is the \"Final Coverage Probability.\" Figure 4B-C plot the differences in these two quantities, which should be 0 for a perfectly calibrated flow probability and are very small ($\\mathcal{O}(10^{-3})$ or $\\mathcal{O}(10^{-2})$). Again, this is expected mathematically because of the deterministic ODE; it is only intended as a check on the accuracy of the score function approximation and numerical integration. \n\nWe hope this explanation clarifies things. If accepted, we plan to revise Section 5 and Appendix B.6 accordingly. \n\n[1] Rodriguez-Santana and Hernandez-Lobato. Adversarial α-divergence minimization for Bayesian approximate inference. Neurocomputing, 471:260–274, 2022. \n\n[2] Brehmer and Cranmer. Flows for simultaneous manifold learning and density estimation. NeurIPS 2020.\n\n[3] Caterini et al. Rectangular flows for manifold learning. NeurIPS 2021.\n\n[4] Cornish et al. Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows. ICML 2020.\n\n[5] Flouris and Konukoglu. Canonical normalizing flows for manifold learning. NeurIPS 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
First, we appreciate the reviewers' thoughtfulness in assessing our paper.  While all reviewers agreed that the work is novel, they also noted weaknesses and requested clarification on several shared themes:

## Problem setup

**First,** we are concerned here with _unconditional generation_ in cases where **the generative model is unknown**. This is the most common problem setup in work on, e.g., VAEs, normalizing flows, and GANs, and is the one we adopt here. While conditional generation is an important problem, we restrict ourselves here to sampling data $\mathbf{x}$ based on some latent variables $\mathbf{z}$.

**Second,** we are interested in learning models with **structured latent representations.** This is typical of both VAEs and injective flows but not of GANs and diffusion models. As Figure 1 (left) illustrates, the diffusion procedure necessarily mixes data in the latent space, resulting in an unstructured latent representation. Thus, _we aim to construct models that combine the latent representations of NFs or VAEs and the generative power of DBMs._

**Finally,** we are interested in the **calibration of inferences.** We particularly apologize for confusion on this point, since it appears to be at the root of several questions posed by reviewers. To be more specific: in the models we consider, one could speak of both _predictive_ calibration --- how well the marginal $p(\mathbf{x})$ matches the data --- and _posterior_ calibration --- how the model posterior $q(\mathbf{z}|\mathbf{x})$ matches the true posterior $p(\mathbf{z}|\mathbf{x})$. Our primary focus is on the latter, though we will have more to say about the former below.

## Other approaches and our contribution
Please see our detailed reply to Reviewer 13gp.

## Model comparisons
All reviewers asked for more quantitative comparisons with other models. 

**First,** for predictive calibration, we can easily calculate measures like FID and, e.g., the maximum mean discrepancy (MMD) between samples from our model and training data. In Table 1 of the attached pdf, we report FID scores for $\alpha$-VAEs, MMD-VAEs, and vanilla VAEs trained on CIFAR-10. Since VAEs are known to have low generative performance, it is no surprise that our model outperforms these by a wide margin. While injective flow methods also require substantial training time (cf. [5], Table 4), we plan to include these additional comparisons in a revised version.

**Second,** For posterior calibration, the situation is tricky. For cases where the generative model is known and parameters are shared across the data, MCMC methods can yield samples from the posterior for low-dimensional models. But for methods like ours, where the generative model is learned, the latent spaces are not necessarily comparable across models, and so "ground truth" is not shared. That is, the true $p(\mathbf{z}|\mathbf{x})$ is conditioned on the generative model, and the latent variables are distinct for each data point. 
    
In this case, apart from the geometric approach we have taken in Figure 4 (see clarification in our rebuttal to Reviewer 13gp), the best method we have found for assessing calibration comes from [2], which generated $\mathbf{x}$s from a parameterized prior over $\mathbf{z}$ and then performed inference on the parameters of this prior using both their model and MCMC. Again, because of the deterministic nature of normalizing flows and our model, we expect both to perform very well on this task, with VAEs faring somewhat worse. If accepted, we plan to include this comparison for a selection of models from both classes.

## Connection with Flow Matching
Finally, since the initial submission of this article, we have established that our method can be linked to recent flow-matching approaches [e.g., 6, 7, 8]. Essentially, our DBM noising schedule can be recast as an "interpolant" in the terminology of [8], meaning that our proposal is essentially a novel, dimension-reducing form of flow-matching.

More importantly, experiments with our toy data examples (Figure 2 of the attached pdf) have found that training using flow matching is not only feasible, but also significantly speeds up learning --- as much as _two orders of magnitude_. This suggests that models like ours could be learned using drastically fewer resources than reported in our initial submission, a direction we intend to continue exploring.

[1] Flouris and Konukoglu. Canonical normalizing flows for manifold learning. NeurIPS 2023.

[2] Brehmer and Cranmer. Flows for simultaneous manifold learning and density estimation. NeurIPS 2020.

[3] Lipman et al. Flow matching for generative modeling. ICLR 2023.

[4] Liu et al. Flow straight and fast: Learning to generate and transfer data with rectified flow. ICLR 2023.

[5] Albergo and Vanden-Eijnden. Building normalizing flows with stochastic interpolants. ICLR 2023.

Meta-review: This paper presents a new class of generative models and associated inference algorithm that maps high-dimensional data to a lower-dimensional Gaussian via ODE integration. The main claimed contribution of the paper is that the method is well calibrated and identifiable, i.e., is uniquely defined on the low-dimensional space. 

Overall, the reviewers agree that the proposed approach is novel but raised several important concerns including: (1) Lack of comparisons with other approaches (including injective normalizing flows); (2) clarity regarding evaluation of calibrated uncertainties; and (3)  somewhat unintuitive results when more dimensions are retained (as the metrics get worse). The authors have, to some extent, addressed these issues satisfactorily in their comments by clarifying where their approach lies (unconditional generative modeling and learning structured representations) and provided additional results. However, the comparison with injective flows was not done due to time constraints, so I’d expect to see this in the final version of the paper. The authors have also clarified the relation of their work to flow matching by stating that it is essentially a “novel, dimension-reduction form of flow-matching”. 

One additional point I’d like to see discussed in the final version is concerned with calibration. Usually in the literature this refers to predictive calibration. However, how well the approximate posterior matches the true posterior is simply an evaluation of the posterior approximation method. While posterior comparisons across different models does not make sense, one can still evaluate (predictive) calibration, which the authors have done. I also agree that uncovering structured latent representations is an important problem, but in my view this has more to do with the underlying model than with the approximation method (while I recognise these two may be conflated sometimes). I think more clarity in the terminology and evaluation should be provided. 

 Given the positive comments and the recognition of the novelty of this work, I recommend acceptance.
Decision: Accept (poster)