Density estimation on low-dimensional manifolds: an inflation-deflation approach
Authors: Christian Horvat
Venue:  (2021)
Keywords: Normalizing Flow
Density Estimation
low-dimensional manifolds
noise
normal space

Abstract: Normalizing Flows (NFs) are universal density estimators based on Neuronal Networks. However, this universality is limited: the density's support needs to be diffeomorphic to a Euclidean space. In this paper, we propose a novel method to overcome this limitation without sacrificing the universality. The proposed method inflates the data manifold by adding noise in the normal space, trains an NF on this inflated manifold and, finally, deflates the learned density. Our main result provides sufficient conditions on the manifold and the specific choice of noise under which the corresponding estimator is exact. Our method has the same computational complexity as NFs, and does not require to compute an inverse flow. We also show that, if the embedding dimension is much larger than the manifold dimension, noise in the normal space can be well approximated by some Gaussian noise. This allows using our method for approximating arbitrary densities on non-flat manifolds provided that the manifold dimension is known. 

Under review as a conference paper at ICLR 2021
DENSITY ESTIMATION ON LOW-DIMENSIONAL MANI- FOLDS: AN INFLATION-DERLATION APPROACH
Anonymous authors
Paper under double-blind review
ABSTRACT
Normalizing Flows (NFs) are universal density estimators based on Neuronal Net- works. However, this universality is limited: the density's ss support needs to be diffeomorphic to a Euclidean space. In this paper, we propose a novel method to overcome this limitation without sacrificing the universality. The proposed method inflates the data manifold by adding noise in the normal space, trains an NF this inflated manifold and, finally, deflates the learned density. Our main re- sult provides sufficient conditions on the manifold and the specific choice of noise under which the corresponding estimator exact. Our method has the same com- putational complexity as NFs, and does not require to compute an inverse flow. We also show that, if the embedding dimension much larger than the manifold dimension, noise in the normal space can be well approximated by some Gaus- sian noise. This allows using our method for approximating arbitrary densities on non-flat manifolds provided that the manifold dimension is known.
INTRODUCTION
Many modern problems involving high dimensional data are formulated probabilistically. Key con- cepts, such as Bayesian Classification, Denoising or Anomaly Detection, rely on the data generating density p" (I:). Therefore, main research area and of crucial importance is learning this data gen- erating density p" (x) from samples.
For the case where the corresponding random variable X RRP takes values on manifold diffeo- morphic to R,, a Normalizing Flow (NF) can be used to learn p" (xr) exactly (Huang et al. 2018) Recently, few attempts have been made overcome this topological constraint. However, to do
so, all of these methods either need to know the manifold beforchand (Gemici et al. (2016), Rezende
(2020) or they sacrifice the exactness of the estimate (Cornish et al. (2019) Dupont et al. (2019)
Our goal in this paper is to overcome both the aforementioned limitations of using NFs for density estimation on Riemannian manifolds. Given data points from a d d-dimensional Riemannian mani- fold embedded in RR,, d < D, we first inflate the manifold by adding specific noise the normal space direction of the manifold, then train an NF on this inflated manifold, and, finally, deflate the trained density by exploiting the choice of noise and the geometry of the manifold. See Figure I for schematic overview of these points.
Our main theorem states sufficient conditions on the manifold and the type of noise we use for the inflation step such that the deflation becomes exact. To guarantee the exactness, we do need to know the manifold as in e.g. Rezende et al. (2020) because we need to be able to sample in the manifold's normal space. However, as will show, for the special case where D > d, the usual Gaussian noise is an excellent approximation for a noise in the normal space component. This allows using our method for approximating arbitrary densities on Riemannian manifolds provided that the manifold dimension is known. In addition, our method is based on a single NF without the necessity to invert it. Hence, we don't add any additional complexity to the usual training procedure of NFs.
Notations: We denote the determinant of the Gram matrix of f as g (x)) == det (J)( Jx(x)) where (x) is the Jacobian of f. We denote the Lebesque measure in R" as X, Random variables will be denoted with capital letter, say X, and its corresponding state space with the calligraphical

---

Under review as a conference paper at ICLR 2021
R
x rh (x)
N ((3)
to?
C Rd
X R
Theorem 1
2 Fr
R
R2
m Mx)
p2)(1)
No (2)
X C Rd
Figure 1: Schematic overview of our method. 1. A density p" (x) with support on a d-dimensional manifold X (top left) is inflated by adding noise r² in the normal space (top right). 2. We have an NF F (z) learn this inflated density ((ū) using a well-known reference measure pu(u). 3.
We deflate the learned density to obtain estimate f(x) for p" (z). 4. Our main result provides sufficient conditions for the manifold and the choice noise such that p(x) = (z).
version, i.e. X. Small letters correspond to vectors with dimensionality given by context. The letters d, D, n, and N are always natural numbers.
2 BACKGROUND AND PROBLEM STATEMENT
An NF transforms known auxiliary random variable by using bijective mappings parametrized by Neuronal Networks such that the given data points are samples from this transformed random variable, see Paaamakarios et al. (2019). Formally, an NF is diffeomorphism Fg u - X and induces density on X' through po (z) (gf (u) (u) where per (u) is known and u F3 (x)) The parameters 0 are updated such that the KL-divergence between p (x) and po()..
(K (x)llpo (z)) = Eewp (z) [log po (x:] + const.
(I)
is minimized. If Fg is expressive enough, was proven that in the limit of infinitely many samples, updating 0 to minimize this objective function converges to a 0* such that it holds Px-almos. surely p" (x) = Pp- (z), see (Huang et al. 2018).
More generally, let X X C R be generated by an unobserved random variable Z Z C Rd with density x=), that is X = f(Z) for some function f Z - X' where typically < D. In Gemici al. (2016), is embedding[] and it was shown that
calculate probabilities such as (x for measurable C X' using a density p² (z) with respect to the volume form dV,
induced by f. that is
P(X A) { (-1(11) T(2)d2= = 1 f/((t)aVVs(0)
(2)
with p0(x) (2)) and (z) Vg where - f- Hence, given an explicit mapping f and samples from p" (x). we can learn the unknown density (²2) using a usual NF in Rd. However, in general, the generating function f is either unknown or not an embedding creating numerical instabilities for training inputs close to singularity points.
In Brehmer & Cranmer 2020). and the unknown density æ are learned simultaneously. The main idea is to define J as level set of a usual flow in RD and train it together with the flow in I used to learn π (2). To evaluate the density, one needs to invert f and thus this approach may be very slow for high-dimensional data. Besides, ad hoc training strategies. We tie in with the idea to use an NF for learning p" (x) with unknown f and study the following problem.
guarantee that f learns the manifold they proposed several
'Thus, regular continuously differentiable mapping (called immersion) which is, restricted to its image, homcomorphism.
2

---

Under review as a conference paper at ICLR 2021
Problem 1 Let X be d- dimensional manifold embedded in RD. Let X f(Z) be random variable generated by an embedding f : R + IR" and a random variable Z (2)) in R Given N samples from p" (z) described above. find an estimator p of such that in the limit of infinitely many samples we have that p(x:) = p" (x), Px- almost surely.
METHODS
To solve Problem ! we want to exploit the universality of NFs. We want inflate X such that the inflated manifold X becomes diffeomorphic set U on which simple density exists. Doing so, allows us to learn the inflated density p(2) exactly using a single NF, see Section Then, given such an estimator for the modified density, we approximate p" (x) and give sufficient conditions when this approximation is exact.
3.1 THE INFLATION STEP
Given a sample x of X, if we add some noise E IR. to it, the resulting new random variable X X + e has the following density
( = (((2))dPx(x)
(3)
Denote the tangent space in z as Ta and the normal space as N₂. By definition, N₂ is the orthogonal complement of T. Therefore, we can decompose the noise into its tangent and normal component, Et + En- In the following, we consider noise in the normal space only, i.e. & 0, and denote the density of the resulting random variable as (n()). The corresponding noise density qo (+/rr) has mean x and domain Ng. We denote the support of qn (-(r) by N (-12a) The random variable
X = X + En is now defined on X = Unex Na.(+la). We want to be diffcomorphic to a set U on which known density can be defined.
Example (a) Let X = S1 = far E R22
||||| 1} be the unit circle. For each E S there cx(2) (cos(z), sinz)))² To sample point i in N.,
exists E 2T) such that which is spanned by e, (z), we sample a scalar value y and set = x + (x)()) With 7 Uniform[-1, 1), we have that
X = Ui E
E [-1,1)] = (x | < 2}
(4)
SEX
which is the open disk with radius? 2. The open disk is diffeomorphic to (0,1) (0, 1). Thus, (T) can be learned by a single NF F-1 and pu(u) = Uniform ((0, 1) x (0, 1)) as reference.
(b) As in (a), we consider the unit circle. Now we set to be x2 distribution with support [- 1, 10). Then
= + ter E [-1, = RR2
(5)
xEX
Thus, qu( can be learned by a single NF F-1 and pu(()) Niu; Ip) as reference. Both cases can be analogously extended higher dimensions.
Remark To be precise, the random variable en is generated by a random variable in RD-d say I, with measure Pr. Then, qn (.(x)) ) the density of the pushforward of the noise measure Pr with regard to the mapping h : IR-d - Nr. Hence, formally, the density q(ā(xr) is with respect to the form
induced volume form dVv, see Section 2 However, if we choose an orthonormal basis for N say
(1)).... ...,,!(D-d,, then we have that 5 h(7)) A + x where the columns of A E DDD(DD)) given by these basis vectors, i.e. A (1) ..... no (D-4)]. Thus, the Gram determinant of h and we have that dV,(()) = dr where dr denotes the volume form with respect ÀD-d- In this case, we can think of qu (f(xz) as density with respect to AD-d-
is g" det(A""A) A)
If qu (ā]z) depends only on || - x|1. as it is for the Gaussian distribution, we have that ga (a Ix) qu (1 z²I) qu 9n((1 (1)l) because h is an isomorphism. Thus, for this case it holds that qn q dVV = qu (llr.))dry Then. for convenience, we may abuse notation by writing 7y ~ q(ā(z) or en r(7) where r(7) ) is the density of Prr with respect AD-d-
3

---

Under review as a conference paper at ICLR 2021
3.2 THE DEFLATION STEP
Our main idea is to find conditions such that
(2) On (i/x)p" (x)
(6)
for almost surely all E X and for an almost surely unique E X. Because then, given an exact estimator of 4n (Z), say (Z), we have for ã that p" (z) c
For equation (6) to be true, we need guarantee that almost every I corresponds to only one E X. This is certainly the case whenever all the normal spaces have no intersections all (think of a simple line in R2). We can relax this assumption by allowing null-set intersections. Moreover, only those subsets of the normal spaces are of interest which are generated by the specific choice of noise qn (#)z). Thus, only the support of qn ( Na.(+1>), matters. The key concept for our main result expressed in the following definition:
Definition Let X be d- dimensional manifold and N₂ the normal space in E X. Let (n(-(a)) be a density defined on N₂ and denote by N Ngu(-la) the domain of qu (-(x)x). Denote the collection of For I E X, we define the set of all possible generators of all such densities as Q =gn((1z))2EA
as A()) = {o' E X 0n (-1x)) 3 f). We say X is Q -normally separated if for all x E X holds that IP Xxx [ N₂ A(T) >
where # A(ā) is the cardinality of the set A(T). words, every & E N₂ is P xxx== almost surely determined by x.
To familiarize with this concept, consider Figure 2 and the following example:
Example 2 For the circle in example | we choose En to uniformly distributed on the half-open interval 11.. The point (0,0)* is contained in Na.(11)) for all x E X and thus N Naal-le) O (a)(1a)) ((0,0)7) for all + z', see Figure 2
(middle). Hence, for any given &t N₂ have 80 i (0,0)
that A(2) = X if O)", and therefore #A(
else,
else.
Thus, Px PxIx== ã ( >
P P (0, 0) for all X. What follows is
that X is Q Qonomlllyy seperated.
were choose en to be uniformly distributed spaces would overlap and we would have that TP x|x==
11.5, Higure 2 (right), the normal
T X A(T)
> O. In this case, x
would not be Q-normally seperated.
Es, - Uniforn(--...5
Ea - Unnifor([-1.1
E,, - Uniforn[fmm1111 -1.5.
-2
2
22
22
Figure 2: Q-normal separability for different noise distributions (n(E(z) used to inflate X' = St (black line). Left: A' is Q-normally separable since every point in the inflated space X (red shaded area) has a unique generator. Middle: X' is Q-normally separable since IP x -almost every point in X' has unique generator. Right: X' is not Q-normally separable since every point in the dark shaded area has two generators.
Theorem Let X be a dimensional manifold. For each E X, let qn (-(x)) denote a distribution the normal space of Let X be Q-normally separated where Q == (4r (+(x)))x) Assume

---

Under review as a conference paper at ICLR 2021
that can learn the density qu (z) defined in equation 33. by using single NF F=), thus On (5) ( "pu(F" (x)) for some known density pu- Then, for x--almost all
It E X holds that yu (t) = p" (z:) (5 x), therefore this equation, when evaluated at T = I, yields
9a(x))
p" (=(x)=
In (x(xr)
The proof can be found in Appendix A.I
3.3 GAUSSIAN NOISE AS NORMAL NOISE AND THE CHOICE OF
Our proposed method depends on three critical points. First, we need to be able sample in the normal space of X. Second, we need to determine the magnitude and type of noise. Third, we need to make sure that the conditions of Theorem arre fulfilled. We address (partially) those three points.
I. For the special case where D > d, we show that a full Gaussian noise is an excellent approxi- mation for Gaussian noise restricted to the normal space. Consider 6, Eus of TD)- Then, the expected absolute squared error when approximating normal noise with full Gaussian noise is E [|e En|²] E
[|st]_] do². The expected relative squared error is therefore
E En 12
E
d
E
d
(8)
cn
E 2
2
because e, and en are independent and [F,
D-d follows scaled inverse x² disttrbbbtt with D
degrees of freedom and scale parameter 1/o². Thus. if D > d, Gaussian noise is an excellent approximation for a Gaussian in the normal space. We denote the inflated density with Gaussian noise by go (f) in the following.
The inflation must not garble the manifold too much. For instance, adding Gaussian noise with magnitude 2 to S will blur the circle. Since the curvature of the circle is 1/r, intuitively, we want a to scale with the second derivative of the generating function f. Additionally, we not want to lose the information of pz (1:) by inflating the manifold. If the generating distribution x<)) makes a sharp transition at 20, (20 A < T + Az,) for | < 1, adding to much noise in zo f(oo will smooth out that transition. Hence.
want a to inversely scale with (=)=). We formalize these intuitions in Proposition | and prove them in Appendix A.2 In accordance with Theorem [ say po (5) approximates well p" (x) if lima2-0 Po (x:)// qn x p" (a)) for all I E X' where qn (x(x) is the normalization constant of a ( - dimenssiol Gaussian distribution.
Proposition 1 Let X E R be generated by Z T(2) through an embedding f I2d + E.,, iee. f(2) X. Let π E (IR)). For qod (i) to approximate well p" (x)) in the sense that p" (z) E X, a necessary condition is that:
lim 2o
(z0) (JF <1
(9)
2T (20)
where ||A|+ (7"(2)))
Eij=1 Aij for Rdd Ted and & denotes the elementwise product, and == is the Hessian of the prior evaluated at 2g = f-))) d2,0z
(p5(2)
Intuitively, a second necessary condition is that the noise magnitude should be much smaller than the radius the curvature of the manifold which directly depends on the second-order derivatives of f. This can be illustrated in the following example:
Example 3 For the circle in IR2 generated by f(=) = =(cos(z), sin(z))² and von Mises distri- 2n bution (2) & exp(x (=)(,)) we get that o² & min K(K sin" (2)- .²2 where the first condition comes from Proposition Aand the second comes from the curvature argument.
2
Even though this bound may not be usefull as such in practice when f and æ are unknown, it can still be used if and are estimated locally with nearest neighbor statisties.
Technically, the circle does not fulfill the conditions of Proposition since the domain of is R.
5

---

Under review as a conference paper at ICLR 2021
From numerical perspective, inflating a manifold using Gaussian noise circumvents degeneracy problems when training an vanilla NF for low-dimensional manifolds. In particular, the flows Ja- cobian determinant becomes numerically unstable, see equation (.). This determinant is essentially a volume changing factor for balls. From a sampling perspective, these volumes can be estimated with the number of samples falling into the ball divided by the total number of points. Therefore, suggest to lower bound a with the average nearest neighbor obtained from the training set to make sure that these volumes are not emply and thus avoid numerical instabilities.
3. Intuitively, if the curvature of the manifold is not too high and if the manifold is not too entangled, Q-normal separability is satisfied for a sufficiently small magnitude of noise. Also in the manifold learning literature, the entangling must not be too strong. Informally, the reach number provides a necessary condition on the manifold such that it is learnable through samples, see Chapter 2.3 in Berenfeld & Hoffmann (2019). Formally, the reach number is the maximum distance Tx such that for all x in a 7x - neighbourhood of X' the projection onto X' is unique. In Appendix [A.3] we prove Theorem 2 which states that any closed manifold X' with 7x > 0 is Q Q-normally separable.
Theorem 2 Let X C RDD be closed d-dimensional manifold. If X has positive reach number Tx, then X Q -normally separable where Q == {4n (-[=)) is the collection of uniform dis- tributions a ball with radius Tx, i.e. qn (z/(z) = Uniform(ā B(x, Tx) O N2) where B(x, Tx) denotes a D dimensional ball with radius Tx and center x.
4 RESULTS
We have three goals in this section: First, we numerically confirm the scaling factor in equation 77) Second, we verify that Gaussian noise can be used to approximate a Gaussian noise restricted to the normal space. Third, we numerically test our bound for o² derived in Section (.33 Finally, we show that we can learn complicated distributions on S without using explicit charts. For training details, refer to Appendix B.I and B.2 respectively.
4.1 VON MISES ON A CIRCLE
Let X' be circle with radius 3 and let (22) o exp(8 cos(2)) be 1D von Mises distribution. Given
2 ~ TT2). we generate a point in X' according to the mapping f2) = 3(cos(2)),snn( sin(2)). We want to
learn the induced density p" (a:). Note that 3p" (x) = π (2) since 1/3 is the volume form induced by f. To benchmark our performance, we use the idea in Gemici et al. (2016) to first embed the circle into I, using c.g. f-1 learn the density there with an NF, and transform this learned density back to St. Brerheerr&& & Cranmer (2020), this method is named Flow manifolds (FOM) and we stick to this notation in the following. Note that f is not injective and to illustrate the benefit our method choose the singularity point to be (1, 0)T. By moving points close to 0) slightly away from TT, we numerically ensure that f is an embedding.
We inflate X using 3 types of noise: Gaussian in the normal space (NG), Gaussian in the full ambient space (FG), and X2-noise as described in examples (Tb) with scale parameter 3. Technically, NG violates the Q-normal separability assumption. However, if o² is small and the scale parameter for the von Mises distribution is large enough, this is practically fulfilled.
1. The Inflation step:
2. Learning yn (Z): For the NFs we use Block Neural Autoregressive Flow (BNAF) (De Cao et al. 2019). We use the same NF architecture and training procedures across all models.
3. Deflation: Given an estimator for qu (i), we use equation 7 to calculate p" (x). For FG and NG,
we have that go (x((x) 1/ /2T0=2 and for the normal x noise is yn (2rlx)) / ////////11111/ (VBT(2))
In Figure 3 we show the results for x2 = 0.01 and 0² = 1. In the respective plot, the first row shows training samples from the inflated distributions (ā) (left), and (ā) (middle), respectively. We color code a sample ã = x + E according p" (x) to illustrate the impact of noise on the inflated density. Note that the FOM model (top right) does not need any inflation and therefore is trained on samples from p² (²) only. In the respective plot, the second row shows the learned density for the different models and compares it to the ground truth von Mises distribution T22) depicted in black. As we see, for a² 0.01 all models perform very well, although the FOM model

---

Under review as a conference paper at ICLR 2021
slightly fails capture p(2) for close 0 which corresponds to the chosen singularity point. For o² ===1, 1, we see significant drop in the performance of the Gaussian model. Although the manifold is significantly disturbed, the normal noise model still learns the density almost perfectly so does the normal x² -nois model, as predicted by Theorem
o² = 0.01
full Gaussian noise
normal Gaussian noise
FOM
-2
-2
-2
-3
-3
-3
2
touc
true
DNCC FDM
1.0
1.0
NG
0.8
0.8
0.8
0.4
0.2
0.2
0.2
0.0
0.0
0.0
a² 1.0
full Gaussian noise
normal Gaussian noise
normal x² noise
2-
-2
-2
-2
-3
-3
2
Duu
1.0
1.0
0.8
0.8
0.8
0.4
0.2
0.2
0.2
2
Figure 3: Learned densities for 0² = 0.01 (above) and 0² = 1 (below), respectively. First row: Samples used for training the respective model: FG (left), NG (middle), FOM/ X2 (right). The black line depicts the manifold X' (a circle with radius 3) and the color codes the value of p" (x)) Second row: Colored line: Learned density ((x)) according to equation multiplied by 3. Blackline: ground truth von Mises distribution.
To measure the dependence of our method on the magnitude of noise, we iterate this experiment for various values of 2 and estimate the Kolmogorov-Sming (KS) statistics. The KS statistic is
Note that method still depends how well NF learn the inflated distribution.

---

Under review as a conference paper at ICLR 2021
defined as KS sup,ex [F(x) - G(x)), where F and G are the cumulative distribution functions
associated with the probability densities p(æ) and ((z). respectively. By definition, KS E [0, and KS = 0 if and only if p(z) g(x) for almost every E X. However, equation (6 is only valid the conditions of Theorem are fulfilled. There is no reason why using equation 7 for the full Gaussian noise would lead density the manifold X. The KS statistics is ill-posed in this case. Nevertheless, we are interested in measuring the sensitivity to the noise, and thus consider the KS statisties as relative performance measure.
In Figure we display the KS values depending on different levels of noise, for the NG (blue) and FG (orange) noise compared with the ground truth von Mises distribution. Also, we embed the circle into higher dimensions D 5, 10, 15, 20 and repeat this experiment. The result for D = 2 and D are shown the first row (left and right) For
2, we add the FOM model (which independent of o²) horizontally. Besides, we depict the lower and upper bound for Φ² from Chapter 3.3 with dashed vertical lines. In the lower-left image, we show the optimal KS values obtained for both models depending on D. The lower-right image shows the corresponding o² for those optimal KS. bright, the optimal average o² shown whereas the dark regions are the minimum respectively maximum values for o² such that outperformed the FOM benchmark. We note that for both cases, the averaged optimal o² is within the predicted bounds for o².
D=2
D=20
0.8
NS
NS
0.6
12 0.4
12 0.4
0.2
0.0
0.0
16a
10
10
10:2
105
10a4
16a4
10
16
18F
FC'
NS"
10-1
45"
FOM
10
0.015
0.010
10
0.005
16-
0.000
15
D
D
Figure 4: KS values for the NG- (blue) and FG-noise method (orange) depending on σ²
[10-9, 10] and the embedding dimension D = 5, 10, 15, 20 in log-scale. For D = 2 and D = 20 (top right), the two vertical lines represent the lower and upper bound for x² estimated according to with
ChapterB.33 with 10K samples. We plot horizontally the KS value obtained from FOM. Bottom left: Optimal KS values depending on D. Bottom right: Optimal averaged o² such that optimal KS obtained (bright). The maximum and minimum x² such that the FOM benchmark is outperformed (dark). The dashed horizontal lines are again the theoretical bounds. We used 10 seeds for the error bars and plot in log-scale.
Several aspects are remarkable. The flat course of the KS vs. o² plot is an indicator that the method not very sensitive to noise and this does not change with the dimensionality of the embedding space. Also, the optimal KS values do not change much depending on D and the NS and FG model approach each other, as predicted.
Interestingly, the onset for the increase in the KS value for the NS-noise is roughly 3 which is the radius of the circle. For increasing o², X resembles more and more double cone which is not
NNoe that the scaling factor depends D, (z(r)) = 1/ 1/(2mo²) Rad
8

---

Under review as a conference paper at ICLR 2021
diffeomorphic to R and thus the NF used to train the inflated distribution may not be able to capture the density close to the circle's 's center correctly.
4.2 MIXTURES ON 8
We show that we can learn complicated distribution, a mixture of 2-dimensional von Mises dis- tributions on sphere with radius without using any knowledge about the manifold except for its intrinsic dimension. For certain magnitudes of o², we obtain similar estimates as the FOM bench- mark as we can see in the direct comparison of the learned densities, see Figure 5 (top right), and the KS statistics (bottom right). As for the circle, the Gaussian restricted in the normal space allows for greater range of noise magnitude without sacrificing the quality of the estimate.
Terget density
FOM
Gaussion
50
10
2.5
2.4
15
2.
1A
12
1.8
2.3
3A
32
26t4 22-4
12"
12'
Figure 5: Left: Target density. Upper right: Learned densities using FOM and our method with Gaussian noise and 0² = 0.01. Lower right: KS vs. σ² plot of the Gaussian noise model (full and in normal space) compared to the FOM with the theoretical bounds from Chapter for x² depicted in vertical dashed lines (with 10K samples used to approximate these bounds).
DISCUSSION
To overcome the limitations of NFs to learn density p²(x:) defined on low-dimensional manifold, we proposed to embed the manifold into the ambient space such that becomes diffcomorphic IR. learn this inflated density using NF, and, finally, deflate the inflated density according to Theorem [ There, we provided sufficient conditions on the choice of inflation such that we can compute pz (x) exactly. Notably, we don't need assume that p²(x) is supported on flat manifold. Our method depends on some critical points which we addressed in Section 3.3 So far, the magnitude of noise o when using NFs real-world data is somewhat chosen arbitrarily. As a first step to overcome this arbitrariness, we derived an upper bound for o² in Proposition and established an interesting connection to the manifold learning literature in Theorem
We hope that our theoretical results motivate some new research directions. Using full Gaussian noise to learn the inflated distribution smears information on p" (x)) in particular if p" (x)) has many local extrema. This loss of information may be especially impactful in out of distribution (OOD) detection or when it comes to adversarial robustness. Therefore, developing methods which allow to generate noise in the manifold's 's normal space could improve the performance of NFs on such tasks. Another interesting direction is to exploit the product form of equation (6) and learn low dimennsiioi representations by forcing the NF to be noise insensitive in the first d-components and noise sensitive in the remaining ones. Inverting the corresponding flow allows to sample directly from the manifold which has the potential to improve the generative ability of NFs.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Clément Berenfeld and Mare Hoffmann. Density estimation on an unknown submanifold. arXiv preprint arXiv: 1910.084777 2019.
Johann Brehmer and Kyle Cranmer. Flows for simultaneous manifold learning and density estima- tion. arXiv preprint arXiv:2003.13913, 2020.
Rob Cornish, Anthony L Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. arXiv preprint arXiv 13833, 2019.
Nicola De Cao, Ivan Titov, and Wilker Aziz. Block neural autoregressive flow. arXiv preprint arXiv:1904. 04676, 2019.
Emilien Dupont, Arnaud Doucet, and Yee Whye Tch. Augmented neural odes. In H. Wallach, H. Larochelle, A. Beygelzimer, d' Alché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 3140-3150. Curran Associates, Inc., 2019. URL htth:
7/papers nips.
pdf
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems, pp. 7511-7522, 2019.
Mevlana C Gemici, Danilo Rezende, and Shakir Mohamed. Normalizing flows riemannian man- ifolds. arXiv preprint 1rX.v:1611.02344 2016.
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregres- sive flows. volume 80 of Proceedings of Machine Learning Research, pp. 2078-2087, Stock- holmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http::/pproeeeddiio
mlr. www.ppeesssss.. v80 htt////hhaggga...c html
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint a6Xiv: 10080088,, 2016.
George Papamakarios, Eric Nalisnick. Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lak- shminarayanan. Normalizing flows for probabilistic modeling and inference. arXiv preprint 11Xiv:1912.02762, 2019.
Danilo Jimenez Rezende, George Papamakarios, Sébastien Racanière, Michael S Albergo, Gurtej Kanwar, Phiala E Shanahan, and Kyle Cranmer. Normalizing flows on tori and spheres. arXiv preprint arXiv:2002.02448. 2020.
A APPENDIX
A.1 PROOF OF THBOREM 1
We denote the probability measure of the random variable X as Px and is defined on (X, B(X)) where ((X)) the set of borel measure in RD intersected with X. For realisation X, say x, we denote the probability measure of the shifted random variable x t. IP xxxx== and it is defined on (Nz, (.. We extend both measures to (R., B(RP)) by setting the probabilities to whenever set A E B(RP) has no intersection with X' or NN, respectively. For instance, that means for I E N₂ that
P[r + En E (ū, di)] =P/x + t E (ž, di) n N = =I Pxx==l(ē, ā dā) n NNJ (10)
where (š, + dì) denotes an infinitesimal volume element around ã.
The mapping (x, e) + + e is B(R²)
- because R is topological vector space and B)BD) the Borel a alageraa. What follows is that X X + en is random variable on (RR ()) and has the pushforward of P(..,.) with regard to the mapping (I, En) + x + En as probability measure where P(xx.s.) is the joint measure of X and &n. Thus, for E B,Y), we have that
P (A) = P(x...) (1(x,e) E D x RD tEE A))
(11)
10

---

Under review as a conference paper at ICLR 2021
Now let N₂ for X. Since X' is Q -onommllly separated, Px almost all i are uniquely determined by (z, e) such that I = x + c. Therefore, we have for Px -almosst all + e that
(x((ū, dā) n Xx ((x,s)) (((x,e) E RP x + ( (G)
P E E (ū,ā+
P (X (I, dx)
(or &n (2.
Px ((x, x + dr) n nx) IP xxxx= ((T,
(12)
where for the first equality we used equation and for the third the fact that z and are almost surely uniquely determined by B.
Both probability measures on the right-hand side have a density. For Ix with respect to dV, f. see
Section 2 this density is p" (x). Similarly, since N₂ is a linear subspace of RD, 9n (o|x) is the density of P x| with respect to the volume form dV, where h is the mapping from RD- D-d N
see Remark 1
Then, the corresponding density of Pxx is with respect to the product measure dV.. dV,. However, this product measure is equivalent to Ap when restricted to subsets of X and thus qu is the density of P & with respect to Xp Therefore, we can write equation (12 in terms of densities as follows:
9n (σ) p" (z))(((
(13)
and it holds that
/ (ī)dū = / (((3)a(((
No
p² (z))
(14)
as needed for density X. By setting ž x in equation we obtain equation 7 This ends the proof.
Remark 2 Note that in Theorem 1 we need that X is diffeomorphic to RRP. This requires that the noise distribution qu (f(x) is continuous for all x.
A.2 PROOF OF PROPOSITION II
The generating function f is embedding for X' and X f(2) has the density p (z) for E X. We may extend the domain of p² (x) to include all points x R2D using the Dirac-delta function as follows
/ 8(x - (())
x²(x)=
(15)
After inflating X, we have that
ps ( = ()))
(16)
with covariance matrix Σ DxD Assume f x for some X. We Taylor expand f(x)
around 2g = f (x)) up to first order,
f(z) re f(z0) + Jr(20)(2 20).
(17)
and (=) up to second order.
(2) (20) (20) -20) + 1 1 20)TT T (zo) 20).
(18)
This because the column vectors of Jy and Jn form basis of R
11

---

Under review as a conference paper at ICLR 2021
where x22) denotes the gradient and (20) the Hessian of т evaluated 2u. thus T(2)) R
and (20) E Raxd Then, we can approximate pr (z) as follows:
(x( (2 det(E) exp( - / -2)) Jf -
20)
Jz(z z0))
1
(T(2)) (z0) (2 20)
=0) x >===())== 20))dz.
(19)
2
Now define -- JJ 13/+___j1 Jg. Then,
V det(EE)
exp( 1ap--20 1 21(0(=22220)""
px(z) R V (2 D det(E) 2 (27)ddel(S)
-==)==)²² 1 20))
(π(()) + T(20) (z 20) 1(2 20) ==== (20)((2 20))
(20)
Thus, we can exploit this Gaussian in Z-space and get
V (e)
(x
(T(20) + _ [[z - 20) 20) 20)))
(d)
V det(E)
(m(2 + (10= @ Ei1++)
(21)
v de)
where @ stands for the elementwise multiplication and ||A||+ Σij=1 Ag for a Rd Rd matrix
A.
For the special case where Σ. = T²Tp, we can simplify this expression by exploiting that
V det (
1 D D-d 2 o
(2 D a-d det(E)
(2T)
d og
(22)
(2TT²) D-d Vg
Thus, in total, we get for this special choice of Σ
1
(T()) + - 1 3 (J] -1)+))
0²
(x(
(2
2x" Vo
2
1
(220
( || |km (20) o J1)
2
(23)
We assume (2
dnd Vg
g
| (zo) J |1- 1
(24)
2m (=))
Note that 1/(2mo²)) 0r from equation 23 is exactly the normalization constant obtained when R=d
inflating the manifold with Gaussian noise the normal space, qu (x|x) 1/(2702)
What
follows is that lim, (x)/q. = p p²(x)) we wanted to show.
A.3 PROOF OF THEOREM 2
The theorem follows directly from the definition of the reach number Tx of X. is defined as the supremum of all r 0 such that the orthogonal projection prx on X' is well-defined on the neiiggbbourhood x of X.
x= = (6 T dist (7, x) S
(25)
12

---

Under review as a conference paper at ICLR 2021
where (ist( () denotes the distance of ē to X. Thus,
Tx sup {r 2 VI E R., dist(E, X) r = Eie E & s.t. dist(E, = | x x||}
(26)
see Definition 2.1. in Berenfeld & Hoffmann 2019). By assumption 7x > 0. Thus for all E A have that x pr x is unique. Since X' is manifold, it must hold that E N₂ where Nz denotes the normal space in æ. Let the noise generating distributions be a uniform distribution on
the ball with radius 7x, thus
4u (Z|z) Uniform( (x)
(27)
where B(x, Tx) denotes a D-dimensionall ball with radius Tx and center z.. Then, we have for X UsEx (a)(-la) that
x= xx
(28)
Thus, X' is Q-normally separable where Q = {-u(1l=))xxxx
B EXPERIMENTS
B. TECHNICAL DETAILS FOR CIRCLE EXPERIMENTS
For the Normalizing Flow T (z) we BNAF (Block Neural Autoregressive Flow) for the circle experiment. The number hidden dimensions was adapted to the dimensionality of the data and the difficulty of the target density. These details reported in the corresponding tabular. For the optimization scheme, we used Adam optimizer with an initial learning rate 0.1, a learning rate decay of 0.5 after 2000 optimization steps without improvement (learning rate patience). The batch size was set to 200. The total number of iterations (one iteration corresponds to updating the parameters using one batch sample) used is also reported in the tabular. No hyperparameter fine-tuning was done.
For the FOM and x² noise models, we use the same architecture as for the D case.
Data dimension
hidden layers
hidden dimension
total parameters
iterations
2
3
100
31,204
70000
3
250
192,010
70000
10
3
500
764,000
70000
15
3
750
1,716,033
100000
20
3
1000
3,048,040
100000
Table 1: BNAF details for circle experiments.
B.2 TECHNICAL DETAILS FOR MIXTURE OF VON MISE DISTRIBUTIONS ON $2
For the Normalizing Flow (=) we use rational-quadratic neural spline flows, alternating cou- pling layers and random feature permutations, see Durkan et al. (001)). For the optimization scheme, we used AdamW optimizer with an initial learning rate 0.0003, learning rate cosine decay to after every 2000 optimization steps, see see|Loshee Loshchilov & Hutter (016), weight decay of 0.0001 and dropout probability of O. The batch size was set to 200. No hyperparameter fine-tuning was done. See table 2 for more details. We use the same architecture for the FOM model and seeds for the error bars.
Coupling
residual blocks
hidden features
bins
spline range
total parameters
iterations
10
3
50
8
3
171,845
50000
Table 2: BNAF details for mixture of von Mises S2
13

---

Under review as a conference paper at ICLR 2021
The target distribution is mixture of four von Mises distributions, PE), 82), P3 (0s,, 0z) and pi (1,, 01). Each of those distributions has the same product form
p (oi, 0.) exp(x cos(O; ps)) exp(n cos(c(ss m.))) (xxx(()
(29)
Tlo(k)
We 6. However, they differ in their mean values μu, and my see table seeds in total to obtain the confidence intervals.
We used different
Mi
m4
2
48
3x
3
3
3x
4+
3
Table 3: Mean values for the mixture of von Mises distributions.
14


Reviews:
[{"review_id": "l4KXKqe9FV8", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tackles the problem of recovering a probability distribution, which is supported in a low-dimensional manifold. When the dimension of that manifold is full, the problem can be solved by the Normalizing Flow method. \n\nSince the dimension is not full, it is suggested to add a gaussian noise to the data points (this is equivalent to a certain convolution of the initial distribution function, ie equation 3). Then full dimensional manifold is recovered and subsequently deconvolution is applied (equation 7).\n\nCertain mathematical aspects of the narrative are vague. Eg \\tilde{X} is defined as X x Noise, but then in equations 4 and 5, it becomes a subset of the plane. \n\nThe notion of Q-normally separated is never satisfied in practice. It is only slightly stronger than \"all the normal spaces have no intersections at all\". The deconvolution made by equation 7 is rarely is a good solution.\n\nAlso, the experimental part is purely dealing with synthetic examples. This could be because the deconvolution by equation 7 is not the best way to recover the signal. \n\nIt is claimed that the first goal of synthetic experiments is to confirm the scaling factor in equation (7) numerically. But experiments deal with highly specific synthetic data - a curve (ie, 1D manifold) on the plane. It is doubtful that such a simple experimental framework can justify eq (7). It is not computationally difficult to check the quality of deconvolution in a style of eq (7) for higher-dimensional data.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Rooj6vKrGUx", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors address main limitations of Normalizing Flows (NFs) method for estimation of density functions on manifolds. Since NFs requires the support of density function to cover the whole Euclidean space, they propose to add noise (inflate) to apply NF. The paper is nicely written with clear introduction of basic concepts very useful for non-expert readers. They provide theoretical guarantees on the variance and type of added noise that make the method work and illustrate with synthetic experiments.\nMy only concern is about the applicability and usefulness on real data and specific machine learning. I have found some issues and also have some questions. See below:\nMajor issues:\n-\tHow this method can be used for a real problem with real datasets. Could you please provide some example or give some guidance about the potential of this method for the machine learning community?\n-\tIt is shown that when D>>d, Gaussian noise is an excellent approximation of noise restricted in the normal direction and the experiments seem to confirm this. What should be the criterion to say that D is large enough to make this approximation useful? Can you give some way to test it in practice?\n-\tExample 2: a graphical description of this example would help, for example, showing that variance of noise can help to make the Q-normally separated condition to be met.\n-\tPage 5: I didn’t understand the following sentence “A potential future avenue is to \\sigma^2A and this reach number”. Could you please elaborate it?\n\nMinor issues\n-\tIn Example 1: “an z” -> “a z”\n-\tReferences have missing information. For example, for Cornish et al, Gemici et al and Rezende et al papers, there is not information about the source. Did they works published in conferences, journals or ArXiV?\n-\tPage 12: Please correct the number of table in “Table ??” \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PHwy6LtjTrh", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for estimating the probability deinsity distribution on a low-dimensional manifold embedded in a high-dimensional space using Normalizing Flow (NF). The problem is that the universality of NF is limited because low-dimensional manifolds are not diffeomorphic with respect to high-dimensional Euclidean space. The proposal of this study is to make NF applicable by inflating low-dimensional manifolds with Gaussian noise. Then, after the transformation is obtained, the probability distribution on the original low-dimensional manifold can be obtained by deflation.\n\nThere is a detailed discussion about the addition of Gaussian noise. Theoretically, any Gaussian noise $ e $ at coordinate x can be decomposed into projections on tangent space $ T_x $ and normal space $ N_x $ as $ e = e_t + e_n $. Only normal noise $ e_n $ is ideal to add. However, it is not realistic to find this at each coordinate x. This paper argues that Gaussian noise $ e $ is a good approximation of $ e_n $ when the manifold dimension $ d $ is sufficiently smaller than the higher-dimensional Euclidean space dimension $ D $. This also argues that the noise variance $ \\ sigma $ at that time should be set according to the inverse of the density. \n\nIn computer experiments, simple simulation results using the von mises distribution on a circle and a sphere are shown.\n\nThe gist of the manuscript is well-written, and the issues it deals with are also important and interesting.\nTheoretically, interesting discussions are being developed, but discussions and experimental results are somewhat weak regarding the merits of practical application.\n\nI have some questions.\n1. I couldn't understand the description about scalability well, \"our method scales to high dimensions because it is based on one NF and does not require to compute its inverse.\" Isn't the scalability the same as normal NF because the proposed method basically uses normal NF?\n\n2. In this research, Authors propose to add noise to the data sampled from low-dimensional manifolds in a pseudo manner, but in actual measurement, noise has been included in data naturally without adding it in a pseudo manner. Is it necessary to add pseudo noise even when applying it to real world data?\n\n3. The denoising auto-encoder is famous as a manifold learning method that adds pseudo noise. Is there any relation to this? Also, I would like to know if there are any advantages of the proposed method over the denoising auto-encoder.\n\n4. VAE is a well-known method for finding the mapping of low-dimensional space to the normal distribution. Is there any relation to this? Also, I would like to know if there are any advantages of the proposed method over VAE.\n\n5. In relation to the above, it may be good to have a comparison experiment with denoising auto-encoder and VAE.\n\n6. The lack of experimental results with actual data makes that the paper is unconvincing. Especially for the description of scalability, it is better to prove it experimentally.\n\n7. This is a pure question, is it possible to know the dimensions of the manifold through this technique?", "extra_scores": {}, "extra_text": {}}, {"review_id": "1ZPdkqRIYqP", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents a novel theoretical development to tackle the problem of estimating normalising flows (NF) for data with support on complex manifolds. Motivated by the fact that NF are diffeomorphic  transformations from a simple space, ideally Euclidean, the author address the problem of modeling data which distribution is defined on more complex and unknow manifolds.  \nThe idea consists in inflating the data manifold with suitable noise (normal noise) in order to make it diffeomorphic to a simpler space where a NF can be estimated. The data density can be therefore approximated from the inflated space through an opportune deflation operation.\n\nThe main contribution of the work consists in formalising the conditions on the inflating noise to obtain the results of Theorem 1. In practice, Proposition 1 shows that the noise can be drawn from a Gaussian with variance depending from the curvature of the manifold. The experimental section demonstrates the theory on simple synthetic cases on S2.\n\nThe idea proposed in the paper is interesting, and of potentially utility. The derivation is however unclear in some aspects, especially concerning the definition of Q-separation. In particular, Definition 1 implies null measure for the set of manifold points spanning the normal subspace. However, to my understanding, this definition is not sufficient to guarantee the unicity of the mapping between normal space and manifold point. For example, any finite collection of manifold points would still have zero measure. In this case however, the unique correspondence between normal space and manifold is broken. Back to Example 2, this non-unicity can be achieved by increasing the noise along the normal direction to cross the origin (0,0). In this case each point of the inflated space has two generators, making the isomorphic mapping  impossible. I can understand that this aspect is related to the choice of the noise discussed in Section 3.3, but this issue still undermines the conclusion of Theorem 1. I would encourage the authors to clarify this point.\n\nThere are  further technical aspects deserving clarification. In Example 2, the definition of the set A(\\tilde{x}) should not contemplate the empty set. Rather, A(\\tilde{x}) should be the point x’ if \\tilde{x} is not (0,0). Moreover, in Proof A.1, equation (13), the relation between the q_n and \\hat{q}_n in the two first integrals, and the reason why they can be exchanged, is not clear. \n\nFinally, some questions arise concerning the practical feasibility of the proposed theory. Computing the curvature of the data manifold may be extremely challenging in presence of complex data, especially in the low sample-size regime. In this case, the estimation of the noise parameter in formula (9) may be practically impossible.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides an interesting set of theoretical ideas to improve the estimation of normalizing flows on datasets that fail to be fully dimensional. Although the method is appealing, I believe the paper falls a bit short of acceptance at the conference. Too many practical issues are left out, as discussed by reviewers, and the method seems promising but not fully connected to the rest of the literature on estimating low-dimensional distributions living in high dimensional spaces. We encourage the authors to use the feedback contained in this round of reviews to improve their work.
Decision: Reject