Year: 2020

Abstract:
Normalizing Flows (NFs) are universal density estimators based on Neuronal Networks. However, this universality is limited: the density’s support needs to be diffeomorphic to a Euclidean space. In this paper, we propose a novel method to overcome this limitation without sacrificing the universality. The proposed method inflates the data manifold by adding noise in the normal space, trains an NF on this inflated manifold and, finally, deflates the learned density. Our main result provides sufficient conditions on the manifold and the specific choice of noise under which the corresponding estimator is exact. Our method has the same computational complexity as NFs, and does not require to compute an inverse flow. We also show that, if the embedding dimension is much larger than the manifold dimension, noise in the normal space can be well approximated by some Gaussian noise. This allows using our method for approximating arbitrary densities on non-flat manifolds provided that the manifold dimension is known.

text: Normalizing Flows (NFs) are universal density estimators based on Neuronal Networks. However, this universality is limited: the density’s support needs to be diffeomorphic to a Euclidean space. In this paper, we propose a novel method to overcome this limitation without sacrificing the universality. The proposed method inflates the data manifold by adding noise in the normal space, trains an NF on this inflated manifold and, finally, deflates the learned density. Our main result provides sufficient conditions on the manifold and the specific choice of noise under which the corresponding estimator is exact. Our method has the same computational complexity as NFs, and does not require to compute an inverse flow. We also show that, if the embedding dimension is much larger than the manifold dimension, noise in the normal space can be well approximated by some Gaussian noise. This allows using our method for approximating arbitrary densities on non-flat manifolds provided that the manifold dimension is known.

heading: 1 INTRODUCTION
text: Many modern problems involving high dimensional data are formulated probabilistically. Key concepts, such as Bayesian Classification, Denoising or Anomaly Detection, rely on the data generating density p∗(x). Therefore, a main research area and of crucial importance is learning this data generating density p∗(x) from samples.
For the case where the corresponding random variable X ∈ RD takes values on a manifold diffeomorphic to RD, a Normalizing Flow (NF) can be used to learn p∗(x) exactly (Huang et al., 2018). Recently, a few attempts have been made to overcome this topological constraint. However, to do so, all of these methods either need to know the manifold beforehand (Gemici et al. (2016), Rezende et al. (2020)) or they sacrifice the exactness of the estimate (Cornish et al. (2019), Dupont et al. (2019)).
Our goal in this paper is to overcome both the aforementioned limitations of using NFs for density estimation on Riemannian manifolds. Given data points from a d−dimensional Riemannian manifold embedded in RD, d < D, we first inflate the manifold by adding a specific noise in the normal space direction of the manifold, then train an NF on this inflated manifold, and, finally, deflate the trained density by exploiting the choice of noise and the geometry of the manifold. See Figure 1 for a schematic overview of these points.
Our main theorem states sufficient conditions on the manifold and the type of noise we use for the inflation step such that the deflation becomes exact. To guarantee the exactness, we do need to know the manifold as in e.g. Rezende et al. (2020) because we need to be able to sample in the manifold’s normal space. However, as we will show, for the special case where D d, the usual Gaussian noise is an excellent approximation for a noise in the normal space component. This allows using our method for approximating arbitrary densities on Riemannian manifolds provided that the manifold dimension is known. In addition, our method is based on a single NF without the necessity to invert it. Hence, we don’t add any additional complexity to the usual training procedure of NFs.
Notations: We denote the determinant of the Gram matrix of f as gf (x) := |det ( Jf (x) TJf (x) ) | where Jf (x) is the Jacobian of f . We denote the Lebesque measure in Rn as λn. Random variables will be denoted with a capital letter, say X , and its corresponding state space with the calligraphical
version, i.e. X . Small letters correspond to vectors with dimensionality given by context. The letters d,D, n, and N are always natural numbers.

heading: 2 BACKGROUND AND PROBLEM STATEMENT
text: An NF transforms a known auxiliary random variable by using bijective mappings parametrized by Neuronal Networks such that the given data points are samples from this transformed random variable, see Papamakarios et al. (2019). Formally, an NF is a diffeomorphism Fθ : U → X and induces a density on X through pθ(x) = (gF (u)− 1 2 pU (u) where pU (u) is known and u = F−1θ (x). The parameters θ are updated such that the KL-divergence between p∗(x) and pθ(x),
DKL(p ∗(x)||pθ(x)) = Ex∼p∗(x)[log pθ(x)] + const. (1)
is minimized. If Fθ is expressive enough, it was proven that in the limit of infinitely many samples, updating θ to minimize this objective function converges to a θ∗ such that it holds PX -almost surely p∗(x) = pθ∗(x), see (Huang et al., 2018).
More generally, let X ∈ X ⊂ RD be generated by an unobserved random variable Z ∈ Z ⊂ Rd with density π(z), that is X = f(Z) for some function f : Z → X where typically d < D. In Gemici et al. (2016), f is an embedding1, and it was shown that one can calculate probabilities such as PX(A) for measurable A ⊂ X using a density p∗(x) with respect to the volume form dVf induced by f , that is
P(X ∈ A) = ∫ f−1(A) π(z)dz = ∫ A p∗(x)dVf (x) (2)
with p∗(x) = π(z)gf (z)− 1 2 and dVf (x) = √ gf (z)dz where z = f−1(x). Hence, given an explicit mapping f and samples from p∗(x), we can learn the unknown density π(z) using a usual NF in Rd. However, in general, the generating function f is either unknown or not an embedding creating numerical instabilities for training inputs close to singularity points.
In Brehmer & Cranmer (2020), f and the unknown density π are learned simultaneously. The main idea is to define f as a level set of a usual flow in RD and train it together with the flow in Rd used to learn π(z). To evaluate the density, one needs to invert f and thus this approach may be very slow for high-dimensional data. Besides, to guarantee that f learns the manifold they proposed several ad hoc training strategies. We tie in with the idea to use an NF for learning p∗(x) with unknown f and study the following problem.
1Thus, a regular continuously differentiable mapping (called immersion) which is, restricted to its image, a homeomorphism.
Problem 1 Let X be a d−dimensional manifold embedded in RD. Let X = f(Z) be a random variable generated by an embedding f : Rd → RD and a random variable Z ∼ π(z) in Rd. Given N samples from p∗(x) as described above, find an estimator p̂ of p∗ such that in the limit of infinitely many samples we have that p̂(x) = p∗(x), PX − almost surely.

heading: 3 METHODS
text: To solve Problem 1, we want to exploit the universality of NFs. We want to inflate X such that the inflated manifold X̃ becomes diffeomorphic to a set U on which a simple density exists. Doing so, allows us to learn the inflated density p(x̃) exactly using a single NF, see Section 2. Then, given such an estimator for the modified density, we approximate p∗(x) and give sufficient conditions when this approximation is exact.

heading: 3.1 THE INFLATION STEP
text: Given a sample x of X , if we add some noise ε ∈ RD to it, the resulting new random variable X̃ = X + ε has the following density
q(x̃) = ∫ X q(x̃|x)dPX(x). (3)
Denote the tangent space in x as Tx and the normal space asNx. By definition, Nx is the orthogonal complement of Tx. Therefore, we can decompose the noise ε into its tangent and normal component, ε = εt + εn. In the following, we consider noise in the normal space only, i.e. εt = 0, and denote the density of the resulting random variable as qn(x̃). The corresponding noise density qn(x̃|x) has mean x and domain Nx. We denote the support of qn(·|x) by Nqn(·|x). The random variable X̃ = X + εn is now defined on X̃ = ⋃ x∈X Nqn(·|x). We want X̃ to be diffeomorphic to a set U on which a known density can be defined.
Example 1 (a) Let X = S1 = {x ∈ R2 | ||x|| = 1} be the unit circle. For each x ∈ S1 there exists z ∈ [0, 2π) such that x = er(x) = (cos(z), sin(z))T . To sample a point x̃ in Nx, which is spanned by er(x), we sample a scalar value γ and set x̃ = x + γer(x). With γ ∼ Uniform[−1, 1), we have that
X̃ = ⋃ x∈X {x+ γer(x)|γ ∈ [−1, 1)} = {x ∈ R2 | ||x||2 < 2} (4)
which is the open disk with radius 2. The open disk is diffeomorphic to (0, 1) × (0, 1). Thus, qn(x̃) can be learned by a single NF F−1 and pU (u) = Uniform ((0, 1)× (0, 1)) as reference.
(b) As in (a), we consider the unit circle. Now we set γ to be a χ2− distribution with support [−1,∞). Then
X̃ = ⋃ x∈X {x+ γer(x)|γ ∈ [−1,∞)} = R2. (5)
Thus, qn(x̃) can be learned by a single NF F−1 and pU (u) = N (u; 0, ID) as reference. Both cases can be analogously extended to higher dimensions.
Remark 1 To be precise, the random variable εn is generated by a random variable in RD−d, say Γ, with measure PΓ. Then, qn(x̃|x) is the density of the pushforward of the noise measure PΓ with regard to the mapping h : RD−d → Nx. Hence, formally, the density q(x̃|x) is with respect to the induced volume form dVh, see Section 2. However, if we choose an orthonormal basis for Nx, say n(1), . . . , n(D−d), then we have that x̃ = h(γ) = Aγ + x where the columns of A ∈ RD×(D−d) are given by these basis vectors, i.e. A = [n(1), . . . , n(D−d)]. Thus, the Gram determinant of h is gh = det(ATA) = 1 and we have that dVh(x̃) = dγ where dγ denotes the volume form with respect to λD−d. In this case, we can think of qn(x̃|x) as a density with respect to λD−d. If qn(x̃|x) depends only on ||x̃ − x||, as it is for the Gaussian distribution, we have that qn(x̃|x) = qn(||x̃ − x||) = qn(||γ||) because h is an isomorphism. Thus, for this case it holds that qn(x̃|x)dVh(x̃) = qn(||γ||)dγ. Then, for convenience, we may abuse notation by writing γ ∼ q(x̃|x) or εn ∼ r(γ) where r(γ) is the density of PΓ with respect to λD−d.

heading: 3.2 THE DEFLATION STEP
text: Our main idea is to find conditions such that
qn(x̃) = qn(x̃|x)p∗(x) (6)
for almost surely all x̃ ∈ X̃ and for an almost surely unique x ∈ X . Because then, given an exact estimator of qn(x̃), say q̂n(x̃), we have for x̃ = x that p∗(x) = q̂n(x)/qn(x|x). For equation (6) to be true, we need to guarantee that almost every x̃ corresponds to only one x ∈ X . This is certainly the case whenever all the normal spaces have no intersections at all (think of a simple line in R2). We can relax this assumption by allowing null-set intersections. Moreover, only those subsets of the normal spaces are of interest which are generated by the specific choice of noise qn(x̃|x). Thus, only the support of qn(x̃|x), Nqn(·|x), matters. The key concept for our main result is expressed in the following definition:
Definition 1 Let X be a d−dimensional manifold and Nx the normal space in x ∈ X . Let qn(·|x) be a density defined on Nx and denote by Nqn(·|x) the domain of qn(·|x). Denote the collection of all such densities as Q := {qn(·|x)}x∈X . For x̃ ∈ X̃ , we define the set of all possible generators of x̃ as A(x̃) = {x′ ∈ X |Nqn(·|x′) 3 x̃}. We say X is Q−normally separated if for all x ∈ X holds that PX̃|X=x [x̃ ∈ Nx|#A(x̃) > 1] = 0 where #A(x̃) is the cardinality of the set A(x̃). In words, every x̃ ∈ Nx is PX̃|X=x-almost surely determined by x.
To familiarize with this concept, consider Figure 2 and the following example:
Example 2 For the circle in example 1, we choose εn to be uniformly distributed on the half-open interval [−1, 1). The point (0, 0)T is contained in Nqn(·|x) for all x ∈ X and thus Nqn(·|x′) ∩ Nqn(·|x) = {(0, 0)T } for all x 6= x′, see Figure 2 (middle). Hence, for any given x̃ ∈ Nx we have
that A(x̃) = { X if x̃ = (0, 0)T , x else, and therefore #A(x̃) = { ∞ if x̃ = (0, 0)T , 1 else.
Thus, PX̃|X=x [ x̃ ∈ X̃ |#A(x̃) > 1 ] = PX̃|X=x [ x̃ = (0, 0)T ] = 0 for all x ∈ X . What follows is that X is Q−normally seperated. If we were to choose εn to be uniformly distributed on [−1.5, 1), see Figure 2 (right), the normal spaces would overlap and we would have that PX̃|X=x [ x̃ ∈ X̃ |#A(x̃) > 1 ] > 0. In this case, X would not be Q−normally seperated.
Theorem 1 Let X be a d−dimensional manifold. For each x ∈ X , let qn(·|x) denote a distribution on the normal space of x. Let X be Q−normally separated where Q := {qn(·|x)}x∈X . Assume
that we can learn the density qn(x̃) as defined in equation (3), by using a single NF F−1, thus qn(x̃) = (g F (F−1(x̃)))− 1 2 pU (F
−1(x̃)) for some known density pU . Then, for PX̃−almost all x̃ ∈ X̃ holds that qn(x̃) = p∗(x)qn(x̃|x), therefore this equation, when evaluated at x̃ = x, yields
p∗(x) = qn(x)
qn(x|x) . (7)
The proof can be found in Appendix A.1.
3.3 GAUSSIAN NOISE AS NORMAL NOISE AND THE CHOICE OF σ2
Our proposed method depends on three critical points. First, we need to be able to sample in the normal space of X . Second, we need to determine the magnitude and type of noise. Third, we need to make sure that the conditions of Theorem 1 are fulfilled. We address (partially) those three points.
1. For the special case where D d, we show that a full Gaussian noise is an excellent approximation for a Gaussian noise restricted to the normal space. Consider ε = εt + εn, ε ∼ N (0, σ2ID). Then, the expected absolute squared error when approximating normal noise with full Gaussian noise is E [ |ε− εn|2 ] = E [ |εt|2 ] = dσ2. The expected relative squared error is therefore
E [ |ε− εn|2
|εn|2
] = E [ |εt|2
|εn|2
] = dσ2E [ 1
|εn|2
] =
d
D − d− 2 (8)
because εt and εn are independent and D−d|εn|2 follows a scaled inverse χ 2−distribution with D − d degrees of freedom and scale parameter 1/σ2. Thus, if D d, Gaussian noise is an excellent approximation for a Gaussian in the normal space. We denote the inflated density with Gaussian noise by qσ2(x̃) in the following.
2. The inflation must not garble the manifold too much. For instance, adding Gaussian noise with magnitude σ ≥ r to S1 will blur the circle. Since the curvature of the circle is 1/r, intuitively, we want σ to scale with the second derivative of the generating function f . Additionally, we do not want to lose the information of p∗(x) by inflating the manifold. If the generating distribution π(z) makes a sharp transition at z0, π(z0 −∆zo) π(z0 + ∆zo) for |∆zo| 1, adding to much noise in x0 = f(z0) will smooth out that transition. Hence, we want σ to inversely scale with π′′(z). We formalize these intuitions in Proposition 1 and prove them in Appendix A.2. In accordance with Theorem 1, we say pσ(x̃) approximates well p∗(x) if limσ2→0 pσ(x)/qn(x|x) = p∗(x) for all x ∈ X where qn(x|x) is the normalization constant of a (D−d)−dimensional Gaussian distribution.
Proposition 1 Let X ∈ RD be generated by Z ∼ π(z) through an embedding f : Rd → RD, i.e. f(Z) = X . Let π ∈ C2(Rd). For qσ2(x̃) to approximate well p∗(x), in the sense that limσ2→0 qσ2(x)/qn(x|x) = p∗(x) for x ∈ X , a necessary condition is that:∣∣∣∣ σ22π(z0) ||π′′(z0) (JTf Jf )−1||+
∣∣∣∣ 1 (9) where ||A||+ = ∑d i,j=1Aij for A ∈ Rd × Rd and denotes the elementwise product, and (π′′(z0))ij = ∂2π(z) ∂zi∂zj |z=z0 is the Hessian of the prior evaluated at z0 = f−1(x).
Intuitively, a second necessary condition is that the noise magnitude should be much smaller than the radius of the curvature of the manifold which directly depends on the second-order derivatives of f . This can be illustrated in the following example:
Example 3 For the circle2 in R2 generated by f(z) = (cos(z), sin(z))T and a von Mises distribution π(z) ∝ exp(κ cos(z)), we get that σ2 min (∣∣∣ 2r2κ(κ sin2(z)−cos(z)) ∣∣∣ , r2) where the first condition comes from Proposition 1 and the second one comes from the curvature argument.
Even though this bound may not be usefull as such in practice when f and π are unknown, it can still be used if f and π are estimated locally with nearest neighbor statistics.
2Technically, the circle does not fulfill the conditions of Proposition 1 since the domain of f is not R.
From a numerical perspective, inflating a manifold using Gaussian noise circumvents degeneracy problems when training an vanilla NF for low-dimensional manifolds. In particular, the flows Jacobian determinant becomes numerically unstable, see equation (1). This determinant is essentially a volume changing factor for balls. From a sampling perspective, these volumes can be estimated with the number of samples falling into the ball divided by the total number of points. Therefore, we suggest to lower bound σ with the average nearest neighbor obtained from the training set to make sure that these volumes are not empty and thus avoid numerical instabilities.
3. Intuitively, if the curvature of the manifold is not too high and if the manifold is not too entangled, Q−normal separability is satisfied for a sufficiently small magnitude of noise. Also in the manifold learning literature, the entangling must not be too strong. Informally, the reach number provides a necessary condition on the manifold such that it is learnable through samples, see Chapter 2.3 in Berenfeld & Hoffmann (2019). Formally, the reach number is the maximum distance τX such that for all x̃ in a τX−neighbourhood of X the projection onto X is unique. In Appendix A.3 we prove Theorem 2 which states that any closed manifold X with τX > 0 is Q−normally separable.
Theorem 2 Let X ⊂ RD be a closed d-dimensional manifold. If X has a positive reach number τX , then X is Q−normally separable where Q := {qn(·|x)}x∈X is the collection of uniform distributions on a ball with radius τX , i.e. qn(x̃|x) = Uniform(x̃;B(x, τX ) ∩ Nx) where B(x, τX ) denotes a D−dimensional ball with radius τX and center x.

heading: 4 RESULTS
text: We have three goals in this section: First, we numerically confirm the scaling factor in equation (7). Second, we verify that Gaussian noise can be used to approximate a Gaussian noise restricted to the normal space. Third, we numerically test our bound for σ2 derived in Section 3.3. Finally, we show that we can learn complicated distributions on S2 without using explicit charts. For training details, we refer to Appendix B.1 and B.2, respectively.

heading: 4.1 VON MISES ON A CIRCLE
text: Let X be a circle with radius 3 and let π(z) ∝ exp(8 cos(z)) be a 1D von Mises distribution. Given z ∼ π(z), we generate a point in X according to the mapping f(z) = 3(cos(z), sin(z)). We want to learn the induced density p∗(x). Note that 3p∗(x) = π(z) since 1/3 is the volume form induced by f . To benchmark our performance, we use the idea in Gemici et al. (2016) to first embed the circle into R, using e.g. f−1, learn the density there with an NF, and transform this learned density back to S1. In Brehmer & Cranmer (2020), this method is named Flow on manifolds (FOM) and we stick to this notation in the following. Note that f is not injective and to illustrate the benefit of our method we choose the singularity point to be (1, 0)T . By moving points close to (1, 0)T slightly away from (1, 0)T , we numerically ensure that f is an embedding.
1. The Inflation step: We inflate X using 3 types of noise: Gaussian in the normal space (NG), Gaussian in the full ambient space (FG), and χ2-noise as described in examples 1(b) with scale parameter 3. Technically, NG violates the Q−normal separability assumption. However, if σ2 is small and the scale parameter for the von Mises distribution is large enough, this is practically fulfilled.
2. Learning qn(x̃): For the NFs we use a Block Neural Autoregressive Flow (BNAF) (De Cao et al., 2019). We use the same NF architecture and training procedures across all models.
3. Deflation: Given an estimator for qn(x̃), we use equation (7) to calculate p∗(x). For FG and NG, we have that qn(x|x) = 1/ √ 2πσ2 and for the normal χ2−noise is qn(x|x) = √ 3e−3/2/( √ 8Γ( 32 )).
In Figure 3, we show the results for σ2 = 0.01 and σ2 = 1. In the respective plot, the first row shows training samples from the inflated distributions qσ2(x̃) (left), and qn(x̃) (middle), respectively. We color code a sample x̃ = x + ε according to p∗(x) to illustrate the impact of noise on the inflated density. Note that the FOM model (top right) does not need any inflation and therefore is trained on samples from p∗(x) only. In the respective plot, the second row shows the learned density for the different models and compares it to the ground truth von Mises distribution π(z) depicted in black. As we can see, for σ2 = 0.01 all models perform very well, although the FOM model
slightly fails to capture p(z) for z close to 0 which corresponds to the chosen singularity point. For σ2 = 1, we see a significant drop in the performance of the Gaussian model. Although the manifold is significantly disturbed, the normal noise model still learns the density almost perfectly3, so does the normal χ2−noise model, as predicted by Theorem 1.
To measure the dependence of our method on the magnitude of noise, we iterate this experiment for various values of σ2 and estimate the Kolmogorov-Smirnov (KS) statistics. The KS statistic is
3Note that our method still depends on how well an NF can learn the inflated distribution.
defined as KS = supx∈X |F (x)−G(x)|, where F and G are the cumulative distribution functions associated with the probability densities p(x) and q(x), respectively. By definition, KS ∈ [0, 1] and KS = 0 if and only if p(x) = q(x) for almost every x ∈ X . However, equation (6) is only valid if the conditions of Theorem 1 are fulfilled. There is no reason why using equation (7) for the full Gaussian noise would lead to a density on the manifold X . The KS statistics is ill-posed in this case. Nevertheless, we are interested in measuring the sensitivity to the noise, and thus consider the KS statistics as a relative performance measure.
In Figure 4, we display the KS values depending on different levels of noise, for the NG (blue) and FG (orange) noise compared with the ground truth von Mises distribution. Also, we embed the circle into higher dimensions D = 5, 10, 15, 20 and repeat this experiment. The result for D = 2 andD = 20 are shown in the first row (left and right).4 ForD = 2, we add the FOM model (which is independent of σ2) horizontally. Besides, we depict the lower and upper bound for σ2 from Chapter 3.3 with dashed vertical lines. In the lower-left image, we show the optimal KS values obtained for both models depending on D. The lower-right image shows the corresponding σ2 for those optimal KS. In bright, the optimal average σ2 is shown whereas the dark regions are the minimum respectively maximum values for σ2 such that we outperformed the FOM benchmark. We note that for both cases, the averaged optimal σ2 is within the predicted bounds for σ2.
Several aspects are remarkable. The flat course of the KS vs. σ2 plot is an indicator that the method is not very sensitive to noise and this does not change with the dimensionality of the embedding space. Also, the optimal KS values do not change much depending on D and the NS and FG model approach each other, as predicted.
Interestingly, the onset for the increase in the KS value for the NS-noise is roughly 3 which is the radius of the circle. For increasing σ2, X̃ resembles more and more a double cone which is not
4Note that the scaling factor depends on D, qn(x|x) = 1/(2πσ2) D−d 2 .
diffeomorphic to R2 and thus the NF used to train the inflated distribution may not be able to capture the density close to the circle’s center correctly.

heading: 4.2 MIXTURES ON S2
text: We show that we can learn a complicated distribution, a mixture of 2-dimensional von Mises distributions on a sphere with radius 1, without using any knowledge about the manifold except for its intrinsic dimension. For certain magnitudes of σ2, we obtain similar estimates as the FOM benchmark as we can see in the direct comparison of the learned densities, see Figure 5 (top right), and the KS statistics (bottom right). As for the circle, the Gaussian restricted in the normal space allows for a greater range of noise magnitude without sacrificing the quality of the estimate.

heading: 5 DISCUSSION
text: To overcome the limitations of NFs to learn a density p∗(x) defined on a low-dimensional manifold, we proposed to embed the manifold into the ambient space such that it becomes diffeomorphic to RD, learn this inflated density using an NF, and, finally, deflate the inflated density according to Theorem 1. There, we provided sufficient conditions on the choice of inflation such that we can compute p∗(x) exactly. Notably, we don’t need to assume that p∗(x) is supported on a flat manifold. Our method depends on some critical points which we addressed in Section 3.3. So far, the magnitude of noise σ2 when using NFs on real-world data is somewhat chosen arbitrarily. As a first step to overcome this arbitrariness, we derived an upper bound for σ2 in Proposition 1 and established an interesting connection to the manifold learning literature in Theorem 2.
We hope that our theoretical results motivate some new research directions. Using full Gaussian noise to learn the inflated distribution smears information on p∗(x), in particular if p∗(x) has many local extrema. This loss of information may be especially impactful in out of distribution (OOD) detection or when it comes to adversarial robustness. Therefore, developing methods which allow to generate noise in the manifold’s normal space could improve the performance of NFs on such tasks. Another interesting direction is to exploit the product form of equation (6) and learn low-dimensional representations by forcing the NF to be noise insensitive in the first d-components and noise sensitive in the remaining ones. Inverting the corresponding flow allows to sample directly from the manifold which has the potential to improve the generative ability of NFs.

heading: A APPENDIX
text: A.1 PROOF OF THEOREM 1
We denote the probability measure of the random variable X as PX and it is defined on (X ,B(X )) where B(X ) is the set of borel measure in RD intersected with X . For a realisation of X , say x, we denote the probability measure of the shifted random variable x+ εn as PX̃|X=x and it is defined on (Nx,B(Nx)). We extend both measures to (RD,B(RD)) by setting the probabilities to 0 whenever a set A ∈ B(RD) has no intersection with X or Nx, respectively. For instance, that means for x̃ ∈ Nx that
P[x+ εn ∈ (x̃, x̃+ dx̃)] =P[x+ εn ∈ (x̃, x̃+ dx̃) ∩Nx] = PX̃|X=x[(x̃, x̃+ dx̃) ∩Nx] (10)
where (x̃, x̃+ dx̃) denotes an infinitesimal volume element around x̃.
The mapping (x, ) 7→ x + is B(RD) × B(RD)−measurable because RD is a topological vector space and B(RD) the Borel σ−algebra. What follows is that X̃ = X + εn is a random variable on (RD,B(RD)) and has the pushforward of P(X,εn) with regard to the mapping (x, εn) → x + εn as probability measure where P(X,εn) is the joint measure of X and εn. Thus, for A ∈ B(X̃ ), we have that
PX̃(A) = P(X,εn) ( {(x, ) ∈ RD × RD|x+ ∈ A} ) . (11)
Now let x̃ ∈ Nx for an x ∈ X . Since X is Q−normally separated, PX̃−almost all x̃ are uniquely determined by (x, ) such that x̃ = x+ . Therefore, we have for PX̃−almost all x̃ = x+ that
PX̃((x̃, x̃+ dx̃) ∩ X̃ ) = P(X,εn) ( {(x, ) ∈ RD × RD|x+ ∈ (x̃, x̃+ dx̃) ∩ X̃} ) = P ( X + εn ∈ (x̃, x̃+ dx̃) ∩ X̃
) = P (X ∈ (x, x+ dx) ∩ X ) · P (x+ εn ∈ (x̃, x̃+ dx̃) ∩Nx) = PX ((x, x+ dx) ∩ X ) · PX̃|X=x ((x̃, x̃+ dx̃) ∩Nx) (12)
where for the first equality we used equation (11) and for the third the fact that x and are almost surely uniquely determined by x̃.
Both probability measures on the right-hand side have a density. For PX with respect to dVf , see Section 2, this density is p∗(x). Similarly, since Nx is a linear subspace of RD, qn(x̃|x) is the density of PX̃|X=x with respect to the volume form dVh where h is the mapping from RD−d to Nx, see Remark 1.
Then, the corresponding density of PX̃ is with respect to the product measure dVh · dVf . However, this product measure is equivalent to λD when restricted to subsets of X̃ and thus qn is the density of PX̃ with respect to λD.5 Therefore, we can write equation (12) in terms of densities as follows:
qn(x̃) = p ∗(x)qn(x̃|x) (13)
and it holds that ∫ X̃ qn(x̃)dx̃ = ∫ X ∫ Nx p∗(x)qn(x̃|x)dVh(x̃)dVf (x)
= ∫ X p∗(x)dVf (x)
=1, (14)
as needed for a density on X̃ . By setting x̃ to x in equation (13), we obtain equation (7). This ends the proof.
Remark 2 Note that in Theorem 1, we need that X̃ is diffeomorphic to RD. This requires that the noise distribution qn(·|x) is continuous for all x.
A.2 PROOF OF PROPOSITION 1
The generating function f is an embedding for X and X = f(Z) has the density p∗(x) for x ∈ X . We may extend the domain of p∗(x) to include all points x ∈ RD using the Dirac-delta function as follows
p∗(x) = ∫ Z δ(x− f(z))π(z)dz (15)
After inflating X , we have that
pΣ(x̃) = ∫ Z N (x̃; f(z),Σ)π(z)dz (16)
with covariance matrix Σ ∈ RD×D. Assume x̃ = x for some x ∈ X . We Taylor expand f(x) around z0 = f−1(x) up to first order,
f(z) ≈ f(z0) + Jf (z0)(z − z0), (17)
and π(z) up to second order,
π(z) ≈ π(z0) + π(z0)′(z − z0) + 1
2 (z − z0)Tπ′′(z0)(z − z0). (18)
5This is because the column vectors of Jf and Jh form a basis of RD .
where π(z0)′ denotes the gradient and π′′(z0) the Hessian of π evaluated at z0, thus π(z0)′ ∈ Rd and π′′(z0) ∈ Rd×d. Then, we can approximate pΣ(x) as follows:
pΣ(x) ≈ 1√
(2π)D det(Σ) ∫ Z exp(−1 2 (z − z0)TJTf Σ−1Jf (z − z0))·
· (π(z0) + π(z0)′T (z − z0) + 1
2 (z − z0)Tπ′′(z0)(z − z0))dz. (19)
Now define Σ̂−1 = JTf Σ −1Jf . Then,
pΣ(x) ≈
√ det(Σ̂)√
(2π)D−d det(Σ) ∫ Z 1√ (2π)d det(Σ̂) exp(−1 2 (z − z0)T Σ̂−1(z − z0))·
· (π(z0) + π(z0)′T (z − z0) + 1
2 (z − z0)Tπ′′(z0)(z − z0))dz. (20)
Thus, we can exploit this Gaussian in Z-space and get
pΣ(x) ≈ √ det(Σ̂)√ det(Σ) (π(z0) + 1 2 E [ (z − z0)Tπ′′(z0)(z − z0) ] )
= √ det(Σ̂)√ det(Σ) (π(z0) + 1 2 ||π′′(z0) Σ̂||+), (21)
where stands for the elementwise multiplication and ||A||+ = ∑d i,j=1Aij for a Rd × Rd matrix A.
For the special case where Σ = σ2ID, we can simplify this expression by exploiting that
√ det(Σ̂)√
(2π)D−d det(Σ) =
1
(2π) D−d 2
σ−D σ−d √ gf
= 1
(2πσ2) D−d 2 √ gf . (22)
Thus, in total, we get for this special choice of Σ
pσ(x) ≈ 1
(2πσ2) D−d 2 √ gf (π(z0) + σ2 2 ||π′′(z0) (JTf Jf )−1||+)
= 1
(2πσ2) D−d 2 √ gf π(z0)(1 + σ2 2π(z0) ||π′′(z0) (JTf Jf )−1||+) (23)
We assume now ∣∣∣∣ σ22π(z0) ||π′′(z0) (JTf Jf )−1||+ ∣∣∣∣ 1. (24) Note that 1/(2πσ2) D−d
2 from equation (23) is exactly the normalization constant obtained when inflating the manifold with Gaussian noise in the normal space, qn(x|x) = 1/(2πσ2) D−d 2 . What follows is that limσ2→0 pσ(x)/qn(x|x) = p∗(x) as we wanted to show.
A.3 PROOF OF THEOREM 2
The theorem follows directly from the definition of the reach number τX of X . It is defined as the supremum of all r ≥ 0 such that the orthogonal projection prX on X is well-defined on the r−neighbourhood X r of X ,
X r := {x̃ ∈ RD| dist(x̃,X ) ≤ r} (25)
where dist(x̃,X ) denotes the distance of x̃ to X . Thus, τX = sup { r ≥ 0 | ∀x̃ ∈ RD, dist(x̃,X ) ≤ r =⇒ ∃!x ∈ X s.t. dist(x̃,X ) = ||x̃− x|| } , (26)
see Definition 2.1. in Berenfeld & Hoffmann (2019). By assumption τX > 0. Thus for all x̃ ∈ X τX we have that x := prX (x̃) is unique. Since X is a manifold, it must hold that x̃ ∈ Nx where Nx denotes the normal space in x. Let the noise generating distributions be a uniform distribution on the ball with radius τX , thus
qn(x̃|x) = Uniform(x̃;B(x, τX ) ∩Nx), (27)
where B(x, τX ) denotes a D−dimensional ball with radius τX and center x.. Then, we have for X̃ = ⋃ x∈X Nqn(·|x) that X̃ = X τX . (28) Thus, X is Q−normally separable where Q := {qn(·|x)}x∈X .

heading: B EXPERIMENTS
text: B.1 TECHNICAL DETAILS FOR CIRCLE EXPERIMENTS
For the Normalizing Flow T−1(x) we use a BNAF (Block Neural Autoregressive Flow) for the circle experiment. The number of hidden dimensions was adapted to the dimensionality of the data and the difficulty of the target density. These details are reported in the corresponding tabular. For the optimization scheme, we used Adam optimizer with an initial learning rate 0.1, a learning rate decay of 0.5 after 2000 optimization steps without improvement (learning rate patience). The batch size was set to 200. The total number of iterations (one iteration corresponds to updating the parameters using one batch sample) used is also reported in the tabular. No hyperparameter fine-tuning was done.
For the FOM and χ2−noise models, we use the same architecture as for the D = 2 case.
B.2 TECHNICAL DETAILS FOR MIXTURE OF VON MISE DISTRIBUTIONS ON S2
For the Normalizing Flow T−1(x) we use rational-quadratic neural spline flows, alternating coupling layers and random feature permutations, see Durkan et al. (2019). For the optimization scheme, we used AdamW optimizer with an initial learning rate 0.0003, a learning rate cosine decay to 0 after every 2000 optimization steps, see Loshchilov & Hutter (2016), a weight decay of 0.0001 and a dropout probability of 0. The batch size was set to 200. No hyperparameter fine-tuning was done. See table 2 for more details. We use the same architecture for the FOM model and 3 seeds for the error bars.
The target distribution is a mixture of four von Mises distributions, p∗1(φ1, θ1), p ∗ 2(φ2, θ2), p ∗ 3(φ3, θ3) and p∗4(φ4, θ4). Each of those distributions has the same product form
p∗i (φi, θi) = exp(κ cos(θi − µi)) 2πI0(κ) · exp(κ cos(2(φi −mi))) πI0(κ) . (29)
We set κ = 6. However, they differ in their mean values µi and mi, see table 3. We used 3 different seeds in total to obtain the confidence intervals.

Review summaries:
In this paper, the authors address main limitations of Normalizing Flows (NFs) method for estimation of density functions on manifolds. Since NFs requires the support of density function to cover the whole Euclidean space, they propose to add noise (inflate) to apply NF. The paper is nicely written with clear introduction of basic concepts very useful for non-expert readers. They provide theoretical guarantees on the variance and type of added noise that make the method work and illustrate with synthetic experiments.

This paper proposes a method for estimating the probability deinsity distribution on a low-dimensional manifold embedded in a high-dimensional space using Normalizing Flow (NF). The problem is that the universality of NF is limited because low-dimensional manifolds are not diffeomorphic with respect to high-dimensional Euclidean space. The proposal of this study is to make NF applicable by inflating low-dimensional manifolds with Gaussian noise. Then, after the transformation is obtained, the probability distribution on the original low-dimensional manifold can be obtained by deflation.