On the Latent Space of Flow-based Models
Authors: Mingtian Zhang
Yitong Sun
Steven McDonagh
Chen Zhang
Venue:  (2021)
Keywords: flow-based mode
generative model
intrinsic dimension
manifold learning

Abstract:  Flow-based generative models typically define a latent space with dimensionality identical to the observational space. In many problems, however, the data does not populate the full ambient data-space that they natively reside in, but rather inhabit a lower-dimensional manifold. In such scenarios, flow-based models are unable to represent data structures exactly as their density will always have support off the data manifold, potentially resulting in degradation of model performance. In addition, the requirement for equal latent and data space dimensionality can unnecessarily increase model complexity for contemporary flow models. Towards addressing these problems, we propose to learn a manifold prior that affords benefits to both the tasks of sample generation and representation quality. An auxiliary product of our approach is that we are able to identify the intrinsic dimension of the data distribution.

Under review as a conference paper at ICLR 2021
ON THE LATENT SPACE OF FLOW-BASED MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Flow-based generative models typically define latent space with dimensionality identical to the observational space. In many problems, however, the data does not populate the full ambient data-space that they natively reside in, but rather inhabit lower-dimensional manifold. In such scenarios, flow-based models to represent data structures exactly as their density will always have support off the data manifold, potentially resulting in degradation of model performance. In addition, the requirement for equal latent and data space dimensionality can un- necessarily increase model complexity for contemporary flow models. Towards addressing these problems, we propose to learn a manifold prior that affords bene- fits to both the tasks of sample generation and representation quality. An auxiliary product of our approach is that we are able to identify the intrinsic dimension of the data distribution.
unable
INTRODUCTION
Normalizing flows (Rezende and Mohamed 2015 Kobyzev et al. 2020) have shown considerable
potential for the tasks of modelling and inferring expressive distributions through the learning of
well-specified probabilistic models. Contemporary flow-based approaches define latent space with dimensionalitty identical to the data space, typically by parameterizing complex model px (x]0) using an invertible neural network fo. Samples drawn from an initial, simple distribution pz (z) (e.g- Gaussian) can be mapped to a complex distribution as x = fo(z). The process results in a tractable density that inhabits the full data space. However, contemporary flow models may make for an inappropriate choice to represent data that resides a diver--ienssioall manifold and thus does not populate the full ambient space. In such cases, the estimated model will necessarily have mass lying off the data manifold, which may result in muder-fitting and poor generation qualities. Furthermore, principal objectives such Maximum Likelihood Estimation (MLE) and Kullback-Leibler (KI.) divergence minimization are ill-defined, bringing additional challenges for model training.
In this work, propose principled strategy model data distribution that lies continuous manifold and we additionally identify the intrinsic dimension of the data manifold. Specifically, by using the connection between MLE and KL divergence minimization in Z space, we can address the important problem of ill-defined KL divergence under typical flow based assumptions.
Flow models are based on the idea of "change of variable". Assume random variable Z with distribution Iz and probability density Pz(z). We can transform Z to get random variable X: X = f(), where f R - TR. is an invertible function with inverse f-1 g. Suppose X has distribution Px and density function px (z), then log (r) will have the following form
Og Ox
(og loggz(g(z)) log det
where log det preserving function the log determinant is equal to 0.
(2g) Uz
is the log determinant of the Jacobian matrix. We call f (or g) volume-
Training of flow models typically makes use of MLE. We denote Xa as the random variable of the data with distribution P. and density pa(x). ). In addition to the well-known connection between MLE and minimization of the KL divergence (L(pp())) MLE is also (approximately) equivalent to minimizing the KL divergence in Z space, this is due to the KI. divergence is invariant under invertible transformation (Yeung) 2008 Papamakarios al.
(z)) in X space (see Appendix [ for detail),

---

Under review as a conference paper at ICLR 2021
2019). Specifically, we define Zg Zo ((xa) with distribution Oz and density function g(z),
the KL divergence in space can be written as
(L((())lo(=))) - f a(=) logglz))z / a((2) log
(())))) log
(2)
- -- palxx) (ogg2 (g(x)) + log det
Ög dr
dr + const.,
(3)
The full derivation can be found in Appendix Since we can only access samples I1, 22,22,.... NN from Pa(z), we approximate the integral by Monte Carlo sampling
N KL(g())llp()) ~ N Σ
1
(x + const..
(4)
We thus highlight the connection between MLE and KI. divergence minimization, in space, for flow based models. The prior distribution p(z) is usually chosen be D-dimensional Gaussian distribution. However, if the data distribution Pa is singular, for example a measure on a low dimen- sional manifold, the induced latent distribution Qz is also singular. In this case, the KL divergence in equation is typically not well-defined under the considered flow based model assumptions. This issue brings both theoretical and practical challenges that we will discuss in the following section.
FLOW MODELS WITH MANIFOLD DATA
We assume a data sample x Pa to be a D dimensional vector X TRD and define the ambient dimensionality of Fa. denoted by Amdim( (PPa). to be D. However for many datasets of interest, e.g. natural images, the data distribution P. is commonly believed to be supported on a lower dimen- sional manifold (Beymer and Poggio 1996). We assume the dimensionality of the manifold to be K where K < D, and define the intrinsic dimension of Pa. denoted by Indim(Fa) a). to be the dimen- sion of this manifold. Figure [La provides 2D space. Specifically, each data sample x ~ Pa is a 2D vector x = (I1, 22) where Ii ~ (O 1) and 222 sin (2n1). Therefore, this example results in Amdim(Pa
example of this setting where Pa a ID distribution in
2 and Tnd m mMP2) = 1.
In flow-based models, function f is constructed such that it is both bijective and differentiable. When the prior Pz is a distribution whose support is RDD (e.g. Multivariate Gaussian distribution), the marginal distribution Ixx will also have support IR D and Amdim((Px) = [ndim((Px) = D. When the support of the data distribution lies on K -ddimensional manifold and K < D, Pa and Fx are constrained to have different support. That is, the intrinsic dimensions of Ix and Pa are always different; Indim(Ixx (Ixx) * Indi m a). In this case is impossible to learn a model distribution Fx identical to the data distribution Pa- Nevertheless, flow-hased models have shown strong empiri- cal success in real-worke problem domains such as the ability to generate high quality and realistic images [Kingma and Dhariwal, 2018). Towards investigating the cause and explaining this dispar- ity between theory and practice, we employ a toy example to provide intuition for the effects and consequences resulting from model and data distributions that possess differing intrinsic dimension.
Consider the toy dataset introduced previously; a 1D distribution lying in a 2D space (Figure Ia The prior density p(z) is standard 2D Gaussian p(z) (O Iz) and the function f is volume preserving flow with two coupling layers (see Appendix (CT)). In Figure [b] we plot samples from the flow model; the sample is generated by first sampling 2D datapoint
NN10, Tz) and then letting x = fzz). Figure Ic shows samples from the prior distributions Iz and Qz. Qz is defined as the transformation of Pa using the bijective function g. such that Qz is constrained to support ID manifold in 2D space, and Indim(
Indim(PP.) = 1. Training of Oz to match Fz (which has intrinsic dimension 2), can be seen in Figure result in curling up of the manifold in the latent space, contorting it towards satisfying distribution that has intrinsic dimension 2. This ill-behaved phenomenon causes several potential problems for contemporary flow models:
1. Poor sample quality. Figure 1b shows examples where incorrect assumptions in turn result in the model generating bad samples.
2. Low quality data representations. The discussed characteristic that results in "curling up" of the latent space may cause degradations of the representation quality.
2

---

Under review as a conference paper at ICLR 2021
3. Inefficient use of network capacity. Neural network capacity is spent contorting the distribution Oz to satisfy imposed dimensionality constraints.
A natural solution to the problem of intrinsic dimension mismatch is to select prior distribu- tion Pz with the same dimensionality as the intrinsic dimension the data distribution such that: Indim(Pz) Indim(Pa involves to instead learn it from the data distribution. In the following section, we will introduce a parameterization approach that enables us learn Indim(Pa..
However, since do not know Indim(Pu)) explicitly, option
0
(a) Data distribution Pd
(b) Samples space
(c) Samples space
Figure 1: Samples and latent visualization from flow based model with fixed Gaussian prior when the intrinsic dimension strictly lower than the true dimensionality the data space.
JO
Pz
Px
LEARNING A MANIFOLD PRIOR
Consider a data vector x IR., then flow based model prior Izz is usually given by D- dimensional Gaussian distribution or alterna- tive simple distribution that is also absolutely continuous (a.c.) in RD, Therefore, the intrin- sic dimension Indim(Fz) prior to have intrinsic dimension strictly less than D, we let Pz have the "generalized den- siry""ppe with the form
s())
D. To allow
Figure 2: The data (black dots) lies on 2D man- ifold in 3D space. To have a model Ixx with Indim(Pxx)
2, we learn prior Pz that 2D Gaussian in 3D space and an invertible func- tion f which maps from Px to Px.
p(z) = -NO0, AA"),
(5)
where z E D and A is a DxD lower triangular matrix with D(D+1) 2 parameters, such that AAT is constrained to be a positive semi-definite matrix. When AA has full rank D, then I' is a (non- degenerate) multivariate Gaussian R. When Rank(AA") degenerate to Gaussian supported on Ki-dimensional manifold, such that the intrinsic dimension Indim(P2) = K. Figure illustrates sketch of this scenario. practice, we initialize identity matrix, so AA" will also be an identity matrix and P is initialized as standard Gaussian.
K and K D, then Pz will
be an
When Rank(AA") < D, the degenerate covariance AAT is no longer invertible and are unable to evaluate the density value of p(z) for given random vector z. Furthermore, when the data distribution Pa is supported on K -ddimensional manifold, Q will also supported on K- dimensional manifold and longer has valid density function. Using equation to train the flow model then becomes impossible as the KL divergence between Pz and Qz is not well defined family of divergence address this problem.
Recent work by Zhang al. 2020) proposed In the following section we briefly review the key concepts pertaining to this family of divergences.
I use generalized density to include the case AA" not full rank. 2The KI. divergence (L(Q) P) is well defined when Q and P. have valid densities and their densities have the support (Ali and Silvey 1966..
3

---

Under review as a conference paper at ICLR 2021
4 FIX THE ILL-DEFINED KL DIVERGENCE
random Let Zo and Zp be two random variables with distribution Qz and Pz. The KL divergence between U and Pz is not defined if Qz Pz does not have valid density function. Let K be an a.c. random variable that is independent of Zo and Zp and has density Pk, We define Zp Zp + K; Lg Ze + K with distributions Iz and Q7 respectively. Then Pz and Oz a.c. with density functions
| z)dQ2 p(ž) pk z)dP2.
(2)
(6)
We can thus define the spread KL divergence between Qz and IPz as the KL divergence between Q and Pz as:
(Q()))
In this work we let K be a Gaussian with diagonal covariance o31 / to satisfy the sufficient conditions such that KL is valid divergence (see Zhang al. (2020) for details) and has the properties:
PL( 2 0,
KL(Q2 = 0 = Qz Pz
(8)
Since Qz and Pz are transformed from Fa and Px using invertible function g, have
Q = Pz + Pa Px
Therefore, the spread KL divergence can be used train flow based models with a manifold prior in order to fit a dataset that lies on a tower-dimensional manifold.
5 IDENTIFIABILITY OF INTRINSIC DIMENSION
A byproduct of our model is that the intrinsic dimension of the data manifold can be identified. Section 4 shows that when (T<(Qz (Pz) 0 < PPx Pa, the supports of Px and Pa will also have the same intrinsic dimension: Indim(Px)) = Indim(Pa). The flow function q. and its inverse is a diffcomorphism (Kobyzev al. 2020). Due to the invariance of dimension property of diffeomorphisms (Lee) 2013) Theorem 2.17), the manifold that
g f-1 are bijective and continuous
supports Pz will have the same dimension the manifold that supports Px. Therefore, have
Indim(Pz) = indim(px)) = Indim(P2x..
(10)
Since the intrinsic dimension of Pz is equal to the rank of the matrix AAT. we calculate Rank(AA") by counting the number of non-zero cigenvalues of the matrix AAT.. This allows for identification of the intrinsic dimension of the data distribution as
Indim(Pa) = Rank(AA"))
(11)
We have shown that we can identify the intrinsic dimension of P. using the spread KI. divergence. In the next section, we will discuss how to estimate the spread KI. divergence in practice.
6 ESTIMATION OF THE SPREAD KL DIVERGENCE
Our goal is to minimize the spread KL divergence between Qz and Pz. Using our definition of spread divergence (equation 7). this is equivalent minimizing
equivalen
KL (g(z))lp()))
((z) log q(ž)dž
q(( log
(12)
Term
Term
where q(ž) and p(ž) are defined in equation 6 We separate the objective into two terms and now discuss the estimation for each of them.
Term 1:
We H(.) denote the differential entropy. Term 1 is the negative entropy -H(Z)) For volume preserving & and Xa is a.c., the entropy H(Zg) H(Xa) and is independent of the
4

---

Under review as a conference paper at ICLR 2021
model parameters and can be ignored during training. However, the entropy H(Z6)
H( + K)
will still depend on g. see Appendix [.1 for an example. We claim that when the variance of K is small, the dependency between the H(Zz) and volume preserving function g weak, thus we can approximate equation 12 by leaving out term I and will not affect the training.
To build intuitions, we first assume Xd is a.c., so Zạ = (XXx) is also a.c.. Using standard entropic properties (Kontoyiannis and Madiman 2014), we can pose the following relationship
H Oz) S (2 K) H(Zo) + T(Z + K),
(13)
where I(-.-) denotes the mutual information. Since Zg is independent of function 9 and if 02 + O, then 1(2g + K, K) - 0 (see Appendix B.2 for proof), the contribution of the 1(2 + K, K) term, with respect to training y. becomes negligible in the case of small oz.
Unfortunately, equation 13 no longer valid when Pa lies on manifold since Z will be singular random variable and the differential entropy H (c) is not defined. In Appendix B.3 we show that leaving out the entropy Hz)z) corresponds to minimizing an upper bound of the spread KL divergence. To further find out the contribution of the negative entropy term, we compare between leaving out -H ZZ)) and approximating - -H(Zg) during training. In Appendix B.4 we discuss the approximation technique and give the empirical evidence which shows that the ignoring - -H( will not affect the training of our model. Therefore, we make use of volume preserving and small variance o3 = 1 x 10-4 our experiments.
In contrast to other volume preserving flows, that utilize fixed prior Pz, our method affords addi- tional flexibility by way of allowing for changes to the 'volume" of the prior towards matching the distribution of the target data. In this way, decision employ volume-preserving flow func- tions does not limit the expressive power or abilities of the model, in principle. Popular non-volume preserving flow structures, e.g. affine coupling flow, may also easily be normalized to become vol- ume preserving, thus further extending the applicability of our approach (see Appendix [C.1 for an example).
Term 2: The noisy prior p(z) defined to be degenerate Gaussian NNO, AAT), convolved with a Gaussian noise (0, o2 7), and has closed form density
p 2 \/ AAT + +02
(14)
Therefore, the log density ((g)(2) is well defined, we can approximate term by Monte Carlo
N
/ ( logp(2)dž Rc
log
(15)
N
where (2) = f p( p(2 To sample from ((), we first get a data sample x ~ Pa. use function g get z x(x) (so is sample of Q2) and finally sample á p(2/z).
EXPERIMENTS
7.4 report experiments on four datasets; toy 2D and 3D data, the fading square dataset and synthesized adaptation of MNIST. We use the Adam optimizer (Kingma and Bal 2014) in all our experiments. Our flow networks consisted of incompressible affine coupling layer introduced by SSorresoon et al. (200)): Dinh et al. (0016). We compare volume preserving flow with learnable prior (our method) non-volume preserving flow with fixed prior, so both models have the ability to adapt their 'volume' to fit the target distribution and retains fair comparisons. See Appendix [C. for detailed discussion of the incompressible affine coupling layer and the network structures we use for all the experiments.
To demonstrate the effectiveness of approach, Sections 7.1
7.1 2D TOY DATA
We firstly verify our method using the toy dataset described in Section 2 and Figure la The flow function has two coupling layers. We train our model using learning rate 3 10 and batch size 100 for 10k iterations. Figure Jhsoows the samples from the model, the learned prior and the eigenvalues of AAT. We observe the sample quality is better than that in Figure 1b and the prior has learned
5

---

Under review as a conference paper at ICLR 2021
degenerate Gaussian with Indim(P2) 1, which matches Indim(Pa). In Appendix C.2 show that our model can not only learn the manifold support of target distribution but also capture the 'density' allocation the manifold.
8
Pd
P2
92
(a) X space samples
(b) Z space samples
(c) Eigenvalues of A
Figure 3: (a) shows the samples from the data distribution Fa and our model Px. (b) shows the sample from the learned prior Pz and the distribution Qz. (C) shows the eigenvalues of AAT
7.2 S-CURVE DATASET
We fit our model to the S-curve dataset shown in Figure 4a The data distribution lies on 2D manifold in a 3D space, therefore Indim(PP2) = 2. Specific network structure and training details can be found in Appendix C.I After training, model learns nonlinear function g to transform Pa to Q2. where the latter lies on a 2D linear subspace in 3D space (see Figure (b). Following this, be conducted generate 2D data representations,
linear dimensionality reduction briefly outline a general procedure for this.
For Qz with Amdin (Qz) D and Indim(Q2) K, we first find the eigenvectors e¹ ..... e of AAT, sorted by their cigenvalues. When Rank(AA") K < D, there exist eigenvectors with positive eigenvalues. We select the first K eigenvectors and form the matrix E |e
e with dimension D K. We then transform each data sample x E R into Z space: z = g(x), such that E P_ Afterwards, linear projection carried out rroj representation prooj E k This procedure can be seen as nonlinear dimensionality reduction, where the nonlinear component is solved using the learned function g.
zE obtain the lower dimensional
We plot the resulting representations in Figure 4b The colormap indicates correspondence between the data in 3D space and the representation 2D space. We observe that our method can success- fully (1) identify that the intrinsic dimension of the data is two and (2) project the data into 2D space that faithfully preserves the structure of the original data distribution. We also compare the sample generation quality with flow that has fixed Gaussian prior, see Appendix C.3 for details.
A
(a) x Pa
(b) Oz
(c) Eigenvalues
(d) Proj z
Figure 4: (a) S-curve data samples x ~ Pa- (b) The latent representation z = gxx), points be observed to lie a linear subspace. Eigenvalues of the matrix AAT'. deduce that Indim(Pa) = 2. (d) Our representation after the dimensionality reduction proj zE.

---

Under review as a conference paper at ICLR 2021
7.3 FADING SQUARE DATASET
The fading square dataset (Rubenstein et al. 2018) was proposed in order to diagnose model behav- ior when data distribution and model possess differing intrinsic dimension and therefore constitutes further relevant test bed for current work. The dataset consists of 32 pixel images with 6x6 grey squares on a black background. The grey scale values are sampled from a uniform distri- Figure [5 shows the data samples. We fit our model to the dataset, the network structure and the training details can be found in Appendix
bution with range 10, 1], Indim(Pa)
Figure 5b) shows samples from trained model. Figure 5d shows the first 20 eigenvalues of the AAT (ranked from high to low), we can see only one eigenvalue is larger than zero and the others have converged zero. This illustrates that have successfully identified the intrinsic dimension of Pa- We further carry out the dimensionality reduction process that was introduced in Section 772 the latent representation z is projected onto ID line and we visualize the correspondence between the projected representation and the data in Figure Se Pixel grey-scale values can be observed to decay as the ID representation is traversed from left to right, indicating that our representations are consistent with the properties of the original data distribution. In contrast, we find that the traditional flow model, with a fixed 1024D Gaussian p(z), fails to learn such data distribution, see Figure 5c
52
01-
(a) X ~V Pa
(b) Our model
(c) Traditional flow
(d) Eigenvalues of AA"
(e) 2 proy
zE
Figure 5: (a) and (b) show the samples from the data distribution and our model, respectively. (c) shows traditional flow based model with fixed Gaussian prior fails to generate any valid samples. (d) shows the first 20 eigenvalues of the matrix AAT. (e) visualization of the representation after applying dimensionality reduction. See text for further discussion.
7.4 MNIST DATA
We further investigate training of model using images of digits. However, for datasets like MNIST (LeCun) 1998). the true intrinsic dimension is unknown. In order to verify the correctness of our model's ability to identify the intrinsic data dimension, we construct synthetic datasets by first
fitting an implicit model (x( =f f(x o(x g(z) (z)))dd to the MNIST dataset, and then use samples from the trained model
(x) training data. The intrinsic dimension of the training dataset dim(z). 10 such that Indim(Pa) 5 and
is the same as the dimension of the latent variable in the implicit model Indim(Pa) We construct two datasets with dim(z) 5 and dim(z) Ind indimm mnP.)) = 10, respectively. Further details on the implicit model, flow network structure, training method and samples from the learned models are found in Appendix C
In contrast to the fading square dataset, we find that in order to successfully train the model (i.e. such that valid image samples are generated), is necessary to add small Gaussian noise to the training data. This trick is commonly used in the training of flow based models for image data (Sorrenson et al. 2020). We note that adding Gaussian noise breaks the assumption that the data lies on a man- ifold and, alternatively, the intrinsic dimension of the training data distribution will be equal to its ambient dimension. Towards alleviating this undesired effect, we firstly add Gaussian noise with standard deviation æ 0.05 and anneal ag after 2000k iterations with factor of 0.9 every 10k:

---

Under review as a conference paper at ICLR 2021
iterations. However, to help retain successful model training, disallow the annealing procedure to reach state where zero noise is added due to the outlined model behavior observed when con- sidering this image dataset. Experimentally,
cap a lower-bound Gaussian noise level of 0.01 and leave further investigation of this phenomenon to future work.
plot the first 20 eigenvalues of the AAT (ranked from high low) after training on two synthetic MNIST datasets with intrinsic dimension and 10. It can be observed that 5 and 10 eigenvalues are significantly larger than the remaining values, respectively. It can thus be concluded that the intrinsic dimension of the two datasets are 5 and 10. Remaining non-zero eigenvalues can be attributed the small Gaussian noise added to the training data.
In Figure 6a and 6b
For the original MNIST dataset, it was shown that digits have different intrinsic dimension (Costa and Hero 2006). This suggests the distribution of MNIST may lie on several, disconnected mani- folds with differing intrinsic dimension. Although our model assumes that Pa lies on one continuous manifold, is interesting to investigate the case when this model assumption fit our model to the original MNIST data and plot the eigenvalues in Figure [6c] In contrast to Fig- ures 6aa and [bb] the gap between eigenvalues can be seen to be less pronounced, with no obvious step change. However the values suggest that the intrinsic dimension of MNIST is between 11 and 14. This result is consistent with previous estimations stating that the intrinsic dimension of MNIST is between 12 and 14 (Facco al., 2017 Hein and Audibert, 2005 Costa and Hero 2006). Recent work Cornish et al. (2019) discusses fitting flow models to a IP that lies on disconnected compo- nents, by introducing mixing prior. Such techniques may be easily combined with method towards constructing more powerful flow models; a promising direction for future work.
fulfilled. We thus
(a) Indim(Pa) =
(b) Indim(Pa) = 10
(c) MNIST
Figure 6: This figure shows the eigenvalues of the AAT after fitting the (a) synthetic MNIST with intrinsic dimension 5; (b) synthetic MNIST with intrinsic dimension 10; (c) original MNIST dataset.
8 RELATED WORK
Classic latent variable generative models assume that data distributions lie around low-dimensional manifold, for example the Variational Auto-Encoder (Kingma and Welling 2013) or, recently intro- duced, Noisy Injective Flows (Cunningham et al. 2020). Such methods typically assume that obser- vational noise is not degenerated (e.g. a fixed Gaussian distribution). therefore the model distribution absolutely continuous and maximum likelihood learning is thus well defined. However, common distributions such as natural images usually don't 't have Gaussian observational noise (Zhao et al.) 2017]. Therefore, in this work, we focus on modeling distributions that on a low-dimensional manifold.
The study of manifold learning for nonlinear dimensionality reduction Cayton (2005) and intrinsic dimension estimation Camastra and Staiano (2016) is rich field with an extensive set of tools. However, most methods commonly do not model data density on the manifold and are thus not used for the same purpose as the models introduced here. There however number of recent works that introduced normalizing flows on manifolds that we now highlight and relate to our approach.
Several works define flows manifolds with preseribed charts. Gemici et al.) 2016) generalized flows from Euclidean spaces to Riemannian manifolds by proposing to map points from the manifold M to R apply normalizing flow in this space and then map back M. The technique has been
8

---

Under review as a conference paper at ICLR 2021
further extended to Tori and Spheres (Rezende al., 2020). These methods require knowledge of the intrinsic dimension K and parameterization of the coordinate chart of the data manifold.
Without providing chart mapping priori, M-flow (Brehmer and Cranmer; 2020) recently pro- posed an algorithm that learns the chart mapping and distribution density simultancously. However, their method still requires that the dimensionality of the manifold is known. They propose that the dimensionality can be learned either by a brute-force solution or through a trainable variance in the density function. The brute-force solution is clearly infeasible for data embedded in extremely high dimensional space, as is often encountered in deep learning tasks. Use of a trainable variance is natural and similar to our approach. However, as discussed at the beginning of this paper, without carefully handling the KL or MLE term in the objective, a vanishing variance parameter will result wild behavior of the optimization process since these terms are not well defined.
While the GIN model considered in [Sorensonn et al. (020)) could recover the low dimensional generating latent variables following their identifiability theorem, the assumptions therein require knowledge of auxiliary variable, e,g. the label, which is not required in our model. Behind this superficial difference is the essential discrepancy between the concept of informative dimensions and intrinsic dimensions. The GIN model discovers the latent variables that are informative in given context, defined by the auxiliary variable u instead of the true intrinsic dimensions. In their synthetic example, the ten dimensional data is nonlinear transformation of ten dimensional latent variables where two out of ten are correlated with the labels of the data and the other eight are not. In this example, there are informative dimensions, but there ten intrinsic dimensions. Nevertheless, our method for intrinsic dimension discovery can be used together with informative dimension discovery methods to discover finer structures of data.
CONCLUSION
We presented principled strategy to learn the data distribution that lies manifold and identify its intrinsic dimension. We fix the ill-defined KL divergence and show, across multiple datasets, the resulting benefits for flow based models under both sample generation and representation quality. There remain a number of open questions and interesting directions for future work. Namely; further exploration the effects of the entropy term in the case of non- non-volume preserving networks and, additionally, investigation of the phenomenon concerning the apparent necessity of noise addition in cases pertaining to complex real-world distributions e.g. image data.
REFERENCES
M. Ali and S. D. Silvey. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society: Series B (Methodological), 28(1):131-142, 1966.
D. Beymer and T. Poggio. Image representations for visual learning. Science, 272(5270):1905- 1909, 1996.
J. Brehmer and K. Cranmer. Flows for simultaneous manifold learning and density estimation. arXiv preprint arXiv:2003. 13300003..39,,,.. 2020.
F. Camastra and A. Staiano. Intrinsic dimension estimation: Advances and open problems. Infor- mation Sciences, 328:26-41, 2016.
L. Cayton. Algorithms for manifold learning. Univ. of California at San Diego Tech. Rep, 12(1-17): 1, 2005.
R. Cornish, A. L. Caterini, G. Deligiannidis, and A. Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. arXiv preprint 1909. 2019.
J. A. Costa and A. O. Hero. Determining intrinsic dimension and entropy of high-dimensional shape spaces. In Statistics and Analysis of Shapes, pages 231-252, Springer, 2006.
E. Cunningham, R. Zabounidis, A. Agrawal, I. Fiterau, and D. Sheldon. Normalizing flows across dimensions. arXiv preprint arXiv:2006. 13070, 2020.

---

Under review as a conference paper at ICLR 2021
L. Dinh, J. Sohl-Dickstein and S. Bengio. Density estimation using real nvp. arXiv preprint arXiv:160S.08803, 2016.
E. Facco, M. d'Errico, A. Rodriguez, and A. Laio. Estimating the intrinsic dimension of datasets by minimal neighborhood information. Scientific reports, 7(1):1-8 2017.
M. C. Gemici, D. Rezende, and S. Mohamed. Normalizing flows on riemannian manifolds. arXiv preprint ar Xrxiv: 1.02304, 2016.
M. Hein and J.-Y. Audibert. Intrinsic dimensionality estimation of submanifolds in rd. Proceed- ings of the 22nd international conference on Machine learning, pages 289-296, 2005.
M. F. Huber, T. Bailey, H. Durrant-Whyy Whyte, and U. D. Hanebeck. On entropy approximation for gaussian mixture random vectors. In 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, pages 181-188. IEEE, 2008
J. Jacobsen, A. Smeulders, and Oyallon. i-revnet: Deep invertible networks. arXiv preprint 1802.0080, 2018.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv: 14X1::1412.6980 2014.
arXiv preprint
D. P. Kingma and Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In Advances in neural information processing systems, pages 10215-10224, 2018.
D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint 1rXiv:1312.6114 2013.
Kobyzev, S. Prince, and M. Brubaker. Normalizing flows: introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
I. Kontoyiannis and M. Madiman. Sumset and inverse sumset inequalities for differential entropy and mutual information. IEEE transactions on information theory, 6014)44503445,, 2014.
Y. LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/nmist// 1998.
M. Lee. Smooth manifolds. Introduction to Smooth Manifolds, pages 1-31. Springer, 2013.
G. Papamakarios, Nalisnick, D. Rezende, Mohamed, and B. Lakshminarayanan. Normaliz- ing flows for probabilistic modeling and inference. urXiv preprint arXiv: 1922.0/1122.2776: 2019.
A. Papoulis and S. U. Pillai. Probability, random variables, and stochastic processes. processe.. Tata McGraw- Hill Education, 2002.
D. Rezende and S. Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:1505.05770, 2015,
D. J. Rezende, G. Papamakarios, S. Racanière, M. S. Albergo, G. Kanwar, P. E. Shanahan, and K. Cranmer. Normalizing flows on tori and spheres. arXiv preprint arXiv:2002.02428, 2020.
K. Rubenstein, B. Schoelkopf, and I. Tolstikhin. On the latent space of wasserstein auto-encoders. arXiv preprint arXiv: 18X2::880..03666 2018.
P. Sorrenson, C. Rother, and U. Kothe. Disentanglement by nonlinear ica with general incompressible-font networks (gin). arXiv preprint arXiv:2001.0483 2020.
I. Tolstikhin, O. Bousquet, S. Gelly, and B. Schoelkopf. Wasserstein auto-encoders. arXiv preprint 1711:0171.00155: 2017.
R. W. Yeung. Information theory and network coding. Springer Science & Business Media, 2008
M. Zhang, P. Hayes, T. Bird, R. Habib, and D. Barber. Spread Divergences. In Proceedings of the 37th International Conference on Machine Learning, ICML, 2020.
Zhao, J. Song, and S. Ermon. Towards deeper understanding of variational autoencoding models. arXiv preprint arXiv: 7002 08658, 2017.
10

---

Under review as a conference paper at ICLR 2021
A MAXIMUM LIKELIHOOD ESTIMATION AND KL DIVERGENCE
Given data 31, -2,..., N sampled independently from the true data distribution Pa. with density function Pa(æ), we want to fit the model density p(a: I
to the data. A popular choice to achieve this
involves minimization of the KI. divergence where:
(L(pa(<)llx(z) = pa(z) log (z)xdd / pal(z) logd(r))
(16)
= - pa(x) loogpz (g(x)) + log det
Og
dr - const.. (17)
Since we can only access samples from pa(a), we approximate the integral by Monte Carlo sampling
1 N
KL(Pa(=)ll<()))
pgg,( + const..
(18)
Therefore, minimizing the KL divergence between the data distribution and the model (approxi- mately) equivalent to Maximum Likelihood Estimation (MLE).
When p(x) is flow-based model with invertible flow function f - X.9 = f- minimizing the KL divergence in X space is equivalent to minimizing the KL divergence in the Z space. We let Xa be random variable data distribution and define Za Zo = g(Xa) with density g(z), so ((2) can be represented as
q(z) / 6(2 10==0(020(2)1)1- - g(z)))p(z)dd.
(19)
Let p(z) be the density of the prior distribution Pz, the KL. divergence in space can be written as KL(a())llx<))) / <(x)) [(2)) log ((=)d= log
(20)
Term 1: using the properties of transformation of random variable (Papoulis and Pillai 2002 pp. 660), the negative entropy can be written as
f ((z)) log ((2)d== =
Ög
pa(x) palx) log
la(x))og det
dx.
(21)
Ocr
const.
Term2: the cross entropy can be written as
J (())))ogr logg()d))
/ 8(2 g(x))pa(x) log (zz)
(22)
f pa(x) log pl)
(23)
Therefore, the KI. divergence in Z space is equivalent to the KL divergence in X space
KL(g(=) Ip(z)) = KL(p)())llp(z))
(24)
We thus build the connection between MLE and minimizing the KI. divergence in Z space.
B ENTROPY
B.1 AN EXAMPLE
0 1
Assume 2D Gaussian random variable X with covariance 0
is a volume preserving flow
2 0 0
with parameter 0. Let 2, = 9o. (Z) be Gaussian with covariance
1 and Z, (X) be
2
For simplicity, we notation p(x) represent the model px (xx unless otherwise specified.
11

---

Under review as a conference paper at ICLR 2021
3
a Gaussian with covariance not depend on 0. Let K be an Gaussian with zero mean and covariance
Therefore the entropy H( = H((2) H(X) and doesn't
0
[1
so Z1 + K is
3
Gaussian with covariance
and Zz + K is a Gaussian with covariance
Therefore
0
3J
H K) + H( + K) and Hg (X) + K) depends on 6. A similar example can be constructed when X is not absolutely continuous.
B.2 Z IS AN ABSOLUTELY CONTINUOUS RANDOM VARIABLE
For two mutually independent absolutely continuous random variable and K, the mutual infor- mation between Z + K and K is
I K, K)
H(Z K) H(Z K|K)
(25)
H(Z + K) K[
(26)
H(ZZ + K) H(Z).
(27)
The last equality holds because Z and K are independent. Since mutual information 1(2 + 1,7 K, K) 2 O, we have
H(Z) < H(Z K) H(Z) + I(Z K)
(28)
Assume K has Gaussian distribution with 0 mean and variance o 2
When 03 0, K degenerates
to delta function, so Z + K = Z and
I + K, K) = H(Z + K) H(Z) =
(29)
this is because the mutual information between a.c. random variable and singular random variable is still well defined, see (Yeung) 2008 Theorem 10.33). Assume K1 K₂ are Gaussian random variables with 0 mean and variances of and of respectively. Without loss of generality, assume of > oं and of = of + o and let K3 be the random variable of a Gaussian that has mean and variance of such that K₁ K₂ Kg. By the data-processing inequality, we have
I(2 + K2, K2) < I(Z + K2 + K5, K2 + Ks) = I(2 + K1).
(30)
Therefore, I(Z + K, K) is monotonically decreasing function when o decreases and when o 0, [Z + K, K) - O.
B.3 UPPER BOUND OF THE SPREAD KL DIVERGENCE
In this section, show that leaving out the entropy term H(Zg) in equation 12 is equivalent to minimizing upper bound of the spread KL divergence.
For singular random variable Zo = g(Xa) and absolutely continuous random variable K that are independent, we have
H(Zg K) H(K) H(Z4 + - H(Z KlZe)
(31)
= 1(2ç + K,) 0.
(32)
The second equation is from the definition of Mutual Information (MI); the MI between a.c. random variable and a singular random variable is well defined and always positive, see (Yeung) 2008 Theorem 10.33) for proof.
Therefore, we can construct an upper bound of the spread KL objective in equation 12
/
(Tl()))) = (()) log (())d2
q(() log p(2)dž
(33)
H(Zo+K)
< H(K) f q(z)) log p(g
(34)
coust.
Therefore, ignoring the negative entropy term during training is equivalent to minimizing upper bound of the spread KL objective.
*The extension higher dimensions straightforward.
12

---

Under review as a conference paper at ICLR 2021
B.4 EMPIRICAL EVIDENCE FOR IGNORING THE NEGATIVE ENTROPY
In this section, we first introduce the approximation technique compute the negative entropy term, and then discuss the contribution of this term for the training. The negative entropy of random variable Z is
H(Za) ((g) (()ddž
(35)
where
g()) = pK z))Q2z
(36)
and Px is the density of a Gaussian with diagonal covariance o2L. We first approximate ((ž) by
mixture of Gaussians
q(ž) Σ N(z:2",0220 " (z)
(37)
where z is the nith sample from distribution Q2 by first sampling x" ~ Pa and letting z" = g(x").
We denote the random variable of this Gaussian mixture as 2 and approximate
H(Z - )
(38)
However, the entropy a Gaussian mixture distribution does not have a closed form, so we further conduct first order Taylor expansion approximation (Huber et al. 2008)
N log @N (z z²),
(39)
this approximation is accurate for small o2.. Finally we have approximation
log g x²).
(40)
N
To evaluate the contribution of the negative entropy, train flow model on both low dimen-
sional data (Toy datasets: 2D) and high dimensional data (Fading square dataset: 1024D) by op- timization that uses two objectives: (1) ignoring the negative entropy term in equation 7 and (2) approximating the negative entropy term in equation 7 using the approximation discussed above. During training, we keep tracking of the value of the entropy H(Zg) (using the approximation value) in both objectives. We let N equal the batch size when approximating the entropy. Addi- tional training details remain consistent with those described Appendix
To evaluate the contribution of the negative entropy, train flow model on both low dimen- sional data (Toy datasets: 2D) and high dimensional data (Fading square dataset: 1024D) by op- timization that uses objectives: (1) ignoring the negative entropy term equation 7 and (2) approximating the negative entropy term in equation 7 using the approximation discussed above. During training, we keep tracking of the value of the entropy H(Zş) (using the approximation value) in both objectives. We let N equal the batch size when approximating the entropy. Addi- tional training details remain consistent with those described in Appendix
In Figure 7 plot the (approximated) entropy value H(Zg) during training for both experiments. We find the difference between having the (approximated) negative entropy term and ignoring the negative entropy to be negligible. We leave theoretical investigation on the effects of leaving out the entropy term during training to future work.
13

---

Under review as a conference paper at ICLR 2021
-11.98
-12.00
-2.02
12.04
-0.56
with entropy
without entropy
-0 0.85
without entropy
with entropy
2000
4000
6000
8000
10000
2500
5000
7500 10000 12500
15000
17500 20000
(a) Toy dataset
(h) Fading square dataset
Figure 7: Figure (a) and (b) show the (approximated) entropy value using two different training objectives, for two different experiments.
C EXPERIMENTS
C.I NETWORK ARCHITECTURE
consists of incompressible affine coupling layers (Sorrenson al. Dinh et al. 2016). Each coupling layer splits D-dimensional input x into two parts X1:d and Xd+1:D. The output of the coupling layer is
The flow network
2020
Yi:d = X1:d
(41)
Yd+1:D = Xd+1:D exp(s(x1=a) t(x1a))
(42)
where 8 R - RDD and Te. + RD-d are scale and translation functions parameterized by neural networks, O is the element-wise product. The log-determinant of the Jacobian of coupling layer is the sum of the scaling function E; s(Xi.d)j. To make the coupling transform volume preserving, we normalize the output of the scale function, so the i-th dimension of the output is
((11a); = s(X1:d); D d Σ s(Xi=d)s>
(43)
and the log-determinant of the Jacobian is E, ((1:aa); 0. We compare volume preserving
flow with a Iearnable prior (normalized s(-)) to a non-volume preserving flow with fixed prior (un- normalized s(-)). In this way both models have the ability to adapt their 'volume' to fit the target distribution, retaining comparison fairness.
In affine coupling layer, the scale function and the translation function t have two types of structure: fully connected net and convolution net. Each fully connected network a 4-layer neural 3-layer
network with hidden-size 24 and Leaky ReLUU with slope 0.2. Each convolution convolutional neural network with hidden channel size 16, kernel size 3x3 and padding size 1. The activation is Leaky ReLU with slope 0.2. The downsampling decreases the image width and height by a factor of 2 and increases the number of channels by 4 in a checkerboard-like fashion (Sorrenson al. 2020 Jacobsen et al. 2018). When multiple convolutional nets are connected together, randomly permute the channels of the output of each network except the final one. In Table we show the network structures for our four main paper experiments.
Type of block
Number
Input shape
Affine coupling layer widths
Fully connected
2
2
1 + 24
24
24 ->
Table The network structure for toy datasets.
14

---

Under review as a conference paper at ICLR 2021
Type of block
Number
Input shape
Affine coupling layer widths
even: 2 24 + 24 + 24 -
& Fully connected
6
3
odd: 1 - 24
24 24 - 2
Table 2: The network structure for S- S-Curve dataset. If denote the input vector of each coupling is x, the function 8 and takes x(0] in the first coupling layer and x(1] in the second coupling layer.
Type of block
Number
Input shape
Affine coupling layer widths
Downsampling
(1, 32, 32)
Convolution
(4, 16, 16)
2 - 16 - +
Downsampling Convolution Flattening
(4, 16)
2
(16,8,8)
8 - 16 + 16 16
(16, 8,8)
Fully connected
1024
512 + 24 7 24 - 24 - 512
Table 3:
The network structure for our fading square dataset experiments. If we denote the input vector of each coupling to be the functions and t take values x[1:2] in the even coupling layer and x x[3] in the odd coupling layer.
Type of block
Number
Input shape
Affine coupling layer widths
Downsampling
(1, 28, 28)
Convolution
4
(4, 14, 14)
2 - 16 - +4
Downsampling
(4, (4, 14)
Convolution
4
(16,7,7)
8 16
16
Flattening
(16, 7,7)
Fully connected
2
784
392
24
24 24 392
Table 4: Network structure for synthetic MNIST dataset experiment.
C.2 TOY DATASET
We also construct second dataset and train flow model using the same training procedure dis- cussed in Section 7.] Figure Sal shows the samples from the data distribution IP,, each data point is 2D vector
[2] where I1 (O, 1) and 22 Ij. Indim(Pa))
1. Figure 81
shows that the prior Pz has learned the true intrinsic dimension Indim(F2) = Indim(Pa) = I. We compare to samples drawn from a flow model that uses a fixed 2D Gaussian prior, with results shown in Figure 8 We can observe, for this simple dataset, flow with a fix Gaussian prior can gen- erate reasonable samples, but the 'curling up' behavior, discussed in the main paper, remains highly evident in the Z space, see Figure 8c
We also plot the 'density allocation' the manifold for the toy datasets. For example, for the data generation process x [I1, where Ij p -( 1) and Ø2 = Ii, we use the density value p( x1i indicate the 'density allocation' on the ID manifold. To plot the 'density allo- cation' of our learned model, we first sample x, uniformly from the support of the data distribution, the subscript 's' here means that they only contain the information of the support. Specifically, since U( 33,,3r) (a is the standard deviation of (O 1), U is the
x, [=]. we sample of uniform distribution) and zÿ = o or ag sin (a depending on which dataset used. We use the projection procedure that was described in Section 7.2 to obtain the projected samples zprod so zoros R Tndim(Pa), We also project the learned prior Pz to I TrdiimPPa) by constructing peroj as Gaussian with zero mean and diagonal covariance contains the non-zeros eigenvalues of AAT. Therefore peros is in R Innimm(Fa) and we denote its density function p (z). We then use the density value projj (z 2.ros) indicate the 'density allocation' at the location of x, the manifold support. Figure [ we compare our model with the ground truth and find that we can successfully capture the 'density allocation' the manifold.
15

---

Under review as a conference paper at ICLR 2021
Pa(x)
3
3
-3
(a) Data distribution
(b) Fix Gaussian prior: X space
(c) Fix Gaussian prior: Z space
Px
02
000
(d) Our method: X space
(e) Our method:
space
Eigenvalues of A.A
Figure 8: (a) shows the samples from the data distribution. (b) and (d) show samples from flow with a fixed Gaussian prior and our method, respectively. (c) and (d) show the latent space both models. In (f), we plot the eigenvalues of the matrix AAT.
C.3 S-CURVE DATASET
To fit model the data, the Adam optimizer with learning rate 5 10-1 and batch size 500 and train the model for 2001 iterations. We anneal the learning rate with factor of 0.9 every 10k iterations.
We compare our method with a traditional normalizing flow with a fixed 3D Gaussian prior. Both models have the same network architecture and training procedure. Figure T0a and 10b show the samples form our model and the traditional flow with fixed 3D Gaussian prior. We can observe more samples lying outwith the true data distribution in Figure [Tbb] than in Figure [0al We can conclude that our model has better sample quality considering this S-Curve dataset. We also compare the latent representation for both models, see Figure T0c and [Od] We can see the representation distribution Qz captures the structure of the data distribution well whereas the distribution Qz in Figure 10d is unable to so.
C.4 FADING SQUARE DATASET
To fit the data, train model for 20k iterations with batch-size 100 using the Adam optimizer. The learning rate initialized to 10- and decays with a factor of 0.9 at every 1 iterations. We additionally use an L2 weight decay with factor 0.1.
C.5 MNIST DATASET
C.5.1 IMPLICIT DATA GENERATION MODEL.
To fit implicit model to the MNIST dataset, first train Variational Auto-Encoder (VAE) (Kingma and Welling (003)) with Gaussian prior p(z)=N(00, I). The encoder is q(z[x) = N (uo(x), (x)) where Σ is diagonal matrix. Both μg and Eg are parameterized by 3-layer
16

---

Under review as a conference paper at ICLR 2021
0-
-2
2
3
(a) Ground truth
(b) Our method
1.00
100
0.5
675
800
0.00
0.25
0.75
-1.00
-1.00
(c) Ground truth
(d) Our method
Figure 9: (a) and (c) shows the ground truth 'density allocation' on the manifold for two toy datasets, (b) and (d) shows the "density allocation' learned by our models.
feed-forward neural network with ReLU activation and the size of the two hidden outputs are 400 and 200. We use a Gaussian decoder p(x (z) N (go(2), r with fixed variance ox = 0.3. The ge is parameterized by 3-layer feed-forward neural network with hidden layer sizes 200 and 400. The activation of the hidden output uses a ReLU and we utilize a Sigmoid function in the final layer output to constrain the output between 0 and 1. The training objective is to maximize the lower bound of the log likelihood
logp(x) 2 / log) KL(g(zi))(p(()),
see Kingma and Welling (2013) for further details. We use an Adam optimizer with learning rate 1 x 10 and batch size 100 to train the model for 100 epochs. After training, we sample from the model by first taking a sample z ~ p(z) and letting x = (z).. This is equivalent to taking 8(x go go(z))d(z)dz. further discussion regarding this implicit model construction. In Figure 11 we plot samples from the trained implicit model with dim(z) dim(z)= 10 and the original MNIST data.
sample from an implicit model (x()
Tolstikhin et al. (2017) for
C.5.2 FLOW MODEL TRAINING
We train our flow models to fit the synthetic MNIST dataset with intrinsic dimensions and 10 and the original MNIST dataset. In all models, we use the Adam optimizer with learning rate 10-4 and batch size 100. We train the model for 3000k iterations and, following the initial 1000k iterations, the learning rate decays every 10k iterations by factor 0.9. In Figure 12 plot the samples from our models trained on three different training datasets. In Figure 13, we also plot the samples from traditional flow models that trained these three datasets using the same experiment settings, we found the samples from our models are sharper than that from traditional flow models.
17

---

Under review as a conference paper at ICLR 2021
Pd
Px
Px
0
-1
2
-2
2
Our model
(h) Fix Gaussian prior
Qz
Pz
0
(c) Our model
(d) Fix Gaussian prior
Figure 10: Figure (a) and (b) plot the data distribution Pz of the S-Curve dataset and the samples from model and traditional flow with fixed Gaussian prior. Figure (c) shows the representa- tion distribution Qz and the learned prior Iz of our model. In Figure (d), we plot the representation distribution Qz using the flow with fixed Gaussian prior.
0 1 4
4
3
7
3
7
1
8
0
8
0
2
5
6
5 04
/
9
2
3
4
0 3
8
6
6
2
7
9
9
6
9
3
2
9
9
6
8
3
3
3
6
1
7
2
8
6
9
9 4
2
0
8
[
4
8
4
0
5
7
9
5
4
3
M
0 9 /
1
2
4
3
2 7 9
5 6 5
9
4
2
0
7
6
9
2
0
3
0
8
8
7
<
1
6
3
8 6
9
0
5
6
0
6
4 7 9
6
6
9
6
1
9
3
8
9
0
0
7
9 6 3 e
9
0
1
8
1 9
3
9
8
5
3
2 0
0
0
7
7
b
4
6
4
8
0
3
0
7
9
8
0
9
4
/
8
5
1
|
5
3
0
4
3
3
3
0
8
0
3
4
4
6
0
4
5
6
0
0 7
7 2
1
3
2
3
7
0
9
3
3 3
5
8
a
4
5 4
4
1
7
6
3
0
2
/
3
2
9
/
2
5
3
1
0
9
I
9
6
8
0
3
9
0
2
6
7
8
3
9
0
4
6
Q
2
9
9
7
8
6
2
8
2
4
J
5
0
9
3
8
6
7
4
6
8
0
7
8
3
/
(a) (
(b) dim(z) 10
(c) MNIST Dataset
Figure 11: Training data for the flow model. Figure (a) and (b) synthetic MNIST samples from two implicit models with latent dimension and 10. Figure (c) are samples from the original MNIST dataset.
18

---

Under review as a conference paper at ICLR 2021
7 9
1
8
/
0
8
0
9
7
4
2
2
2
9 0 9
4
8
8
4
9
0
3
0
0
1
3
1
2
9
0
9
6
47
3
2
1
8
4
6
8
3
4 Φ
4
8
4
5
9
9 8
5
7
4
9
29
8
5
9
8
8
9
4
7 5 7
6
0
2
3
3
9
4
3
839
R
0
0
0
0
7 L
0
3
9
/
G
3
8
9
7
2
9
0
9
3
8
9
0
5
Q 7
0
1
8
0
0 9 0
6
9
0
0
3
9 8
<
0
7
0
4 4
0
9
9 6
9
2,
9
1
/
9 6
0
8
7
1
0
D
7
7
9
9
9
0
4 7
7
3
4
5
8
2
7
7
8
9
7
8
4
6
0
f
5
5
0
0
7
3
7
9
4
8
9
8
0
9
D
9 8
9 0
8
2
4
9
3
D
8
8
5
5
3
J
8
6
Q
8
(a) dim(z) 5
(b) dim(u) 10
(c) MNIST
Figure 12: Samples from our methods. Figure (a) and (b) are samples flow models trained on
synthetic MNIST data with intrinsic dimension 5 and 10. Figure (c) samples from flow model that trained on original MNIST data.
9
9
7
6
1
3
9
9
6
4
9
7
9
0
0 2 4
0
5
8
9
(
2
3
O
3
0
0
0 9 1
3
9
9
9
2
7
0
G 8 7
3
&
2
9
2
7
4
9
5
6
3
0
2
9
?
9 / S
0
/
7
9
5
3
4
2
4
0
0
8
9
7
@ O R
69
9
2
5
2
8
7
7
8
6
3
9
7
9
2
8
/
9
U 1 1 9
4
4
9 4
7 6 4
0
0
7
3
8
4
9
0
2
5
R
0
Q
9
9
2
6
7
/
Q
7
9
0
7
4
6
0
3
G
7
9
8
(
9
d
5 0 3 6
0
0
0
8
4
0
/
/
8
9
9
4
9
3 9
9
8
8
0
7
0
9 6 0
7
9
9
4
0
4
9
/
8
9
0
7
5
5
1
3
5
6
9
5
0
0
0
3
4
5
4
3
9
4
7
(a) dim(z)
(b) dim(z) 10
(c) MNIST
Figure 13: Samples from traditional non-volume preserving flow models with fixed Gaussian prior.
19


Reviews:
[{"review_id": "oyXFucOT_eg", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a modification to the latent distribution of a flow model, replacing the commonly used full-rank Normal with a low-rank one which has the form N(0, AA^T). To train this degenerate model, the spread divergence from Zhang et al. (2020) is used and some approximation is made (e.g., ignoring the first entropy term). Experiments on toy data (with 1d intrinsic dimension) and mnist digits generated from a GAN (with 5-10 intrinsic dimension) demonstrated that the proposed model can identify the intrinsic dimension through the rank of AA^T.\n\n##### Originality & Significance\nAs far as I know, the proposal of modeling the latent distribution of a flow model with a low-rank Normal is original .\nOther parts of the paper are more like combination of existing methods and known results (e.g., the invariance of KL under invertible transformation, spread divergence). \n\n##### Clarity\nThe clarity is generally good, although I don't feel the result in equation (4) is worth highlighting in the introduction. It can be a bit misleading as this is a well-known result.\n\n##### Strengths\n* Figure 1(c) is very illustrative as it shows how the distribution is curling up in the latent space to behave more like that it has two intrinsic dimensions, and the conclusion that neural network capacity is wasted by doing this.\n* The experiment results are good and it is clever to use the data generated from a GAN to control the intrinsic dimension of image data, although the data is still very toy.\n\n##### Weaknesses\n* The authors mentioned several times that they \"establish a connection between MLE and KL minimization in Z space\". I guess they are not aware of this result thus trying to make it a contribution of this work. However, the invariance of KL under reparameterization is a well-known result, and it immediately follows that doing MLE = minimizing KL in X space = minimizing KL in Z space.\n* The entropy term in equation (12) can only be ignored given two conditions\n * g is volume preserving, so that the log jacobian term disappears.\n * The variance of K is close to zero.\ndid the authors all use volume preserving flows in experiments? And the second condition can never be true in practice. In the paper an experiment that approximates this term has been conducted to justify the choice. But the approximation is very rough and I'd encourage the authors to try more accurate gradient estimators for the entropy term. For example, there is nonparametric score estimators (https://github.com/miskcoo/kscore) which can be easily plugged in here if you want to make a fast attempt, or you could try parametric estimators like sliced score matching (which requires training a neural network).\n* All experiments are sort of toy and has only <10 intrinsic dimensions. Among them the most complex data is MNIST. Therefore, it is  a question if there exist other easier methods that could also find out the intrinsic dimension given the problem is very simple (e.g., nonlinear PCA).\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "dc96UxRoBqJ", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method of training flow models, instead of minimizing KL divergence in the data space X, the paper proposes to minimize the KL divergence in the latent space Z. However, the problem of this is dimension mismatch, so the KL divergence is ill-defined. The proposed solution is to add noise, such that the KL divergence can be well defined again. In addition, this paper proposes learning a low dimensional prior p(Z) by a Gaussian distribution with low rank covariance matrix. \n\nPro\n\nThe proposed method addresses a very important concern about flow models, which is that the latent space dimension has to be identical to the data space dimension. \n\nThe paper proposes learning the dimensionality of the latent space (prior) distribution p(Z) by parameterizing it as a Gaussian with low rank covariance. This seems like a reasonable design choice. \n\nThe experiments show that for simple synthetic data, the proposed approach is able to recover the true dimensionality of the data. \n\nCon:\n\nThe major caveat is that the method seems to only work for volume preserving flow models (since the term 1 of Eq (12) can only be approximately neglected if the flow is volume preserving); most modern flow architectures are not volume preserving. I think this is quite a major limitation that needs to be explicitly stated when introducing the scope of the work. \n\nIt is unclear why minimizing the KL divergence in latent space Z is better than minimizing the KL divergence in data space X (which is current standard practice) \n\nI think the severity of the dimension mismatch problem is somewhat overstated. If one applies standard training procedure for flow models (i.e. maximum likelihood on data space X) instead of the alternative proposed in the paper (minimizing KL divergence in latent space Z), then the problem in Figure 1 will not occur? I think this is because the dimension of the model distribution is greater than the true data distribution (which is conjectured to lie on a manifold), so log likelihood is well defined. \n\nThe experiments are somewhat weak. Only synthetic datasets are used. If the claimed benefit is better sample generation quality, then at least readers would expect standard benchmarks such as CIFAR. \n\nMinor issues: \n\nThe title is not very informative\n\nThe major contribution of the paper is not stated early on (for example, the introduction seems like a background section). \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PcAtp6pWgdp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method to train flow models on data from low dimensional manifolds embedded in high dimensional ambient spaces. The basic idea is based on minimizing the KL divergence in the latent space, which is equivalent to maximizing expected log-likelihood over the data distribution. Since the KL between two low-dimensional distributions are often undefined, authors propose to use spread divergence as a surrogate. Experiments demonstrate that the authors' technique can successfully model the distributions on low-dimensional manifolds without knowing the manifold beforehand. In particular, the proposed method can recover the intrinsic dimensionality of data manifolds.\n\n#### Pros\n\n* The proposed method simultaneously learns the manifold of a data distribution and the density on it, without the need of manually specifying the manifold.\n\n* The method can give the intrinsic dimensionality of the data manifold. Authors have carried out extensive experimental study to verify this claim using both toy and natural datasets. The method can also be used as an approach for dimensionality reduction.\n\n#### Cons\n\n* The theoretical part is weak. As admitted by the authors, their analysis is not applicable to data distributions that are not absolutely continuous w.r.t. the Lebesgue measure in the ambient space. A stronger analysis should prove that $H(Z_\\mathbb{Q} + K)$ is a constant for fixed $\\sigma_Z$ and volume-preserving $g$. \n\n* Experiments only show that the flow model can learn the manifold correctly. However, data points on the manifold can distribute differently. It seems all experiments in this paper assume that data points lie uniformly on the manifold. It is necessary to see experimental results where data have different densities in different regions of the manifold to check whether the proposed flow model can fit them accurately.\n\n* The proposed method requires using a volume-preserving flow model. This is a big limitation to the expressivity of model architectures.\n\n* In Figure 1, $p_d$ and $p_X$ should be swapped in the legend.\n\n--------------\nPost-rebuttal\n\nI would like to thank the authors for the response. However, my concern on the theoretical part remains. For densities within the manifold, I suggest reviewers consider experiments similar to those in [1][2]. I also agree with R3, R4 on the reinvention of the connection between KL in the data space and KL in the latent space. In addition, I agree with R1, R3, R4 on the scale of their experiments and scalability of the approach. As such, I will lower my score from 6 to 5.\n\n\n[1] Mathieu, Emile, and Maximilian Nickel. 2020. “Riemannian Continuous Normalizing Flows.” arXiv [stat.ML]. arXiv. http://arxiv.org/abs/2006.10605.\n[2] Lou, Aaron, Derek Lim, Isay Katsman, Leo Huang, Qingxuan Jiang, Ser-Nam Lim, and Christopher De Sa. 2020. “Neural Manifold Ordinary Differential Equations.” arXiv [stat.ML]. arXiv. http://arxiv.org/abs/2006.10254.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5I-AHiXrR9", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n1. this work propose to learn a manifold prior, by doing so, it can be used to improve the generation and representation quality.\nIntrinsic dimension is applied in the method.\n2. to fix the ill-defined KL, the authors proposed to use a \"bridge\" distribution, so that Q and P can have overlap on their support.\n\n\nCons:\neven though I love the idea of this work, in the experiment section, the authors fail to compare their method with other flow-based method with quantitative results.\nSince the authors claim that their method can improve the generation and representation quality, without any comparison, it lacks of evidence.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1RMcrDZpYE", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n### Summary:\n\nThis paper uses the idea of spread KL divergence to learn densities for data that lies on a lower-dimensional manifold. The proposed method does not require the knowledge of this dimension and even offers a mechanism to estimate this intrinsic dimensionality. The authors validate their ideas on several toy-experiments.  \n\n### Strength: \n\nThe paper's idea is elegant; it is extremely well written for most parts, and the visualizations are informative. Authors use several compelling examples and make their case favorably.  The notation is pinpoint, and the paper is overall an easy read. \n\n### Concerns:\n\nOne primary problem with the paper is the reinvention of the connection between MLE and KL divergence minimization in Z-space. This is well-documented (see [Papamakarios et al. 2017, Appendix A]; [Papamakorios et al. 2019, Appendix A])--not only authors skipped these citations, but the paper in its current form depicts this as a novel contribution (see Section 1, paragraph 2.)  \n\nI think the extension of this method to large-scale problems is not as trivial. The use parameters of the lower-triangular A scale as $O(D^2)$ can quickly become prohibitive.  Similarly, posthoc computation of eigenvalues of $AA^T$ can become expensive. Further, as noted by the authors in their experiments with MNIST, the real-world data may lie on a noisy lower-dimensional manifold--a modeling challenge that the current theory does not account for. I believe the easiest way to convince me would be to do experiments similar to Brehmer and Cranmer, 2020 on Style-GAN images (see Brehmer and Cranmer, 2020, Section 4, subsection E.) Alternatively, authors can explain why scalability is not an issue and provide other evidence for real-data performance.\n\n#### Minor concern:\n\nIn Cunnigham et al. 2020, the authors consider learning densities for data that lie on lower-dimensional manifolds using normalizing flows. Their approach starts with a low-dimensional latent variable and uses affine transformations to change dimensions; they also use additive Gaussian noise such that the final density is in the ambient space. I believe there are interesting parallels in this work and their work--the authors should consider drawing out this comparison in the revision. \n\n### References:\n\n- Edmond Cunningham, Renos Zabounidis, Abhinav Agrawal, Ina Fiterau, Daniel Sheldon. Normalizing Flows Across Dimensions. ICML workshop, 2020.\n- George Papamakarios, Theo Pavlakou, and Iain Murray. Masked autoregressive flow for density estimation. NeurIPS, 2017.\n- George Papamakarios, Eric T. Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. http://arxiv.org/abs/1912.02762. 2019\n\n### Updates after the rebuttal\n\nI found the approach clean and believe it has some merit among competing approaches. However, as detailed in discussions, I am skeptical of the scalability of the approach. I understand that an approach with scale limitations is acceptable and believe authors can benefit from an honest discussion about tradeoffs--which is currently missing. Further, I would suggest adding non-trivial experiments in the same vein as the ones in $\\mathcal{M}-$flow paper. I think with these revisions, the paper can make for a welcome addition to the flows-for-density-on-manifold-literature.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an algorithm for training flow models by minimizing the KL divergence in the latent space Z. The paper addresses an important problem in training flow models. However, some major concerns remain after the discussion among the reviewers. The scale of the experiments and the scalability of the approach appear limited in the current version of the paper. Moreover, the applicability of the current theoretical analysis to general distributions is quite limited. 
Decision: Reject