Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows
Authors: Brendan Leigh Ross
Jesse C Cresswell
Venue: NeurIPS 2021 Poster (2021)
Keywords: generative modelling
normalizing flows
injective flows
image generation
manifold learning
conformal

Abstract: Normalizing flows are generative models that provide tractable density estimation via an invertible transformation from a simple base distribution to a complex target distribution. However, this technique cannot directly model data supported on an unknown low-dimensional manifold, a common occurrence in real-world domains such as image data. Recent attempts to remedy this limitation have introduced geometric complications that defeat a central benefit of normalizing flows: exact density estimation. We recover this benefit with Conformal Embedding Flows, a framework for designing flows that learn manifolds with tractable densities. We argue that composing a standard flow with a trainable conformal embedding is the most natural way to model manifold-supported data. To this end, we present a series of conformal building blocks and apply them in experiments with synthetic and real-world data to demonstrate that flows can model manifold-supported distributions without sacrificing tractable likelihoods.

Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows
Brendan Leigh Ross Layer 6 Al brendan@layer6.
Jesse Cresswell Layer 6 Al j jesse@layer6.c
Abstract
Normalizing flows are generative models that provide tractable density estimation via an invertible transformation from a simple base distribution to complex target distribution. However, this technique cannot directly model data supported on an unknown low-dimensional manifold, common occurrence in real-world domains such image data. Recent attempts to remedy this limitation have introduced geometric complications that defeat a central benefit of normalizing flows: exact density estimation. We recover this benefit with Conformal Embedding Flows, a framework for designing flows that learn manifolds with tractable densities. We argue that composing a standard flow with trainable conformal embedding is the most natural way to model manifold-supported data. To this end, we present a series of conformal building blocks and apply them in experiments with synthetic and real- world data to demonstrate that flows can model manifold-sin -uppooed distributions without sacrificing tractable likelihoods.
Introduction
Deep generative modelling is the task of modelling complex, high-dimensional data distribution from a sample set. Research has encompassed major approaches such as normalizing flows (NFs) [6,
59),
generative adversarial networks (GANs) +
variational autoencoders (VAEs) 3d autoregressive
models (52), energy-based models T8] score-based models [4], and diffusion models (2
63] NFs
known base density. This approach provides the unique combination of efficient inference, efficient sampling, and exact density estimation, but in practice generated images have not been detailed or realistic as those of those of other methods
in particular describe distribution by modelling change-of-variables mapping
68
One limitation of traditional NFs is the use of base density with the same dimensionality as the data. This stands contrast to models such as GANs and VAEs, which generate data by sampling from low-dimensional latent prior and mapping the sample data space. In many application domains, it is known or commonly assumed that the data of interest lives on a Lower-dimensional manifold embedded in the higher-dimensional data space 21 For example, when modelling images, data samples belong to [0. 1]", where n is the number of pixels in each image and each pixel has a brightness in the domain [0, 1j. However, most points in this data space correspond meaningless noise, whereas meaningful images of objects lie on a submanifold of dimension m < n. A traditional NF cannot take advantage of the lower-dimensional nature of realistic images.
There is growing research interest in injective flows, which account for unknown manifold structure by incorporating base density of lower dimensionality than the data space 66
12
39
A1l
Flows with oow-dimensional latent spaces could benefit from making better use of fewer parameters, being more memory efficient, and could reveal information about the intrinsic structure of the data. Properties of the data manifold, such as its dimensionality or the semantic meaning of latent directions,
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

can be of interest as well [3 58 However, leading injective flow models still suffer from drawbacks including intractable density estimation or reliance on stochastic inverses
In this paper we propose Conformal Embedding Flows (CEFs), a class of flows that use conformal embeddings transform from low high dimensions while maintaining invertibility and an efficiently computable density. We show how conformal embeddings can be used to learn a lower dimensional data manifold, and we combine them with powerful NF architectures for learning densities. The overall CEF paradigm permits efficient density estimation, sampling, and inference. We propose several types of conformal embedding that can be implemented as composable layers of a flow, including three new invertible layers: the orthogonal k x & convolution, the conditional orthogonal transformation, and the special conformal transformation. Lastly, we demonstrate their efficacy on synthetic and real-world data.
Background
2.1 Normalizing Flows
In the traditional setting of a normalizing flow [6 55 an independent and identically distributed sample (x,) C X'= R" from an unknown ground-truth distribution with density (x(x) is used to learn an approximate density Px (x) via maximum likelihood estimation. The approximate density modelled using a diffcomorphism f Z + X' which maps a base density Pz (z) over the space Z R", typically taken to be multivariate normal, to px(x) via the change of variables formula
(x(x) = P2 (f- (x)) Jdct Jr (x))
where (z( is the Jacobian matrix of at the point z. In geometric terms, the probability mass in an infinitesimal volume dz of Z must be preserved in the volume of X' corresponding to the image f(dz), and the magnitude of the Jacobian determinant is exactly what accounts for changes in the coordinate volume induced by f. By parameterizing classes of diffeommrphisms fg, the flow model can be fitted via maximum likelihood on the training data {x.} C X. Altogether, the three following operations must be tractable: sampling with f f(z), inference with f (x), and density estimation with the |det Jr (z) factor. To scale the model, one can compose many such layers f fg o o f,, and the jdet Jr, (z) factors multiply in Eq.
Generally, there no unifying way to parameterize an arbitrary bijection satisfying these constraints. Instead, normalizing flow research has progressed by designing new and more expressive component bijections which can be parameterized, learned, and composed. In particular, progress has been made by designing invertible layers whose Jacobian determinants are tractable by construction. A significant theme has been to structure flows to have a triangular Jacobian 16 59 37 17 Kingma and Dhariwal [35 introduced invertible 1 x 1 convolution layers for image modelling; these produce block-diagone Jacobians whose blocks are parameterized in a PLU-decomposition, so that the determinant can be computed in O(c), the number of input channels. See Papamakarios et al. for a thorough survey of normalizing flows
2.2 Injective Flows
The requirement that f be a diffcomorphism fixes the dimensionality of the latent space. In turn, px(x) must have full support over X, which is problematic when the data lies on submanifold M C X' with dimension m < n. Dai and Wipf 14] observed that if a probability model with full support is fitted via maximum likelihood to such data, the estimated density can converge towards infinity on M while ignoring the true data density p" (x) entirely. Behrmann et al. [ point out that analytically invertible neural networks can become numerically -nveerett,, especially when the effective dimensionality of data and latents are mismatched. Correctly learning the data manifold along with its density may circumvent these pathologies.
Injective flows seek to learn an explicitly low-dimensional support by reducing the dimensionality of the latent space and modelling the flow smooth embedding, or an injective function which is

---

22 u
u2
Z
g
X
21
M
h
21
32
Figure I: A normalized base density in the 2 space is mapped by bijective flow h to a more complicated density in The injective component g maps this density onto a manifold M in X. For inference, data points from M follow the reverse path through g¹ and h to the latent space Z where their densities can evaluated and combined with the determinant factors in Eq.
diffeomorphic to its image This formula for densities as follows [22].
be accommodated with generalized change of variables
Let 2 + X' be smooth embedding from latent space U onto the data manifold M C X. That is, M g(44) the range of g. Accordingly, g has a left-inverse
X + which is smooth for all u E U. Suppose Pu(u is a density on U described using
on M and satisfies g' (g(u)) coordinates The same density can be described in the ambient space X' using coordinates by pushing through g.
The quantity that accounts for changes in the coordinate volume at each point u is V det. (w) where the Jacobian Jg now a n x m matrix 4 Hence, using the shorthand u = gx(),, the generalized change of variables formula defined for x E M can be written
(x) Pu (u) det [J (
2
While g describes how the data manifold M is embedded the larger ambient space, the mapping g alone may be insufficient represent normalized base density. As before, it is helpful to introduce a latent space Z of dimension m along with a diffcomorphism h + l4 representing a bijective NF between Z and U [ Taking the overall injective transformation g h and applying the chain rule J Jghh~ = Jn simplifies the determinant Eq. ? since the outer Jacobian Jn is square,
det P (det Jn,)² 2 del [T
Finally, writing z = -u)). the data density is modelled by
(x(x) = Pa (z) |det Jn (z)l |et
(4)
with the entire process depicted in Fig.
Generating samples from (x( is simple; a sample Px is drawn from the base density and passed through g o h. Inference on data sample Px Px(x) achieved by passing it through h oog. evaluating the density according Pa (z), and computing both determinant factors.
Notably, the learned density Px (x()) only has support on low-dimensional subset M of X, as per the manifold hypothesis. This formulation leads the learned manifold M to be diffeomorphic to Euclidean space, which can cause numerical instability when the data's 's support differs in topology [T] but we leave this issue to future work.
In practice, there will be off-manifold points during training or if g(u) cannot perfectly fit the data, in which case the model's 's log- likelihood will be co. Cunningham et al. [ remedy this by adding
Throughout this work dwweemmeniionll space can sit inside dimeeniimmnii space. This not to be confused with other uses term machine learning, namely a low-dimensional representation high-dimensional or discrete data. 2 denotes left-inverse function, not necessarily the matrix pseudoinverse.
"embedding" in the topological
function which describes how

---

an off-manifoll noise term to the model, but inference requires a stochastic inverse, and the model must be optimized using an ELBO-like objective. Other work 6 39 has projected data to the manifold via g o prior to computing log-likeliin log-likelihoods and optimized using the reconstruction loss Ex=pg X g (x)) We prove in App. A that minimizing the reconstruction loss brings the learned manifold into alignment with the data manifold.
When computing oogllikeihhodds the determinant term log det g presents a computational challenge. Kumar et al. [I] maximize it using an approximate lower bound, while Brehmer and Crammer and Kothari et al. [3]] circumvent its computation altogether by only maximizing the other terms in the log- ligliikliioodd. In concurrent work, Caterini et al. [9] optimize injective flows using stochastic estimate of the log-determinant' gradient. They are also able to optimize log det [ g exactly for smaller datasets, but this procedure involves the explicit construction of Jg. which would be memory-intensive to scale to larger data such as CelebA. In line with research to build expressive bijective flows where det. J is tractable, our work focuses on designing and parameterizing injective flows where log det R as a whole efficiently computable. In contrast to past injective flow models, our approach allows for straightforward evaluation and optimization of log del [J in the same way standard NFs do for log (det Jr|- As far as we can find, ours is the first approach to make this task tractable at scale.
3 Conformal Embedding Flows
In this section we propose Conformal Embedding Flows (CEFs) as a method for learning the low- dimensional manifold M C X and the probability density of the data on the manifold.
Modern bijective flow work has produced tractable log det Jrl terms by designing layers with For injective flows, the combination JTJ is symmetric, so it is triangular if and only if it is diagonal. In turn, J being diagonal is equivalent to Jg. having orthogonal columns. While this restriction feasible for single layer g. it not composable. gi and g2 are both smooth embeddings whose Jacobians have orthogonal columns, it need not follow that J gagg: has orthogonal columns. Additionally, since the Jacobians are not square the determinant in Eq. @2.. det [J' J JT JR. cannot be factored into a product of individually computable terms g: g2
triangular Jacobians [I. 17.
as in Eq. B.. To ensure composability we propose enforcing the slightly stricter criterion that each J J be a scalar multiple of the identity. This is precisely the condition that g is conformal embedding.
Formally, g : + X is conformal embedding if it is a smooth embedding whose Jacobian satisfies [ = (m)
(5) where x: 41 R is smooth non-zero scalar function, the conformal factor factor + In other words, Jg has orthonormal columns up to a smoothly varying non-zero multiplicative constant. Hence g locally preserves angles.
From Eq. 5 it is clear that conformal embeddings naturally satisfy our requirements as an injective flow. In particular, let 1 + X' be conformal embedding and h 2 - U be standard normalizing flow model. The injective flow model g h: 2 -> X' satisfies
xx(x)= P2 (z) |det Jn (z) (z)1-²x (u).
We call g o h Conformal Embedding Flow.
CEF' provide a new way to coordinate the training dynamics of the model's manifold and density. It is important to note that not all parameterizations g of the learned manifold are equally suited to density estimation [9]. Prior injective flow models [6 39 have been trained sequentially by first optimizing g using the reconstruction loss Ex~p; ||x g(g (x)) ? then training h for maximum likelihood with g fixed. This runs the risk of initializing the density Pu (u) in configuration that is challenging for to learn. Brehmer and Cranmer [ also alternate training g and h. but this does not prevent g from converging to poor configuration for density estimation. Unlike previous injective flows, CEFs have tractable densities, which allows and h to be trained jointly by optimizing the loss function
L= Exopp [- log + al|x g(g (x) This mixed loss provides more flexibility in how g is learned, and is unique to our model because is the first for which log (x) is tractable.
(7)

---

3.1 Designing Conformal Embedding Flows
Having established the model's high-level structure and training objective, it remains for us to design conformal embeddings which are capable of representing complex data manifolds. For g to be useful in a CEF we must be able to sample with g(u), perform inference with g' (x), and compute the conformal factor X(u). In general, there is no unifying way to parameterize the entire family of such conformal embeddings (see App. B for more discussion). As when designing standard bijective flows, we can only identify subfamilies of conformal embeddings which we parameterize and compose to construct an expressive flow. To this end, we work with conformal building blocks g: -14-1 - U (where Un U and U₁ X), which compose produce the full conformal embedding g:
gA
In turn, g is conformal because
J - (Jz .... ge (Jg. J...gg)))) 17 .
Our goal in the remainder of this section is to design classes of conformal building blocks which can be parameterized and learned in a CEF.
3.1.1 Conformal Embeddings from Conformal Mappings
Consider the special case where the conformal embedding maps between Euclidean spaces U4 C R" and V C Rd of the same dimensiorf In this special case g; is called a conformal mapping. Liouville's theorem 27 states that any conformal mapping can be expressed as a composition of translations, orthogonal transformations, scalings, and inversions, which are defined in Table I (see App. [B. for details on conformal mappings). We created conformal embeddings primarily by composing these layers. Zero-padding 3 is another conformal embedding, with Jacobian g = (Im 0) and can be interspersed with conformal mappings to provide changes in dimensionality.
Table I: Conformal Mappings
TYPE
FUNCTIONAL FORM
PARAMS
INVERSE
X(u)
Translation
u - u a
a E Rd
V
Orthogonal
u 1-> Qu
Q E (d)
V Q
Scaling
u r-> Au
À E R
V A ''vv
A
Inversion
u + u/|lu|"
V v/lvv|
|u||
SCT
+ 1 u-
||||||//1111
|
e Rd
V 142b 61 Ivj
v+Ilvl
1-2b u+ u+||b||* |ull
Stacking translation, orthogonal transformation, scaling, and inversion layers is sufficient to learn any conformal mapping in principle. However, the inversion operation is numerically unstable, replaced it with the special conformal transformation (SCT), a transformation of interest in conformal field theory 15). It can be understood as inversion, followed by translation by -b, followed by another inversion. In contrast to inversions, SCTs have a continuous parameter and include the identity when this parameter is set 0.
The main challenge to implementing conformal mappings was writing trainable orthogonal layers. We parameterized orthogonal transformations in different ways: by using Householder matrices [67.. which are cheaply parameterizable and easy to train, and by using GeoTorch, the API provided by 44, which parameterizes the special orthogonal group by taking the matrix exponential of skew-symmetric matrices. GeoTooch also provides trainable non-square matrices with orthonormal columns, which conformal embeddings (not conformal mappings) and which incorporate change the data's dimensionality.
To scale orthogonal transformations to image data, we propose new invertible layer: the orthogonal x k convolution. In the spirit of the invertible
convolutions of Kingma and Dhariwal [35], we
We consider conformal mappings between spaces of dimension d >> 2. Conformal mappings in d = 2 are much less constrained, while the case = 1 is trivial since there is no notion of an angle.
5

---

note that a k x & convolution with stride k has block diagonal Jacobian. The Jacobian is orthogonal and only these blocks are orthogonal. It suffices then to convolve the imput with set of filters that together form an orthogonal matrix. Moreover, by modifying these matrices to be non-square with orthonormal columns (in practice, reducing the filter count), we can provide conformal changes in dimension. It is also worth noting that these layers can be inverted efficiently by applying transposed convolution with the same filter, while a standard invertible matrix inversion. This facilitates quick forward and backward passes when optimizing the model's reconstruction loss.
convolution requires a
3.1.2 Piecewise Conformal Embeddings
To make the embeddings more expressive, the conformality condition g can be relaxed to the point of being conformal almost everywhere. Formally, the latent spaces 2 and l/ are redefined as 2 = {z g conformal h(z)} and U h(2). Then g remains conformal embedding U. and as long as (x : g is nonconformal at g' (x)) also has measure zero, this approach poses no practical problems. Note that the same relaxation is performed implicitly with the diffeomorphism used in coupling layers | and can be
property of standard flows when rectifier nonlincarities
justified by generalizing the change of variables formula 38
Table 2: Piecewise Conformal Embeddings
TYPE
FUNCTIONAL FORM
PARAMS
LEFT INVERSE
X(u)
Conformal
Qu
ReLU
Q O(d)
Q" (v1 v2)
ReLu
Qu
Conditional
Q1u
if
< [[]] 2 1
[|u]]]
Q if [v] <1
Orthogonal
Qau
if
Q Q2 E O(d)
v
Qu if [v] 2
We considered the two piecewise conformal embeddings defined in Table 2 Due to the success of ReLU in standard deep neural networks 500) try ReLU-like layer that piecewise conformal. conformal ReLU is based on the injective ReLU proposed by Kothari et al.
39 We believe it to be of general interest as dimension-changing conformal nonlinearity, but it provided no performance improvements in experiments.
More useful was the conditional orthogonal transformation. which takes advantage of the norm- preservation of orthogonal transformations to create an invertible layer. Despite being discontinuous, it provided a substantial boost in reconstruction ability image data. The idea behind this transfor- mation can be extended to the other parameterized mappings in Table [ For each type of mapping can identify hypersurfaces in IR" such that each hypersurface is mapped back to itself; i.e., each hypersurface an orbit of its points under the mapping. Applying the same type conformal mapping piecewise either side remains an invertible operation long as trajectories do not cross the hypersurface, and the result is conformal almost everywhere. The conditional orthogonal layer was the only example of these that provided performance improvements.
4 Related Work
Flows preseribed manifolds. Flows be developed for Riemannian manifolds M C X which are known in advance and can be defined as the image of some fixed & 14 + X, where In particular, Rezende et al. [60] model densities on spheres and tori with convex combinations of Möbius transformations, which are cognate to conformal mappings. For known manifolds ø is fixed, and the density's ' Jacobian determinant factor may be computable in closed form. Our work replaces & with a trainable network but the log-determinant still has a simple closed form.
U C R
22
4
Sz!
Flows on learnable manifolds. Extending flows to learnable manifolds brings about two main challenges: handling off-manifold points, and training the density the manifold.

---

When the distribution is manifold-supporree it will assign zero density to off- manifold points. This has been addressed by adding an off off-manifold noise term
1E or by projecting the data onto the We opt for the latter approach.
manifold and training it with a reconstruction loss 6 39
Training the density the manifold is challenging because the log-determinant term is typically intractable. Kumar et al. A use series of lower bounds to train the fog-determinant, while Brehmer and Cranmer 3 and Kothari et al.
33 separate the flow into two components and train only the low-
dimensional component. Caterini et al. maximize log-likelihood directly by either constructing the embedding's s Jacobian explicitly or using stochastic approximation of the rog-determmnant's gradient, but both approaches remain computationally expensive for high-ding sigh--imensional data. Our approach the first injective model to provide learnable manifold with exact and efficient determmmrnn computation.
Conformal networks. Numerous past works have imposed approximate conformality or its special 57 770 but it has been less common to enforce conformality strictly. To maintain orthogonal weights, one must optimize along the Stiefel manifold orthogonal matrices. Past work to achieve this has either trained with Riemannian gradient descent or directly parameterized subsets of orthogonal matrices. Riemannian gradient descent algorithms typically require singular value QR decomposition at each training step [26
cases as a regularizer 4 31
56
We found that orthogonal matrices trained more quickly when directly parameterized. In particular, Lezcano-Casado and Martínez-Rute Rubio 45 and Lezcano-Casas < parameterize orthogonal matrices as the matrix exponential of skew-symmetric matrix, and Tomczak and Welling 66 use Householder matrices.
30 53
We used mix of both.
5 Experiments
To implement CEFs, we worked off of the nflows github repo 20 which is derived from the code of Durkan et al. 19 Our code is available httpss//gitthub com com/layer6ai-labar /CEF Full model and training details are provided in App. C while additional reconstructions and generated images are presented in App.
5.1 Spherical Data
To demonstrate how a CEF can jointly learn a manifold and density, we generated a synthetic dataset from known distribution with support spherical surface embedded in IR as described in App.
The distribution is visualized Fig. along with the training dataset of 10 sampled points.
We trained the two components of the CEF jointly, using the mixed loss function in Eq. 7 with an
end-to-end log-likelihood term. The resulting model density plotted in Fig. allon with generated samples. It shows good fidelity to the true manifold and density.
55
25
-30
-10
30
10
-10
-13
1.0
-10
-25
1.0
(a)
(b)
(c)
(d)
Figure 2: (a) A density px(x) with support on the sphere, and (b) 10 samples comprising the training
dataset (xi), ) (c) The density learned by CEF, and (d) 10 generated samples.
5.2 Image Data
We now evaluate manifold flows on image data. Our aim is to show that, although they represent a strict architectural subset of mainstream injective flows, CEFs remain competitive in generative performance ( (39] In doing so, this work is the first to include end-to-end maximum likelihood
7

---

Table 3: Synthetic CIFAR-10 Ship Manifolds
64 DIMENSIONS
512 DIMENSIONS
MODEL
RECON
FID
DENSITY
Cov
RECON
FID
DENSITY
Cov
J-CEF
0.000695
36.5
0.0491
0.0658
0.000568
76.2
0.398
0.266
S-CEF
0.000717
35.3
0.0548
0.0640
0,000627
74.7
0,421
0.251
S-MF
0.000469
28.7
0.0756
0.1103
0.000568
53.6
0.570
0.446
Figure 3: Synthetic 512-dimensional Ship Manifold Reconstructions. From top to bottom: groundtruth samples joint CEF, sequential CEP, and sequential ME.
training with injective flow on image data. Three approaches were evaluated on each dataset: jointly trained CEF, a sequentially trained CEF, and for a baseline a sequentially trained injective flow, as in Brehmer and Cranmer labelled manifold flow (MF).
Injective models cannot be compared on the basis of log-likellin likelihoodd since each model may have different manifold support. Instead, evaluate generative performance in terms of fidelity and diversity 661. The FID score (28) a single metric which combines these factors, whereas density and coverage 4 measure them separately. For FID,, lower is better, while for density and coverage, higher is better. We use the PyTorch-Ii) package 62 and the implementation of density and coverage from Naeem et
149
Synthetic image manifolds. Before graduating to natural high-dimensional data, we test CEFs on high-dimensional synthetic data manifold whose properties are better understood. We generate data using a GAN pretrained on CIFAR-I0 40 by sampling from a selected number of latent space dimensions (64 and 512) with others held fixed. Specifically, we sample a single class from the class-conditional StyleGAN2-AD
provided by Karras et [+ This setup reflects our model design in that (1) the true latent dimension is known and (2) since single class is used, the resulting manifold is more likely to be connected. On the other hand, the GAN may not be completely injective,
its support may not technically be manifold. Results are shown in Table Figs. and and App.
Figure 4: Uncurated Synthetic diz-eimnns Ship Manifold Samples. From top to bottom: joint CEF, sequential CEF, and sequential MF.

---

Table 4: Natural Image Data
MNIST
CELEBA
MODEL
RECON
FID
DENSITY
Cov
RECON
FID
DENSITY
Cov
J-CEF
0.003222
38.5
0.0725
0.1796
0.001016
118
0.05581
0.00872
S-CEF
0,003315
37.9
0,0763
0,1800
0,001019
171
0,00922
0.00356
S-MF
0.000491
16.1
0.5003
0.7126
0.000547
142
0.02425
0.00576
7
2
/
4
7
2
4
9
2
7
7
/
0
4
4
a
Figure 5: MNIST Reconstructions, From top to bottom: groundtruth samples, joint CEF, sequential CHF, and sequential MF.
All models achieve comparable reconstruction losses with very minor visible artifacts. showing that conformal embeddings can learn complex manifolds with similar efficacy to state-of-the-ar flows, despite their restricted architecture. The manifold flow achieves better generative performance based on our metrics and visual clarity. Between the CEFs, joint training allows the learned manifold to adapt better, but this did not translate directly to better generative performance.
Natural image data. We scale CEFs to natural data by training on the MNIST 42 and CelebA 4o datasets, for which a low-dimensional manifold structure postulated but unknown. Results are given in Table g Figs. 5 Fand 5 and App. D
As expected, since the MF's embedding is more flexible, it achieves smaller reconstruction losses than the CEFs. On MNIST, this visible as faint blurriness in the CEF reconstructions Fig. 5 and translates better sample quality for the MF as per the metries in Table Interestingly however, the jointly-trained CEF obtains substantially better sample quality on CelebA, both visually (Fig. σ) and by every metric. We posit this due the phenomenon observed by Caterini et al. ? in concurrent work: for complex distributions, the learned manifold parameterization has significant influence on the difficulty of the density estimation task. Only the joint training approach, which maximizes likelihoods end-to-end can train the manifold parameterization to an optimal starting point for density estimation, while sequential training optimizes the manifold solely on the basis of reconstruction loss. CelebA is the presumably quite complex, so one can reasonably expect joint training to provide better results. On the other hand, the sequentially trained CEF's performance suffers from the lack of both joint training and the expressivity afforded by the more general MF architecture.
dataset tested here, and its distribution is
Limitations and Future Directions
Expressivity. Just as standard flows trade expressivity for tractable likelihoods, so must injective flows. Our conformal embeddings in particular are less expressive than state-of-the-ee flow models; they had higher reconstruction loss than the neural spline flow-based embeddings we tested. The con- formal embeddings we designed were limited in that they mostly derive from cimeeniion-preervvii conformal mappings. which a naturally restrictive class by Liouville's theorem [27] Just as early work on NFs [6 59 introduced limited classes of parameterizable bijections, which were later im- proved substantially (e.g. [19 35 ) our work introduces several classes parameterizable conformal embeddings. We expect that future work will uncover more expressive conformal embeddings.

---

Figure 6: Uncurated CelebA Samples. From top to bottom: joint CEP, sequential CEF, and sequential MF.
Manifold learning. Strictly manifold-supported probability models such as ours introduce a bi-
objective optimization problem. How balance these objectives is unclear and, thus far, empirical [6] The difference in supports between two manifold models also makes their likelihoods incomparable. Cunningham et al. [ have made progress in this direction by convolving the manifold manifold-supportee supported distribution with noise, but this makes inference stochastic and introduces density estimation chal-
may make density estimation more tractable in this setting, but further research is needed in this direction.
lenges. We suspect that using conformal manifold-Icane
Broader impact. As deep generative models become more advanced, researchers should carefully consider some accompanying ethical concerns, Large-scale, natural image datasets carry social biases which are likely to be codified in turn by the models trained on them [65]. For instance, CelebA does not accurately represent the real-world distribution of human traits, and models trained on CelebA should be vetted for fairness before being deployed to make decisions that can adversely affect people. Deep generative modelling also lends itself to malicious practices impersonation using deepfakes (69]
S such as disinformation and
Our work seeks endow normalizing flows with more realistic assumptions about the data they model. While such improvements may invite malicious downstream applications, they also encode better understanding of the data, which makes the model more interpretable and thus more transparent. We hope that a better understanding of deep generative models will synergize with current lines of research aimed at applying them for fair and explainable real-world use [ 5
7 Conclusion
This paper introduced Conformal Embedding Flows for modelling probability distributions on low- dimensional manifolds while maintaining tractable densities. We showed that conformal embeddings naturally match the framework of normalizing flows by providing efficient sampling, inference, and density estimation, and they are composable so that they can be scaled to depth. Furthermore, appears conformality is minimal restriction in that any looser condition will sacrifice one more of these properties. As we have reviewed, previous instantiations of injective flows have not maintained all of these properties simultancously.
Normalizing flows are still outperformed by other generative models such as GANs and VAEs in the arena of realistic image generation. Notably, these two alternatives benefit from a low-dimensional latent space, which better reflects image data's manifold structure and provides for more scalable model design. By equipping flows with a low-dimensional latent space, injective flow research has made progress towards VAE- or GAN-level performance. The CEF paradigm is way to match these strides while maintaining the theoretical strengths of NFs.
10

---

Acknowledgments and Disclosure of Funding
We thank Gabriel Doaiza-Ganem and Anthony Caterini for their valuable discussions and advice. We also thank Parsa Torabian for sharing his experience with orthogonal weights and Maksims Volkovs for his helpful feedback.
The authors declare no competing interests or thiry-party funding sources.
References
[1] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In International conference machine learning, pages 214-223, PMLR, 2017.
[2] S. Arora, R. Ge, Y. Liang. T. Ma, and Y. Zhang. Generalization and equilibrium in generative adversarial nets (GANs). In D. Precup and Y. W. Tch, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 224-232. PMLR, 06-11 Aug 2017. URL htttps:/pproceedings .mlr press/v70/ars aroraa7 html
[3] M. Balunović, A. Ruoss, and M. Vechev, Fair normalizing flows. arXiv:2106.05937, 2021. 2021.
[4] N. Bansal, X. Chen, and Z. Wang. Can We Gain More from Orthogonality Regularizations in Training Deep Networks? In Advances in Neural Information Processing Systems, volume 31, 2018.
[5] J. Behrmaann, P. Vicol, K.-C. Wang, R. Grosse, and J.-H. Jacobsen. Understanding and mitigating exploding inverses in invertible neural networks. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130, pages 1792-1808 PMLR, 13-15 Apr 2021.
[6] Brehmer and K. Cranmer. Flows for simultaneous manifold learning and density estimation. In Advances in Neural Information Processing Systems, volume 33, 2020.
[7] A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
[8] M. Brundage, Avin, J. Clark, H. Toner, Eckersley, B. Garfinkel, A. Dafoe, P. Scharre, T. Zeit- zoff, B. Filar, H. Anderson, H. Roff, G. C. Allen, J. Steinhardt, C. Flynn, S. O. hEigcartaigh, S. Beard, H. Belfield, S. Farquhar, C. Lyle, R. Crootof, O. Evans, M. Page, J. Bryson, R. Yam- polskiy, and D. Amodei. The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. /.di://002.7225 2018
[9] A. L. Caterini, G. Loaiza-Ganem, G. Pleiss, and J. P. Cunningham. Rectangular flows for manifold learning. arXiv:2106.014411 2021.
[10] R. Child. Very deep VAEs generalize autoregressive models and outperform them images. In International Conference on Learning Representations, 2021.
[11] R. Cornish, A. Caterini, G. Deligiannidis, and A. Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. In Proceedings the 37th International Conference on Machine Learning, volume 119, pages 1143,,,3, 2020.
[12] E. Cunningham and M. Fiterau. change of variables method for rectangular matrix-vector products. In Proceedings of The 24th International Conference Artificial Intelligence and Statistics, volume 130. PMLR, 2021.
[13] E. Cunningham, R. Zabounidis, A. Agrawal, I. Fiterau, and D. Sheldon. Normalizing Flows Across Dimensions. 1XXiv22006113700 2020.
[14] B. Dai and D. Wipf. Diagnosing and Enhancing VAE Models. In International Conference Learning Representations, ICLR 2019, 2019.
[15] Di Francesco, P. Mathieu, and D. Sénéchal. Conformal field theory. Springer Science Business Media, 2012.

---

[16] L. Dinh, D. Krueger, and Y. Bengio. Nice: Non-linear independent components estimation. arXiv:1410.856,, 2014.
[17] L. Dinh, Sohl-Dickstein, and Bengio. Density estimation using Real NVP. International Conference on Learning Representations, ICLR 2017, 2017.
[18] Y. Du and I. Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems, volume 32, 2019.
[19] C. Durkan, Bekasov, I. Murray, and G. Papamakarios. Neural spline flows. Advances in Neural Information Processing Systems, 32:7511-7522 2019.
[20] C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios. nflows: normalizing flows in PyTorch, Nov. 2020. URL httpps:/dooiii org wwwporrg...........o www.12222oeooooooo. 4296287
[21] C. Fefferman, S. Mitter, and H. Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 2944)99331, 2016.
[22] M. C. Gemici, D. Rezende, and S. Mohamed. Normalizing Flows Riemannian Manifolds, 2016.
[23] I. J. Goodfellow, J. Abudet-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. arXiv:1406. 2661, 2014.
[24] A. Grover, M. Dhar, and S. Ermon. Flow-gan: Combining maximum likelihood and adversarial learning in generative models. In AAAI Conference on Artificial Intelligence, 2018.
[25] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of wasserstein gans. In NIPS, 2017.
[26] M. Harandi and B. Fernando. Generalized BackPropagation, Étude De Cas: Orthogonality. arXiv:1611.0592. 2016.
[27] P. Hartman. On isometries and theorem of Liouville. Mathematische Zeitschrift. 69: 202-210, 1958.
[28] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge Local Nash Equilibrium. In Advances in Neural Information Processing Systems, volume 30, 2017.
[29] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages
2020.
[30] I.. Huang, X. Liu, B. Lang, A. W. Yu, and B. Li. Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks. In AAAI Conference on Artificial Intelligence, 2018.
[31] K. Jia, D. Tao, S. Gao, and X. Xu. Improving training of deep neural networks via singular value bounding. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3994 4002. 2017.
[32] Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[33] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila. Training generative adversarial networks with limited data. In Advances in Neural Information Processing Systems, volume 33, pages 12104-12114, 2020.
[34] D. Kingma and Ba. Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations, ICLR 2015, 2015. URL //app://://ii....//// abs/
1412. 6980
[35] D. Kingma and Dhariwal. Glow: Generative Flow with Invertible 1x1 Convolutions. In Advances in Neural Information Processing Systems, volume 31, 2018.
12

---

[36] D. P. Kingma and M. Welling. Auto-encoding Variational Bayes. arXiv:1312.6114, 2013.
[37] D. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, volume 29, 2016.
[38] N. Koenen, M. N. Wright, P. Maass, and J. Behrmann. Generalization of the change of variables formula with applications to residual flows. ICML. Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021.
[39] K. Kothari, A. Khorashadizadeh, M. de Hoop. and I. Dokmanić. Trumpets: Injective Flows for Inference and Inverse Problems. arxiv:2102.10461 2021.
[40] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. URL https://www. cs. toronto.oo............ Baaronto.eduu"HHH edu/"kriz/
featminn--eatuues-220000 2009--TT---
[41] A. Kumar, B. Poole, and K. Murphy. Regularized Autoencoders via Relaxed Injective Probability Flow. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108, pages 4292-4301, 2020.
[42] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffnner Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.
[43] J. M. Lee. Introduction to Riemannian manifolds. Springer, 2018.
[44] M. Lezcano-Casado. Trivializations for gradient-baser optimization on manifolds. In Advances in Neural Information Processing Systems, NeurlPS, pages 9154-9164, 2019.
[45] M. Lezcano-Casade and D. Martinez-Rubio. Cheap Orthogonal Constraints Neural Networks:
A Simple Parametrization of the Orthogonal and Unitary Group. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 3794-3803, 2019.
[46] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[47] I. Loshchilov and F. Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th
International Conference on Learning Representations, ICLR 2017, 2017. URL https:// wwtp://eee///it/etttsoe/er.s
[48] E. Mathieu and M. Nickel. Riemannian Continuous Normalizing Flows. In Advances in Neural Information Processing Systems, volume 33, pages 2503-2515, 2020.
[49] M. F. Naeem, S. J. Oh, Y. Uh, Y. Choi, and Yoo. Reliable fidelity and diversity metrics for generative models. In Proceedings of the 37th International Conference on Machine Learning, volume 1 11 of Proceedings of Machine Learning Research, pages 7176-7185. PMLR, 13-18 Jul 2020.
[50] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML'10, 10, page 807-814, 2010.
[51] E. Nalisnick, A. Matsukawa, Y. W. Teh, D. Gorur, and Lakshminarayaman. Do deep generative models know what they don't it know? In International Conference on Learning Representations, 2019. URL /ttpss://openrevieew htt.://r..e..ooooorrroaoaaaaoaaa
[52] A. V. Oord, N. Kalchbrenner, and K. Kavukeuoglu. Pixel recurrent neural networks. In Proceed- ings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1747-1756. PMLR, 2016.
[53] M. Ozay and T. Okatani. Optimization on Submanifolds of Convolution Kernels in CNNs. 1701v:1610.07008, 2016.
13

---

[54] G. Papamakarios, E. Nalismick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan. Normal- izing Flows for Probabilistic Modeling and Inference. Journal of Machine Learning Research, 22(57):1 2021.
[55] X. Pennec. Intrinsic statistics on riemannian manifolds: Basic tools for geometric measurements. Journal of Mathematical Imaging and Vision, 25(1):127-154 2006.
[56] E. Peterfreund, O. Lindenbaum, F. Dietrich, T. Bertalan, M. Gavish, I. G. Kevrekidis, and R. R. Coifman, Local conformal autoencoder for standardized data coordinates. Proceedings of the National Academy of Sciences, 1 117(49)330118 10-333997, 2020.
[57] H. Qi, C. You, X. Wang, Y. Ma, and J. Malik. Deep isometric learning for visual recognition. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 7824-7835, 2020.
[58] A. Radford, L. Metz, and S. Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. In International Conference on Learning Representations, 2016.
[59] D. Rezende and S. Mohamed. Variational Inference with Normalizing Flows. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 1530-1538, 2015.
[60] D. J. Rezende, G. Papamakarios, S. Racaniere, M. Albergo, G. Kanwar, Shanahan, and K. Cranmer. Normalizing Flows on Tori and Spheres. In Proceedings of the 37th International Conference on Muchine Learning, volume 119, pages 8083-8092, 2020.
[61] M. S. M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly, Assessing generative models via precision and recall. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[62] M. Seitzer. pytorch-fid: FID Score for PyTorch. https://github.comm pytorch-fid August 2020. Version 0.1.1.
[63] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 2256-2265, 2015.
[64] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data dis- tribution. Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL http:: htttps://proceediggg wtwpeeeeddinnn...n.s.s....e. /eurr/ps.cc/ppppr//0//////
3501e1277447655378888844447 Paper. pdf
[65] R. Steed and A. Caliskan. Image representations learned with unsupervised pre-training contain human-like biases. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 701-713, 2021.
[66] I. Tolstikhin, O. Bousquet, S. Gelly, and B. Schoelkopf. Wasserstein auto-encoders. In International Conference on Learning Representations, 2018. URL http:://ppeeerriiee wiw.orrummidddLLini..
[67] J. M. Tomezak and M. Welling. Improving Variational Auto-Encoders using Householder Flow. arXiv:1611.09633 2016.
[68] A. Vahdat and J. Kautz. NVAE: A Deep Hierarchical Variational Autoencoder. In Advances in Neural Information Processing Systems, volume 33, pages 19667-19679, 2020.
[69] M. Westerlund. The emergence of deepfake technology: A review. Technology Innovation Management Review, 911)), 2019.
[70] L. Xiao, Y. Bahri, J. Sohl-Dickstein S. Schoenholz, and Pennington. Dynamical isometry and a mean field theory of CNNs: How to train 10,000-layer vanilla convolutional neural networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 5393-5402, 2018
14


Reviews:
[{"review_id": "m-tENARNMZA", "reviewer": "Reviewer_YERV", "summary": "Many datasets of interest that are modelled with normalizing flows are assumed to lie on a lower-dimensional manifold embedded in a higher-dimensional ambient space. Standard normalizing flows that are restricted to learn a *bijective* transformation are not well suited for modelling such data, as the target transformation between the lower-dimensional manifold space and the ambient space (in which said manifold is embedded) is *injective*.\n\nIn theory, an *injective flow* can be defined, where we'd parametrize an injective transformation that would only be invertible *for points on the manifold*. In practice there are two problems with this, however. First is that historically there's been much less work on designing flexible *injective* transformations, hence we don't have the same arsenal of invertible layers as we do for bijective flows. Second, the change-of-variable formula for an injective transformation includes a $\\det J_f^T J_f$ term, where $J_f$ is the (rectangular) Jacobian matrix of the injective transformation $f$. This determinant, even if tractable, is typically overly expensive to compute exactly.\n\nAuthors of this work make an observation that in the bijective flow literature the game has always been to design parametric transformations $f$ that are restricted to be invertible, but *also* ones that have and easy-to-compute $\\det J_f$. In line with this, authors propose a class of transformations that would have an easy-to-compute $\\det J_f^T J_f$, specifically *conformal embeddings*. A conformal embedding $f$ is a transformation that preserves *angles*, but more importantly has a property that $J_f^T J_f$ is a scalar multiple of the identity, the determinant of which is simply the product of diagonal terms (i.e. an exponent of said scalar). \n\nAuthors consider a few examples of conformal embeddings (translation, scaling, an orthogonal transformation, zero-padding, etc.), which could be stacked in the same way as invertible layers are in a bijective flow. In addition, authors consider *piecewise* conformal embeddings, which are conformal *almost everywhere* (i.e. for all points except for a subset of measure zero), and can increase the expressivity of the otherwise simple conformal embeddings.\n\nBy using the conformal embeddings as injective transformations authors come up with the first injective flow with a tractable and *cheap-to-compute* density on the learned manifold. This allows said density to be used as a part of the training objective, even though an additional regularization term still needs to be included in the objective to prevent degenerate solutions. The method demonstrates excellent results on a synthetic 3D problem where the target density lies on a sphere, as well as the CelebA image dataset, where the visual fidelity of the samples is improved in comparison to the more expressive baseline flow.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "What an interesting idea, and what a well-written paper --- very much enjoyed reading it! It could be one of these papers that kick-starts a new sub-field within the normalizing flow community, one where researchers would be working on designing novel types of expressive conformal embeddings.\n\nAuthors motivate the proposed method very well, taking the reader on a journey that arrives at the conformal embeddings as almost the one logical solution, which of course it might only be in retrospect.\n\nThe mathematical exposition is excellent: authors manage to keep things precise without drowning the reader in equations.\n\nIn my view the weakest part of the paper is the experiments. While what is done is definitely enough to get the reader interested, it's probably not sufficient to get an idea of whether the method is ready for real-world use. A comparison with alternative injective flow implementations would be of great help, as well exploring the hyper-parameters of the method (like the latent dimensionality) and additional datasets. In similar work datasets produced by sampling from a generative model with a lower-dimensional latent space (itself trained on real data) are sometimes used, where  the \"real\" intrinsic dimensionality of the dataset is known.\n\nNevertheless, I think the paper will be of great interest to the community, and the theoretical part is sufficiently novel to give authors some leeway in regards to the empirical part. I recommend acceptance, but also encourage authors to focus their efforts on the experiments for future iterations of the paper. \n\nMinor points:\n- Line 164: while it might indeed be trivial that zero-padding is conformal, I suggest including it in Table 1 for completeness (and perhaps writing a sentence summarizing the proof of the transform being conformal). This is the part that allows the flow to change dimensionality, hence it's worth being particularly precise here.\n- In two places authors refer to a normalizing flow (I think?) as a \"stump\": this doesn't seem to be defined anywhere, and I've not seen this terminology before.\n- Table 3: it might be useful to have another baseline where the \"Sequential Baseline\" would be run with $g$ that would have the number of parameters that is similar to $g$ used in CEF, just to make the runs more comparable. \n\n---\n\nI thank the authors for their response. These additional results and clarifications in the exposition will definitely strengthen the paper. Having read the other reviews/responses, I maintain my recommendation to accept the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors carefully discuss both the limitations of the method and its potential societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "K3h_BPV7rTJ", "reviewer": "Reviewer_3bMb", "summary": "This work proposes to leverage conformal embedding flows for modeling probability distributions on low-dimensional manifolds. Differing from other manifold-based flows, the proposed methods maintain tractable densities for low-dimensional latent space. The unique contribution enables fast sampling, invertibility for inference, and efficient likelihood estimation, which inherits from normalizing flows. \n\nExisting approaches share the drawback that exact densities cannot be calculated through the dimensionality reducing step. In the paper,\nthe authors present an approach that offers tractable density estimation and is straightforward to train by constraining this reduction to the class of conformal embeddings. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Followed by recent work on manifold flows, the paper poses an attractive research direction for normalizing flows.  Overall, the paper is well-written and easy to follow. The mathematical derivations are rigorous and careful so I did not see any unclear part in the thought process. Through two experiments, the authors demonstrate the idea clearly and also give a detailed discussion about the current limitation and future direction, which would be helpful. I just have few concerns listed as follows:\n\n1. The experiments are not very strong. I think it would be better if the authors could show more synthetic examples, for example, a more challenging one with complex manifolds. For the synthetic example, there are no other baselines for comparison such that we may not get a straightforward insight why the proposed methods are better.  For the image example, could you show more examples, maybe from a simpler one with less computational cost?  To compare the performance of generative models, I know FID is commonly used but recently some other metrics are proposed, for example, density and coverage metrics [1], which may help to evaluate the performance better. \n\n[1] \"Reliable fidelity and diversity metrics for generative models.\" In International Conference on Machine Learning, pp. 7176-7185. PMLR, 2020.\n\n2. The baseline and ablation study.  The authors discussed two important baselines [3] and [27], is that possible to compare them quantitatively with some experiments, even toy examples?  It might be helpful to highlight the original contribution and unique strength of the proposed flows. As for the size of low-dimensional space, have you investigated the effect of various latent dimensionality? It is interesting to see the performance if you have the results in experiments. \n\n3. Implementation and details. I expect to see the implementation if the authors would like to provide the code in the supplementary materials. Unfortunately, I can not reproduce the code and see more details. I just worried about the efficiency, scalability, and flow expressivity as the authors mentioned.  For the flow-based architecture, the authors only use the affline coupling layers in RealNVP and Glow, does it get improvement if some advanced flow-based models, like neural spline flows, are used?  Is the easy to incorporate the conformal embedding operator into other flow-based models? If so, that would increase the general applicability of this method. \n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No potential negative societal impact", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "9.5"}, "extra_text": {}}, {"review_id": "IDtnpf_64L", "reviewer": "Reviewer_wnDz", "summary": "The paper proposes classes of conformal building blocks that can be used to parameterize a homomorphic flow network that can map from low dimensional space to a continuous manifold in high dimension space.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The paper restricts function g to be a conformal embedding. The k*k convolution is new, the SCT is new to the machine learning field.\n\nClarity: The paper is clearly written. But I think the novel building blocks like k*k convolution should be detailed discussed.\n\nQuality: The submission is technically sound.\n\nSignificance and Concerns:\n1. My first concern is more general. The traditional flow models aim to have a tractable likelihood for maximum likelihood estimation and achieves success in a lot of applications. The paper interests in modeling distributions that lie on a low-dimensional manifold. As the author mentioned, the density is no longer well defined (the distribution is not absolutely continuous with respect to Leabage measure), so the maximum likelihood estimation (MLE)  and the KL divergence are not well defined,  so why we need to construct a function with tractable log density in the first place when the density is not defined? Furthermore, the method has to use L2 reconstruction loss as a regularization to alleviate the ill-defined MLE (equation 7), does this objective provide a consistent estimation, and what distance it is minimizing? Why not just directly use a distance which is well defined for the singular distribution like e.g. Wasserstein distance. Although I understand there are several papers that use this setting but the benefit of such a setting is unclear to me which makes me less confident about the motivation of the paper.\n\n2. The paper restricted the function g to be conformal, and propose several building blocks. However, the experiments are limited and cannot support the significance of the proposed methods. The samples of the CelebA experiments are far from good, which maybe because of the limited function power or the training objective with the l2 norm. The reconstruction quality is less meaningful for generative models, it can only verify the invertibility.  I think more experiments and comparisons should be conducted to support the usefulness of the framework. I can understand the  CelebA dataset may not satisfies the model assumption (lies exactly on a low-dimensional manifold ) and the intrinsic dimension is unknown. So I suggest the training data can be generated from a trained GAN model, so the intrinsic dimension is (approximately) equal to the dimension of the latent space and the generated samples will lie on a low dimensional manifold.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the author mentioned the major limitation is expressive power.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes to use conformal mapping to implement injective flow to map into a lower dimensional embedding space. 
The proposed idea is original, well explained, and well motivated. While the experiments were lacking in the initial submission, the authors have properly engaged with the reviewers: they offered the appropriate additional experiments to improve the paper and address reviewers concerns. 

I recommend this paper for acceptance.
Decision: Accept (Poster)