Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows
Year: 2021

Abstract:
Normalizing flows are generative models that provide tractable density estimation via an invertible transformation from a simple base distribution to a complex target distribution. However, this technique cannot directly model data supported on an unknown low-dimensional manifold, a common occurrence in real-world domains such as image data. Recent attempts to remedy this limitation have introduced geometric complications that defeat a central benefit of normalizing flows: exact density estimation. We recover this benefit with Conformal Embedding Flows, a framework for designing flows that learn manifolds with tractable densities. We argue that composing a standard flow with a trainable conformal embedding is the most natural way to model manifold-supported data. To this end, we present a series of conformal building blocks and apply them in experiments with synthetic and realworld data to demonstrate that flows can model manifold-supported distributions without sacrificing tractable likelihoods.

heading: 1 Introduction
text: Deep generative modelling is the task of modelling a complex, high-dimensional data distribution from a sample set. Research has encompassed major approaches such as normalizing flows (NFs) [16, 59], generative adversarial networks (GANs) [23], variational autoencoders (VAEs) [36], autoregressive models [52], energy-based models [18], score-based models [64], and diffusion models [29, 63]. NFs in particular describe a distribution by modelling a change-of-variables mapping to a known base density. This approach provides the unique combination of efficient inference, efficient sampling, and exact density estimation, but in practice generated images have not been as detailed or realistic as those of those of other methods [7, 10, 29, 32, 68].
One limitation of traditional NFs is the use of a base density with the same dimensionality as the data. This stands in contrast to models such as GANs and VAEs, which generate data by sampling from a low-dimensional latent prior and mapping the sample to data space. In many application domains, it is known or commonly assumed that the data of interest lives on a lower-dimensional manifold embedded in the higher-dimensional data space [21]. For example, when modelling images, data samples belong to [0, 1]n, where n is the number of pixels in each image and each pixel has a brightness in the domain [0, 1]. However, most points in this data space correspond to meaningless noise, whereas meaningful images of objects lie on a submanifold of dimension m ⌧ n. A traditional NF cannot take advantage of the lower-dimensional nature of realistic images.
There is growing research interest in injective flows, which account for unknown manifold structure by incorporating a base density of lower dimensionality than the data space [6, 12, 13, 39, 41]. Flows with low-dimensional latent spaces could benefit from making better use of fewer parameters, being more memory efficient, and could reveal information about the intrinsic structure of the data. Properties of the data manifold, such as its dimensionality or the semantic meaning of latent directions,
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
can be of interest as well [36, 58]. However, leading injective flow models still suffer from drawbacks including intractable density estimation [6] or reliance on stochastic inverses [13].
In this paper we propose Conformal Embedding Flows (CEFs), a class of flows that use conformal embeddings to transform from low to high dimensions while maintaining invertibility and an efficiently computable density. We show how conformal embeddings can be used to learn a lower dimensional data manifold, and we combine them with powerful NF architectures for learning densities. The overall CEF paradigm permits efficient density estimation, sampling, and inference. We propose several types of conformal embedding that can be implemented as composable layers of a flow, including three new invertible layers: the orthogonal k ⇥ k convolution, the conditional orthogonal transformation, and the special conformal transformation. Lastly, we demonstrate their efficacy on synthetic and real-world data.

heading: 2 Background

heading: 2.1 Normalizing Flows
text: In the traditional setting of a normalizing flow [16, 59], an independent and identically distributed sample {xi} ⇢ X = Rn from an unknown ground-truth distribution with density p⇤x(x) is used to learn an approximate density px(x) via maximum likelihood estimation. The approximate density is modelled using a diffeomorphism f : Z ! X which maps a base density pz(z) over the space Z = Rn, typically taken to be a multivariate normal, to px(x) via the change of variables formula
px(x) = pz f 1(x) detJf f 1(x) 1 , (1)
where Jf (z) is the Jacobian matrix of f at the point z. In geometric terms, the probability mass in an infinitesimal volume dz of Z must be preserved in the volume of X corresponding to the image f(dz), and the magnitude of the Jacobian determinant is exactly what accounts for changes in the coordinate volume induced by f . By parameterizing classes of diffeomorphisms f✓, the flow model can be fitted via maximum likelihood on the training data {xi} ⇢ X . Altogether, the three following operations must be tractable: sampling with f(z), inference with f 1(x), and density estimation with the |detJf (z)| factor. To scale the model, one can compose many such layers f = fk · · · f1, and the |detJfi (z)| factors multiply in Eq. (1).
Generally, there is no unifying way to parameterize an arbitrary bijection satisfying these constraints. Instead, normalizing flow research has progressed by designing new and more expressive component bijections which can be parameterized, learned, and composed. In particular, progress has been made by designing invertible layers whose Jacobian determinants are tractable by construction. A significant theme has been to structure flows to have a triangular Jacobian [16, 59, 37, 17]. Kingma and Dhariwal [35] introduced invertible 1⇥ 1 convolution layers for image modelling; these produce block-diagonal Jacobians whose blocks are parameterized in a PLU -decomposition, so that the determinant can be computed in O(c), the number of input channels. See Papamakarios et al. [54] for a thorough survey of normalizing flows.

heading: 2.2 Injective Flows
text: The requirement that f be a diffeomorphism fixes the dimensionality of the latent space. In turn, px(x) must have full support over X , which is problematic when the data lies on a submanifold M ⇢ X with dimension m < n. Dai and Wipf [14] observed that if a probability model with full support is fitted via maximum likelihood to such data, the estimated density can converge towards infinity on M while ignoring the true data density p⇤(x) entirely. Behrmann et al. [5] point out that analytically invertible neural networks can become numerically non-invertible, especially when the effective dimensionality of data and latents are mismatched. Correctly learning the data manifold along with its density may circumvent these pathologies.
Injective flows seek to learn an explicitly low-dimensional support by reducing the dimensionality of the latent space and modelling the flow as a smooth embedding, or an injective function which is
diffeomorphic to its image1. This case can be accommodated with a generalized change of variables formula for densities as follows [22].
Let g : U ! X be a smooth embedding from a latent space U onto the data manifold M ⇢ X . That is, M = g(U) is the range of g. Accordingly, g has a left-inverse2 g† : X ! U which is smooth on M and satisfies g†(g(u)) = u for all u 2 U . Suppose pu(u) is a density on U described using coordinates u. The same density can be described in the ambient space X using x coordinates by pushing it through g.
The quantity that accounts for changes in the coordinate volume at each point u isq det ⇥ JTg (u)Jg(u) ⇤ , where the Jacobian Jg is now a n⇥m matrix [43]. Hence, using the shorthand u = g†(x), the generalized change of variables formula defined for x 2 M can be written
px(x) = pu (u) det ⇥ JTg (u)Jg(u) ⇤ 1 2 . (2)
While g describes how the data manifold M is embedded in the larger ambient space, the mapping g alone may be insufficient to represent a normalized base density. As before, it is helpful to introduce a latent space Z of dimension m along with a diffeomorphism h : Z ! U representing a bijective NF between Z and U [6]. Taking the overall injective transformation g h and applying the chain rule Jg h = JgJh simplifies the determinant in Eq. (2) since the outer Jacobian Jh is square,
det ⇥ JThJ T gJgJh ⇤ = (detJh) 2 det ⇥ JTgJg ⇤ . (3)
Finally, writing z = h 1(u), the data density is modelled by
px(x) = pz (z) |detJh (z)| 1 det ⇥ JTg (u)Jg(u) ⇤ 1 2 , (4)
with the entire process depicted in Fig. 1.
Generating samples from px(x) is simple; a sample z ⇠ pz(z) is drawn from the base density and passed through g h. Inference on a data sample x ⇠ px(x) is achieved by passing it through h 1 g†, evaluating the density according to pz(z), and computing both determinant factors.
Notably, the learned density px(x) only has support on a low-dimensional subset M of X , as per the manifold hypothesis. This formulation leads the learned manifold M to be diffeomorphic to Euclidean space, which can cause numerical instability when the data’s support differs in topology [11], but we leave this issue to future work.
In practice, there will be off-manifold points during training or if g(u) cannot perfectly fit the data, in which case the model’s log-likelihood will be 1. Cunningham et al. [13] remedy this by adding
1Throughout this work we use “embedding” in the topological sense: a function which describes how a low-dimensional space can sit inside a high-dimensional space. This is not to be confused with other uses for the term in machine learning, namely a low-dimensional representation of high-dimensional or discrete data.
2† denotes a left-inverse function, not necessarily the matrix pseudoinverse.
an off-manifold noise term to the model, but inference requires a stochastic inverse, and the model must be optimized using an ELBO-like objective. Other work [6, 9, 39] has projected data to the manifold via g g† prior to computing log-likelihoods and optimized g using the reconstruction loss Ex⇠p⇤x ||x g(g
†(x))||2. We prove in App. A that minimizing the reconstruction loss brings the learned manifold into alignment with the data manifold. When computing log-likelihoods, the determinant term log det ⇥ JTgJg ⇤ presents a computational challenge. Kumar et al. [41] maximize it using an approximate lower bound, while Brehmer and Cranmer [6] and Kothari et al. [39] circumvent its computation altogether by only maximizing the other terms in the log-likelihood. In concurrent work, Caterini et al. [9] optimize injective flows using a stochastic estimate of the log-determinant’s gradient. They are also able to optimize log det ⇥ JTgJg ⇤ exactly for smaller datasets, but this procedure involves the explicit construction of Jg, which would be memory-intensive to scale to larger data such as CelebA. In line with research to build expressive bijective flows where detJf is tractable, our work focuses on designing and parameterizing injective flows where log det ⇥ JTgJg ⇤ as a whole is efficiently computable. In contrast to past injective flow
models, our approach allows for straightforward evaluation and optimization of log det ⇥ JTgJg ⇤ in the same way standard NFs do for log |detJf |. As far as we can find, ours is the first approach to make this task tractable at scale.

heading: 3 Conformal Embedding Flows
text: In this section we propose Conformal Embedding Flows (CEFs) as a method for learning the lowdimensional manifold M ⇢ X and the probability density of the data on the manifold.
Modern bijective flow work has produced tractable log | detJf | terms by designing layers with triangular Jacobians [16, 17]. For injective flows, the combination JTgJg is symmetric, so it is triangular if and only if it is diagonal. In turn, JTgJg being diagonal is equivalent to Jg having orthogonal columns. While this restriction is feasible for a single layer g, it is not composable. If g1 and g2 are both smooth embeddings whose Jacobians have orthogonal columns, it need not follow that Jg2 g1 has orthogonal columns. Additionally, since the Jacobians are not square the determinant in Eq. (2), det ⇥ JTg1J T g2Jg2Jg1 ⇤ , cannot be factored into a product of individually computable terms as in Eq. (3). To ensure composability we propose enforcing the slightly stricter criterion that each JTgJg be a scalar multiple of the identity. This is precisely the condition that g is a conformal embedding.
Formally, g : U ! X is a conformal embedding if it is a smooth embedding whose Jacobian satisfies JTg (u)Jg(u) =
2(u)Im , (5) where : U ! R is a smooth non-zero scalar function, the conformal factor [43]. In other words, Jg has orthonormal columns up to a smoothly varying non-zero multiplicative constant. Hence g locally preserves angles.
From Eq. (5) it is clear that conformal embeddings naturally satisfy our requirements as an injective flow. In particular, let g : U ! X be a conformal embedding and h : Z ! U be a standard normalizing flow model. The injective flow model g h : Z ! X satisfies
px(x) = pz (z) |detJh (z)| 1 m(u) . (6)
We call g h a Conformal Embedding Flow.
CEFs provide a new way to coordinate the training dynamics of the model’s manifold and density. It is important to note that not all parameterizations g of the learned manifold are equally suited to density estimation [9]. Prior injective flow models [6, 39] have been trained sequentially by first optimizing g using the reconstruction loss Ex⇠p⇤xkx g(g
†(x))k2, then training h for maximum likelihood with g fixed. This runs the risk of initializing the density pu (u) in a configuration that is challenging for h to learn. Brehmer and Cranmer [6] also alternate training g and h, but this does not prevent g from converging to a poor configuration for density estimation. Unlike previous injective flows, CEFs have tractable densities, which allows g and h to be trained jointly by optimizing the loss function
L = Ex⇠p⇤x ⇥ log px(x) + ↵kx g(g †(x))k2 ⇤ . (7)
This mixed loss provides more flexibility in how g is learned, and is unique to our model because it is the first for which log px(x) is tractable.

heading: 3.1 Designing Conformal Embedding Flows
text: Having established the model’s high-level structure and training objective, it remains for us to design conformal embeddings g which are capable of representing complex data manifolds. For g to be useful in a CEF we must be able to sample with g(u), perform inference with g†(x), and compute the conformal factor (u). In general, there is no unifying way to parameterize the entire family of such conformal embeddings (see App. B for more discussion). As when designing standard bijective flows, we can only identify subfamilies of conformal embeddings which we parameterize and compose to construct an expressive flow. To this end, we work with conformal building blocks gi : Ui 1 ! Ui (where U0 = U and Uk = X ), which we compose to produce the full conformal embedding g:
g = gk · · · g1 . (8)
In turn, g is conformal because
JTgJg = JTg1 · · ·J T gk (Jgk · · ·Jg) = 2 1 · · · 2 kIm . (9)
Our goal in the remainder of this section is to design classes of conformal building blocks which can be parameterized and learned in a CEF.

heading: 3.1.1 Conformal Embeddings from Conformal Mappings
text: Consider the special case where the conformal embedding maps between Euclidean spaces U ✓ Rd and V ✓ Rd of the same dimension3. In this special case gi is called a conformal mapping. Liouville’s theorem [27] states that any conformal mapping can be expressed as a composition of translations, orthogonal transformations, scalings, and inversions, which are defined in Table 1 (see App. B.1 for details on conformal mappings). We created conformal embeddings primarily by composing these layers. Zero-padding [6] is another conformal embedding, with Jacobian Jg = (Im 0)
T , and can be interspersed with conformal mappings to provide changes in dimensionality.
Stacking translation, orthogonal transformation, scaling, and inversion layers is sufficient to learn any conformal mapping in principle. However, the inversion operation is numerically unstable, so we replaced it with the special conformal transformation (SCT), a transformation of interest in conformal field theory [15]. It can be understood as an inversion, followed by a translation by b, followed by another inversion. In contrast to inversions, SCTs have a continuous parameter and include the identity when this parameter is set to 0.
The main challenge to implementing conformal mappings was writing trainable orthogonal layers. We parameterized orthogonal transformations in two different ways: by using Householder matrices [67], which are cheaply parameterizable and easy to train, and by using GeoTorch, the API provided by [44], which parameterizes the special orthogonal group by taking the matrix exponential of skew-symmetric matrices. GeoTorch also provides trainable non-square matrices with orthonormal columns, which are conformal embeddings (not conformal mappings) and which we incorporate to change the data’s dimensionality.
To scale orthogonal transformations to image data, we propose a new invertible layer: the orthogonal k⇥ k convolution. In the spirit of the invertible 1⇥ 1 convolutions of Kingma and Dhariwal [35], we
3We consider conformal mappings between spaces of dimension d > 2. Conformal mappings in d = 2 are much less constrained, while the case d = 1 is trivial since there is no notion of an angle.
note that a k⇥ k convolution with stride k has a block diagonal Jacobian. The Jacobian is orthogonal if and only if these blocks are orthogonal. It suffices then to convolve the input with a set of filters that together form an orthogonal matrix. Moreover, by modifying these matrices to be non-square with orthonormal columns (in practice, reducing the filter count), we can provide conformal changes in dimension. It is also worth noting that these layers can be inverted efficiently by applying a transposed convolution with the same filter, while a standard invertible 1⇥ 1 convolution requires a matrix inversion. This facilitates quick forward and backward passes when optimizing the model’s reconstruction loss.

heading: 3.1.2 Piecewise Conformal Embeddings
text: To make the embeddings more expressive, the conformality condition on g can be relaxed to the point of being conformal almost everywhere. Formally, the latent spaces Z and U are redefined as Z = {z : g is conformal at h(z)} and U = h(Z). Then g remains a conformal embedding on U , and as long as {x : g is nonconformal at g†(x)} also has measure zero, this approach poses no practical problems. Note that the same relaxation is performed implicitly with the diffeomorphism property of standard flows when rectifier nonlinearities are used in coupling layers [17] and can be justified by generalizing the change of variables formula [38].
We considered the two piecewise conformal embeddings defined in Table 2. Due to the success of ReLU in standard deep neural networks [50], we try a ReLU-like layer that is piecewise conformal. conformal ReLU is based on the injective ReLU proposed by Kothari et al. [39]. We believe it to be of general interest as a dimension-changing conformal nonlinearity, but it provided no performance improvements in experiments.
More useful was the conditional orthogonal transformation, which takes advantage of the normpreservation of orthogonal transformations to create an invertible layer. Despite being discontinuous, it provided a substantial boost in reconstruction ability on image data. The idea behind this transformation can be extended to the other parameterized mappings in Table 1. For each type of mapping we can identify hypersurfaces in Rn such that each hypersurface is mapped back to itself; i.e., each hypersurface is an orbit of its points under the mapping. Applying the same type of conformal mapping piecewise on either side remains an invertible operation as long as trajectories do not cross the hypersurface, and the result is conformal almost everywhere. The conditional orthogonal layer was the only example of these that provided performance improvements.

heading: 4 Related Work
text: Flows on prescribed manifolds. Flows can be developed for Riemannian manifolds M ✓ X which are known in advance and can be defined as the image of some fixed : U ! X , where U ✓ Rm [22, 48, 54]. In particular, Rezende et al. [60] model densities on spheres and tori with convex combinations of Möbius transformations, which are cognate to conformal mappings. For known manifolds is fixed, and the density’s Jacobian determinant factor may be computable in closed form. Our work replaces with a trainable network g, but the log-determinant still has a simple closed form.
Flows on learnable manifolds. Extending flows to learnable manifolds brings about two main challenges: handling off-manifold points, and training the density on the manifold.
When the distribution is manifold-supported, it will assign zero density to off-manifold points. This has been addressed by adding an off-manifold noise term [12, 13] or by projecting the data onto the manifold and training it with a reconstruction loss [6, 39, 41]. We opt for the latter approach.
Training the density on the manifold is challenging because the log-determinant term is typically intractable. Kumar et al. [41] use a series of lower bounds to train the log-determinant, while Brehmer and Cranmer [6] and Kothari et al. [39] separate the flow into two components and train only the lowdimensional component. Caterini et al. [9] maximize log-likelihood directly by either constructing the embedding’s Jacobian explicitly or using a stochastic approximation of the log-determinant’s gradient, but both approaches remain computationally expensive for high-dimensional data. Our approach is the first injective model to provide a learnable manifold with exact and efficient log-determinant computation.
Conformal networks. Numerous past works have imposed approximate conformality or its special cases as a regularizer [4, 31, 56, 57, 70], but it has been less common to enforce conformality strictly. To maintain orthogonal weights, one must optimize along the Stiefel manifold of orthogonal matrices. Past work to achieve this has either trained with Riemannian gradient descent or directly parameterized subsets of orthogonal matrices. Riemannian gradient descent algorithms typically require a singular value or QR decomposition at each training step [26, 30, 53]. We found that orthogonal matrices trained more quickly when directly parameterized. In particular, Lezcano-Casado and Martínez-Rubio [45] and Lezcano-Casado [44] parameterize orthogonal matrices as the matrix exponential of a skew-symmetric matrix, and Tomczak and Welling [67] use Householder matrices. We used a mix of both.

heading: 5 Experiments
text: To implement CEFs, we worked off of the nflows github repo [20], which is derived from the code of Durkan et al. [19]. Our code is available at https://github.com/layer6ai-labs/CEF. Full model and training details are provided in App. C, while additional reconstructions and generated images are presented in App. D.

heading: 5.1 Spherical Data
text: To demonstrate how a CEF can jointly learn a manifold and density, we generated a synthetic dataset from a known distribution with support on a spherical surface embedded in R3 as described in App. C. The distribution is visualized in Fig. 2, along with the training dataset of 103 sampled points.
We trained the two components of the CEF jointly, using the mixed loss function in Eq. (7) with an end-to-end log-likelihood term. The resulting model density is plotted in Fig. 2 along with generated samples. It shows good fidelity to the true manifold and density.

heading: 5.2 Image Data
text: We now evaluate manifold flows on image data. Our aim is to show that, although they represent a strict architectural subset of mainstream injective flows, CEFs remain competitive in generative performance [6, 39]. In doing so, this work is the first to include end-to-end maximum likelihood
training with an injective flow on image data. Three approaches were evaluated on each dataset: a jointly trained CEF, a sequentially trained CEF, and for a baseline a sequentially trained injective flow, as in Brehmer and Cranmer [6], labelled manifold flow (MF).
Injective models cannot be compared on the basis of log-likelihood, since each model may have a different manifold support. Instead, we evaluate generative performance in terms of fidelity and diversity [61]. The FID score [28] is a single metric which combines these factors, whereas density and coverage [49] measure them separately. For FID, lower is better, while for density and coverage, higher is better. We use the PyTorch-FID package [62] and the implementation of density and coverage from Naeem et al. [49].
Synthetic image manifolds. Before graduating to natural high-dimensional data, we test CEFs on a high-dimensional synthetic data manifold whose properties are better understood. We generate data using a GAN pretrained on CIFAR-10 [40] by sampling from a selected number of latent space dimensions (64 and 512) with others held fixed. Specifically, we sample a single class from the class-conditional StyleGAN2-ADA provided by Karras et al. [33]. This setup reflects our model design in that (1) the true latent dimension is known and (2) since a single class is used, the resulting manifold is more likely to be connected. On the other hand, the GAN may not be completely injective, so its support may not technically be a manifold. Results are shown in Table 3, Figs. 3 and 4, and App. D.
All models achieve comparable reconstruction losses with very minor visible artifacts, showing that conformal embeddings can learn complex manifolds with similar efficacy to state-of-the-art flows, despite their restricted architecture. The manifold flow achieves better generative performance based on our metrics and visual clarity. Between the CEFs, joint training allows the learned manifold to adapt better, but this did not translate directly to better generative performance.
Natural image data. We scale CEFs to natural data by training on the MNIST [42] and CelebA [46] datasets, for which a low-dimensional manifold structure is postulated but unknown. Results are given in Table 4, Figs. 5 and 6, and App. D.
As expected, since the MF’s embedding is more flexible, it achieves smaller reconstruction losses than the CEFs. On MNIST, this is visible as faint blurriness in the CEF reconstructions in Fig. 5, and it translates to better sample quality for the MF as per the metrics in Table 4. Interestingly however, the jointly-trained CEF obtains substantially better sample quality on CelebA, both visually (Fig. 6) and by every metric. We posit this is due to the phenomenon observed by Caterini et al. [9] in concurrent work: for complex distributions, the learned manifold parameterization has significant influence on the difficulty of the density estimation task. Only the joint training approach, which maximizes likelihoods end-to-end, can train the manifold parameterization to an optimal starting point for density estimation, while sequential training optimizes the manifold solely on the basis of reconstruction loss. CelebA is the highest-dimensional dataset tested here, and its distribution is presumably quite complex, so one can reasonably expect joint training to provide better results. On the other hand, the sequentially trained CEF’s performance suffers from the lack of both joint training and the expressivity afforded by the more general MF architecture.

heading: 6 Limitations and Future Directions
text: Expressivity. Just as standard flows trade expressivity for tractable likelihoods, so must injective flows. Our conformal embeddings in particular are less expressive than state-of-the-art flow models; they had higher reconstruction loss than the neural spline flow-based embeddings we tested. The conformal embeddings we designed were limited in that they mostly derive from dimension-preserving conformal mappings, which is a naturally restrictive class by Liouville’s theorem [27]. Just as early work on NFs [16, 59] introduced limited classes of parameterizable bijections, which were later improved substantially (e.g. [19, 35]), our work introduces several classes of parameterizable conformal embeddings. We expect that future work will uncover more expressive conformal embeddings.
Manifold learning. Strictly manifold-supported probability models such as ours introduce a biobjective optimization problem. How to balance these objectives is unclear and, thus far, empirical [6]. The difference in supports between two manifold models also makes their likelihoods incomparable. Cunningham et al. [13] have made progress in this direction by convolving the manifold-supported distribution with noise, but this makes inference stochastic and introduces density estimation challenges. We suspect that using conformal manifold-learners may make density estimation more tractable in this setting, but further research is needed in this direction.
Broader impact. As deep generative models become more advanced, researchers should carefully consider some accompanying ethical concerns. Large-scale, natural image datasets carry social biases which are likely to be codified in turn by the models trained on them [65]. For instance, CelebA does not accurately represent the real-world distribution of human traits, and models trained on CelebA should be vetted for fairness before being deployed to make decisions that can adversely affect people. Deep generative modelling also lends itself to malicious practices [8] such as disinformation and impersonation using deepfakes [69].
Our work seeks to endow normalizing flows with more realistic assumptions about the data they model. While such improvements may invite malicious downstream applications, they also encode a better understanding of the data, which makes the model more interpretable and thus more transparent. We hope that a better understanding of deep generative models will synergize with current lines of research aimed at applying them for fair and explainable real-world use [3, 51].

heading: 7 Conclusion
text: This paper introduced Conformal Embedding Flows for modelling probability distributions on lowdimensional manifolds while maintaining tractable densities. We showed that conformal embeddings naturally match the framework of normalizing flows by providing efficient sampling, inference, and density estimation, and they are composable so that they can be scaled to depth. Furthermore, it appears conformality is a minimal restriction in that any looser condition will sacrifice one or more of these properties. As we have reviewed, previous instantiations of injective flows have not maintained all of these properties simultaneously.
Normalizing flows are still outperformed by other generative models such as GANs and VAEs in the arena of realistic image generation. Notably, these two alternatives benefit from a low-dimensional latent space, which better reflects image data’s manifold structure and provides for more scalable model design. By equipping flows with a low-dimensional latent space, injective flow research has made progress towards VAE- or GAN-level performance. The CEF paradigm is a way to match these strides while maintaining the theoretical strengths of NFs.
Acknowledgments and Disclosure of Funding
We thank Gabriel Loaiza-Ganem and Anthony Caterini for their valuable discussions and advice. We also thank Parsa Torabian for sharing his experience with orthogonal weights and Maksims Volkovs for his helpful feedback.
The authors declare no competing interests or third-party funding sources.

Review summaries:
Many datasets of interest that are modelled with normalizing flows are assumed to lie on a lower-dimensional manifold embedded in a higher-dimensional ambient space. Standard normalizing flows that are restricted to learn a *bijective* transformation are not well suited for modelling such data, as the target transformation between the lower-dimensional manifold space and the ambient space (in which said manifold is embedded) is *injective*.  In theory, an *injective flow* can be defined, where we'd parametrize an injective transformation that would only be invertible *for points on the manifold*. In practice there are two problems with this, however. First is that historically there's been much less work on designing flexible *injective* transformations, hence we don't have the same arsenal of invertible layers as we do for bijective flows. Second, the change-of-variable formula for an injective transformation includes a $\det J_f^T J_f$ term, where $J_f$ is the (rectangular) Jacobian matrix of the injective transformation $f$. This determinant, even if tractable, is typically overly expensive to compute exactly.  Authors of this work make an observation that in the bijective flow literature the game has always been to design parametric transformations $f$ that are restricted to be invertible, but *also* ones that have and easy-to-compute $\det J_f$. In line with this, authors propose a class of transformations that would have an easy-to-compute $\det J_f^T J_f$, specifically *conformal embeddings*. A conformal embedding $f$ is a transformation that preserves *angles*, but more importantly has a property that $J_f^T J_f$ is a scalar multiple of the identity, the determinant of which is simply the product of diagonal terms (i.e. an exponent of said scalar).   Authors consider a few examples of conformal embeddings (translation, scaling, an orthogonal transformation, zero-padding, etc.), which could be stacked in the same way as invertible layers are in a bijective flow. In addition, authors consider *piecewise* conformal embeddings, which are conformal *almost everywhere* (i.e. for all points except for a subset of measure zero), and can increase the expressivity of the otherwise simple conformal embeddings.  By using the conformal embeddings as injective transformations authors come up with the first injective flow with a tractable and *cheap-to-compute* density on the learned manifold. This allows said density to be used as a part of the training objective, even though an additional regularization term still needs to be included in the objective to prevent degenerate solutions. The method demonstrates excellent results on a synthetic 3D problem where the target density lies on a sphere, as well as the CelebA image dataset, where the visual fidelity of the samples is improved in comparison to the more expressive baseline flow.

This work proposes to leverage conformal embedding flows for modeling probability distributions on low-dimensional manifolds. Differing from other manifold-based flows, the proposed methods maintain tractable densities for low-dimensional latent space. The unique contribution enables fast sampling, invertibility for inference, and efficient likelihood estimation, which inherits from normalizing flows.   Existing approaches share the drawback that exact densities cannot be calculated through the dimensionality reducing step. In the paper, the authors present an approach that offers tractable density estimation and is straightforward to train by constraining this reduction to the class of conformal embeddings.