Title: Canonical normalizing flows for manifold learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: manifold learning flows

normalizing flows

optimization

orthogonalization

sparsity

sparse learning

generative modeling

Riemannian manifold

geometry

metric tensor

orthogonal basis
Abstract: Manifold learning flows are a class of generative modelling techniques that assume a low-dimensional manifold description of the data. The embedding of such a manifold into the high-dimensional space of the data is achieved via learnable invertible transformations. Therefore, once the manifold is properly aligned via a reconstruction loss, the probability density is tractable on the manifold and maximum likelihood can be used to optimize the network parameters. Naturally, the lower-dimensional representation of the data requires an injective-mapping. Recent approaches were able to enforce that the density aligns with the modelled manifold, while efficiently calculating the density volume-change term when embedding to the higher-dimensional space. However, unless the injective-mapping is analytically predefined, the learned manifold is not necessarily an \emph{efficient representation} of the data. Namely, the latent dimensions of such models frequently learn an entangled intrinsic basis, with degenerate information being stored in each dimension. Alternatively, if a locally orthogonal and/or sparse basis is to be learned, here coined canonical intrinsic basis, it can serve in learning a more compact latent space representation. Toward this end, we propose a canonical manifold learning flow method, where a novel optimization objective enforces the transformation matrix to have few prominent and non-degenerate basis functions. We demonstrate that by minimizing the off-diagonal manifold metric elements $\ell_1$-norm, we can achieve such a basis, which is simultaneously sparse and/or orthogonal. Canonical manifold flow yields a more efficient use of the latent space, automatically generating fewer prominent and distinct dimensions to represent data, and consequently a better approximation of target distributions than other manifold flow methods in most experiments we conducted, resulting in lower FID scores.

Canonical normalizing flows for manifold learning

Abstract

Manifold learning flows are a class of generative modelling techniques that assume a low-dimensional manifold description of the data. The embedding of such a manifold into the high-dimensional space of the data is achieved via learnable invertible transformations. Therefore, once the manifold is properly aligned via a reconstruction loss, the probability density is tractable on the manifold and maximum likelihood can be used to optimize the network parameters. Naturally, the lower-dimensional representation of the data requires an injective-mapping. Recent approaches were able to enforce that the density aligns with the modelled manifold, while efficiently calculating the density volume-change term when embedding to the higher-dimensional space. However, unless the injective-mapping is analytically predefined, the learned manifold is not necessarily an efficient representation of the data. Namely, the latent dimensions of such models frequently learn an entangled intrinsic basis, with degenerate information being stored in each dimension. Alternatively, if a locally orthogonal and/or sparse basis is to be learned, here coined canonical intrinsic basis, it can serve in learning a more compact latent space representation. Toward this end, we propose a canonical manifold learning flow method, where a novel optimization objective enforces the transformation matrix to have few prominent and non-degenerate basis functions. We demonstrate that by minimizing the off-diagonal manifold metric elements ℓ 1 -norm, we can achieve such a basis, which is simultaneously sparse and/or orthogonal. Canonical manifold flow yields a more efficient use of the latent space, automatically generating fewer prominent and distinct dimensions to represent data, and consequently a better approximation of target distributions than other manifold flow methods in most experiments we conducted, resulting in lower FID scores. 1

1 Introduction

Many emerging methods in generative modeling are based on the manifold hypothesis, i.e., higherdimensional data are better described by a lower-dimensional sub-manifold embedded in an ambient space. These methods replace the bijective normalizing flow in the original construction [1-5] (NF) with an injective flow. NFs are constructed as a smooth bijective mapping, i.e. a homeomorphism, where the learned distribution density is supported on the set of dimension D equal to the data dimension, where R D is the data space. In order to fulfill the manifold hypothesis, an emerging class of methods, namely manifold learning flow models (MLF) [6-9], model the latent distribution as a random variable in R d where d < D , i.e. realize injective flows. Consequently, the density lives on some d-dimensional manifold M d embedded in R D as required. If M d is known a-priori, [10, 11], the density transformation to R D in MLF is a relatively straight forward affair. If the manifold is to be

learned, the change-of-variable formula, i.e. the volume element, can be computationally inhibiting to calculate. Caterini et al [7] have recently solved this problem and established tractable likelihood maximization.

MLF methods have thus evolved to the point where the general manifold hypothesis is properly incorporated and the likelihood is efficiently calculated. However, it has been observed that the learned manifold can be suboptimal; for example, due to a nonuniform magnification factor [12] and the inability to learn complex topologies [13]. Furthermore, pushforward models such as MLFs, i.e., d < D , are designed to avoid over-fitting, but d in R d is still arbitrary unless determined by some meta-analysis method. As well as, there is often degeneracy in the information stored in each of these latent dimensions d . For instance, as we show in Section 4.1, consider modeling a two-dimension line with considerable random noise, i.e. a 'fuzzy line'. If d = 2 , both latent dimensions will attempt to encompass the entire manifold in a degenerative use of latent dimensions. Particularly, this behavior of storing degenerate representations is inefficient and can lead to over-fitting, topological, and general learning pathologies. Alternatively, we postulate that motivating sparsity [14] and local orthogonality during learning can enhance the manifold learning process of MLFs.

To this end, we propose an optimization objective that either minimizes or finds orthogonal gradient attributions in the learned transformation of the MLF. Consequently, a more compact [15] and efficient latent space representation is obtained without degenerate information stored in different latent dimensions, in the same spirit as automatic relevance determination methods [16] or network regularization [17]. The relevant dimensions correspond to locally distinct tangent vectors on the data manifold, which can yield a natural ordering among them in the generations. Borrowing the term 'canonical basis', i.e., the set of linearly independent generalized eigenvectors of a matrix, we introduce Canonical manifold learning flows (CMF). In CMF, the normalizing flow is encouraged to learn a sparse and canonical intrinsic basis for the manifold. By utilizing the Jacobians calculated for likelihood maximization, the objective is achieved without additional computational cost.

2 Theoretical background

2.1 Normalizing and rectangular normalizing flows

First, we define a data space as X with samples { x 1 , ..., x n } , where x i ∈ R D , and a latent space Z ⊂ R d . If, d = D , then a normalizing flow is a ϕ parameterized diffeomorphism q ϕ : R D → R D , i.e., a differentiable bijection with a differentiable inverse, that transforms samples from the latent space to the data space, X := q ϕ ( Z ) . Optimization of ϕ is achieved via likelihood maximization, where due to the invertibility of q ϕ the likelihood, p X ( x ) , can be calculated exactly from the latent probability distribution p Z ( z ) using the transformation of distributions formula [18],

J denotes the Jacobian functional such that J q ϕ = ∂x/∂z , i.e., the Jacobian of q ϕ at x = q ϕ ( z ) . Here, Z is modelled by a random variable z ∼ p Z for a simple density p Z , e.g., a normal distribution. Therefore, for large n , a normalizing flow approximates well the distribution that gave rise to the samples in a dataset { x i } n i =1 by transforming the samples z ∼ p Z with a trained q ϕ ∗ if ϕ ∗ := argmax ϕ ∑ n i =1 log p X ( x i ) [4].

̸

In the context of manifold learning flows, or rectangular normalizing flows, d < D implies there is a lower dimensional manifold M⊂ R d that is embedded into the R D with a smooth and injective transformation g ϕ : R d → R D . In practice, g ϕ = h η ◦ pad ◦ f θ , where h η : R d → R d , f θ : R D → R D and pad : R d → R D . Padding can be used for trivial embedding, i.e., pad(˜ z ) = (˜ z 0 . . . ˜ z d -1 , 0 . . . 0) . The rectangular Jacobian of J g ϕ is not invertible and thus, J T g ϕ J g ϕ = J 2 g ϕ , which implies that the volume element needs to be calculated fully, i.e., the square root of the determinant of J T g ϕ J g ϕ which is an equivalent expression for the general metric tensor in this context - see the appendix. This leads to a more generic formulation

However, note that maximizing p X ( x ) on its own would only maximize the likelihood of the projection of x on the latent space, not ensuring that the model can reconstruct x from z , i.e., the

modeled manifold may not be aligned with the observed data. To encourage x ∈ M , namely to align the learned manifold to the data space, a reconstruction error needs to be minimized during optimization. The following loss is thus added to the maximum-likelihood loss function, as explained in [6]:

Combining the logarithm of Equation (2) and using the hyperparameter β > 0 to adjust Equation (3) we arrive at the total Lagrangian to be maximized:

2.2 Riemannian geometry and the metric tensor

A d dimensional curved space is represented by a Riemannian manifold M⊂ R d , which is locally described by a smooth diffeomorphism h , called the chart. The set of tangential vectors attached to each point y on the manifold is called the tangent space T y M . All the vector quantities are represented as elements of T y M . The derivatives of the chart h are used to define the standard basis ( e 1 , ..., e d ) = ∂ h , ..., ∂ h .

The metric tensor g can be used to measure the length of a vector or the angle between two vectors. In local coordinates, the components of the metric tensor are given by

where · is the standard Euclidean scalar product.

The metric tensor G ij describing the manifold M ⊂ R D learned in manifold learning flows is equivalent to the section of the rectangular Jacobian-transpose-Jacobian J T J that describes M . As explained in [7] and the appendix, the J T J can be very efficiently approximated in the context of MLFs. Thus, it can be seen that no additional computational cost is need to calculate the metric tensor.

3 Related Work and motivation

Generative models with lower-dimensional latent space have long being established as favorable methods due to their efficiency and expressiveness. For instance, variational autoencoders [19] have been shown to learn data manifolds M with complex topologies [20, 21]. However, as seen in [22], variational approaches exhibit limitations in learning the distribution p X on M . Manifold learning flows [6] (Mflow) form a bridge between tractable density and expressiveness. Their support ∈ R d can inherently match the complex topology of R D , while maintaining a well-defined change of variables transformation. Furthermore, state-of-the-art methods can efficiently calculate the determinant of the Jacobian dot product, as in rectangular normalizing flow [7] (RNF). Denoising normalizing flow [9] (DNF) represents a progression beyond Mflow, where density denoising is used to improve on the density estimation. RNFs can be seen as a parallel approach to DNF but with a more direct methodology, circumventing potential ambiguities stemming from heuristic techniques like density denoising.

Despite the success of manifold learning flows in obtaining a tractable density whilst maintaining expressivity, the learned manifold is not optimized within its latent space representation [23], an outstanding problem for most generative models. For example, due to lack of constrains between latent variables it can be the case that some of them contain duplicate information, i.e., form a degenerate intrinsic basis to describe the manifold, also demonstrated in Section 4.1. Sparse learning [14] was an early attempt to limit this degeneracy and encourage learning of global parameters, introducing solutions such as the relevance vector machine [24, 25] (RVM).The RVM attempts to

limit learning to its minimum necessary configuration, while capturing the most relevant components that can describe major features in the data.

Even stricter approaches are principal [26, 27] and independent component analyses [28, 29], PCA and ICA, respectively. PCA and ICA can obtain a well-defined and optimized representation when convergent; however, PCA and ICA are linear in nature and their non-linear counterparts require kernel definitions. Here, the network-based approach is more general, reducing the need for handcrafting nonlinear mappings. Furthermore, pure PCA and ICA can be limited in modelling some data manifold topologies where strict global independent components are not necessarily a favorable basis [30]. Furthermore, [15] presents a method for efficient post-learning structuring of the latent space, showcasing the advantages of model compactness. We propose a canonical manifold learning flow where the model is motivated to learn a compact and non-degenerate intrinsic manifold basis.

Furthermore, it has been shown that tabular neural networks can show superior performance when regularized via gradient orthogonalization and specialization [17]. They focus on network regularization, which is not specific to one method but requires certain network structure and costly attribution calculations.

Recently, NF methods with flow manifested in the principal components have been proposed [31, 32]. Cranmer et al. [31] focus on datasets that have distinct PCAs that can be efficiency calculated and, therefore, utilize the computed PCAs for efficient density estimation. Cunningham et al. [32] (PCAflow) relies on a lower-bound estimation of the probability density to bypass the J T J calculation. This bound is tight when complete principal component flow, as defined by them, is achieved. The fundamental distinction here is that our method does not confine itself to a pure PCAflow scenario, which has the potential to restrict expressivity. In contrast, our method only loosely enforces orthogonality, while encouraging sparsity, i.e. promoting learning of a canonical manifold basis.

4 Method

4.1 Canonical intrinsic basis

In this section, in order to motivate the proposed development, we first demonstrate how rectangular manifold learning flows assign a latent space representation in comparison to canonical manifold learning flow. We generated synthetic data on a tilted line with noise in the perpendicular and parallel directions, i.e., a fuzzy line. We sample 1000 x i 's such that x 1 ∼ U ( -2 . 5 , 2 . 5) and x 2 = x 1 + ϵ , where ϵ ∼ U ( -0 . 5 , 0 . 5) . In Figure 1(a), a two-dimensional manifold M is learned with the vanillaRNF method using the samples by minimizing the loss given in Equation 4. As expected and seen on the left of Figure 1(a), the probability density is learned satisfactorily, i.e., the model fit well the samples and can generate samples with accurate density. However, when samples on the manifold are generated from the two latent dimensions, z i , individually, shown in Figure 1 (ii) and (iii), the sampling results are very similar. The implication being that the intrinsic basis is almost degenerate and information is duplicated amongst the latent variables. In contrast, when the same M is learned with the proposed CMF, shown in Figure 2, the sampled latent dimensions are close to perpendicular to each other. Each basis is capturing non-degenerate information about M , see Figure 1(b), i.e., the two major axes of the dataset.

4.2 Canonical manifold learning

As canonical manifold is not a standard term in literature, let us start by defining a canonical manifold for manifold learning flows.

̸

Definition 4.1. Here, a canonical manifold, M , is a manifold that has an orthogonal and/or sparse basis e i such that e i · e j = 0 ∀ y ∈ M and whenever i = j .

The name in Definition 4.1 is inspired from the 'canonical basis', i.e., the set of linearly independent generalized eigenvectors of a matrix, as defined in linear algebra. We hypothesize that enforcing learning a canonical manifold as defined in Definition 4.1 during manifold flows will lead to a less degenerate use of the latent space, and consequently a better approximation of the data distribution. As in sparse learning [14], during training necessary dimensions in the latent space will be automatically determined, and those that are necessary will be used in such a way to model a manifold with an

(i)

(ii)

(iii)

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

log

p

(

x

)

(i)

(ii)

(iii)

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

log

p

(

x

)

Figure 1: Comparison of density plots for a fuzzy line learned with RNF ( a ) and CMF ( b ). Sampled 1000 x i 's such that x 1 ∼ U ( -2 . 5 , 2 . 5) and x 2 = x 1 + ϵ with ϵ ∼ U ( -0 . 5 , 0 . 5) . The black dots represent samples from the real data manifold. (i) Samples from the model using the full latent space where colors correspond to log p ( x ) . (ii), (iii) Samples from the model using only the first and the second components z 1 / 2 ∈ Z while setting the other to zero, respectively. Colors correspond to log p ( x ) once more. All densities have been normalized to [ -1 , 1] for visualization purposes. Note that compared to RNF, with CMF different latent variables could capture almost an orthogonal basis for this manifold.

orthogonal local basis to fit the data samples. This, we believe, will enhance the manifold learning process, avoiding overfitting and other learning pathologies associated with correlated attributions to dimensions of the latent space.

In order to obtain a canonical basis we first realize that, as described in Section 2 and the appendix, the metric tensor can be used as concise language to describe the manifold i.e. the transformation of the chart.

This provides an opportunity to regularize the learned representation directly with a single optimization objective. The dependencies on G are dropped for brevity.

Leveraging the metric tensor representation, we propose to minimize the off-diagonal elements of the metric tensor, enforcing learning of a canonical manifold

̸

̸

̸

where ∥·∥ 1 1 is the ℓ 1 norm. Using ∥ G i = j ∥ 1 1 as a cost to minimize serves both the sparsity and generating orthogonal basis. While the ℓ 2 norm could be used to enforce a specific embedding [8], it is not in the scope of this work. In order to minimize this cost, the network will have to minimize the dot products ∂x/∂z i · ∂x/∂z j . There are two ways to accomplish this. The first is to reduce the magnitudes of the basis vectors ∂x/∂z i . Minimizing the magnitudes with the ℓ 1 norm would lead to sparser basis vectors. The second is to make the basis vectors as orthogonal as possible so the dot product is minimized, which serves towards learning a canonical manifold as well. As detailed in [7] and the appendix, the Jacobian product, the bottleneck for both RNF and our method, can be efficiently approximated. With complexity O ( id 2 ) , it is less than O ( d 3 ) when i << d , where i is the conjugate gradients method iterations. Thus, encouraging a canonical manifold through G i = j incurs minimal extra computational cost.

̸

One can imagine an alternative optimization schemes. For instance, the diagonal elements of the metric tensor G kk can be used to access the transformation of individual z i ∈ Z . Minimizing the ℓ 1 norm ∑ k ∥ G kk ∥ 1 1 will encourage sparse learning, akin to a relevance vector machine [14] or network specialization [17]. Nevertheless, minimizing the diagonal elements will clearly not motivate perpendicular components and notably there is no mixing constraint on these components. Consequently, only the magnitude of the transformation can be affected.

Furthermore, one can imagine minimizing the cosine similarity between the basis vectors to promote orthogonality. However, this would not serve towards automatically determining the necessary dimensions in the latent space. Combining the ℓ 1 of the diagonal elements and the cosine similarity may also be a solution; however, this would bring an additional weighing factor between the two

losses. ℓ 1 loss of the off-diagonal elements of the metric tensor brings together these two objectives elegantly.

Note that the injective nature of the flow, the arbitrary topology of the image space and the finite dimensionality of the chart imply that a canonical solution may not exist for the complete learned manifold. However, this is not prohibitive as the local nature of the proposed method should allow even for isolated, sparse and independent component realizations. Even such solutions can be more preferable when representing complex multidimensional image dataset manifolds. Additionally, the canonical solution is not absolutely but rather statistically enforced, i.e. it is not a strict constraint. This is similar to encouraging orthogonality and specialization of gradient attributions for network regularizations [17].

To this end, combining Equation (7) with the established manifold learning flow log likelihood and reconstruction loss from Section 2.1, i.e., Equation (4), we arrive at the following total optimization objective of the canonical manifold learning flow loss:

̸

where γ is a hyperparameter. In summary, for the transformation parameters ϕ , the log-likelihood of NFs is maximized, while taking into consideration the rectangular nature of the transformation. Additionally, a reconstruction loss with a regularization parameter β is added to ensure that the learned manifold, M⊂ R d , is aligned to the data manifold, M ⊂ R D . The ℓ 1 loss of the off-diagonal metric tensor ensures that a canonical manifold M is learned.

5 Experiments

We compare our method with the rectangular normalizing flow (RNF) and the original Brehmer and Cranmer manifold learning flow (MFlow). No meaningful comparison can be made to linear PCA/ICA as they are not suitable for data on non-linear manifolds. While non-linear PCA/ICA can be used, they require a specific feature extractor or kernel. Our method, a type of manifold learning, directly learns this non-linear transformation from data. We use the same architectures as [7], namely real NVP [7] without batch normalization, for the same reasons explained in the nominal paper 2 . Further experimental details can be found in the appendix. In short, we use the same hyperparameters for all models to ensure fairness. For example, we use a learning rate of 1 × 10 -4 across the board, and we have only carried out simple initial searches for the novel parameter γ from Equation (8). Namely, from trying γ = [0 . 001 , 0 . 01 , 0 . 1 , 1 , 10 , 50] , we concluded in using γ = β = 1 for the low-dimensional datasets, β = 5 and γ = 0 . 1 for the tabular datasets and β = 5 and γ = 0 . 01 for the image datasets to ensure stable training. γ = 1 also produced good results in our experiments with different datasets, but it was not always stable and long training was required. Overall, the method is not overly sensitive to the hyperparameter γ . Likelihood annealing was implemented for all image training runs. Approximate training times can be found in the appendix.

5.1 Simulated data

We consider some simulated datasets. Specifically, we implement uniform distributions over a two-dimensional sphere and a Möbius band embedded in R 3 . Then we generate 1000 samples from each of these distributions and use the samples as the simulated data to fit both RNF and CMF. Such datasets have a straightforward and visualizable canonical manifold representation. The samples generated by the trained networks are shown in Figure 2(a) and Figure 2(b) for the RNF and CMF methods, respectively. From the fully sampled density plots Figure 2(a)(i) and Figure 2(b)(i), both methods learn the manifold and the distribution on it. However, the manifold learning of CMF is superior to RNF as the sphere is fully encapsulated, and the density is more uniform. Furthermore, when the three z i dimensions are sampled individually in Figure 2(b) (ii),(iii) and (iv), the canonical

intrinsic basis learned by CMF can be seen. Specifically, two distinct perpendicular components are obtained while the third component approaches zero, i.e. reduced to something small as it is not necessary. In contrast, from Figure 2(a) (ii),(iii) and (iv) it can be seen that in RNF all three components attempt to wrap around the sphere without apparent perpendicularity, and the second component shrinks partially. Even more striking results can be seen for the Möbius band, Figure 2(c) and Figure 2(d). Additionally, the non-trivial topology of the Möbius manifold, the twist and hole, is proving harder to learn for the RNF as compared to CMF. These findings are in direct alignment with the expected canonical manifold learning, where the network is motivated to learn an orthogonal basis and/or implement sparse learning, as explained in Section 4.2. For CMF and RNF methods, the log likelihoods are 1.6553 vs. 1.6517 and KS p-values are 0.17 vs. 0.26. On the sphere, the log likelihoods are 1.97 vs. 1.16. CMF demonstrates better quality than RNF, both quantitatively and qualitatively.

(i)

(ii)

(iii)

(iv)

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

log

p

(

x

)

(a)

Hallow sphere learned with RNF.

(i)

(ii)

(iii)

(iv)

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

log

p

(

x

)

(c)

Moebius band learned with RNF.

(i)

(ii)

(iii)

(iv)

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

log

p

(

x

)

(i)

(ii)

(iii)

(iv)

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

log

p

(

x

)

(b)

Hallow sphere learned with CMF.

(d)

Moebius band learned with CMF.

Notably, for these simulated datasets we employ full dimensional latent representations, D = d . First, it allows us to visualize what all the latent dimensions are doing in relation to what we expect them to do. In particular, CMF correctly uses only 2 latent dimensions for representing 2D surfaces even though they have 3 latent dimensions. Second, it clearly shows the advantage to RNFs for the case where dimensions of the embedded manifold are known. Such scenario is equivalent to a standard NF. RNF is based on [33], which is an NF that already tries to solve the learning pathologies of complicated manifolds. Therefore, there is no practical limitation in choosing D = d , as also seen empirically. Considering the above, the CMF method is showcased to outperform these previous methods, which also aligns with the theoretical intuition.

5.2 Image Data

First, Mflow, RNF and CMF are also compared using image datasets MNIST, Fashion-MNIST and Omniglot ∈ R 784 . As mentioned, the hyperparameters were set to β = 5 and γ = 0 . 01 for all CMF runs, β = 5 and γ = 0 for RNF and Mflow methods, and the network architectures are all equivalent. Furthermore, to emphasize the ability and importance of CMF, we train for different latent dimensions d . The test FID scores of the trained models are shown in Table 1. We use R d =20 initially as in [7] for consistency but also run for lower latent dimensions, R d =10 , in order to showcase the advantages of CMF relative to the established methods. Namely, a learned canonical intrinsic basis can lead to a more efficient latent space representation and therefore advantageous performance when the latent space is limited. From Table 1, CMF appears to outperform the other methods.

Then, we also experimented using the CIFAR-10 and SVHN datasets with R 3072 , and CelebA with R 12288 . Again we set β = 5 and γ = 0 . 01 , however, this time with latent dimensions, R d =30 , and R d =40 . The extra dimensions were necessary for the higher dimensional data. The table Table 1

presents the FID scores. CMF outperforms the other alternatives, consistent with the results obtained on lower dimensional imaging data sets.

In order to visualize the sparsity and orthogonalization encouraged by the CMF, the pair-wise absolute cosine similarity between the basis vectors, i.e., { ∂x/∂z i } d i =1 (bottom) and the diagonal of the metric tensor (top) are visualized in Figure 3. The CMF clearly induces more sparsity as compared to the RNF, seen from the sparse diagonal elements of the metric tensors, see Figure 3(top), for both the Fashion-MNIST and Omniglot examples. Similarly, the cosine similarity for the CMF is evidently lower for both data sets, see Figure 3(bottom); the MACS stands for the mean absolute cosine similarity. Similar results were observed for the tabular datasets, see appendix.

Additionally, in order to demonstrate the more efficient use of latent space by CMF, we trained both CMF and RNF using latent-dimension d = 40 on both the MNIST and the Fashion-MNIST datasets. Then we choose to vary the number of prominent latent dimensions from Z ∈ R 40 as those with the largest | G kk | . We first generate samples from the models using only the prominent dimensions by sampling from the prominent dimensions and setting all others to zero. We compute FID scores for the generated samples. Later, we also reconstruct real samples using only the prominent dimensions and compute reconstruction loss using mean squared error. Here, we take the dimensions, the larger G ii as the ones with the highest weights; this simple interpretation is used for analysis. We repeat the experiment for different number of prominent latent dimensions (1 , 8 , 16 . . . , 40) non-zero z i ∈ Z . In Figure 4(a) and Figure 4(b) the FID score and the mean-squared-error for the different number of prominent dimensions are plotted respectively. From Figure 4, it is evident that the CMF (solid lines) shows better results as compared with RNF (dotted lines) consistently, when using fewer prominent latent dimensions. This suggests that the CMF is able to use the latent space more efficiently. Furthermore, the RNF lines exhibit some oscillation whereas the CMF lines are smoother, indicating that there is also some additional inherent importance ordering in the latent dimensions as expected from sparse learning. Samples generated with different numbers of latent dimensions can be found in the appendix.

1.0

0.5

0.0

0.5

1.0

Gi

=

j

0.100

0.075

0.050

0.025

0.000

0.025

0.050

0.075

0.100

Cosine similarity

1.0

0.5

0.0

0.5

1.0

Gi

=

j

0.100

0.075

0.050

0.025

0.000

0.025

0.050

0.075

0.100

Cosine similarity

1.0

0.5

0.0

0.5

1.0

Gi

=

j

0.100

0.075

0.050

0.025

0.000

0.025

0.050

0.075

0.100

Cosine similarity

1.0

0.5

0.0

0.5

1.0

Gi

=

j

0.100

0.075

0.050

0.025

0.000

0.025

0.050

0.075

0.100

Cosine similarity

5.3 Tabular data

The standard normalizing flow benchmarks of tabular datasets from [3] are used to compare the RNF, MFlow and CMF methods. We fit all models with latent dimension d =2, 4, 10 and 21 to POWER, GAS, HEMPMASS and MINIBOONE datasets, respectively. The experiments here are similar to those in [7], however, we keep the latent dimension the same for all the methods. Further details can be found in the appendix. We then computed FID-like scores, as was done in compared works [7] to compare the three models. The results are shown in Table 2. CMF appears to have lower FID-like metric scores for POWER, HEMPMASS and MINIBOONE datasets. The GAS dataset

10

20

30

40

effective

d

0

50

100

150

200

FID

Fashion-MNIST with CMF

Fashion-MNIST with RNF

MNIST with CMF

MNIST with RNF

(a) Frechet Inception Distance.

10

20

30

40

effective

d

1000

2000

3000

4000

5000

6000

7000

8000

||

x

x

||

2

2

Fashion-MNIST with CMF

Fashion-MNIST with RNF

MNIST with CMF

MNIST with RNF

(b)

Mean-squared-error.

showed mixed results in the second run set, as seen in Table 2. The performance ordering has shifted, suggesting challenges in learning this dataset, possibly leading to random outcomes. However, the other tabular datasets showed consistent performance rankings across run sets. See the appendix for further investigation into the tabular dataset training pathologies. Training on tabular datasets with doubled latent dimensions yielded no significant differences, Table 2.

6 Limitations

Our method optimizes the learned manifold M⊂ R d of manifold learning flows by constraining the learning to a canonical manifold M as in Definition 4.1 at no significant additional computational cost as compared to RNFs [7]. A table of indicative training times can be found in the appendix. Nevertheless, the nature of likelihood-based training, where the full J T J needs to be calculated, dictates that such a class of methods is computationally expensive for higher dimensions, despite the efficient approximate methods implemented here and in RNFs. Therefore, higher dimensional datasets can be prohibitively computationally expensive to train with this class of methods. Further computational efficiency advancements are expected to be achieved in future works. A notable fact is that the original Mflow method [6] relies on the fundamental approximation such that the full J T J calculation can be avoided, resulting in inferior log-likelihood calculation and therefore generations, albeit it being noticeably more computationally efficient.

Additionally, CMF can only model a manifold that is a homeomorphism to R d , and when such mapping preserves the topology. Therefore, if, for example, there is a topological mismatch between

the components of the image space R D described directly by M⊂ R d , it could lead to pathological training. This drawback, however, is shared across all manifold learning flow methods. We also acknowledge the possibility that an orthogonal basis may not always be an optimal representation and difficult to learn: enforcing strict complete orthogonality can restrict the expressivity of the transformation.

Furthermore, other families of state-of-the-art generative modeling methods working in full dimensional representations, for example denoising diffusion models [34], can obtain superior FID scores overall as compared to the class of methods described in this work. However, by default they use a latent space as large as the image space. Notably, for any future use that entails isolating prominent components, it would be beneficial to perform a proper intrinsic dimensionality estimation.

7 Summary and Conclusions

In this work, we attempt to improve on the learning objective of manifold learning normalizing flows. Having observed that MLFs do not assign always an optimal latent space representation and motivated from compact latent space structuring [15] and sparse learning [14], while taking advantage of the Riemannian manifold nature of homeomorphic transformations of NFs, we propose an effective yet simple to implement enhancement to MLFs that does not introduce additional computational cost.

Specifically, our method attempts to learn a canonical manifold, as defined in Section 4.1, i.e., a compact and non-degenerate intrinsic manifold basis. Canonical manifold learning is made possible by adding to the optimization objective the minimization of the off-diagonal elements of the metric tensor defining this manifold. We document this learned basis for low-dimensional data in Figure 2 and for image data in Figure 3. Furthermore, we showcase for image data that higher quality in generated samples stems from the preferable, compact latent representation of the data, Table 1.

Distinctly, the off-diagonal manifold metric elements when minimized by an ℓ 1 loss facilitates sparse learning and/or non-strict local orthogonality. This means that the chart's transformation can be significantly non-linear while retaining some of the benefits of orthogonal representations. This idea can be used in other optimization schemes, which mitigates many of the drawbacks of previous methods in a simple yet theoretically grounded way. Indeed, standard approaches often focus solely on the diagonal elements or strictly enforcing the off-diagonals to be zero, as in [32], limiting expressivity. For instance, in [8], an isometric embedding, essentially a predefined constrained transformation, is presented. Although it allows for direct density estimation and is a form of PCA, it is a strict constraint. Furthermore, as indicated by the synthetic data examples, our present method can be implemented in principle as a full dimensional flow, naturally assigning dimensions accordingly.

The learned manifold representations can be useful as a feature extraction procedure for a downstream task. For example, improved out-of-distribution detection could be a consequence of the feature extraction. Additionally, data that require orthogonal basis vectors, like solutions of a many-bodyphysics quantum Hamiltonian, can show improved learning performance with the current method.

Manifold learning flows are noteworthy due to their mathematical soundness, particularly in the case of methods such as RNF and CMF, which do not implement any fundamental approximations in the density calculations using lower dimensional latent space.We anticipate that the method will stimulate further research into the nature and importance of the latent space representation of MLFs, enhancing our understanding of generative modelling and paving the way to general unsupervised learning.

References

Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning , pages 1530-1538. PMLR, 2015.

George Papamakarios, Eric T Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research , 22(57):1-64, 2021.

George Papamakarios, Theo Pavlakou, and Iain Murray. Masked autoregressive flow for density estimation. Advances in neural information processing systems , 30, 2017.

Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516 , 2014.

Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803 , 2016.

Johann Brehmer and Kyle Cranmer. Flows for simultaneous manifold learning and density estimation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 442-453. Curran Associates, Inc., 2020.

Anthony L Caterini, Gabriel Loaiza-Ganem, Geoff Pleiss, and John P Cunningham. Rectangular flows for manifold learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 30228-30241. Curran Associates, Inc., 2021.

Eike Cramer, Felix Rauh, Alexander Mitsos, Raúl Tempone, and Manuel Dahmen. Nonlinear isometric manifold learning for injective normalizing flows. arXiv preprint arXiv:2203.03934 , 2022.

Christian Horvat and Jean-Pascal Pfister. Denoising Normalizing Flow. In Advances in Neural Information Processing Systems , volume 34, pages 9099-9111. Curran Associates, Inc., 2021.

Danilo Jimenez Rezende, George Papamakarios, Sebastien Racaniere, Michael Albergo, Gurtej Kanwar, Phiala Shanahan, and Kyle Cranmer. Normalizing flows on tori and spheres. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 8083-8092. PMLR, July 2020.

Brendan Ross and Jesse Cresswell. Tractable density estimation on learned manifolds with conformal embedding flows. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 26635-26648. Curran Associates, Inc., 2021.

Nutan Chen, Alexej Klushyn, Francesco Ferroni, Justin Bayer, and Patrick Van Der Smagt. Learning flat latent manifolds with V AEs. In Proceedings of the 37th International Conference on Machine Learning , ICML'20. JMLR.org, 2020.

Brendan Leigh Ross, Gabriel Loaiza-Ganem, Anthony L. Caterini, and Jesse C. Cresswell. Neural implicit manifold learning for topology-aware generative modelling, 2022.

Michael E Tipping. Sparse Bayesian learning and the relevance vector machine. Journal of machine learning research , 1(Jun):211-244, 2001.

Elena Trunz, Michael Weinmann, Sebastian Merzbach, and Reinhard Klein. Efficient structuring of the latent space for controllable data reconstruction and compression. Graphics and Visual Computing , 7:200059, 2022.

David Wipf and Srikantan Nagarajan. A new view of automatic relevance determination. Advances in neural information processing systems , 20, 2007.

Alan Jeffares, Tennison Liu, Jonathan Crabbé, Fergus Imrie, and Mihaela van der Schaar. TANGOS: Regularizing tabular neural networks through gradient orthogonalization and specialization. In The Eleventh International Conference on Learning Representations , 2023.

Maurice S Bartlett. The use of transformations. Biometrics. Journal of the International Biometric Society , 3(1):39-52, 1947.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Yiping Sun, Zhaoyu Li, Jiadui Chen, Xuefeng Zhao, and Meng Tao. Variational autoencoderbased topological optimization of an anechoic coating: An efficient- and neural network-based design. Materials Today Communications , 32:103901, 2022.

Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 7045-7054. PMLR, July 2020.

Bin Dai and David Wipf. Diagnosing and enhancing VAE models. arXiv preprint arXiv:1903.05789 , 2019.

Carlos Lassance, Vincent Gripon, and Antonio Ortega. Representing deep neural networks latent space geometries with graphs. Algorithms , 14(2):39, 2021.

Michael Tipping. The relevance vector machine. Advances in neural information processing systems , 12, 1999.

Christopher M Bishop and Michael Tipping. Variational relevance vector machines. arXiv preprint arXiv:1301.3838 , 2013.

P Thomas Fletcher, Conglin Lu, Stephen M Pizer, and Sarang Joshi. Principal geodesic analysis for the study of nonlinear statistics of shape. IEEE transactions on medical imaging , 23(8):995-1005, 2004.

Guy Bresler, Sung Min Park, and Madalina Persu. Sparse PCA from sparse linear regression. Advances in Neural Information Processing Systems , 31, 2018.

Jonatan Henríquez and Werner Kristjanpoller. A combined Independent Component Analysis-Neural Network model for forecasting exchange rate variation. Applied Soft Computing , 83:105654, 2019.

Rizwan A Choudrey. Variational Methods for Bayesian Independent Component Analysis . PhD thesis, University of Oxford Oxford, UK, 2002.

Isadore Manuel Singer and Jack Thorpe. Lecture notes on elementary topology and geometry., 1967.

Eike Cramer, Alexander Mitsos, Raúl Tempone, and Manuel Dahmen. Principal component density estimation for scenario generation using normalizing flows. Data-Centric Engineering , 3, 2022.

Edmond Cunningham, Adam D Cobb, and Susmit Jha. Principal component flows. In International Conference on Machine Learning , pages 4492-4519. PMLR, 2022.

Rob Cornish, Anthony Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. In International Conference on Machine Learning , pages 2133-2143. PMLR, 2020.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 6840-6851. Curran Associates, Inc., 2020.

Kaare Brandt Petersen, Michael Syskind Pedersen, et al. The matrix cookbook. Technical University of Denmark , 7(15):510, 2008.

M.F. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics - Simulation and Computation , 19(2):433-450, 1990.

Stephen Wright, Jorge Nocedal, et al. Numerical optimization. Springer Science , 35(67-68):7, 1999.

A Riemannian geometry

The Latin indices run over the spatial dimensions and Einstein summation convection is used for repeated indices.

A D dimensional curved space is represented by a Riemannian manifold M, which is locally described by a smooth diffeomorphism h , called the chart. The set of tangential vectors attached to each point y on the manifold is called the tangent space T y M . In the fluid model, all the vector quantities are represented as elements of T y M . The derivatives of the chart h are used to define the standard basis ( e 1 , ..., e D ) = ∂ h 1 , ..., ∂ h D .

The metric tensor g can be used to measure the length of a vector or the angle between two vectors. In local coordinates, the components of the metric tensor are given by

where · is the standard Euclidean scalar product.

For a given metric tensor, the vector v = v i e i ∈ T y M has a norm || v || g = √ v i g ij v j and a corresponding dual vector v ∗ = v i e i ∈ T ∗ y M in the cotangent space, which is spanned by the differential 1-forms dx i = g ( e i , · ) . The coefficients v i of the dual vector are typically denoted by a lower index and are related to the upper-index coefficients v i by contraction with the metric tensor v i = g ij v j or equivalently, v i = g ij v j , where g ij denotes the inverse of the metric tensor. The upper-index coefficients v i of a vector v are typically called contravariant components , whereas the lower-index coefficients v i of the dual vectors v ∗ are known as the covariant components .

A necessary feature for the description of objects moving on the manifold is parallel transport of vectors along the manifold. The tangent space is equipped with a covariant derivative ∇ (Levi-Civita connection), which connects the tangent spaces at different points on the manifold and thus allows for the transportation of a tangent vector from one tangent space to the other along a given curve γ ( t ) . The covariant derivative can be viewed as the orthogonal projection of the Euclidean derivative ∂ onto the tangent space, such that the tangency of the vectors is preserved during the transport. In local coordinates, the covariant derivative is fully characterized by its connection coefficients Γ i jk (Christoffel symbols), which are defined by the action of the covariant derivative on the basis vector, ∇ j e k = Γ i jk . In the standard basis, e i = ∂ h ∂x i , the Christoffel symbols are related to the metric by

Acting on a general vector v = v i e i , the covariant derivative becomes:

where the product rule has been applied, using that the covariant derivative acts as a normal derivative on the scalar functions v i . Extended to tensors of higher rank, for example the second order tensors T = T ij , it becomes

; in this work, the basis vectors e i are generally dropped. Compatibility of the covariant derivative with the metric tensor implies that ∇ k g ij = ∇ k g ij = 0 . This property allows us to commute the covariant derivative with the metric tensor for the raising or lowering of tensor indices in derivative expressions.

The motion of the particle can be described by the curve γ ( t ) , which parameterizes the position of the particle at time t . The geodesic equation, ∇ ˙ γ ˙ γ = 0 , in local coordinates γ ( t ) = γ i ( t ) e i is defined by

The geodesic equation can be interpreted as the generalization of Newtons law of inertia to curved space. The solutions of Eq. (13) represent lines of constant kinetic energy on the manifold, i.e. the geodesics. The Riemann curvature tensor R can be used to measure curvature, or more precisely, it measures curvature-induced change of a tangent vector v when transported along a closed loop.

In a local coordinate basis e i , the coefficients of the Riemann curvature tensor are given by

Contraction of R i jkl to a rank 2 and 1 tensor yields the Ricci-tensor R ij = R k ikj and the Ricci-scalar R = g ij R ij respectively, which can also be used to quantify curvature.

The gradient is defined as ∇ i f = g ij ∂ j f , the divergence as ∇ i v i = 1 √ g ∂ i ( √ gv i ) , and the integration over curved volume as V = ∫ V QdV , where dV = √ gdx 1 ...dx D =: √ gd D x denotes the volume element. √ g denotes the square root of the determinant of the metric tensor.

B Extension to Related Work and further discussions

Normalizing flows have direct access to the latent distribution due to homeomorphism defining the transformation [4]. Nevertheless, the very nature of the homeomorphism can result in pathological behaviour when manifolds with complex topology are learned. For instance, a two-dimensional data space p X can have different numbers and types of connected components, 'holes' and 'knots', and images have complicated shapes and topology due to the sheer number of dimensions. Therefore, as usually p Z is modelled by a simple distribution, like a Gaussian, it cannot trivially match the topology of p X . [33] demonstrates a stable solution to this topological pathology of NFs, albeit with R D support.

Effectively learning manifolds with complicated topologies remains a challenge, even with the full-dimensional latent space. However, the RNF is based on CIF [33] which is an NF that already tries to solve the learning pathologies of complicated manifolds: "CIFs are not subject to the same topological limitations as normalizing flows" [33]. Furthermore, in that regard, we estimate there is no practical limitation in setting, as also seen empirically. In principle, CMF can be a solution for higher-dimensional data. However, in practice it is very computationally expensive to train a full latent dimension (e.g. J T J calculation) and the solution will take long to converge. Additionally, empirical knowledge suggests that a lower-dimensional representation can enhance expressivity, as discussed in the context of Mflows. Consequently, in practice using the current method, one can start from a predefined lower dimension and let the network optimize at will.

PCAflow [32] relies on a lower-bound estimation of the probability density to bypass the J t J calculation. This bound is tight when complete principal component flow, as defined by them, is achieved. The fundamental distinction here is that CMF does not confine itself to a pure PCAflow scenario, which has the potential to restrict expressivity. Since our method only loosely enforces orthogonality, or mutual information → 0 , we anticipate the contour lines to lie somewhere between NF and PCAflow. Furthermore, the introduction of sparsity adds an extra layer of complexity, making direct comparison challenging. As an outlook, we acknowledge that DNF [9] and PCAflow, can undergo quantitative comparison with our method. However, we estimate that RNF is precisely in alignment with CMF, providing a more representative comparison.

Generally, the meaning of "canonical manifold" can vary depending on the context in which it is used, therefore we use Definition 4.1 to precisely define the term "canonical manifold". Definition 4.1 aligns closely with the assumption of PCAflow. However, it is vital to note that our method does not enforce this strictly. To put it in perspective, our approach seeks a 'partly canonical manifold,' if you will. Additionally, our method encompasses sparsity-meaning the diagonal elements can also approach zero.

In principle, full-dimensional flow d = D can be also solution for higher-dimensional datasets, as the CMF loss will ensure that the number of prominent latent dimensions should correspond to the true intrinsic dimensionality of the data. However, in practice it is very computationally expensive to train a full latent dimension (e.g. J t J calculation) and the solution will take long to converge. Additionally, empirical knowledge suggests that a lower-dimensional representation can enhance expressivity, as discussed in the context of Mflows. Consequently, in practice, one can start from a predefined lower-dim and let the network optimize at will. Nevertheless, the efficient approximate methods indicate that CMF could be implemented for a full-dimensional flow. Further exploration of this potential could be undertaken in future studies.

C FID score

For a dataset { x 1 , . . . x n } ⊂ R D and sampled generated data { ˜ x 1 , . . . ˜ x n } ⊂ R D for a statistic F the mean and variance are:

The FID score takes F as the last hidden layer of a pre-trained network and evaluates generated sample quality by comparing generated moments against data moments. This is achieved by calculating the squared Wasserstein-2 distance between Gaussian and corresponding moments by:

This would reach zero if they match. For tabular data, F is taken to be the identity F ( x ) = x , as in [7].

D Optimizing the objective and calculating the metric tensor

The third term of canonical manifold learning optimization objective, (Equation (8)):

̸

is not so trivial to backpropagate through. Here we use the method suggested by [7]. First, J T θ J θ is calculated efficiently with the Forward-Backward automatic differentiation trick as explained in Section 4.3 of [7]. For the simulated low-dimensional datasets the exact method is used, and for images the approximate stochastic gradient method with K = 1 is used. Second, to back-propagate for the approximate method, a stochastic unbias estimator is used. Namely, from matrix calculus [35] it is known that:

where tr is the trace and θ j is the j -th component. Then the Hutchinson trace estimator is used [36]:

where ε 1 , . . . ε k are sampled from a Gaussian distribution. Access to J T θ J θ as explained above allows for the direct computation of Equation (21). Nevertheless, the first term ϵ T k ( J T θ J θ ) -1 = [( J T θ J θ ) -1 ε ] T is calculated by an iterative method solver of the matrix inverse problem, the conjugate gradient method CG [37].

̸

Lastly, the metric tensor is obtained from J T θ J θ as stated in Section 2.2. Back-propagation of ∥ G j = k ∥ 1 1 is straight forward by taking the derivative w.r.t. θ by an (E.g. Adams) optimizer, similar to the reconstruction loss in Equation (8).

1.0

0.5

0.0

0.5

1.0

Gkk

0.4

0.2

0.0

0.2

0.4

Cosine similarity

1.0

0.4

0.2

0.0

0.2

Cosine similarity

0.4

1.0

0.4

0.2

0.0

0.2

Cosine similarity

0.4

1.0

0.4

0.2

0.0

0.2

Cosine similarity

0.4

E Tabular datasets

In Figure 5, we visualize the metric tensor for the tabular datasets trained with sufficiently high latent dimensions and present the mean absolute cosine similarity (MACS) in the supplementary PDF. For the specific case of GAS with d=2, the plot might not offer significant insights. We observe that the mean absolute cosine similarity is lower for CMF, although, considering the error, there remains some overlap, which is within expectations.

In order to investigate the cause for the performance gap in GAS, we visualize the validation FID-like scores during training, Figure 6. Upon comparing the results of HEMPMASS and GAS, it is apparent that learning converges until a certain point, after which instability may arise for all three methods. Despite this instability, it is important to note that our implementation selects the best validation step checkpoint for testing, mitigating the impact of training instability. However, it is plausible that these instabilities contribute to the variations between different runs. Furthermore, the training curves follow similar behaviors, converging up to a certain point and then displaying a degree of instability. This trend holds true for all three methods when trained on the GAS dataset.

(a) GAS trained with CMF and RNF.

F Image generations

F.1 Samples from prominent dimensions

In order to demonstrate the more efficient use of latent space by CMF, we trained both CMF and RNF using latent-dimension d = 40 on both MNIST and Fashion-MNIST datasets. We choose a varying number of prominent latent dimensions from Z ∈ R 40 as those with the largest | G kk | , this is mathematical intuition that the largest eigenvalues (if an eigen-decomposition is possible) correspond to the dimensions with the highest weights.

We generate samples from the models using only the prominent dimensions, by sampling from the prominent dimensions and setting all others to zero, subsequently FID scores are calculated for the

1.0

0.5

0.0

Gkk

0.5

1.0

0.5

0.0

Gkk

0.5

1.0

0.5

0.0

Gkk

0.5

generated samples. Additionally, we reconstruct real samples using only the prominent dimensions and compute reconstruction loss using mean squared error, as shown in the main text. We repeat the experiment for different number of prominent latent dimensions (1 , 8 , 16 . . . , 40) , or similarly, non-zero z i ∈ Z . Here, in the appendix, we show samples when the number of prominent dimensions sampled from increases.

We would like to note that potentially for any future use that entails isolating prominent components, a proper ID estimation should be carried out.

For example, in Figure 7, different samples are generated along the horizontal. This is repeated for increasing number of prominent dimensions used for sampling, top to bottom along the vertical (4 , 8 , 12 , 16 . . . , 40) . In Figure 7, FashionMNIST models were trained with R d =40 and for the CMF and RNF methods. Similarly to the FID plots in the main text, the CMF shows a bit less variation when going along the columns from top to bottom; this is an indication that fewer prominent dimensions are needed for a faithful generation.

(a)

(b)

To visualize the sparsity of learned manifold, we generate samples from individual latent space dimensions for fully trained models on FashionMNIST with the RNF and CMF. In Figure 8, the trained RNF and CMF models exhibit FIDs of ∼ 302 and 297 respectively, see main text. Across the rows, different samples can be found as sampled from each of 10 prominent component subgroups, i.e. ( { 1 , 2 } , { 3 , 4 } , { 5 , 6 } . . . , { 19 , 20 } ) out of the 20 latent dimensions. Here, only these subgroups are used for sampling at once and the other 18 dimensions are set to zero. The dimensions sampled from at the bottom are the most significant ones; the rest follow in order towards the top. It can be clearly seen that the CMF method manages to capture most of the information of the manifold in the first two dimension, as the generations in the bottom row are good and the rest of the columns seem to have less relevance, indicating strong sparse learning. For the RNF, again there is an importance distinction, as expected, but even dimensions of less prominence seem to be generating variation in the samples. Combining these observations with the higher FID of the CMF, efficient sparse learning is evident.

In Figure 9, we plot a sampled manifold between -3 and 3 for the two most prominent latent dimensions from Z ∈ R 20 as those have the largest | G kk | . This is done for an RNF and CMF trained MNIST model on the left and right of Figure 9. The CMF method appears to split the latent space of z 1 without much variation in the z 2 direction, indicating perpendicularity.

(a)

(b)

(a)

3

2

1

0

1

2

3

z

1

3

2

1

0

1

2

3

z

2

(b)

3

2

1

0

1

2

3

z

1

3

2

1

0

1

2

3

z

2

8

8

9

9

9

7

7

7

8

8

8

9

9

9

7

7

8

8

8

8

8

9

7

8

8

8

8

8

8

8

8

8

8

8

8

8

8

8

8

8

8

8

8

8

8

8

a

a

2

2

2

a

a

2

g

G

8

8

1

1

D

8

8

/

0

0

0

0

8

1

1

0

0

8

0

o

0

8

1

o

0

8

0

8

8

F.2 Out of distribution detection

We evaluate the performance of CMF for out-of-distribution detection (OoD). Table 3 quantifies the results, where accuracy here is the log-likelihood. The results are similar for the previous methods as expected.

G Experimental details

The training was performed on a GPU cluster with various GPU nodes including Nvidia GTX 1080, Nvidia GTX 1080 Ti, Nvidia Tesla V100, Nvidia RTX 2080, Nvidia Titan RTX, Nvidia Quadro RTX 6000, Nvidia RTX 3090. Multiple GPUs were used in parallel across the batch.

FMNIST

700

MNIST

decisionboundary

600

amples

500

400

iS

300

#

200

100

0

-200 -175 -150 -125 -100 -75-50

log p(x)

There exists no substantial computational cost distinction between the RNF and CMF methods, as both encounter the calculation bottleneck of the Jacobian-transpose-Jacobian. Any other option was set to the default as found in [7]. The approximate training times can be found in Table 4. Currently, manifold learning flows are computationally demanding; nevertheless, their potential is noteworthy due to their mathematical soundness, particularly in the case of methods such as RNF and CMF, which do not implement any fundamental approximations in the density calculations using lower dimensional latent space.

G.1 Images

To train for images, 4 or 5 nodes of less than 10GB each were used in parallel across the batch size. The hyperparameters were kept constant for the three different methods compared, unless a distinction was made for the manifold learning flow as explained in [7].

G.2 Simulated data

The parameter choice is also kept consistent for training for simulated data. Training was performed on a single GPU or CPU and training times were observed to be 1-4 hours depending on epochs and if early stopping was used. Again, similar training times between RNF and CMF were observed.

G.3 Tabular data

We train for the tabular datasets as found in Papamakrios et al. [3]. Parameters and settings were chosen as follows, as per RNF.

For each run set, we performed 5 repetitions, of which we report an average FID score and a standard error in Table 2. We train models with latent dimension d=2, 4, 10 and 21 to POWER, GAS, HEMPMASS and MINIBOONE datasets, respectively.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors address the interesting question of how to disentangle the relevant manifold directions properly in the latent space of manifold learning normalizing flows (MLF). After a comprehensive theoretical background, the main motivation is presented in form of a toy example of a simple noisy line embedded in $\\mathbb{R}^{2}$. When varying the latent dimension $z_{i}$ of a RNF and setting the remaining latent variables to $0$, the corresponding contour lines are partially aligned. Doing the same with the proposed method, CMF, leads to orthogonal contour lines and thus to a meaningful latent representation where one latent variable encodes the line and the other the (noisy) off-manifold direction. This intuition leads straightforwardly to enforcing the orthogonality of the flows Jacobian matrix. To do so, the authors propose a new cost function for learning manifold structures using an NF by adding a penalty term which ensures that the off-diagonal entries of the Gram matrix (Jacobian-transpose-Jacobian) are $0$ at the one hand, and are sparse at the other due to the l1-norm. The effecitiveness of the method is tested in experiments on toy-data, tabular data and 32x32x3 image data.\n\nSTRENGTHS:\nThe paper is overall well written. The theoretical background is exhaustive and well-explained. The example in 4.1 serves as a good illustrative motivation. The experiments are comprehensive.\n\nWEAKNESSES:\nThe contribution is somewhat marginal. Though citing [1], they don't mention that the corresponding authors propose a very similar penalty term. To my understanding, the only differences of the penalty term in [1] are\na) that in [1] the diagonal entries are penalized to be $1$ whereas in this work the diagonal entries are unconstrained.\nb) [1] used the L2-norm whereas in this paper the L1-norm is used.\n\nGiven the great similarity, the penalty term suggested by [1] should be included in the comparison. What do we gain by not constraining the flow to be an isometry? In addition, the authors use the M-flow for benchmarking although a better method for overcoming the limitations of M-flows (namely that the log-determinant term is not considered while training) was already proposed in [2]. At least citing and ideally comparing with [2] should be part of the paper. The same is true for comparing with [3]. It would be nice to see how the contour lines differ from the method introduced in [3].\n\nFurthermore, I am not sure to understand the reasoning behind Definition 4.1. To my understanding, every differentiable manifold is a canonical manifold as one is free to choose the basis of the tangent space to be orthogonal. Thus, such a basis always exists which is the only requirement to be a canonical manifold. Definition 4.1. is simply an existence statement and not useful as such. I think what the authors wanted to define is a manifold equipped with a chart s.t. the chart induces an orthogonal basis of the tangent space. However, this is very much the definition of the principal component flow [3]. Please enlighten me in case I totally misunderstood something.\n\nI am also confused about the comparison with rectangular NF (RNF) in Figure 2. By definition for RNF we have that $d<D$. However, in Figure 2 the author uses $d=D$. This is a contradiction. If $d=D$, a standard NF can be used rather than an RNF. In addition, one claimed strength of the proposed CMF is the sparisity induced by the l1 norm. Thus, the relevant dimensions should be found automatically which elevates, in theory, the necessity to estimate $d$ a priori. Then, why not always set $d=D$ and then apply the proposed penalty term? The number of prominent latent dimensions should correspond to the true intrinsic dimensionality of the data.\n\nFinally, the notation for the manifold, data manifold, and learned manifold is confusing. In line 74 or line 90, $\\mathcal{M}$ is a lower dimensional manifold. In line 83, $\\mathcal{M}_{\\theta}$ is introduced without explaination. In line 212, $\\mathcal{M}$ is referred to as the learned manifold and in the same line, the data manifold is introduced without further explanation. In addition, a new notation for a canonical manifold is suggested. I find the various versions for a manifold confusing and don't see the added benefit of it (especially given that Definition 4.1. is not useful in my opinion).\n\n\n[1] Eike Cramer, Felix Rauh, Alexander Mitsos, Raúl Tempone, and Manuel Dahmen. Nonlinear 350 isometric manifold learning for injective normalizing flows. arXiv preprint arXiv:2203.03934, 351 2022.\n\n[2] C. Horvat and J.-P. Pfister. Denoising normalizing flow. In Advances in Neural Information Processing Systems, volume 34, 2021.\n\n[3] Edmond Cunningham, Adam D Cobb, and Susmit Jha. Principal component flows. In Interna410 tional Conference on Machine Learning, pages 4492–4519. PMLR, 2022.\n\nQUESTIONS:\n1. Does Definition 4.1 really makes sense (see above)\n2. Is the comparison with RNF in Figure 2 fair (see above)? By definition of RNF, it must be $d<D$. Please elaborate.\n3. Why not always set $d=D$ and then apply the proposed penalty term (see above)?\n4. Figure 1. Why is the density varying greatly when only varying one dimension? Shouldn't be uniform if the density is properly learned?\n5. What is the quality in terms of density estimation? For instance, what is the KS statistics compared to the true density, or, at least, does it sum to 1?\n6. Why not benchmark with PCA flow? It would be interesting to see if this method can overcome some of its limitations (see above).\n7. Moebius band with RNF seems to be poorly learned. Why?\n8. Why prominent latent dimensions are the ones with the greatest $G_{ii}$? Is this a mathematical statement/proposition or intuition?\n9. line 305: in which case would an orthogonal basis not be an optimal representation?\n10. What happens if we use the L2 rather than L1 norm?\n11. How are prominent latent dimensions defined? Do they correspond to the true intrinsic dimensionality? Is there a clear cut-off in the magnitudes of $|G_{kk}|$ in e.g. MNIST? If not, a somewhat arbitrary threshold needs to be defined, isn't it?\n\n*Minor*:\n+ line 70-72: the statement is only true for large $n$\n+ line 83: What is $\\mathcal{M}_{\\theta}$?\n+ line 83: \"to encorage $x\\in \\mathcal{M}_{\\theta}$\" do you mean $g_{\\phi}(z)$ rather than $x$?\n+ line 114: typo: \"due to lack...\"\n+ line 145: noise is not perpendicular to line\n+ line 217: mention reason (maybe in a footnote)\n+ line 225: $\\gamma$ not crucial but had an impact on training time? What is crucial for you?\n+ line 244: bad wording \"the expected the canonical manifold learning\"\n+ line 297: Inconsistent acronym: M-flow vs. Mflow\n+ line 299: inferior lok-likelihood calculations of Mlow...where shown?\n+ line 309/310: wrong statement: there are diffusion models with lower dimensional latent space. However, by default indeed the latent space has the same dimensionality\n\nLIMITATIONS:\nMajor limitations are addressed in Section 6. However, some questions are unresolved such as: how to find/define the prominent latent dimensions without an intrinsic dimensionality estimator?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the current manifold learning methods. It compares canonical manifold learning flow (CMF) with other manifold learning methods and demonstrates that CMF can learn the orthogonal features existing in data. From synthetic data on Moebius, the paper shows the benefits of using CMF.  Also, the paper shows the generated images from the CMF learned latent space is of higher quality than other methods' learned latent spaces. Lastly, the paper acknowledges some limitations of CMF, for example, high computation cost of the full Jacobian tranpose Jacobian.\n\nSTRENGTHS:\n1. The paper is very clear on the advantages of CMF against other manifold learning methods. \n2. The paper did various types of experiments to showcase the CMF's performance among the methods.\n\nWEAKNESSES:\n1. Although being mentioned in the limitations, the paper does not provide results about the computation costs of CMF. There is no comparison of the computation time of CMF against other methods, also how the dimension affects the computation cost. It is unclear if CMF is scalable to higher-dimensional datasets.\n2. The paper does not compare CMF with a lot of methods, including PCA and ICA or purely deep learning methods. For example, an autoencoder can also extract feature representations from data. How does CMF compare to these approaches?\n\nQUESTIONS:\n1. What is the complexity of the RNF and CMF methods?\n2. For table 2, are there explanations on why M-flow works the best and CMF is the worst on GAS?\n3. Figure 3 is hard to interpret. What are the advantages of CMF in this plot?\n4. What are the generative models used in the image experiments?\n\nLIMITATIONS:\nThe authors have discussed the limitations clearly in the paper. They acknowledge the high computation costs and the limitation that the method only applies when the manifold is homogeneous to Rd space.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a new method for regularising manifold learning flows. Essentially, it attempts to reduce the entanglement between the dimensions of the learned manifold by encouraging non-diagonal elements of the metric tensor to be small. Leveraging the already necessary computation of $J^\\top J$, where $J$ is the Jacobian of the flow transformation, this is done efficiently by minimising the $\\ell-1$ norm of the non-diagonal entries.\n\nExperiments on synthetic and real data show how this is effectively achieved, especially when compared to similar models that do not employ the regularisation scheme. Beyond achieving the desired effect, the proposed method achieves lower FID-like scores on real tabular data.\n\nSTRENGTHS:\nThe paper is very well written and the presented ideas are easy to follow. Experimentally, the authors confirm both in intuitive, synthetic examples and real data. The proposed regularisation scheme is also simple and computationally efficient, which is also a desirable feature. Overall, I consider it a strong contribution and see high potential of being widely adopted as a mechanism for regularising manifold learning flows.\n\nI also appreciate the limitations raised and discussed by the authors, indicating maturity in their analysis and raising important considerations related to the use of their work.\n\nWEAKNESSES:\nAlthough I see the potential mentioned above, I do consider it could be viewed more widely as a weak point, given the relatively niche application. Although it does not affect the merit of the work, the principle might be too specific to manifold learning flows.\n\nI would consider more tabular data experiments are needed, with additional data sets, given the mixed results attained. More concretely, some analysis of what specifically differentiates the data sets enough to cause the difference in performance.\n\nQUESTIONS:\n- Tied to the aforementioned weakness, what do the authors posit is the cause for the performance gap in GAS?\n- Related to the previous question: are the metric tensors found in tabular data also behaving similarly to the ones shown in Figure 3?\n- Is it clear during training when a pathological scenario is reached? How does it behave w.r.t. increasing/decreasing the value of $d$?\n- Are the learned manifold representations also useful as a feature extraction procedure for downstream tasks? Could this be also a scenario for evaluating the quality of the learned models? (As opposed to only raw performance in scores)\n\nMinor comments:\n- line 35: Caterini et al [7] has => Caterini et al. [7] have\n- line 44: Please avoid unnecessary adjectives such as \"tedious\"\n- line 62: why is the set starting with $x_0$?\n\nLIMITATIONS:\nI was pleasantly surprised with the discussion on limitations in the paper. I believe the authors addressed important issues and even highlighted which other approaches could have better raw performance.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of learning a latent representation for data supported on a low-dimensional manifold. It proposes to promote orthogonality of the tangent vectors arising from a learned chart, on top of existing rectangular flow loss. Experiments are provided to demonstrate the effectiveness of the algorithm.\n\nSTRENGTHS:\nPromoting orthogonality of the tangent vectors from a learned chart is interesting.\n\nWEAKNESSES:\n## Presentation:\n1. The paper seems insufficiently prepared and proofread. In particular, there are numerous evident typos even in the first paragraph of the introduction. Consequently, they weaken the credibility of the paper.\n    1. Lines 29, 33 the \"D\" in \"R^D\" uses a mathbb, whereas in lines 31, 32 the \"d\" or \"D\" does not.\n    1. Line 29 “fulfil” -> “fulfill”\n    1. In equation 1, the q_phi(x) should have been q_phi^{-1} (x)\n    1. Section 3 talks about general manifold learning, and is almost detached from the technical sections 2 and 4.\n2. Line 43: A line is one-dimensional. What is a two-dimension line?\n3. If one compares equations (5) and (6), it appears that we assume the learned map q_phi is a chart of manifold M. If that is the case, this should be made explicit.\n4. From (8), it is unclear that where the introduced G term are evaluated. I suppose it is on points x_i’s?\n\nQUESTIONS:\n## Significance\nLooks like the inverse of g_phi is in objective (6), so using line 75 it boils down to the inverses of g_eta and f_theta. How easy is it to invert g_eta and f_theta? The root of this question is, if one takes f_theta to be a square matrix, then inverting the square matrix is needed, leading to high cost if the dimension of the square matrix is large. If f_theta is a square matrix followed by say ReLU, then ReLU is not invertible.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}