Canonical normalizing flows for manifold learning
Authors: Kyriakos Flouris
Ender Konukoglu
Venue: NeurIPS 2023 poster (2023)
Keywords: manifold learning flows
normalizing flows
optimization
orthogonalization
sparsity
sparse learning
generative modeling
Riemannian manifold
geometry
metric tensor
orthogonal basis
TL;DR: We propose an optimization objective which encourage learning of an orthogonal and sparse manifold basis as applied to canonical manifold learning flows

Abstract: Manifold learning flows are a class of generative modelling techniques that assume a low-dimensional manifold description of the data. The embedding of such a manifold into the high-dimensional space of the data is achieved via learnable invertible transformations. Therefore, once the manifold is properly aligned via a reconstruction loss, the probability density is tractable on the manifold and maximum likelihood can be used to optimize the network parameters. Naturally, the lower-dimensional representation of the data requires an injective-mapping. Recent approaches were able to enforce that the density aligns with the modelled manifold, while efficiently calculating the density volume-change term when embedding to the higher-dimensional space. However, unless the injective-mapping is analytically predefined, the learned manifold is not necessarily an \emph{efficient representation} of the data. Namely, the latent dimensions of such models frequently learn an entangled intrinsic basis, with degenerate information being stored in each dimension. Alternatively, if a locally orthogonal and/or sparse basis is to be learned, here coined canonical intrinsic basis, it can serve in learning a more compact latent space representation. Toward this end, we propose a canonical manifold learning flow method, where a novel optimization objective enforces the transformation matrix to have few prominent and non-degenerate basis functions. We demonstrate that by minimizing the off-diagonal manifold metric elements $\ell_1$-norm, we can achieve such a basis, which is simultaneously sparse and/or orthogonal. Canonical manifold flow yields a more efficient use of the latent space, automatically generating fewer prominent and distinct dimensions to represent data, and consequently a better approximation of target distributions than other manifold flow methods in most experiments we conducted, resulting in lower FID scores.

Canonical normalizing flows for manifold learning
Kyriakos Flouris
Ender Konukoglu
Department of Information Technology and Electrical Engineering
Department of Information Technology and Electrical Engineering
ETH Zürich kflouris@vision ee. ethz. ch
ETH Zürich kender@vision ee ethz. ch
Abstract
Manifold learning flows are class of generative modelling techniques that assume low-dimensional manifold description of the data. The embedding of such a manifold into the high-de -inen-immnsionnl space of the data is achieved via learnable invertible transformations. Therefore, once the manifold properly aligned via a reconstruction loss, the probability density is tractable on the manifold and maximum likelihood can be used optimize the network parameters. Naturally, the lover-dimensional representation of the data requires an injective-mapping Recent approaches were able to enforce that the density aligns with the modelled manifold, while efficiently calculating the density volume-change term when embedding to the higher-dimensional space. However, unless the injective-mapping is analytically predefined, the learned manifold not necessarily efficient representation of the data. Namely, the latent dimensions of such models frequently learn entangled intrinsic basis, with degenerate information being stored in each dimension. Alternatively. if a locally orthogonal and/or sparse basis is to be learned, here coined canonical intrinsic basis, it serve in learning compact latent space representation. Toward this end, we propose a canonical manifold learning flow method, where novel optimization objective enforces the transformation matrix to have few prominent and non-degenerate basis functions. We demonstrate that by minimizing the off-diagonal manifold metric elements f--nomm, we can achieve such a basis, which is simultaneously sparse and/or orthogonal. Canonical manifold flow yields more efficient use of the latent space, automatically generating fewer prominent and distinct dimensions to represent data, and consequently a better approximation of target distributions than other manifold flow methods in most experiments we conducted, resulting in lower FID scores.
1 Introduction
Many emerging methods in generative modeling are based on the manifold hypothesis, i.e., higher- dimensional data are better described by lower-dimensional sub-manifold embedded in ambient space. These methods replace the bijective normalizing flow in the original construction 5T5 (NF) with injective flow. NFs constructed where the learned distribution density is supported on the set of dimension equal to the data dimension, where RD is the data space. In order fulfill the manifold hypothesis, emerging class of methods, namely manifold learning flow models (MLF) 6.9, model the latent distribution as a random variable in Rd where d < D. i.e. realize injective flows. Consequently, the density lives some d-dimensional manifold Ma embedded in R as required. If Ma is known a-priori, [TO] the density transformation R in MLF is relatively straight forward affair. If the manifold be
smooth bijective mapping, i.c. homeomorphism.
Code available httpps://githuu..oo//// flouris/emf/
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

learned, the change-of- vfrviiabl formula, i.e. the volume element, can be computationally inhibiting to calculate. Caterini 7 have recently solved this problem and established tractable likelihood maximization.
MLF methods have thus evolved the point where the general manifold hypothesis is properly incorporated and the likelihood is efficiently calculated. However, it has been observed that the learned manifold can be suboptimal; for example, due to a nonuniform magnification factor [12) and the inability to learn complex topologies 13] Furthermore, pushforward models such as MLFs, i.c. D, are designed avoid over-fitting, but d' in I is still arbitrary unless determined by some meta-analyss method. As well as, there is often degeneracy in the information stored in each of these latent dimensions d. For instance. as we show in Section 4.1 consider modeling a wwo-dimension line with considerable random noise, i.c. a 'fuzzy line". If d 2, both latent dimensions will attempt to encompass the entire manifold a degenerative use of latent dimensions. Particularly, this behavior of storing degenerate representations is inefficient and can lead to over-fitting, topological, and general learning pathologies. Alternatively, we postulate that motivating sparsity 14 and local orthogonality during learning can enhance the manifold learning process of MLFs.
To this end, we propose an optimization objective that either minimizes or finds orthogonal gradient attributions in the learned transformation of the MLF. Consequently, a more compact [15] and efficient latent space representation is obtained without degenerate information stored in different latent dimensions, in the same spirit as automatic relevance determination methods [16] or network regularization [7]. The relevant dimensions correspond to locally distinct tangent vectors on the data manifold, which can yield a natural ordering among them in the generations. Borrowing the term 'canonical basis', i.e., the set of linearly independent generalized eigenvectors of a matrix, we introduce Canonical manifold learning flows (CMF). In CMF, the normalizing flow is encouraged to learn a sparse and canonical intrinsic basis for the manifold. By utilizing the Jacobians calculated for likelihood maximization, the objective is achieved without additional computational cost.
2 Theoretical background
2.1 Normalizing and rectangular normalizing flows
First, we define a data space as X' with samples {Ii,..., where x, E RD, and a latent space 2 C Rd. If, d = D, then a normalizing flow is a é parameterized diffeomorphism do : RD. - RR.. i.e., a differentiable bijection with a differentiable inverse, that transforms samples from the latent space to the data space, X' := go (2). Optimization of & is achieved via likelihood maximization, where due to the invertibility of go the likelihood, (x(()) can be calculated exactly from the latent probability distribution pz(z) using the transformation of distributions formula [8]
(x(x) = p2(05"(2)))dett (z))) 40 (0 (z)))=
(1)
J denotes the Jacobian functional such that Ja, = 0x/Oz, i.e., the Jacobian of da at x = (z).
Here, 2 is modelled by random variable = ~ pz for simple density pz, e.g., normal distribution. Therefore, for large n, a normalizing flow approximates well the distribution that gave rise to the samples in dataset (x.) [=] by transforming the samples pz with trained Qø- if o² 1 (ogpx(<;
argmax, E
(4].
In the context of manifold learning flows, rectangular normalizing flows, d < D implies there is a lower dimensional manifold M C Rd that is embedded into the RD with a smooth and injective transformation Io Rdd + R In practice, I = h, opado fe, where h. Rd + Rd fo R - R and pad: Rd + R. Padding can be used for trivial embedding, i.c., pad(2) (2) (2o ... 2. 0...O) 0). The rectangular Jacobian of J go is not invertible and thus, J Jos * J g which implics that the volume element needs to be calculated fully, i.e., the square root of the determinant of JT J. which is an equivalent expression for the general metric tensor in this context- see the appendix. This leads to a more generic formulation
9,
(x(x) = pz g (x)) |detJ" ga (z)( go (9 (z))
However, note that maximizing px px( (4) on its own would only maximize the likelihood of the projection of x on the latent space, not ensuring that the model can reconstruct from i.c., the

---

modeled manifold may not be aligned with the observed data. To encourage 2 E M, namely to align the learned manifold the data space, reconstruction enror needs minimized during optimization. The following loss is thus added to the maximum-likelihod loss function, as explained
in 6
Σ 2, 96 (55' (2) (=))
(3)
Combining the logarithm of Equation 2 and using the hyperparameter | adjust Equation we arrive at the total Lagrangian to be maximized:
"
logpz pz 40 (x²) log kdelJ (x:))
argmax
2 log jdetJ] 30 (Jo Jo (o (x:))) 8
9o 9 (xi))
Riemannian geometry and the metric tensor
A d dimensional curved space is represented by Riemannian manifold M C Rd, which is locally described by a smooth diffeomorphism h, called the chart. The set of tangential vectors attached to each point y on the manifold is called the tangent space Ty.M. All the vector quantities are represented as elements of M. The derivatives of the chart h are used to define the standard basis 8h
(e.,..., Ca)
The metric tensor g can be used measure the length of vector or the angle between two vectors. In local coordinates, the components of the metric tensor are given by
Öh
Öh
Gúj = c;
(5)
0zz'
where is the standard Euclidean scalar product.
The metric tensor Gúj describing the manifold M C R learned in manifold learning flows is equivalent to the section of the rectangular secobian-traasposee transpose-Jacobian Jacobian J" that describes M. As explained in 77] and the appendix, the J can be very efficiently approximated in the context of MLFss Thus, it can be seen that no additional computational cost is need to calculate the metric tensor.
3 Related Work and motivation
Generative models with liwer-dimensional latent space have long being established as favorable methods due to their efficiency and expressiveness. For instance, variational autoencoders [19] have been shown to learn data manifolds M with complex topologies (20 21I. However, as seen in [22] variational approaches exhibit limitations in learning the distribution Px on M. Manifold learning flows 6 (Mflow) form a bridge between tractable density and expressiveness. Their support E IR can inherently match the complex topology of R.,, while maintaining well-defined change of variables transformation. Furthermore, ttate-of-the-arr methods can efficiently calculate the determinant of Jacobian dot product, as in rectangular normalizing flow 7 (RNF). Denoising normalizing flow (DNF) represents progression beyond Mflow, where density denoising is used to improve on the density estimation. RNFs can be seen as a parallel approach to DNF but with a more direct methodology, circumventing potential ambiguities stemming from heuristic techniques like density denoising.
[9]
Despite the success of manifold learning flows in obtaining a tractable density whilst maintaining expressivity, the learned manifold is optimized within latent space representation [23], an outstanding problem for most generative models. For example, due to lack of constrains between latent variables can be the case that some of them contain duplicate information, i.e., form degenerate intrinsic basis to describe the manifold, also demonstrated in Section 4.1 Sparse learning 114 was an early attempt to limit this degeneracy and encourage learning of global parameters, introducing solutions such the relevance vector machine [24] 25 (RVM).The RVM attempts to

---

limit learning to its minimum necessary configuration, while capturing the most relevant components that can describe major features in the data.
Even stricter approaches are principal [26] 27 and independent component analyses 28 29 PCA
and ICA, respectively. PCA and ICA obtain well-defined and optimized representation when convergent; however, PCA and ICA are linear in nature and their non-linear counterparts require kernel definitions. Here, the network-based approach is more general, reducing the need for hand- crafting nonlinear mappings. Furthermore, pure PCA and ICA can be limited in modelling some data manifold topologies where strict global independent components are not necessarily a favorable basis [30 Furthermore, [1] presents a method for efficient post-learning structuring of the latent space, showcasing the advantages of model compactness. We propose a canonical manifold learning flow where the model is motivated to learn compact and non-degenerate intrinsic manifold basis.
Furthermore, it has been shown that tabular neural networks can show superior performance when regularized via gradient orthogonalization and specialization [17]. They focus on network regulariza- tion, which not specific to one method but requires certain network structure and costly attribution calculations.
Recently, NF methods with flow manifested in the principal components have been proposed 31 Cranmer et al. [1] focus on datasets that have distinct PCAS that can be efficiency calculated and, therefore, utilize the computed PCAs for efficient density estimation. Cunningham B2] (PCAflow) relies on a lower-bound estimation of the probability density to bypass the J calculation. This bound is tight when complete principal component flow, defined by them, is achieved. The fundamental distinction here is that our method does not confine itself to a pure PCAflow scenario, which has the potential to restrict expressivity, In contrast, method only loosely enforces orthogonality, while encouraging sparsity, i.e. promoting learning of a canonical manifold basis.
al.
Method
4.1 Canonical intrinsic basis
In this section, in order to motivate the proposed development, we first demonstrate how rectangular manifold learning flows assign latent space representation in comparison to canonical manifold learning flow. We generated synthetic data on tilted line with noise in the perpendicular and parallel directions, i.e., fuzzy line. We sample 1000 x; such that x1 ()(4(((2)) (25,, 2.5) and 22 21 e, where E <~ U( 0.5, 0.5). In Figure [a), a two-dimensional manifold M is learned with the vanilla- RNF method using the samples by minimizing the loss given in Equation 4 As expected and seen on the left of Figure a the probability density is learned satisfactorily. i.e., the model fit well the samples and can generate samples with accurate density. However, when samples on the manifold are generated from the two latent dimensions, 2; individually, shown in Figure I (ii) and (iii,, the sampling results are very similar. The implication being that the intrinsic basis is almost degenerate and information is duplicated amongst the latent variables. In contrast, when the same M is learned with the proposed CMF, shown in Figure the sampled latent dimensions are close to perpendicular to each other. Each basis is capturing non-degenerate information about M, see Figure | lb), i.c., the two major axes of the dataset.
4.2 Canonical manifold learning
As canonical manifold is not a standard term in literature, let us start by defining a canonical manifold for manifold learning flows.
Definition 4.1. Here, a canonical manifold, M. is a manifold that has an orthogonal and/or sparse basis e; such that e; ej = -0% Vy E M1 and whenever i +
The name in Definition 4. is inspired from the "canonical basis", i.e., the set of linearly independent generalized eigenvectors of a matrix, as defined in linear algebra. We hypothesize that enforcing learning a canonical manifold as defined in Definition 4.1 during manifold flows will lead to a less degenerate use of the latent space, and consequently a better approximation of the data distribution. As in sparse learning 14, during training necessary dimensions the latent space will be automatically determined, and those that are necessary will be used in such a way to model manifold with an

---

(a) Density plot for fuzzy line learned with RNF, via Equation 4
(b) Density plot for fuzzy line learned with CMF, via Equation g
Figure I: Comparison of density plots for a fuzzy line learned with RNF (a) and CMF (b). Sampled
1000 x; 's such that 3i 41( (22,,, 2.5) and 22 31 + with c U(-0.5, 0.5). The black dots represent samples from the real data manifold. (i) Samples from the model using the full latent space where colors correspond to log p(x). (ii), (iii) Samples from the model using only the first and the second components 21/2 E Z while setting the other to zero, respectively. Colors correspond to logpl p(z) once more. All densitics have been normalized to [-1, 1] for visualization purposes. Note that compared to RNF, with CMF different latent variables could capture almost an orthogonal basis for this manifold.
orthogonal local basis to fit the data samples. This, we believe, will enhance the manifold learning process, avoiding overfitting and other learning pathologies associated with correlated attributions to dimensions of the latent space.
In order to obtain canonical basis first realize that, as described in Section 2 and the appendix, the metric tensor can be used as concise language to describe the manifold i.. the transformation of the chart.
Gij = Σ 0g (x)
Og 1,k
Sx Ox
(6)
Oz
Ozt
Oz θz1
This provides an opportunity to regularize the learned representation directly with a single optimiza- tion objective. The dependencies on G are dropped for brevity.
Leveraging the metric tensor representation, we propose minimize the off-diagonal elements of the metric tensor, enforcing learning of a canonical manifold
||
ΣΣΣ |G sffi
where ||-11 is the & norm. Using ||Gı as a cost to minimize serves both the sparsity and generating orthogonal basis. While the & norm could be used to enforce specific embedding [8], it is not in the scope this work. In order to minimize this cost, the network will have to minimize the dot products 0r/0z 0x/02'. There are two ways to accomplish this. The first is to reduce the magnitudes of the basis vectors 0rr 2z/0=". Minimizing the magnitudes with the & norm would lead to sparser basis vectors. The second is make the basis vectors orthogonal possible so the dot product is minimized, which serves towards learning a canonical manifold as well. As detailed in [7] and the appendix. the Jacobian product, the bottleneck for both RNF and our method, can be efficiently approximated. With complexity Oii?²), it is less than (d2) when << d, where is the conjugate gradients method iterations. Thus, encouraging canonical manifold through Giøz incurs minimal extra computational cost.
One can imagine an alternative optimization schemes. For instance, the diagonal elements of the metric tensor Gka can be used to access the transformation of individual z; E Z. Minimizing the & norm E. || will encourage sparse learning, akin to a relevance vector machine [4] or network specialization [17]. Nevertheless, minimizing the diagonal elements will clearly not motivate perpendicular components and notably there is no mixing constraint on these components. Consequently, only the magnitude of the transformation can be affected.
Furthermore, one can imagine minimizing the cosine similarity between the basis vectors to promote orthogonality. However, this would not serve towards automatically determining the necessary dimensions in the latent space. Combining the l of the diagonal elements and the cosine similarity may also be a solution; however, this would bring additional weighing factor between the two
5

---

losses. E loss of the off- elements of the metric tensor brings together these two objectives elegantly.
Note that the injective nature of the flow, the arbitrary topology of the image space and the finite dimensionality of the chart imply that canonical solution may not exist for the complete learned manifold. However, this is not prohibitive as the local nature of the proposed method should allow even for isolated, sparse and independent component realizations. Even such solutions can be more preferable when representing complex multidimensional image dataset manifolds. Additionally, the canonical solution is not absolutely but rather statistically enforced, i.e. it not strict constraint. This is similar to encouraging orthogonality and specialization of gradient attributions for network regularizations [17].
To this end, combining Equation (7) with the established manifold learning flow log likelihood and reconstruction loss from Section 2. i.e., Equation 4 we arrive at the following total optimization objective of the canonical manifold learning flow loss:
a Φ- = argmax Σ
logpz g (x:)) - log |detJ (15 (x:)))
(xx))
(8)
log |et.J" " (x)) Uo ( (o.))) - 1 1;
go g (z:)) (1= - 11
- 2
where 7 is hyperparameter. In summary, for the transformation parameters Φ. the log- -igllike of NFs is maximized, while taking into consideration the rectangular nature of the transformation. Additionally, a reconstruction loss with a regularization parameter |3 is added to ensure that the learned manifold, .M C Rd, is aligned to the data manifold, M C D. The &l loss of the off-diagonal metric tensor ensures that a canonical manifold 21 is learned.
5 Experiments
We compare our method with the rectangular normalizing flow (RNF) and the original Brehmer and Cranmer manifold learning flow (MFlow). No meaningful comparison can be made to linear PCA/ICA as they are not suitable for data on non-linear manifolds. While non-linear PCA/ICA can be used, they require a specific feature extractor or kernel. Our method, a type of manifold learning. directly learns this non-linear transformation from data. We use the same architectures as 77, namely real NVP 7 without batch normalization. for the same reasons explained in the nominal paper Further experimental details can be found in the appendix. In short, we use the same hyperparameters for all models to ensure faimess. For example, we use a learning rate of 1 x 10-4 across the board, and we have only carried out simple initial searches for the novel parameter y from Equation (8) Namely, from trying y [0.001, 0.01, 0.1, 1, 10, 50), we concluded in using 7 B 1 for the low-dimensional datasets, 8 = 5 and 7 = 0.1 for the tabular datasets and image datasets to ensure stable training. 7 1 also produced good results in our experiments with different datasets, but was not always stable and long training was required. Overall, the method is not overly sensitive to the hyperparameter y. Likelihood annealing was implemented for all image training runs. Approximate training times can be found in the appendix.
and y = 0.01 for the
5.1 Simulated data
We consider some simulated datasets. Specifically, we implement uniform distributions over wwo-dimensional sphere and a Möbius band embedded in R3. Then we generate 1000 samples from each of these distributions and use the samples as the simulated data to fit both RNF and CMF. Such datasets have straightforward and visualizable canonical manifold representation. The samples generated by the trained networks are shown in Figure 2 (a) and Figure b)) for the RNF and CMF methods, respectively. From the fully sampled density plots Figure a)) and Figure (i), both methods learn the manifold and the distribution on it. However, the manifold learning of CMF is superior to RNF as the sphere is fully encapsulated, and the density is more uniform. Furthermore, when the three z; dimensions are sampled individually in Figure b) (ii) and (iv), the canonical
Batch normalization causes issues with vector-Jacobian product computations.

---

intrinsic basis learned by CMF can be seen. Specifically, two distinct perpendicular components are obtained while the third component approaches zero, i.e. reduced to something small as it is not necessary. In contrast, from Figure a (ii), and (iv) it can be seen that in RNF all three components attempt to wrap around the sphere without apparent perpendicularity. and the second component shrinks partially. Even more striking results can be seen for the Möbius band, Figure 2c) and Figure d). Additionally, the non-trivial topology of the Möbius manifold, the twist and hole, is proving harder to learn for the RNF as compared to CMF. These findings are in direct alignment with the expected canonical manifold learning, where the network motivated to learn an orthogonal basis and/or implement sparse learning, as explained in Section 4.2 For CMF and RNF methods, the log likelihoods are 1. 1.6553 vs. 1.6517 and KS p-values are 0.17 vs. 0.26. On the sphere, the log likelihoods are 1.97 vs. 1.16. CMF demonstrates better quality than RNF, both quantitatively and qualitatively.
NOO
(a) Hallow sphere learned with RNF.
(c) Moebius band learned with RNF
IM
(b) Hallow sphere learned with CMF.
(d) Moebius band learned with CMF.
Density plot for a uniform distribution on a sphere (left) and uniform distribution on a Möbius (right) band learned with RNF (top) and CMF (bottom). the black dots are the data samples, similar to Figure D RNF and CMF are learned via Equations 4 and Equation (8) (i) samples from the model using the full latent space where colors correspond to log pgg))) (ii,, (II), (iv) Samples from
Figure 2:
the model using only the first, the second and third components 21/2/3 E Z while setting the others to zero, respectively. Colors correspond to log)(x) once more. All densities have been normalized to [-1, 1] for visualization purposes.
Notably, for these simulated datasets employ full dimensional latent representations, D d. First, it allows us to visualize what all the latent dimensions are doing in relation to what we expect them to do. In particular, CMF correctly uses only 2 latent dimensions for representing 2D surfaces even though they have 3 latent dimensions. Second, it clearly shows the advantage to RNF's for the case where dimensions of the embedded manifold are known. Such scenario is equivalent to standard NF RNF is based on [33], which is an NF that already tries to solve the learning pathologies of complicated manifolds. Therefore, there is practical limitation in choosing D d, also seen empirically. Considering the above, the CMF method is showcased to outperform these previous methods, which also aligns with the theoretical intuition.
5.2 Image Data
First, Mflow, RNF and CMF are also compared using image datasets MNIST, Fashion-MNIST and Omniglot C R 781 As mentioned, the hyperparameters were set to B = 5 and 0.01 for all CMF runs, 3 3 and = for RNF and Mflow methods, and the network architectures are all equivalent. Furthermore, to emphasize the ability and importance of CMF, d. The test FID scores the trained models are shown in Table 1 We use dg= initially as 77] for consistency but also run for lower latent dimensions, 12 in order to showcase the advantages of CMF relative to the established methods. Namely, a learned canonical intrinsic basis can lead to a more efficient latent space representation and therefore advantageous performance when the latent space is limited. From Table CMF appears to outperform the other methods.
train for different latent dimensions
Then, we also experimented using the CIFAR-10 and SVHN datasets with 30072 and CelebA with 122888 Again we set B and 7 = 0.01, however, this time with latent dimensions, RRd=30 and Rd=10 The extra dimensions were necessary for the higher dimensional data. The table Table
7

---

presents the FID scores. CMF outperforms the other alternatives, consistent with the results obtained on lower dimensional imaging data sets.
In order to visualize the sparsity and orthogonalization encouraged by the CMF, the pair-wise absolute cosine similarity between the basis vectors, i.e., (Ōx (0x/0=²) [== (bottom) and the diagonal of the metric lensor (top) are visualized in Figure The CMF clearly induces more sparsity as compared to the RNF, seen from the sparse diagonal elements of the metrie tensors, see Tigure3top) for both the Fashion-MNIST and Omniglot examples. Similarly, the cosine similarity for the CMF is evidently lower for both data sets, see Figure 3 bottom); the MACS stands for the mean absolute cosine similarity. Similar results were observed for the tabular datasets, see appendix.
Additionally, in order to demonstrate the more efficient use of latent space by CMF, trained both CMF and RNF using latent-dimension d = 40 on both the MNIST and the Fashion-MNIST datasets. Then choose vary the number of prominent latent dimensions from 2 E R:10 those with the largest |Gka|- We first generate samples from the models using only the prominent dimensions by sampling from the prominent dimensions and setting all others zero. We compute FID scores for the generated samples. Later, we also reconstruct real samples using only the prominent dimensions and compute reconstruction loss using mean squared error. Here, take the dimensions, the larger G as the ones with the highest weights; this simple interpretation is used for analysis. We repeat the experiment for different number of prominent latent dimensions (1, 8, 16.......... 40) non-zero 2, E Z. In Figure 4 a) and Figure 4bb the FID score and the meea--quarrd-erra -eroor for the different number prominent dimensions are plotted respectively. From Figure 4 is evident that the CMF (solid lines) shows better results as compared with RNF (dotted lines) consistently, when using fewer prominent latent dimensions. This suggests that the CMF able to use the latent space more efficiently. Furthermore, the RNF lines exhibit some oscillation whereas the CMF lines are smoother, indicating that there is also some additional inherent importance ordering in the latent dimensions as expected from sparse learning. Samples generated with different numbers of latent dimensions can be found in the appendix.
.24
4.1
-1%
0456
2551
3.03
118
-005
-0
Fashion-MNIST (b) Fashion-MNIST (c) Omniglot (d) Omniglot
trained with RNF. trained with CMF. trained with RNF. trained with CME MACS=0.03.
MACS=0.02.
MACS=0.04.
MACS=0.03.
Figure 3: Sparsity and orthogonalization encouraged by CMF. Average diagonal elements of the metric tensor (top) and average cosine similarity between the basis vectors, (0z/0=') (bottom), with 1= Trained for different datasets with RNF and CMF methods. The sparse metric tensor indicates specialization, and cosine similarity closer to zero indicates orthogonalization. MACS stands for mean absolute cosine similarity. The top plots show the CMF method yielding a sparser transformation, while the bottom ones demonstrate its ability to learn more orthogonal transformations, evidenced further by the reduced MACS values.
5.3 Tabular data
The standard normalizing flow benchmarks of tabular datasets from S are used to compare the RNF, MFlow and CMF methods. We fit all models with latent dimension d=2, 4, 10 and 21 to POWER, GAS, HEMPMASS and MINIBOONE datasets, respectively. The experiments here are similar those 7 however, we keep the latent dimension the same for all the methods. Further details can be found in the appendix. We then computed FID-like scores, as was done in compared works [7] to compare the three models. The results are shown Table 2 CMF appears to have lower FID-like metric scores for POWER, HEMPMASS and MINIBOONE datasets. The GAS dataset
8

---

Table 1: Image datasets FID scores, lower is better.
Method
MNIST
FMNIST
OMNIGLOT
SVHN
CIFARIO CELEBA
Rd=10
Rd
MFlow
77.2
663.6
174.9
102.3
541.2
N/A
RNF
68.8
545.3
150.1
95.6
544.0
9064
CMF
63.4
529.8
147.2
88.5
532.6
9060
Rd=20
RR
Mflow
76,211
421.401
141.0
110.7
535.7
N/A
RNF
49.476
302.678
139.3
94.3
481.3
9028
CMF
49.023
297.646
135.6
72.8
444.6
8883
Fachion WIST with OMF Fushen-MMBI with 120
8000
Fashion MIIST OME
200
MIJITT with CMT
7000
HISS with RNF
6000
150
5000
100
4000
3000
50
2000
1000
30
40
10
20
30
40
effective
effective
(a) Frechet Inception Distance.
(b) Mean-squared-error.
Figure 4: FID (a) and rean-squared-errro (b) for fully trained CMF (solid line) and RNF (dotted line) methods on Fashion-MNIST and MNIST with Rd=10 The x e-axii corresponds to sampling from the models using different number of latent dimensions, while setting others The chosen latent dimensions are the most prominent ones, defined as those with the largest |Gal values.
showed mixed results in the second run set, as seen in Table The performance ordering shifted, suggesting challenges in learning this dataset, possibly leading to random outcomes. However, the other tabular datasets showed consistent performance rankings across run sets. See the appendix for further investigation into the tabular dataset training pathologies. Training on tabular datasets with doubled latent dimensions yielded no significant differences, Table
6 Limitations
Our method optimizes the learned manifold M C Rd of manifold learning flows by constraining the learning to a canonical manifold as Definition a. no significant additional computational cost as compared to RNFs (7]. A table of indicative training times can be found in the appendix. Nevertheless, the nature of likelihood-barte
training, where the full J needs to be calculated, dictates that such class of methods is computationally expensive for higher dimensions, despite the efficient approximate methods implemented here and in RNFs. Therefore, higher dimensional datasets can be prohibitively computationally expensive to train with this class of methods. Further computational efficiency advancements are expected to be achieved in future works. A notable fact is that the original Mflow method [6] relies the fundamental approximation such that the full JJ J calculation can be avoided, resulting in inferior log- likelihood calculation and therefore generations, albeit it being noticeably
computationally efficient.
Additionally, CMF can only model a manifold that is homeomorphism to Rd, and when such mapping preserves the topology. Therefore, if, for example, there is topological mismatch between

---

Table 2: Tabular data FID-like metric scores, lower is better.
METHOD
POWER
GAS (IST RUN)
GAS (2ND RUN)
HEMPMASS
MINIBOONE
MFLOW
0.258 0.045
0.219 = 0.016
0.433 + 0.071
0.741 0.052
1.650 0.105
RNF
0.074 = 0.012
0.283 = 0.031
0.470 0.101
0.628 0.046
1.622 0.121
CMF
0.053 + 0.005
0.305 + 0.059
0.373 + 0.065
0.574 + 0.044
1.508 0.082
DOUBLE LATENT DIMENSIONS
RNF
0.432 + 0.022
0.386 = 0.044 0.367 + 0.007
0.608 + 0.023 0.519 + 0.063
CMF
0.205 + 0.098
the components of the image space R2 described directly by M C Rd it could lead pathological training. This drawback, however, is shared across all manifold learning flow methods. We also acknowledge the possibility that an orthogonal basis may not always be an optimal representation and difficult to learn: enforcing strict complete orthogonality can restrict the expressivity of the transformation.
Furthermore, other families of state-of-theeare generative modeling methods working in full dimen- sional representations, for example denoising diffusion models [34,, can obtain superior FID scores overall as compared to the class of methods described in this work. However, by default they use a latent space large the image space. Notably, for any future that entails isolating prominent components, would be beneficial to perform a proper intrinsic dimensionality estimation.
Summary and Conclusions
In this work, we attempt to improve on the learning objective of manifold learning normalizing flows. Having observed that MLFs do not assign always an optimal latent space representation and motivated from compact latent space structuring [15] and sparse learning 14]. while taking advantage of the Riemannian manifold nature homeomorphic transformations of NFs, we propose an effective yet simple to implement enhancement to MLFs that does not introduce additional computational cost.
Specifically, our method attempts to learn canonical manifold, defined in Section 4.1 i.e., compact and non-degenerate intrinsic manifold basis. Canonical manifold learning is made possible by adding the optimization objective the minimization of the off-diagonal elements of the metric tensor defining this manifold. We document this learned basis for low-dimensional data in Figure P and for image data in Figure Furthermore, we showcase for image data that higher quality in generated samples stems from the preferable, compact latent representation of the data, Table
Distinctly, the off-diagonal manifold metric elements when minimized by an é loss facilitates sparse learning and/or non-strict local orthogonality. This means that the chart's s transformation can be significantly non-lincar while retaining some of the benefits of orthogonal representations. This idea can be used in other optimization schemes, which mitigates many of the drawbacks of previous methods in simple yet theoretically grounded way. Indeed, standard approaches often focus solely on the diagonal elements or strictly enforcing the off-diagonals to be zero, as in [32]. limiting expressivity. For instance, in (8), an isometric embedding, essentially predefined constrained transformation, is presented. Although it allows for direct density estimation and a form of PCA, it a strict constraint. Furthermore, as indicated by the synthetic data examples, our present method can be implemented in principle as a full dimensional flow, naturally assigning dimensions accordingly.
The learned manifold representations can be useful feature extraction procedure for downstream task. For example, improved out-of-distribution detection could be a consequence of the feature extraction. Additionally, data that require orthogonal basis vectors, like solutions of many-body- physics quantum Hamiltonian, can show improved learning performance with the current method.
Manifold learning flows are noteworthy due to their mathematical soundness, particularly in the case of methods such as RNF and CMF, which not implement any fundamental approximations the density calculations using lower dimensional latent space. We anticipate that the method will stimulate further research into the nature and importance of the latent space representation of MI.Fs, enhancing understanding of generative modelling and paving the way to general unsupervised learning.
10

---

Acknowledgement
This project was supported by grants #2018-222, #2022-274 and #2022-643 the Strategic Focus Area "Personalized Health and Related Technologies (PHRT)" of the ETH Domain (Swiss Federal Institutes Technology)
References
[1] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, pages 1530-1538. PMLR, 2015.
[2] George Papamakarios, Eric T Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57): 2021.
[3] George Papamakarios, Theo Pavlakou, and lain Murray, Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30, 2017.
[4] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv: /4:i::/440000 10. 16, 2014.
[5] Laurent Dinh, Jascha Soch--Dickttei, and Samy Bengio. Density estimation using real nvp. arXiv preprini arXiv: 1605.08803, 2016.
[6] Johann Brehmer and Kyle Cranmer. Flows for simultaneous manifold learning and density estimation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 442-453. Curran Associates, Inc., 2020.
[7] Authony L Caterini, Gabriel Loaiza-Ganem, Geoff Pleiss, and John Cunningham. Rectangular flows for manifold learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 00228-3024.. Curran Associates, Inc., 2021.
[8] Eike Cramer, Felix Rauh, Alexander Mitsos, Raúl Tempone, and Manuel Dahmen. Nonlinear isometric manifold learning for injective normalizing flows. arXiv preprint 22X1v:2203.03934, 2022.
[9] Christian Horvat and Resn-Pascal Pfister. Denoising Normalizing Flow. In Advances in Neural Information Processing Systems, volume 34, pages
Curran Associates, Inc., 2021.
[10] Danilo Jimenez Rezende, George Papamakarios, Sebastien Racaniere, Michael Albergo, Gurtej Kanwar, Phiala Shanahan, and Kyle Cranmer. Normalizing flows on tori and spheres. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 8083 8092. PMLR, July 2020.
[11] Brendan Ross and Jesse Cresswell. Tractable density estimation on learned manifolds with conformal embedding flows. M. Ranzato, A. Beygelzimer, Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, Curran Associates, Inc., 2021.
pages 26635-26641
[12] Nutan Chen, Alexej Klushyn, Francesco Ferroni, Justin Bayer, and Patrick Van Der Smagt. Learning flat latent manifolds with VAEs. In Proceedings of the 37th International Conference on Machine Learning, ICML, '20. JMI..org, 2020.
[13] Brendan Leigh Ross, Gabriel Loaiza-Ganem, Anthony Caterini, and Jesse Cresswell. Neural implicit manifold learning for topology-aware generative modelling, 2022.
[14] Michael E Tipping. Sparse Bayesian learning and the relevance vector machine. Journal of machine learning research, 10u24)211-244, 2001.

---

[15] Elena Trunz, Michael Weinmann, Sebastian Merzbach, and Reinhard Klein. Efficient structuring of the latent space for controllable data reconstruction and compression. Graphics and Visual Computing, 7:200059, 2022.
[16] David Wipf and Srikantan Nagarajan. A new view of automatic relevance determination. Advances neural information processing systems, 20, 2007.
[17] Alan Jeffares, Tennison Liu, Jonathan Crabbé, Fergus Imrie, and Mihaela van der Schaar. TANGOS: Regularizing tabular neural networks through gradient orthogonalization and special- ization. In The Eleventh International Conference on Learning Representations, 2023.
[18] Maurice S Bartlett. The use of transformations. Biometrics. Journal of the International Biometric Society, 311)399-52, 1947.
[19] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 131iv:1312.61 14, 2013.
[20] Yiping Sun, Zhaoyu Li, Jiadui Chen, Xuefeng Zhao, and Meng Tao. Variational autoencoder- based topological optimization of an anechoic coating: An efficient- and neural network-based design. Materials Today Communications, 32: 32:103901, 2022.
[21] Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7045-7054. PMLR, July 2020.
[22] Bin Dai and David Wipf. Diagnosing and enhancing VAE models. (003::9900.5500 2019.
arXiv preprint
[23] Carlos Lassance, Vincent Gripon, and Antonio Ortega. Representing deep neural networks latent space geometries with graphs. Algorithms, 14(2):39, 2021.
[24] Michael Tipping. The relevance vector machine. Advances in neural information processing systems, 12, 1999.
[25] Christopher M Bishop and Michael Tipping. Variational relevance vector machines. arXiv preprint arXiv:1301.30. 1301.3838, 2013.
[26] P Thomas Fletcher, Conglin Lu, Stephen M Pizer, and Sarang Joshi. Principal geodesic analysis for the study of nonlinear statistics of shape. IEEE transactions on medical imaging, 23(8)9955-1055, 2004.
[27] Guy Bresler, Sung Min Park, and Madalina Persu. Sparse PCA from sparse linear regression. Advances in Neural Information Processing Systems, 31, 2018.
[28] Jonatan Henríquez and Werner Kristjanpoller. combined Independent Component Analy- exchange sis-Neural Network model for forecasting exchange rate variation. Applied Soft Computing, 83:105654, 2019.
[29] Rizwan A Choudrey. Variational Methods for Bayesian Independent Component Analysis. Phl thesis, University of Oxford Oxford, UK, 2002.
[30] Isadore Manuel Singer and Jack Thorpe. Lecture notes on elementary topology and geometry.. 1967.
[31] Eike Cramer, Alexander Mitsos, Raúl Tempone, and Manuel Dahmen. Principal component density estimation for scenario generation using normalizing flows. Data-Centric Engineering, 3. 2022.
[32] Edmond Cunningham, Adam D Cobb, and Susmit Jha. Principal component flows. In Interna- tional Conference on Machine Learning, pages 4492-4519. PMLR, 2022.
[33] Rob Cornish, Anthony Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. In International Conference on Machine Learning, pages 2133-2143. PMLR, 2020.
12

---

[34] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In
H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Associates, Inc., 2020.
[35] Kaare Brandt Petersen, Michael Syskind Pedersen, et al. The matrix cookbook. Technical University of Denmark, 7(15):510, 2008.
[36] M.F. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics Simulation and Computation, 19(2):333 450, 1990.
[37] Stephen Wright, Jorge Nocedal, et al. Numerical optimization. Springer Science, 35(67-68)77. 1999.
13

---

A Riemannian geometry
The Latin indices run over the spatial dimensions and Einstein summation convection used for repeated indices.
A D dimensional curved space is represented by Riemannian manifold M. which is locally described by a smooth diffeomorphism h, called the chart. The set of tangential vectors attached to each point y on the manifold is called the tangent space M. In the fluid model, all the vector quantities represented as elements of Ty M. The derivatives of the chart h are used to define the standard basis 8h
(e1, ... CD) 3x
2h
The metric tensor g can be used to measure the length of a vector or the angle between two vectors. In local coordinates, the components of the metric tensor are given by
8h
0h
9iy (5)(2) (x)) -e()))
dzi
dxi
where is the standard Euclidean scalar product.
For a given metric tensor, the vector v = v'e, E TyM has a norm ||e|| = Vengi giyed and a corresponding dual vector r* = w'e, E M in the cotangent space, which is spanned by the differential I-forms dæ' .). The coefficients e; of the dual vector are typically denoted by lower index and are related to the upper-indes coefficients o² by contraction with the metric tensor مıyud or equivalently, v² = ge where gdd denotes the inverse of the metric tensor. The upper-index coefficients v² of a vector e are typically called contravariant components, whereas the lower-index coefficients v of the dual vectors v² are known as the covariant components.
necessary feature for the description of objects moving on the manifold is parallel transport of vectors along the manifold. The tangent space is equipped with covariant derivative V (Levi-Civita connection), which connects the tangent spaces at different points on the manifold and thus allows for the transportation of a tangent vector from one tangent space the other along a given curve 7t). The covariant derivative can be viewed as the orthogonal projection of the Euclidean derivative onto the tangent space, such that the tangency of the vectors is preserved during the transport. In local coordinates, the covariant derivative is fully characterized by its connection coefficients Ti (Christoffel symbols), which are defined by the action of the covariant derivative on the basis vector, 2h V,e,- = D2x In the standard basis, e; = Dztt the Christoffel symbols are related to the metric by
1 (0, + &rga - Ōrggk). r jk 2
(10)
Acting on general vector v = v'ei, the covariant derivative becomes:
V (0gv² + 1
(11)
where the product rule has been applied, using that the covariant derivative acts as normal derivative on the scalar functions vf. Extended to tensors higher rank, for example the second order tensors T T 7 it becomes
V = ( + T5 + r T" Øe
(12)
; in this work, the basis vectors e; are generally dropped. Compatibility of the covariant derivative with the metric tensor implies that Vxg = V ABii = 0. This property allows us to commute the covariant derivative with the metric tensor for the raising or lowering of tensor indices in derivative expressions.
The motion of the particle can be described by the curve (t), which parameterizes the position of the particle at time f. The geodesic equation, V = 0, in local coordinates (t)) = (t)ee is defined by
7 2o O.
(13)
The geodesic equation can be interpreted as the generalization of Newtons law of inertia to curved space. The solutions of Eq. (13) represent lines of constant kinetic energy on the manifold, i.e. the geodesics. The Riemann curvature tensor R can be used to measure curvature, or more precisely, it measures curvature- inducedd change of tangent vector v when transported along closed loop.
R(e,, e
V.V V, V.v.
(14)
14

---

In local coordinate basis e;, the coefficients of the Riemann curvature tensor are given by
Rigk g(R(e, ey)ek, e1) =0,r'x - OuTi + 1 jm
(15)
G ij. Contraction of Ri to a rank and tensor yields the Ricci-tensor Raj = Ri and the Ricci-scalar R - n"و Raj respectively, which can also be used to quantify curvature.
(16)
The gradient is defined as / = ك the divergence as Viu* 0i ((( / and the integration over curved volume as V = fy QdV, where dV = /gdæ' ...dr =: vgd"x x denotes the volume element. /g denotes the square root of the determinant of the metric tensor.
B Extension to Related Work and further discussions
Normalizing flows have direct access to the latent distribution due to homeomorphism defining the transformation [ Nevertheless, the very nature of the homeomorphism can result in pathological behaviour when manifolds with complex topology are learned. For instance, a two-dimensional data space px can have different numbers and types of connected components, "holes" and "knots', and images have complicated shapes and topology due to the sheer number of dimensions. Therefore, usually pz is modelled by simple distribution, like Gaussian, it cannot trivially match the topology of pxr [33] demonstrates a stable solution to this topological pathology of NFs, albeit with R² support.
Effectively learning manifolds with complicated topologies remains challenge, even with the full-dimensional latent space. However, the RNF is based on CIF [33 which is an NF that already tries to solve the learning pathologies of complicated manifolds: "CIFs are not subject to the same topological limitations as normalizing flows" (33] Furthermore, in that regard, we estimate there is no practical limitation setting, as also seen empirically. In principle, CMF can solution for higher-dimensional data. However, in practice it is very computationally expensive to train full latent dimension (e.g. J"" calculation) and the solution will take long to converge. Additionally, empirical knowledge suggests that diver--dmensional representation discussed in the context of Mflows. Consequently, in practice using the current method, one can start from predefined lower dimension and let the network optimize at will.
enhance expressivity, as
PCAflow [32] relies on a lower-bound estimation of the probability density to bypass the J calculation. This bound is tight when complete principal component flow, as defined by them, is achieved. The fundamental distinction here is that CMF does not confine itself to a pure PCAflow scenario, which has the potential to restrict expressivity. Since our method only loosely enforces orthogonality, or mutual information - O, we anticipate the contour lines lie somewhere between NF and PCAflow Furthermore, the introduction of sparsity adds an extra layer of complexity, making direct comparison challenging. As an outlook, we acknowledge that DNF [9] and PCAflow, can undergo quantitative comparison with our method. However, we estimate that RNF is precisely in alignment with CMF, providing a more representative comparison.
Generally, the meaning of "canonical manifold" can vary depending on the context in which it is used, therefore we use Definition to precisely define the term "canonical manifold", Definition [ aligns closely with the assumption of PCAflow. However, it is vital to note that our method does not enforce this strictly. To put it in perspective, our approach seeks a 'partly canonical manifold, you will. Additionally, our method encompasses mparsity-meaiing the diagonal elements can also approach zero.
In principle, full-dimensional flow d = D can be also solution for nigher-dimensione datasets, as the CMF loss will ensure that the number prominent latent dimensions should correspond the true intrinsic dimensionality of the data. However, in practice it is very computationally expensive to train full latent dimension (e.g. JJ'J calculation) and the solution will take long to converge. Additionally, empirical knowledge suggests that fower-dimensions representation can enhance expressivity, as discussed the context of Mflows. Consequently, practice, one can start from a predefined lower-dim and let the network optimize at will. Nevertheless, the efficient approximate methods indicate that CMF could be implemented for a full-dimensional flow, Further exploration of this potential could be undertaken in future studies.
15

---

C FID score
For a dataset f. Fnd C RD and sampled generated data {a. FnF C R. for statistic F the mean and variance are:
1
Σ= n n-
1
Σ (F(()) 11)
(17)
T
Σ F F(t:),
Σ= n- Σ
p)".
(18)
The FID score takes as the last hidden layer of a pre-trained network and evaluates generated sample quality by comparing generated moments against data moments. This is achieved by calculating the squared Wasserstein-2 distance between Gaussian and corresponding moments by:
|| ~ll + tr(E - Σ 2(E
(19)
This would reach zero if they match. For tabular data, F is taken to be the identity F(x) = 2, as in
D Optimizing the objective and calculating the metric tensor
The third term of canonical manifold learning optimization objective, (Equation 8):
logpz (5 (x;)) -log IdetJ n. (g (2;)))
argmax
- log /dttJ T fo ( (zx)) (z.))) 3 ||o, 96 (5o (o:)) (0=3( -
(x;))
is not trivial to backpropagate through. Here the method suggested by 77. First, J Ja calculated efficiently with the aawaadd-Bakwward automatic differentiation trick as explained in Section 4.3 of 77). For the simulated low-dimensional datasets the exact method is used, and for images the approximate stochastic gradient method with K = used. Second, to back- pack-propagate for the approximate method, stochastic unbias estimator is used. Namely, from matrix calculus [35] is known that:
logdet.J Jg = tr((J] Jo) 08;
JJJJ Jel.
(20)
00,
where tr is the trace and 0 is the j-th component. Then the Hutchinson trace estimator is used 336
i
1
ô (o J Jock.
logdet Jo
(21)
00,
00,
where c1,... Gk are sampled from a Gaussian distribution. Access to JJ. Jo as explained above allows for the direct computation of Equation 21). Nevertheless, the first term of (Jī Jo) = [(]] -tejT calculated by an iterative method solver of the matrix inverse problem, the conjugate gradient method CG (37].
i
1 CG O J Joes. 003
JfgdetJ of Jo
(22)
002
K
Lastly, the metric tensor is obtained from Jf Jo stated in Section 2.2 Back-propagation of |[Gy | is straight forward by taking the derivative w.r.t. by an (F.g. Adams) optimizer, similar to the reconstruction loss in Equation (
16

---

-12
-04
HEMPMASS (b) HEMPMASS (c) Miniboone (d) Miniboone
trained with RNF trained with CMF. trained with RNF. trained with CME MACS=0.33.
MACS=0.30.
MACS=0.43.
MACS-0.25.
Figure 5: Sparsity and orthogonalization encouraged by CMF. Average diagonal elements of the metric tensor (top) and average cosine similarity between the basis vectors, (0z/0z')/ (bottom), with Rd=10 Trained for different tabular datasets with RNF and CMF methods. The sparse metric tensor indicates specialization, and cosine similarity closer indicates orthogonalization. MACS stands for mean absolute cosine similarity.
E Tabular datasets
we visualize metric tensor for the tabular datasets trained with sufficiently high latent dimensions and present the mean absolute cosine similarity (MACS) in the supplementary PDF. For the specific case GAS with d=2, the plot might not offer significant insights. We observe that the mean absolute cosine similarity is lower for CMF, although, considering the error, there remains some overlap, which is within expectations.
Figure 5
In order to investigate the cause for the performance gap in GAS, we visualize the validation FiD- scores during training, Fi Upon comparing the results of HEMPMASS and GAS, it is apparent that learning converges until a certain point, after which instability may arise for all three methods. Despite this instability, is important to note that our implementation selects the best validation step checkpoint for testing, mitigating the impact of training instability. However, it is plausible that these instabilities contribute to the variations between different runs. Furthermore, the training curves follow similar behaviors, converging up certain point and then displaying degree of instability. This trend holds true for all three methods when trained on the GAS dataset.
(b) HEMPMASS trained with CMF and RNF
(a) GAS trained with CMF and RNF,
Figure FID-like validation score training curves. Different colors correspond to different repetitions with both methods, CMF and RNF.
F Image generations
F.1
Samples from prominent dimensions
In order to demonstrate the more efficient use of latent space by CMF, we trained both CMF and RNF using latent-dimension = 40 on both MNIST and MaSHI datasets. We choose a varying number of prominent latent dimensions from 2 E R:40 as those with the largest [G this is mathematical intuition that the largest eigenvalues (if an cigen-decomposition is possible) correspond the dimensions with the highest weights.
We generate samples from the models using only the prominent dimensions, by sampling from the prominent dimensions and setting all others to zero, subsequently FID scores are calculated for the
17

---

generated samples. Additionally, we reconstruct real samples using only the prominent dimensions and compute reconstruction loss using mean squared error, shown the main text. We repeat the experiment for different number of prominent latent dimensions (1, 8, 16..... 40), or similarly, non-zero 2; E 2. Here, in the appendix, we show samples when the number of prominent dimensions sampled from increases.
We would like note that potentially for any future use that entails isolating prominent components, a proper ID estimation should be carried out.
For example, in Figure 7 different samples generated along the horizontal. This is repeated for increasing number of prominent dimensions used for sampling, top to bottom along the vertical (4, 8, 12, 16 40). In Figure 7 FaShionMNIST models were trained with R==40 and for the CMF and RNF methods. Similarly to the FID plots the main text, the CMF shows a bit less variation when going along the columns from top bottom; this is indication that fewer prominent dimensions are needed for a faithful generation.
(a)
(b)
Figure 7: Generated samples for increasing prominent dimensions used for sampling from top to bottom along the columns. The model is trained on MNNSST with Rd=110, (a) RNF, (b) CMF Rows represent different samples. The CMF captures the generations almost fully, even with the first prominent dimensions.
To visualize the sparsity of learned manifold, we generate samples from individual latent space dimensions for fully trained models on FashionMNIST with the RNF and CMF. In Figure X the trained RNF and CMF models exhibit FIDs of ~ 302 and 297 respectively, see main text. Across the rows, different samples can be found as sampled from each of 10 prominent component subgroups. i.e. ([1, 2}, (3, 4), (5, ...... [19, 20}) out of the 20 latent dimensions. Here, only these subgroups ... are used for sampling at once and the other dimensions are set to zero. The dimensions sampled from at the bottom are the most significant ones; the rest follow in order towards the top. It can be clearly seen that the CMF method manages to capture most of the information of the manifold the first two dimension, as the generations in the bottom row good and the rest of the columns seem to have less relevance, indicating strong sparse learning. For the RNF, again there is an importance distinction, as expected, but even dimensions of less prominence seem to be generating variation in the samples. Combining these observations with the higher FID the CMF, efficient sparse learning is evident.
In Figure 9 we plot a sampled manifold between -3 and for two most prominent latent dimensions from 2 E 20 as those have the largest [GEa]. This is done for an RNF and CMF trained MNIST model on the left and right of Figure The CMF method appears to split the latent space of 21 without much variation in the 22 direction, indicating perpendicularity.
18

---

(a)
Figure 8: Generated samples for 10 different hierarchical subgroups of prominent dimensions
([1,2), (3,4), (5, 6} (19, 20}) along the columns from bottom to top, while setting all other dimensions to zero. Models were trained on Fashion-MNIST with IRd= with (a) RNF, (b) CMF Rows represent different samples. Sparse learning is evident for the CMF method.
8
8
8
8
9
7
7
7
8
8
7
8
8
8
8
9
9
7
7
8
7
8
8
8
8
8
8
7
7
0
8
/
8
8
8
8
8
8
8
8
/
8
8
8
8
8
8
8
3
8
8
8
8
8
8
1
8
2
2
8
&
2
2
2
1
O
8
/
2
2
2
2
2
2
2
2
O
-3
-2
-1
i
(a)
(b)
Figure 9: Sampled manifold for the two most prominent dimensions as calculated from GER. (a) RNF,
(b) CMF.
F.2 Out of distribution detection
We evaluate the performance of CMF for out-of-distribution detection (OoD). Table 3 quantifies the results, where accuracy here is the log-likelihood. The results are similar for the previous methods as expected.
G Experimental details
The training was performed GPU cluster with various GPU nodes including Nvidia GTX 1080, Nvidia GTX 1080 Ti, Nvidia Tesla V100, Nvidia RTX 2080, Nvidia Titan RTX, Nvidia Quadro RTX 6000, Nvidia RTX 3090. Multiple GPUs were used in parallel across the batch.
19

---

Table Decision stump for OoD accuracy (higher better)
METHOD
MNIST -> FMNIST
FMNIST
MNIST
M-FU M-FLOW
94 %
84 %
RNF
96 %
78 %
CMF
97 %
77 %
FMNIST
700
MNIST
decision boundary
600
# of samples
500
300
200
0
-200
-175
-150
-125
-100
-75
p(x)
Figure 10: OoD dection with CMF, trained on Fashion-MNIST.
There exists substantial computational cost distinction between the RNF and CMF methods, as both encounter the calculation bottleneck the secobianttranspose-aasoe
Any other option was set to the default found in (7). The approximate training times be found in Table Currently, manifold learning flows are computationally demanding; nevertheless, their potential is noteworthy due their mathematical soundness, particularly in the case of methods such as RNF and CMF, which do not implement any fundamental approximations in the density calculations using lower dimensional latent space.
Table 4: Approximate training times for 2d=10/20 D and H indicate days and hours respectively. MNIST and FMNIST required around 120 epochs, SVHN, around 350, CelebA around 220, CIFARIO around 500 epochs of training.
МETHOD
SVHN
CIFARI0
CELEBA
MNIST
FMNIST
M-FLOW
N/A
N/A
N/A
1D
20H
RNF
12D
20D
28D
3D
2D 18H
CMF
14D
21D
27D
3D 4H
17H
G.1 Images
To train for images, 4 or 5 nodes of less than 10GB each were used in parallel across the batch size. The hyperparameters were kept constant for the three different methods compared, unless distinction made for the manifold learning flow as explained in 77.
20

---

Parameter
Setting
Learning rate
X 10
Optimizer
Adams
Likelihood annealing
between epochs 25 and 50
Early stopping
MNIST, FashionMNIST, Omniglot: min 121 epochs SVHN, CclebA, CIFARIO: min 201 epochs
Batch size
120
Latent dimensions
(5, 10, 20, 30, 40)
Hyperparameters
Equation 8 : 3 = y = 0.1,0.07
Caleulation method
lacobian-transpose waansttrsasspssooooaaaae Jacobian:: Hutchinson, Gaussian, K=1
CG tolerance
0.001
D-dim flow coupler
8x64
d-dim flow layers
10
G.2 Simulated data
The parameter choice is also kept consistent for training for simulated data. Training was performed on a single GPU CPU and training times were observed to be 1-4 hours depending on epochs and if early stopping was used. Again, similar training times between RNF and CMF were observed.
Parameter
Setting
Learning rate
x 10
Optimizer
Adams
Likelihood annealing
False
Epochs
5000 (without early stopping)
Batch size
1000
Latent dimensions
[2,3]
Hyperparameters
Equation 3 = 1
Calculation method
Jacobian- secobian-rransoose-aaat Jaanspos--Jacoiin: 'exact'
D-dim flow coupler
RealNVP
d-dim flow layers
G.3
Tabular data
We train for the tabular datasets found in Papamakrios al. chosen as follows, as per RNF.
3
Parameters and settings were
Parameter
Setting
Learning rate
1 x 10
Optimizer
Adams
Likelihood annealing
between epochs 25 and 50
Early stopping
Yes
Batch size
POWER: 5000, GAS: 2500, HEMPMASS: 750 .MINIBOONE: 400
Hyperparameters
Equation 8 5, 7 0.01
Calculation method
Jacobian aecobian-transpose Jacobiann Hutchinson, Gaussian, K=1
CG tolerance
0.001
D-dim flow coupler
4x 128 with 10 layers
d-dim flow Latent dimension
RealNVP 2x32 with 5 layers
d-D/2
For each run set, we performed repetitions, of which we report an average FID score and a standard error in Table 2 We train models with latent dimension d=2, 4, 10 and 21 POWER, GAS, HEMPMASS and MINIBOONE datasets, respectively.
21


Reviews:
[{"review_id": "h5KRG5rGaZ", "reviewer": "Reviewer_tk9x", "summary": "This paper studies the problem of learning a latent representation for data supported on a low-dimensional manifold. It proposes to promote orthogonality of the tangent vectors arising from a learned chart, on top of existing rectangular flow loss. Experiments are provided to demonstrate the effectiveness of the algorithm.", "questions": "## Significance\nLooks like the inverse of g_phi is in objective (6), so using line 75 it boils down to the inverses of g_eta and f_theta. How easy is it to invert g_eta and f_theta? The root of this question is, if one takes f_theta to be a square matrix, then inverting the square matrix is needed, leading to high cost if the dimension of the square matrix is large. If f_theta is a square matrix followed by say ReLU, then ReLU is not invertible.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "Promoting orthogonality of the tangent vectors from a learned chart is interesting.", "weaknesses": "## Presentation:\n1. The paper seems insufficiently prepared and proofread. In particular, there are numerous evident typos even in the first paragraph of the introduction. Consequently, they weaken the credibility of the paper.\n    1. Lines 29, 33 the \"D\" in \"R^D\" uses a mathbb, whereas in lines 31, 32 the \"d\" or \"D\" does not.\n    1. Line 29 “fulfil” -> “fulfill”\n    1. In equation 1, the q_phi(x) should have been q_phi^{-1} (x)\n    1. Section 3 talks about general manifold learning, and is almost detached from the technical sections 2 and 4.\n2. Line 43: A line is one-dimensional. What is a two-dimension line?\n3. If one compares equations (5) and (6), it appears that we assume the learned map q_phi is a chart of manifold M. If that is the case, this should be made explicit.\n4. From (8), it is unclear that where the introduced G term are evaluated. I suppose it is on points x_i’s?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's suggestions. While we acknowledge that different presentation approaches could have been explored, we believed that, given space constraints, the current explanation best encapsulated the work.  We are encouraged by the fact that the presentation is also accepted by the other reviewers. We apologize for any typos or language errors and commit to rectifying them diligently in the revised paper.\n \n 1.1 Indeed, there was bracket typo in latex there.\n\n 1.2 In British English it is indeed \"fulfil\", but of course the convention is American English, we will fix this.\n\n 1.3 Apologies, for the typo.\n\n 1.4 Our method is based on manifold learning flows; consequently, the first paragraph is dedicated to that. Furthermore, the rest of the section, and indeed the majority of it, addresses the relevant work that inspires our method or explores similar themes. This includes the Relevance Vector Machine, which motivates sparse learning, PCA and ICA methods that emphasize orthogonality, tabular neural networks which implement a similar idea to ours to constrain a specific NN, and PCA flow which explores orthogonal contour learning for flows.\n\n 2. Two-dimension here refers to the length and width of this 'fuzzy line'. Indeed, a line is a 1D object but the \"fuzzy line\" we use here has 2D noise added to the line. \n\n 3. Line 172, just above equation 6 we explicitly specify 'the transformation of the chart'\n\n 4. This is correct, G is depended on x and the parameters of the network, we understand this is explicitly defined by equations (6) and (7), of course we can include this in G or explicitly specify that the dependencies are dropped for brevity. \n\n\n This is indeed a fundamental property of invertible flows, and it has ultimately been tackled through various methods, such as 'coupling layers' [1], in the last 5-10 years. Additionally, the inconsistency with the activation function is an acknowledged limitation of such approaches that has already been addressed over these years [2]. For instance, when the ReLU activation is utilized in a normalizing flow, it is typically combined with an element-wise scale and shift transformation (affine transformation) to ensure invertibility. The original flow part of our work builds upon these prior contributions and more recent developments [3].\n\n[1] Laurent Dinh, David Krueger, Yoshua Bengio NICE: Non-linear Independent Components Estimation\n\n[2] Real NVP: Density Estimation and Inverse Problems Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio\n\n[3] Variational Inference with Continuously-Indexed Normalizing Flows Anthony Caterini, Rob Cornish, Dino Sejdinovic, Arnaud Doucet\n"}, {"review_id": "cP4g4Mb6K3", "reviewer": "Reviewer_7TiS", "summary": "The authors introduce a new method for regularising manifold learning flows. Essentially, it attempts to reduce the entanglement between the dimensions of the learned manifold by encouraging non-diagonal elements of the metric tensor to be small. Leveraging the already necessary computation of $J^\\top J$, where $J$ is the Jacobian of the flow transformation, this is done efficiently by minimising the $\\ell-1$ norm of the non-diagonal entries.\n\nExperiments on synthetic and real data show how this is effectively achieved, especially when compared to similar models that do not employ the regularisation scheme. Beyond achieving the desired effect, the proposed method achieves lower FID-like scores on real tabular data.", "questions": "- Tied to the aforementioned weakness, what do the authors posit is the cause for the performance gap in GAS?\n- Related to the previous question: are the metric tensors found in tabular data also behaving similarly to the ones shown in Figure 3?\n- Is it clear during training when a pathological scenario is reached? How does it behave w.r.t. increasing/decreasing the value of $d$?\n- Are the learned manifold representations also useful as a feature extraction procedure for downstream tasks? Could this be also a scenario for evaluating the quality of the learned models? (As opposed to only raw performance in scores)\n\nMinor comments:\n- line 35: Caterini et al [7] has => Caterini et al. [7] have\n- line 44: Please avoid unnecessary adjectives such as \"tedious\"\n- line 62: why is the set starting with $x_0$?", "limitations": "I was pleasantly surprised with the discussion on limitations in the paper. I believe the authors addressed important issues and even highlighted which other approaches could have better raw performance.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written and the presented ideas are easy to follow. Experimentally, the authors confirm both in intuitive, synthetic examples and real data. The proposed regularisation scheme is also simple and computationally efficient, which is also a desirable feature. Overall, I consider it a strong contribution and see high potential of being widely adopted as a mechanism for regularising manifold learning flows.\n\nI also appreciate the limitations raised and discussed by the authors, indicating maturity in their analysis and raising important considerations related to the use of their work.", "weaknesses": "Although I see the potential mentioned above, I do consider it could be viewed more widely as a weak point, given the relatively niche application. Although it does not affect the merit of the work, the principle might be too specific to manifold learning flows.\n\nI would consider more tabular data experiments are needed, with additional data sets, given the mixed results attained. More concretely, some analysis of what specifically differentiates the data sets enough to cause the difference in performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our gratitude to the reviewer for their precise summary.\n\nW1: We thank the reviewer for the insightful remarks. Indeed, we have developed the method in the context of manifold learning flows in order to solve an existing pathology as well as dis-entagling the latent space. While being specific at the moment, it is our estimate that there is a prototypical idea and thus of some theoretical value as well. Specifically, we propose and show that the off-diagonal manifold metric elements when minimised by an L1 loss allow simultaneous sparse and/or orthogonal basis. This idea can be used in other optimization schemes. \n\nW2: We thank the reviewer for the suggestion. We carried out one more tabular data experiment, experiments with twice the size of latent dimensions and repetitions of the experiments for understanding. We show the results in the supplementary PDF. See also Q1 for discussion on these and their training curves.\n\nQ1: We appreciate the reviewer's suggestion. As indicated in the manuscript, we calculate the average best FID score from an average of 5 simulations for each tabular dataset. In response to your suggestion, we conducted an additional GAS tabular experiment and present the results in an updated FID score table for tabular datasets, accessible in the supplementary PDF. Notably, the performance ordering has shifted, indicating possible inherent challenges in learning the dataset, which might introduce randomness in the outcome. However, it's worth mentioning that all other tabular datasets maintained consistent performance orderings across methods, so we haven't included them here.\n\nTo delve deeper into this matter, we visualize the validation FID-like scores during training, available in the supplementary PDF. Upon comparing the results of Hempmass and GAS, it's apparent that learning converges until a certain point, after which instability may arise for all three methods. Despite this instability, it's important to note that our implementation selects the best validation step checkpoint for testing, mitigating the impact of training instability. However, it's plausible that these instabilities contribute to the variations between different runs. Furthermore, following your suggestion, we have introduced another tabular dataset, namely Miniboone.\n\nQ2: Indeed, they behave similarly. We visualize these  for the tabular datasets trained with sufficiently high latent dimensions and present the mean absolute cosine similarity (MACS) in the supplementary PDF. For the specific case of GAS with d=2, the plot might not offer significant insights. However, we  report here the MACS for RNF as $3.16 \\pm 0.7$ and for CMF as $2.16 \\pm 0.2$. We observe that the MACS is lower for CMF, although, considering the error, there remains some overlap, which is within expectations.\n\nQ3: We conducted training for the tabular datasets with twice the number of their respective latent dimensions, as compared to the original,  see Appendix. The outcomes are available in the provided PDF. Notably, we haven't observed any substantial differences. Furthermore, the training curves follow similar behaviours, converging up to a certain point and then displaying a degree of instability. This trend holds true for all three methods when trained on the GAS dataset.\n\nQ4: That is part of the main outlook of this work. For example, improved out-of-distribution detection could be a consequence of the feature extraction, some preliminary but promising results have been included in the supplementary material of this work. Additionally, data that require orthogonal basis vectors, like solutions of a many-body-physics quantum Hamiltonian, can show improved learning performance with the current method.  \n\nWe acknowledge the minor corrections and will incorporate them into the revised version of the manuscript. The reviewer is correct, the typo $x_0$ should be rectified, and the set should commence with $x_1$ for consistency.\n"}, {"review_id": "rrKUCtXjZS", "reviewer": "Reviewer_8w9Q", "summary": "The paper studies the current manifold learning methods. It compares canonical manifold learning flow (CMF) with other manifold learning methods and demonstrates that CMF can learn the orthogonal features existing in data. From synthetic data on Moebius, the paper shows the benefits of using CMF.  Also, the paper shows the generated images from the CMF learned latent space is of higher quality than other methods' learned latent spaces. Lastly, the paper acknowledges some limitations of CMF, for example, high computation cost of the full Jacobian tranpose Jacobian.", "questions": "1. What is the complexity of the RNF and CMF methods?\n2. For table 2, are there explanations on why M-flow works the best and CMF is the worst on GAS?\n3. Figure 3 is hard to interpret. What are the advantages of CMF in this plot?\n4. What are the generative models used in the image experiments?", "limitations": "The authors have discussed the limitations clearly in the paper. They acknowledge the high computation costs and the limitation that the method only applies when the manifold is homogeneous to Rd space.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is very clear on the advantages of CMF against other manifold learning methods. \n2. The paper did various types of experiments to showcase the CMF's performance among the methods.", "weaknesses": "1. Although being mentioned in the limitations, the paper does not provide results about the computation costs of CMF. There is no comparison of the computation time of CMF against other methods, also how the dimension affects the computation cost. It is unclear if CMF is scalable to higher-dimensional datasets.\n2. The paper does not compare CMF with a lot of methods, including PCA and ICA or purely deep learning methods. For example, an autoencoder can also extract feature representations from data. How does CMF compare to these approaches?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: We appreciate the reviewer for highlighting this matter. A comparative table can be found in Appendix 6 of the supplementary material. To summarize, there exists no substantial computational cost distinction between the RNF and CMF methods, as both encounter the calculation bottleneck of the Jacobian-transpose-Jacobian. Additionally, results from an initial CelebA 64x64 training have been included in the supplementary response PDF, indicating that CMF is scalable to higher-dim datasets with promising results. Currently,  manifold learning flows are computationally demanding; nevertheless, their potential is noteworthy due to their mathematical soundness, particularly in the case of methods such as RNF and CMF, which do not implement any fundamental approximations in the density calculations using lower dimensional latent space. \n\nW2: We believe this question is rather related to general manifold flow literature and not specific to our work. Linear PCA / ICA would fail when you apply them to data living on non-linear manifolds, for instance those shown in the toy examples. Non-linear PCA or non-linear ICA may be applied but one needs to identify the feature extractor or the kernel. Manifold learning methods come to aid at this point. They learn the non-linear transformation from the data. The method presented here falls in this category. Regarding comparing with AutoEncoders, AE's would not learn data distributions, e.g., they cannot be used to sample or compute likelihoods. Manifold flow learning techniques, to which our method belongs to, fall in the category of methods that approximate distributions. Furthermore, manifold learning flow methods are relatively new and in that regard there are only a couple of prominent works.  \n\nQ1: The bottleneck for both methods is the calculation of the Jacobian-transpose-Jacobian, as explained in Appendix 4 and [1]. The complexity of the approximation used is $\\mathcal{O}(id^2) < \\mathcal{O}(d^3) $ if $i << d$, $i$ refers to iterations of the iterative conjugate gradients method and d to the latent dimension.\n\nQ2: We appreciate the reviewer's comment. As indicated in the manuscript, we calculate the average best FID score from an average of 5 simulations for each tabular dataset.  We  have conducted an additional GAS tabular experiment and present the results in an updated FID score table for tabular datasets, accessible in the supplementary PDF. Notably, the performance ordering has shifted, indicating possible inherent challenges in learning the dataset, which might introduce randomness in the outcome. However, it's worth mentioning that all other tabular datasets maintained consistent performance orderings across methods, so we haven't included them here.\n\nTo delve deeper into this matter, we visualize the validation FID-like scores during training, available in the supplementary PDF. Upon comparing the results of Hempmass and GAS, it's apparent that learning converges until a certain point, after which instability may arise for all three methods. Despite this instability, it's important to note that our implementation selects the best validation step checkpoint for testing, mitigating the impact of training instability. However, it's plausible that these instabilities contribute to the variations between different runs. Furthermore, we have introduced another tabular dataset, namely Miniboone,  as well as we run experiments with double the latent dimensions as compared to the original experiments, all with promising results.\n\n\nQ3: Figure 3 demonstrates the realization of sparse and partially orthogonal learning, as per hypothesis of the proposed method, when compared with the original approach. Additionally, when these findings are coupled with the better FID scores for the image datasets and the outcomes from the simulated data, it can be deduced that CMF is opting for a more ``efficient'' use of the latent space, in other words enforcing orthogonality between dimensions and not putting any weight on unnecessary dimensions. We acknowledge that visualizing high-dimensional data always presents a challenge.\n\n\nQ4: We wish to clarify that the CMF framework functions as an independent generative model. It approximates the data distribution with a low dimensional latent space. The learned manifold pertains to the lower-dimensional representation of the data, which is subsequently mapped back into the image space, allowing direct sampling from the latent space for image generation. The essence of this work is to demonstrate that by acquiring a more effective latent representation, we achieve a more successful generative model than its precursors, M-flow and RNF! \n\n[1] Rectangular Flows for Manifold Learning Anthony L. Caterini, Gabriel Loaiza-Ganem, Geoff Pleiss, John P. Cunningham"}, {"review_id": "zihRWaNRc3", "reviewer": "Reviewer_iKjw", "summary": "The authors address the interesting question of how to disentangle the relevant manifold directions properly in the latent space of manifold learning normalizing flows (MLF). After a comprehensive theoretical background, the main motivation is presented in form of a toy example of a simple noisy line embedded in $\\mathbb{R}^{2}$. When varying the latent dimension $z_{i}$ of a RNF and setting the remaining latent variables to $0$, the corresponding contour lines are partially aligned. Doing the same with the proposed method, CMF, leads to orthogonal contour lines and thus to a meaningful latent representation where one latent variable encodes the line and the other the (noisy) off-manifold direction. This intuition leads straightforwardly to enforcing the orthogonality of the flows Jacobian matrix. To do so, the authors propose a new cost function for learning manifold structures using an NF by adding a penalty term which ensures that the off-diagonal entries of the Gram matrix (Jacobian-transpose-Jacobian) are $0$ at the one hand, and are sparse at the other due to the l1-norm. The effecitiveness of the method is tested in experiments on toy-data, tabular data and 32x32x3 image data.\n", "questions": "1. Does Definition 4.1 really makes sense (see above)\n2. Is the comparison with RNF in Figure 2 fair (see above)? By definition of RNF, it must be $d<D$. Please elaborate.\n3. Why not always set $d=D$ and then apply the proposed penalty term (see above)?\n4. Figure 1. Why is the density varying greatly when only varying one dimension? Shouldn't be uniform if the density is properly learned?\n5. What is the quality in terms of density estimation? For instance, what is the KS statistics compared to the true density, or, at least, does it sum to 1?\n6. Why not benchmark with PCA flow? It would be interesting to see if this method can overcome some of its limitations (see above).\n7. Moebius band with RNF seems to be poorly learned. Why?\n8. Why prominent latent dimensions are the ones with the greatest $G_{ii}$? Is this a mathematical statement/proposition or intuition?\n9. line 305: in which case would an orthogonal basis not be an optimal representation?\n10. What happens if we use the L2 rather than L1 norm?\n11. How are prominent latent dimensions defined? Do they correspond to the true intrinsic dimensionality? Is there a clear cut-off in the magnitudes of $|G_{kk}|$ in e.g. MNIST? If not, a somewhat arbitrary threshold needs to be defined, isn't it?\n\n*Minor*:\n+ line 70-72: the statement is only true for large $n$\n+ line 83: What is $\\mathcal{M}_{\\theta}$?\n+ line 83: \"to encorage $x\\in \\mathcal{M}_{\\theta}$\" do you mean $g_{\\phi}(z)$ rather than $x$?\n+ line 114: typo: \"due to lack...\"\n+ line 145: noise is not perpendicular to line\n+ line 217: mention reason (maybe in a footnote)\n+ line 225: $\\gamma$ not crucial but had an impact on training time? What is crucial for you?\n+ line 244: bad wording \"the expected the canonical manifold learning\"\n+ line 297: Inconsistent acronym: M-flow vs. Mflow\n+ line 299: inferior lok-likelihood calculations of Mlow...where shown?\n+ line 309/310: wrong statement: there are diffusion models with lower dimensional latent space. However, by default indeed the latent space has the same dimensionality", "limitations": "Major limitations are addressed in Section 6. However, some questions are unresolved such as: how to find/define the prominent latent dimensions without an intrinsic dimensionality estimator?", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper is overall well written. The theoretical background is exhaustive and well-explained. The example in 4.1 serves as a good illustrative motivation. The experiments are comprehensive.", "weaknesses": "The contribution is somewhat marginal. Though citing [1], they don't mention that the corresponding authors propose a very similar penalty term. To my understanding, the only differences of the penalty term in [1] are\na) that in [1] the diagonal entries are penalized to be $1$ whereas in this work the diagonal entries are unconstrained.\nb) [1] used the L2-norm whereas in this paper the L1-norm is used.\n\nGiven the great similarity, the penalty term suggested by [1] should be included in the comparison. What do we gain by not constraining the flow to be an isometry? In addition, the authors use the M-flow for benchmarking although a better method for overcoming the limitations of M-flows (namely that the log-determinant term is not considered while training) was already proposed in [2]. At least citing and ideally comparing with [2] should be part of the paper. The same is true for comparing with [3]. It would be nice to see how the contour lines differ from the method introduced in [3].\n\nFurthermore, I am not sure to understand the reasoning behind Definition 4.1. To my understanding, every differentiable manifold is a canonical manifold as one is free to choose the basis of the tangent space to be orthogonal. Thus, such a basis always exists which is the only requirement to be a canonical manifold. Definition 4.1. is simply an existence statement and not useful as such. I think what the authors wanted to define is a manifold equipped with a chart s.t. the chart induces an orthogonal basis of the tangent space. However, this is very much the definition of the principal component flow [3]. Please enlighten me in case I totally misunderstood something.\n\nI am also confused about the comparison with rectangular NF (RNF) in Figure 2. By definition for RNF we have that $d<D$. However, in Figure 2 the author uses $d=D$. This is a contradiction. If $d=D$, a standard NF can be used rather than an RNF. In addition, one claimed strength of the proposed CMF is the sparisity induced by the l1 norm. Thus, the relevant dimensions should be found automatically which elevates, in theory, the necessity to estimate $d$ a priori. Then, why not always set $d=D$ and then apply the proposed penalty term? The number of prominent latent dimensions should correspond to the true intrinsic dimensionality of the data.\n\nFinally, the notation for the manifold, data manifold, and learned manifold is confusing. In line 74 or line 90, $\\mathcal{M}$ is a lower dimensional manifold. In line 83, $\\mathcal{M}_{\\theta}$ is introduced without explaination. In line 212, $\\mathcal{M}$ is referred to as the learned manifold and in the same line, the data manifold is introduced without further explanation. In addition, a new notation for a canonical manifold is suggested. I find the various versions for a manifold confusing and don't see the added benefit of it (especially given that Definition 4.1. is not useful in my opinion).\n\n\n[1] Eike Cramer, Felix Rauh, Alexander Mitsos, Raúl Tempone, and Manuel Dahmen. Nonlinear 350 isometric manifold learning for injective normalizing flows. arXiv preprint arXiv:2203.03934, 351 2022.\n\n[2] C. Horvat and J.-P. Pfister. Denoising normalizing flow. In Advances in Neural Information Processing Systems, volume 34, 2021.\n\n[3] Edmond Cunningham, Adam D Cobb, and Susmit Jha. Principal component flows. In Interna410 tional Conference on Machine Learning, pages 4492–4519. PMLR, 2022.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1: We appreciate the reviewer for their insightful comments. We would like to emphasize that calculating L1 for the off-diagonal elements is not a trivial conceptual step while it may seem like a simple implementation detail, and this distinction mitigates many of the drawbacks of previous methods in a simple yet theoretically grounded way. For instance, in [1], an isometric embedding —essentially a predefined constrained transformation— is presented. Although it allows for direct density estimation and is a form of PCA, its expressivity is greatly limited, as evidenced by the authors' attempts to address these limitations using additional components like I-AE. Indeed, standard approaches often involve focusing solely on the diagonal elements or strictly enforcing the off-diagonals to be zero, as in [3]. However, these methods come with constrained expressivity.\n\nAs detailed in the manuscript, the utilization of the L1 norm on the off-diagonal elements facilitates sparse learning and/or variable (non-strict) local orthogonality. This means that the chart's transformation can be significantly non-linear while retaining some of the benefits of orthogonal representations. Note that, the orthogonality does not mean a global coordinate system, only a local one. On the other hand, the L2-norm is used to match the metric to $\\mathcal{I}$, ensuring an isometric embedding—an entirely distinct concept. As a result, we posit that our contribution is substantial not only as an independent method but also as a prototypical idea for optimization problems.\n\nW2: Indeed, a comparison with [1] should be incorporated into the relevant work section in the revised manuscript. Thank you for pointing this out. That said, given that our primary aim was to establish a highly expressive transformation devoid of heuristic solutions, which aligns with the core ideas of manifold learning flows, we focused on the most relevant part of the literature for brevity. \n\nIndeed, [2] represents a progression beyond M-flows, while RNFs directly compute the JtJ term that M-flows neglect, overcoming the aforementioned limitation. RNFs can be seen as a parallel approach to DNF [2] but with a more direct methodology, circumventing potential ambiguities stemming from heuristic techniques like density denoising [2]. [2] will also be cited in the revised version, thank you for pointing this out.\n\nIn reference to [3], PCA flow relies on a lower-bound estimation of the probability density to bypass the JtJ calculation. This bound is tight when complete principal component flow, as defined by them, is achieved. The fundamental distinction here is that CMF does not confine itself to a pure PCA flow scenario, which has the potential to restrict expressivity. Since our method only loosely enforces orthogonality, or mutual information $\\rightarrow 0$, we anticipate the contour lines to lie somewhere between NF and PCA flow. Furthermore, the introduction of sparsity adds an extra layer of complexity, making direct comparison challenging. As an outlook, we acknowledge that [2]  and possibly [3] see Q6, can undergo quantitative comparison with our method. However, at present, due to time constraints and the necessity for GPU cluster maintenance, we have been unable to carry out such comparisons.\n\nW3: It is accurate that any Riemannian manifold can be characterized by an orthogonal local basis. We merely use 4.1 to precisely define the term \"canonical manifold\", given that it is no a standard term in literature. We also include sparsity as part of the definition. \nGenerally, the meaning of \"canonical manifold\" can vary depending on the context in which it is used. It is astutely pointed out that Definition 4.1 aligns closely with the assumption of PCA flow. However, it is vital to note that our method does not enforce this strictly. To put it in perspective, our approach seeks a 'partly canonical manifold,' if you will. Additionally, our method encompasses sparsity—meaning the diagonal elements can also approach zero.\n\nW4: Excellent remark, yes, we do employ the full dimension, D=d, for the low-dimensional simulated data, and this choice serves illustrative purposes. First, it allows us to visualize what all the latent dimensions are doing in relation to what we expect them to do. In particular, CMF correctly uses only 2 latent dimensions for representing 2D surfaces even though it has 3 latent dimensions. Second, it clearly shows the advantage to RNFs for the case where dimensions of the embedded manifold are known.  In principle, that can be a solution for higher-dim data. However, in practice it is very computationally expensive to train a full latent dimension (e.g. JtJ calculation) and the solution will take long to converge. Additionally, empirical knowledge suggests that a lower-dimensional representation can enhance expressivity, as discussed in the context of M-flows. Consequently, in practice, one can start from a predefined lower-dim and let the network optimize at will. \n\nW5: We apologize for any confusion cost. At Line 83 the ``$\\theta$'' is a typo. $\\mathcal{M}$ represents the lower dim manifold as learned by standard manifold learning, and $\\mathfrak{M}$ the manifold learned by our method this convention is used to distinguish them, as done in lines 212 and 213. The data manifold is also a distinct manifold in that regard. %If such notation is confusing, we can simplify and just use words instead of the symbols.  We estimate that 4.1 is useful for the reasons elaborated above.\n\nQ1: 4.1 is used to define the canonical manifold (see W3). \n\n$*$Kindly refer to the global rebuttal for the continuation of the reply\n\n\n\n\n"}]

Author rebuttal:
We express our gratitude to both the reviewer and the chair for their valuable time and insights. We have diligently addressed each of the reviewer's comments individually. Furthermore, we have expanded our testing to encompass additional tabular datasets and incorporated CelebA, 64x64 FID test scores for a more comprehensive evaluation.

$........................................................................................$ 

In order to adequately address the points raised in the extended review by reviewer 4 (8w9Q), we make use of this space for further discussion.

Q2: The comparison is indeed fair as they are both manifold learning flows, nothing restricting the use of $d=D$. Furthermore, similar results can be obtained by embedding a 2-sphere in a 4D space and setting $d=3$, or just the orthogonality can be see by setting $d=2$. Additionally, the arguments about lower-dim expressivity are not so relevant for such low-dim data, see W4. 

Q3: Excellent remark, it is mathematically sound but computationally expensive and possible convergence issues, see W4. 

Q4: Yes, exactly for this reason, CMF is shown to be learning better. Perfect density estimation is not a trivial task from our experience. 

Q5: The log likelihood (L)  is 1.6553 and 1.6517, and the KS p-value is 0.17 and 0.26, for the CMF and RNF methods respectively. For the sphere,  L=1.97 and 1.16 for for the CMF and RNF methods respectively. CMF shows improved quantitative quality over the RNF, as it is also seen qualitatively. 

Q6: PCF [3], mentions that they obtain a similar test set log likelihood to that of the normalizing flow (NF), and that the PCF has almost zero pointwise mutual information. Additionally, the M-Flow method shows already improved image generations over other flow methods, and RNF improves on those even further.  Therefore, we had estimated that RNF is the main competitor in this regard. Furthermore, there can be no real comparison of the mutual information (or similarity etc) as the PCF relies on fully orthogonal contours, which is not necessarily a desirable quality in the CMF framework as explained in W1. Nevertheless, a comparison can always be made.  However, due to time constraints and GPU cluster maintenance, it is currently pending.

Q7: We appreciate you bringing this to our attention. Our methodology was to run about 10 repetitions for each and chose the best. Our estimate is that it is a hard manifold with non-trivial topology that degenerate latent-space representations fail to capture it well.

Q8: An excellent question, it is more an intuition that the largest eigenvalues (if an eigen-decomposition is possible) correspond to the dimensions with the highest weights.  

Q9: Enforcing strict complete orthogonality can restrict the expressivity of the transformation in the case of high dimensional complex data where the latent manifold is completely unknown. 

Q10: L2 with the identity matrix can be used to enforce a specific embedding, e.g. isometric [1] or conformal [4] which is not the purpose of this work, we want the learned transformation to be general as explained in W1 as well. 

Q11: We appreciate you bringing this to our attention and allows us to clarify that the method does not arbitrarily discard any dimensions, rather they would be effectively irrelevant if their weight is close to zero. For the analysis plots, we order the dimensions according to their weights and strictly choose a pre-defined number of them where applicable. Thresholds would be very hard to define indeed. 

Minor: We acknowledge, and we will diligently rectify, all the minor issues in the revised manuscript. Notes: line 83: $\theta$ was a typo. line 225: We mean that the method is not overly sensitive to the hyperparameter $\gamma$, we will rephrase, thank you. line 299: This is the main premise of RNF and Denoising Normalizing flows, also this limitation is accepted in the original M-Flow paper. 

Lim: As elaborated in Q11, there is no need to find the prominent latent dimensions specifically, their weights define their significance as adjusted by sparse learning and/or orthogonality.

[4] Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows
Brendan Leigh Ross, Jesse C. Cresswell


Meta-review: After the rebuttal and some discussion the reviewers agree that the paper is suitable for acceptance.  A number of small issues have been identified during the review process and the authors should revise their manuscript accordingly for the final version.
Decision: Accept (poster)