Minimum Curvature Manifold Learning
Authors: Yonghyeon Lee
Frank C. Park
Venue: Submitted to ICLR 2023 (2023)
Keywords: Autoencoder
Manifold
Curvature
Riemannian geometry

Abstract: It is widely observed that vanilla autoencoders can have low manifold learning accuracy given a noisy or small training dataset. 
Recent work has discovered that it is important to regularize the decoder that explicitly parameterizes the manifold, 
where a neighborhood graph is employed for decoder regularization. However, one caveat of this method is that it is not always straightforward to construct a correct graph. Alternatively, one may consider naive graph-free regularization methods such as minimizing the norm of the decoder's Jacobian or Hessian, but these norms are not coordinate-invariant (i.e. reparametrization-invariant) and hence do not capture any meaningful geometric quantity of the manifold nor result in geometrically meaningful manifold regularization effects. 
Another recent work called the isometric regularization implicitly forces the manifold to have zero intrinsic curvature, resulting in some geometrically meaningful regularization effects. But, since the intrinsic curvature does not capture how the manifold is embedded in the data space from an extrinsic perspective, the regularization effects are often limited. In this paper, we propose a {\it minimum extrinsic curvature principle} for manifold regularization and {\bf Minimum Curvature Autoencoder (MCAE)}, a graph-free coordinate-invariant extrinsic curvature minimization framework for autoencoder regularization. Experiments with various standard datasets demonstrate that MCAE improves manifold learning accuracy compared to existing methods, especially showing strong robustness to noise.

Under review as a conference paper at ICLR 2023
MINIMUM CURVATURE MANIFOLD LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
is widely observed that vanilla autoencoders can have low manifold learning ac- curacy given a noisy small training dataset. Recent work has discovered that it is important to regularize the decoder that explicitly parameterizes the manifold, where a neighborhood graph is employed for decoder regularization. However, one caveat of this method is that it is not always straightforward to construct a cor- rect graph. Alternatively, one may consider naive graph-free regularization meth- ods such as minimizing the norm of the decoder's 's Jacobian or Hessian, but these norms are not coordinate-invariant (i.e. pntrramttrzzationnnvaaiii not capture any meaningful geometric quantity of the manifold nor result in geo- metrically meaningful manifold regularization effects. Another recent work called the isometric regularization implicitly forces the manifold to have zero intrinsic curvature, resulting in some geometrically meaningful regularization effects. But, since the intrinsic curvature does not capture how the manifold is embedded in the data space from an extrinsic perspective, the regularization effects are often lim- ited. In this paper, we propose minimum extrinsic curvature principle for man- ifold regularization and Minimum Curvature Autoencoder (MCAE), a graph- free porddnnateiinvarian extrinsic curvature minimization framework for autoen- coder regularization. Experiments with various standard datasets demonstrate that MCAE improves manifold learning accuracy compared to existing methods, es- pecially showing strong robustness to noise.
and hence do
INTRODUCTION
Autoencoders are widely used to identify, given a set of high-dimensional data, the underlying lower- dimensional manifold structure and its coordinate space, simultancously (Kramer [991]. The de- coder explicitly parameterizes the data manifold as a mapping from a lower-dimensional coordinate space (i.e., latent space) the high-dimensional data space, and the encoder maps data points to their corresponding coordinates (i.e., latent values). However, vanilla autoencoders trained to recon- struct the given training data often learn manifolds that severely overfit to noisy training data or are wrong in regions where there are fewer data, impairing their manifold learning performances.
It has been recently discovered by Lee et al. (202T) that autoencoder regularization methods that focus on regularizing the latent space distributions determined entirely by the encoders (Kingma & Welling 2013; Tolstikhin et al. 2018 Makhzani al., 2015 Rifai al.) (011) are not sufficient to learn correct manifolds, yet it is important to properly regularize the decoders that parameterize the manifolds. In (Lee et al., (021). neighborhood graphs constructed from data are successfully utilized to regularize the local geometry and connectivity of the manifold, significantly improving the manifold learning accuracy. However, the underlying premise behind this method is that the graph has to be accurate, yet constructing correct graph may not be always straightforward.
There are some graph-free methods such as the denoising autoencoder (Vincent et al. 2010) and reconstruction contractive autoencoder (Alain & Bengio 2014) that regularize not only encoder but also a decoder. They can learn manifolds that are robust to noise to some extent, but when the noise level is large, the performance is often sussthaan-deiiabble and they do not always produce correct manifolds, especially in regions where there are fewer data (discussed in more detail in Section
Since the decoder needs to be regularized, one may come up with some naive regularization strate- gies such minimizing the norm of the decoder's Jacobian or Hessian, considering them mea-

---

Under review as a conference paper at ICLR 2023
Same Manifold with Different Parameterizations
Zero Intrinsic Curvature Manifolds
|| 11
Figure 1: Left: Two decoders / and S' parameterize the same data manifold where the norm of Jacobian of f" is smaller than that of f.i.e., || > || Right: curve and developable surface embedded R have zero intrinsic curvatures.
sures of the manifold's smoothness. However, these norms do not properly capture any geometric quantity of the manifold because they are not reparametrization-in As shown in Figure ( just by increasing the volume of the latent space without actually chang- ing the manifold, i.e., re- re-parametrizin the manifold f ~> f', the above norms can minimized.
(or goordinate-invarinn
Just recently, a coordinate ievrinate-invariant geometric distortion measure has been introduced to regularize decoder to be a geometry-preserving mapping, which is called isometric regularization (LEE al. (022), so that the data space geometry is preserved in the latent space. Minimizing this dis- tortion measure implicitly forces the learned manifold to have zero intrinsic curvature which only depends distances measured within the manifold (e.g., cylinder's side surface has intrin- sie curvature unlike the spherical surface) -, resulting some geometrically meaningful manifold regularization effects.
The intrinsic curvature, however, does not capture how the manifold lies in the data space and thus minimizing the manifold's intrinsic curvature may be enough learn correct manifolds. For example, curves and developable surfaces P in IR'S always have zero intrinsic curvatures, e.g., Fig- ure (Right), regardless of how severely they are curved from extrinsic point of view (Do Carmo 2016
The main contribution of this paper is a coordinate-invarian extrinsic curvature minimization framework for autoencoder regularization, which we refer to a Minimum Curvature Autoencoder (MCAE), that is graph-free and effectively improves the manifold learning accuracy given noisy or small training dataset. Specifically, we develop moordinate-invarii extrinsic curvature measure of the learned manifold, by investigating how smoothly tangent space changes on the manifold, and use it as regularization term.
To make things more explicit, let M be a manifold of dimension m embedded in TR.. Consider mapping 7' that maps point in M to its tangent space T,M, linear subspace that has the dimension of m attached at x, i.e., T(x) = T.M. The set of all linear subspaces of dimension m in IRD forms manifold called the Grassmann manifold denoted by Gr(m, PD (Bendokat et al. 2020 and thus the mapping T can be viewed as a mapping between two Riemannian manifolds, Le., Gr(m, PP) By using the Dirichlet energy (Eells & Lemaire 1978), natural smoothness measure of mappings between two Riemannian manifolds defined in a coordinate-invarian way, we formulate extrinsic curvature measure. We also propose practical estimation strategy of the curvature measure that can be used for high- nigh-dimensiinal problems, reducing computation costs.
Experiments on diverse image and motion capture data confirm that, compared existing graph- free regularized autoencoders, our MCAE improves manifold learning accuracy for noisy and small training datasets. In particular, our experiments show that even compared to the methods specially designed to be robust to input perturbations such as the DAE (Vincent et al. (010)) and RCAE (Alain Bengio 2014), the MCAE shows comparable or even some cases significantly higher robust manifold learning performance.
Manifold's intrinsic properties are defined without involving embedding. A developable surface be formed by bending rolling planar surface without stretching tearing.
2

---

Under review as a conference paper at ICLR 2023
2
GEOMETRIC PRELIMINARIES
2.1 GRASSMANN MANIFOLD
In this section, we review the Grassmann manifold and its Riemannian geometry from a matrix- analytic perspective. The Grassmann manifold is defined as the set of all m dimensional linear subspaces of the Euclidean space RDD, denoted by Gr(m, R )) this can be identified with the set of orthogonal rank-m projection matrices as follows:
(1) which is an m(D m)) dimensional manifold; which associates P E Gr(m, RP) with the linear subspace range(P) RRD. This an implicit parametrization of the Grassmann manifold consid- ered as being embedded in the Euclidean space E DDD For more formal and detailed descriptions of the Grassmann manifold, we refer to (Bendokat et al. 2020
Gr(m,, RD) = {P D p P p² rank = m).
Given a rank-m matrix J E Dxm, one may want to consider its range, mn-dimensional lin- ear subspace RP, as an element of the Grassmann manifold. The embedding E R." 1 A Gr(m, RP) such that EJ) (JJ ()=J)--'' " properly converts / to the element of dLp. We note that (i) range(J) = range (()) and (ii) E{J) = F(JA A) for any m x m invertible matrix A E mm since the transformation J ~ JA does change the range.
Next, we introduce the basic Riemannian structure of the Grassmann manifold. At a point E Gr(m,, R) the tangent space is defined follows:
VP+ TpGr(m, R = (V e RXD VP v}. =
(2)
which can be derived from [ by differentiating the constraints. One canonical choice of the Rie- mannian metric is given follows:
(V. 12)
V2 VVVV for V1, V2 E TrGr(m,R"))
(3)
This metric invariant under the orthogonal transformation, i.c. (V1, V2) = (RV1, RV2) for any D x D orthogonal matrix R.
2.2 DIRICHLET ENERGY FOR MAPPINGS BETWEEN RIEMANNIAN MANIFOLDS
This section introduces the Dirichlet energy for mappings between two Riemannian manifolds. Let M and N be Riemannian manifolds of dimension and n; will consider a differentiable map- ping f M + N. We will assume x E M is explicitly parametrized by local coordinates as M is expressed as m X m positive-definite matrix G(x) = (9:5(z)) E mmmm and N is embedded in the Euclidean space of higher dimension as N C. Rd (d >> n) and the Riemannian metric at E N is given as (:-) for E N (c.g. Grass- mann manifold). The mapping is expressed as R" + N C gd such that y = f(x).
E R and the Riemannian metric
The Dirichlet energy, global measure of how much the mapping f changes, defined as follows:
/
m m ddd (x) dr dzl (x)) det G(x) dr dx.
OS (x), 0s
where (xx) denotes (ih element of the inverse of G(z) and / det G(x) dr ... dr is the Riemannian volume form, which corresponds to the integral functional from the theory of harmonic maps; this integral an intrinsic quantity (i.e., coordinate-invariant), We note that the integrand is a local measure of how much the mapping f changes. We refer to the extensive literature on the theory and applications of harmonic maps, e.g., (Bells & Lemaire
1978 1988
Park & Brockett
1994 Jang et al. 2020 LEE et al.) 2022).
3 MINIMUM CURVATURE AUTOENCODERS
In this section, we propose a regularized autoencoder based on the principle of minimum curvature manifold learning. Throughout, consider a data space TR and latent space R (D) > m) and
3

---

Under review as a conference paper at ICLR 2023
denote parametric encoder by 96 RD R such that 2 = [2) and a parametric decoder by fo R RD such that = fo(2). The manifold parametrized by the decoder will be denoted by Mø, and the Jacobian of the decoder by Jo(2) afe az (2). Given set of data points {2x E P
the empirical data distribution will be denoted by p(x) = N + 8(x x;)) and the latent space distribution encoded by 96 by Po(2) == _/r Σi= 8(2 (z()). The subscripts show what variables each function or geometric object depends on, either 0 or Φ.
3.1
COORDINATE- EXTRINSIC CURVATURE MEASURE
In this section, we formulate a coordinate- invariiat (i.e., innnettrizaaionnnnaaaiinn extrinsic cur- vature measure of the manifold MM embedded in R.D. We begin by introducing the notion of coordinate-invarian
Definition 1. Given manifold M of dimension m embedded in RD,, let : + M be its explicit parametrization. A functional F(f) is coordinateinvvariant (i.e., reparametrization-inaat -ntaatiannn-))aaan) if, given any invertible mapping or coordinate transformation (i.e., reparametrization) h: R IR" F(f)=F(( o h h)
The sendiinate-ivvarineee is necessary to properly measure any geometrically meaningful quantity of the manifold. For example, the integration of the Frobenius norm of Ja coordinate space IRm not coordinate-nnvariant, and hence does not capture any geometrically meaningful quantity of Ma-
Now, we define a coordinate-invariant extrinsic curvature measure of Mp- The core idea define local measure of the extrinsic curvature by measuring how fast the tangent space Ta changes within the neighborhood of x, and then integrate it over the manifold to define global curvature measure. For this purpose, let pair of mappings, encoder dø and decoder fo, be coordinate system for Ma. and consider mapping T R - Gr(m, RP) such that (=) is the element of the Grassmann manifold (I) whose range is equal to (=) We note that the range of the Jacobian matrix Je(z) E I8 Dxm is Mg, hence, by using the embedding E' IR Dxm + Gr(m, RD) such that E(Ja) =- Jo(J% Jo) JJ"" we can explicitly write the mapping T T(z) E(Jo(z)).
Let MM be assigned with the Riemannian metric induced from the ambient space Euclidean metric, so that the metric expressed the coordinate space J (z) Jo()). and Gr(m, IR. ") be assigned with the Riemannian metric in 33. We use the dirichlet energy in (4) of the mapping T as a coordinate- invariant extrinsic curvature measure, where the integral replaced by the expectation over pa(2):
Definition 2. Given an encoder 96. decoder Se. and empirical distribution in coordinate space P2(2), the global extrinsic curvature measure of Ma with respect to po(z) is defined as
m
m
C(8, 6) = E. E2-Ao()
Σ
JoY T (E(Jo) dz (E(Jo)))
(5)
Oz
1=1==1
The curvature measure C(B. Φ) in Definition 2 is coordinate-invarion i.e., for an- other pair of encoder gø := h o go and decoder Se To h- with any invertible map or coordinate transformation ht such that z' = l1(z), the measure is invariant, i.e., C(O, ④) = C('','')
Proposition
Proof. The proof is given the Appendix A.2
Our definition of the curvature generalizes classical definition of the curvature of a curve embedded in R from differential geometry (Kühnel 2015) (please see Appendix A.3 for more details).
With the proposed curvature measure, we define a regularized autoencoder where the loss func- tion consists of the following two terms i) reconstruction error term for manifold learning and ii) regularization term C(0, Φ) for curvature minimization:
min Em ||| - fa gx( a C(O, 6.6
where a is the regularization coefficient, which we refer to as the Minimum Curvature Autoen- coder (MCAE).
4

---

Under review as a conference paper at ICLR 2023
3.2 PRACTICAL IMPLEMENTATIONS
This section introduces two practical strategies for computation of the curvature measure (
Augmented Distribution: In (5), the local curvature measure expected over the empirical latent space distribution. However. the influence of the measure is then limited to regions where data available; thus the manifold's curvature in regions where data is no data may not be properly regularized. In practice, we use data augmentation to resolve this issue. Following (Chen et al.) 2020 LEE et al., 2022). we use the modified mix up data-augma aatta-uumentationn method with a parameter > O. where (=) augmented by
dzı + (1 =) such that 2i pa(z), 1,2, where 8 is uniformly sampled from [-77, ,, 1 + T/]. We set = 0.2 throughout.
Stochastic Trace Estimation: At first glance, the curvature measure seems computationally very expensive, because it involves the computation of the full Jacobian 3 of a deep neural network and derivative of the Jacboaidn dJe, and we even need backpropagate through them when using "z
the standard stochastic gradient descent algorithms. To efficiently compute the measure in practice, we use the Hutchinson's trace estimator (Hutchinson) 1989), i.e., Tr(A) = EE-N(O, [v the curvature measure C(O, () has the following expression:
Auj. then
Aw E(Jo)) ((E(J))m) Go'u),
C(O,0) ②) = -R
Oz
Oz
where Is is the & x identity matrix and Gg = JJJJo To implement this computationally efficiently, use the Jacobian-vector and vector-Jacboian products in multiple times: (i) for E(Jo)w = JoGp J/ w. we first use the vector-Jacobian product for J w and the Jacobian-vector product for Ja(G, "J Jf w). and (ii) for [(E(J))") v and ((E())))) (G)²o). we use the Jacobian-vector products. These techniques make the computation of (5) iractable for high-dimensional complex problems. Surprisingly, for the estimation of (7). using one sample of and w each 2 (2) was sufficient to train MCAE in our later experiments. When the latent space is high- nigh-dimensional, the matrix inverse computation Gj takes up most of the computation time. Using an approximate inverse can significantly reduce the computation time, see the Appendix A.6
0z
EXPERIMENTS
4.1 PARAMETER SWEEP
We first provide empirical study the effect of the most important parameter of MCAE, the reg- ularization coefficient a. Intuitively, as a increases, the tendency to minimize the extrinsic curvature of the manifold becomes stronger, so the learned manifold will become closer to linear subspace. And, if o is too small, the learned manifold will not be different from that of the vanilla autoencoder; hence it is important to select appropriate value for depending the dataset.
Figure Sgcure2]showe shows how a affects the learned manifold in MCAE with two examples. the upper figure, given noisy wwo-dimensional data points, we train MCAEs with one-dimensional latent spaces. In the lower figure, given sparse three-dimensional data points constrained on the 2-sphere 5 := {u E R ||-||| = 1}. we train MCAEs with -inedimnnla latent spaces, where the decoder outputs are normalized to be in S As can be seen, a 0.01 and a = 0.0001 are good values for the upper and lower examples, respectively. In practice, we can find the optimal value of with a proper validation criteria (e.g., mean reconstruction error for validation data).
4.2 COMPARISON OTHER REGULARIZATION METHODS
In this section, we compare the proposed MCAE with other regularized autoencoders and highlight the differences.. Please refer to Appendix A.I for more detailed comparisons.
Comparison to Isometrically Regularized Autoencoders: In the Isometrically Regularized Au- toencoder (IRAE) (LEE et al. 2022), the decoder is regularized to be a scaled isometry: similar to (6). a regularization term that measures how far fo from being scaled isometry is added to the reconstruction error term with the regularization coefficient a. This regularization implicitly forces the learned manifold Ma have zero intrinsic curvature, but not the extrinsic curvature; therefore
5

---

Under review as a conference paper at ICLR 2023
0.0001
0.001
0.01
0.000001
0,00001
0.0001
0.001
0.01
points
Ground menilold
Learned manifold (MCAE)
Figure 2: Learned manifold becomes flatter the regularization coefficient increases. Upper: Learned data manifolds 1d sin-curve and noisy training data points. Lower: Learned data mani- folds of 1d S-curve projected to the 2-sphere and sparse training data points.
it is at first glance expected that, when learning a one-dimensional manifold, the IRVE should not have any meaningful manifold regularization effect (since me--iimensional manifolds always have zero intrinsic curvatures).
Counterintuitively. shown in Figure Ba). our experiments show that the extrinsic curvature of the one-dimensional manifold learned by TRAE decreases as a increases. the decoder's hypothesis space set of arbitrary smooth functions, this result would not have been obtained, but since the hypothesis space defined as the set of neural networks is smaller, the isometric regularization seems reduce the extrinsic curvature at the expense of obtaining the isometric representations. Fig- ure ()) shows how the reconstruction MSE for clean test data varies as function of the extrinsic curvature of the learned manifold by IRAE and MCAE. As the curvature decreases or the regular- ization coefficient increases (from left to right), the test reconstruction MSE decreases, reaches minimum, and then increases again. We note that the graph of MCAE lies lower than that of IRAE, implying that the MCAE can learn more accurate manifold than the IRAE.
(a)
(b)
a
0.10
Reconstruction MSE
IRAE
0.05
Data points
Ground truth manifold Learned manifold (IRAE)
0.4
0.2
0.0
Curvature
Figure 3: (a) Learned manifold by IRAE becomes flatter as the regularization coefficient or increases. (b) Test data reconstruction MSE (i.e., manifold learning accuracy) as a function of the extrinsic curvature obtained by IRAE and MCAE.
DAE
ROAE
Comparison to Denoising and Reconstruction Contrac- tive Autoencoders: Denoising Autoencoder (DAE) (Vin- cent et al. 2010) and Reconstruction Contractive Autoencoder (RCAE) (Alain Bengio 2014) are intuitive and straightfor- ward regularization methods for learning manifolds robust to input perturbations. As shown Figure (Upper), the DAE and RCAE learn manifolds robust to noise to some extent. However, as shown in Figure 4 (Lower), for the projected S- curve example in Figure (Lower), they still learn wrong man- ifolds in regions where there are fewer data and do not improve the vanilla autoencoder. On the other hand, the MCAE explic- itly regularizes the learned manifold to have a small curvature globally and improves the manifold learning accuracy.
points
Learned manifold
Figure 4: Learning by DAE and
RCAE for examples in Figure

---

Under review as a conference paper at ICLR 2023
Table 1: Averages and standard errors of the test data reconstruction MSEs (5 times run) for the sincurve example in Figure 2 (Upper) with various Gaussian noise of standard deviations 0.1, 0.2, 0.3, the lower the better. The best results are marked in bold. The numbers are written in
units of 10-3
Noise
VAF
DAE
RCAF
DCAF
DHAE
MCAE
0.1
3.98 = 0.22
2,05 + 0.36
2.23 # 0.26
2,95 : 0.44
281 # 0.21
2.68 0.31
1.63 + 0.25
1,28 = 0.13
0.2
22.7
2.9
1:072
6,99
1.04
12.5
10.9 1.02
1499
2.87
6.59
1,09
4.56 0.69
0.3
68.5 4 17.3
13.5 2.3
17.8 3.4
30.5 4 7.9
30.7 1: 4.8
46.9 4: 12.5
20.2 4: 5.1
9.80 :: 1.55
Quantitative Comparisons of Noise Robustness: As seen from the above examples, besides the proposed MCAE, the IRAE, DAE, RCAE all have the robustness properties noise. We quanti- tatively compare the robust manifold learning performance given noisy input training data with the sincurve example in Figure (Upper) with various noise levels, i.e., Gaussian noise with standard deviations of 0.1, 0.2, 0.3. In addition to the IRAE, DAE, RCAE, we compare the MCAE with the vanilla Autoencoder (AE) and other regularized autoencoders such as the Variational Autoen- coder (VAE) (Kingma& & Welling 2013), Decoder Contractive Autoencoder (DCAE), and Decoder Hessian Contractive Autoencoder (DHAE), where the DCAE and DHAE minimize the decoder's Jacobian norm and the decoder's Hessian norm, respectively. [ the averages and stan- dard errors of the test data set reconstruction MSEs, the lower the better. The MCAE produces the lowest errors regardless of the noise level.
4.3
IMAGE DATA
First, we investigate the manifold learning performance of MCAE compared to the other regularized autoencoders with the standard grayscale im- age data (MNIST, Fashion-MNIST, KMNIST) the num- ber of training + validation data and noise level varies. We use two-layer fully connected neural networks (512 nodes per layer) for both encoder and decoder with ELU activation functions, and the latent space dimensions are 16, 32, 32, respectively.
Grayscale Image:
MNIST
FMNIST
Clean
8
2
9
9 7
Noisy
8
1
AE
0
2
5
7
9
VAE
0
2
9
7
9
DAE
8
2
9
7
9
RCAE
8
2
9
7
9
DCAE
8
2
9
7
9
Figure 6 shows the test reconstruction MSEs as func- tion of the number of training (80%) + validation (20%) data. For all methods, the error decreases as the number of data increases; MCAES mostly produce the lowest er- rors except for some MNIST cases. Figure 7 shows the Peak Signal-to-Noise Ratios (PSNRs) computed with the clean test set data (the higher the better) as a function of the standard deviation of the Gaussian noise added to the training data (the number of training data is 8000). The PSNR decreases as the noise level increases;
DHAE
8
2:
9
7
9
IRAE
2 2.
5
7
9
MCAE
7 9
Figure 5: De-noising examples (noise level 0.3).
MCAES mostly produce the highest PSNRs. Figure 5 shows some de-noising examples with cor- rupted input data of MNIST and FMNIST.
MNIST
FMNIST
KMNIST
Reconstruction MSE
0.0250
0.020
0.06
AE
0.0025
0.018
6.0000
8.0175
0.044
0.036
D-AEE
DEAE
RCAE
0.0150
0.022
0.0125
MEAE
0.010
6000
8000
10000
2000
10000
2000
training validation data
# of training + validation data
of training validation data
Figure 6: Test set MSES as a function of the number of training (80%) + validation (20%) data, the lower the better.
7

---

Under review as a conference paper at ICLR 2023
MNIST
FMNIST
KMNIST
20
14.0
AE
36
VAE
PSNR
38
15.0
DCAE
D-AE
12.0
15
MLAE
0.10
0.15
0.25
0.30
0.05
0.10
0.15
0.30
0.25
0.30
0.05
0.10
0.15
0.20
0.25
Noise level
Noise level
Noise level
Figure 7: Test set Peak Signal-to-Noise Ratios (PSNR) as function of the noise level, the higher the better.
SVHN & CIFARIO Image:
We compare the manifold learning performances of MCAE with other regularized autoencoders for the SVHN and CIFARIO image datasets for both clean and corrupted training datasets. We use the convolutional and transposed convolutional neural net- works for encoder and decoder with ReLU activation functions and the latent space dimensions are 64; the number of training data is 8000. For the corrupted train- ing dataset cases, we add three different types of noise: (i) Gaussian, (ii) Shot, and (iii) Impulse noises adopted from (Hendrycks & Dietterich 2019); see Figure g
SVHN
OIARRO
1919
Clar
1919
1919
2
Impule
Figure FARIO images.
Corrupted SVHN and CI-
Table 2 shows the test set MSEs for experiments with the clean training datasets, and TableB]shows the PSNRs for experiments with the corrupted training datasets, where in both cases the metrics are computed with the clean test data. From the results, we note that (i) MCAE shows the second or third best results, (ii) MCAE does not im- prove the vanilla AE for the SVHN clean training dataset case, and (iii) for the corrupted training dataset cases, RCAE produces better results than the MCAE unlike the grayscale image data. Overall, compared to the grayscale image data, the minimum curvature regularization is less effective for SVHN and CIFAR1O 10. One possible inter- pretation is related to the limitation of MCAE (discussed in the conclusion section), that the SVHN and CIFAR10 10 manifolds have locally very different curvatures and thus is difficult to find a proper constant regularization co- efficient a in 6 because if we use a big enough a to correctly learn low curvature areas of the manifold, then high curvature areas can be overly flattened, and vice versa. Figure g shows the density plots of the log normalized local curvature of the learned manifolds by vanilla autoencoders, i.e., log(ks) - log(k), = 1, ...... N where K; is the local curvature at i-th training data points and log(k) 1/N E, log(ss). which is invariant to the scale of the mean curvature. As shown in Fig- ure 9 the variance of the SVHN manifold's local curvature is bigger than those of the others, which supports the above interpretation.
Log normalized local curvature
MNIST
0.6
FMNIST
KMNIST
Density
SVHN
CIFAR10
0.0
-2
2
Figure 9: Density plots of the log- normalized local curvatures of mani- folds learned by vanilla AEs.
Table 2: Test set MSES of autoencoders trained with clean datasets, the lower, the better. The best and second best results are marked in red and blue, respectively.
Dataset
AE
VAE
DAB
RCAE
DCAE
DHAE
IRAE
MCAE
SVII
0.00228
0.00461
0.00228
0.00252
0.00255
0.00233
0.00213
0.00229
CI
0.01204
0.01533
0.01204
0.01119
0.01303
0.01244
0.01176
0.01125
4.4 HUMAN SKELETON POSE DATA
In this section, evaluate the MCAE with the human skeleton pose data adopted from the NTU RGB+D dataset (Shahroudy et al. 2016). A human pose skeleton data onsists of 25 three- dimensional key points and thus is considered 75-dimensional vector. There are 60 different action
8

---

Under review as a conference paper at ICLR 2023
Table 3: Test data PSNRs with various noise types, the higher, the better. best results are marked in red and blue, respectively.
The best and second
type
RCAE
DCAE
Gaussian
20.13
22.60
22.04
25.39
20.92
19.87
20.74
24.37
Impulse
19.33
19.06
20.28
23.31
19.18
19.25
19.71
20.18
Gaussian
17.06
17.60
17.78
19.51
17.10
16.92
17.35
18.62
CIFARIO
Shot
17.18
17.46
17.87
19.52
17.26
17.18
18.49
18.64
Impulse
16.71
16.03
16.93
18.49
16.58
16.59
16.62
17.04
classes (e.g., drinking water, brushing teeth), and each action data consists a sequence of skeleton randomly-selected 800 and 200 skeleton poses training and
poses. For each action class, validation data, and 9000 poses as test data. We use two-layer fully connected neural networks (512 nodes per layer) for both encoder and decoder with ELU activation functions, and the latent space dimension is 8.
Table shows the averages and standard errors of the data set reconstruction MSEs over 60 different action classes, the lower the better. MCAE mostly produces the lowest errrr,, especially by a significant margin for noisy training data cases. Figure shows some example reconstruction results of noisy input skeleton data (noise level 0.05); MCAE shows the best de-noising results.
Table 4: Averages and standard errors of the test data set reconstruction MSEs with various Gaussian noise of standard deviations 0.05, 0.1, the lower the better. The best and second best results are marked in red and blue, respectively. The numbers written in units of 10
Noise
AE
VAE
DAE
RCAE
DCAE
DHAAE
IRAE
MCAE
0
2.23 = 0.09
295 + 0.13
2.21
= 0.09
2,17
0.09
2.25
0.09
2,22 0.09
2.08 = 0.09
2,11 = 0.09
0.05
+ 0.04
4.32 + 0.02
2,70
0.01
2.98
0.01
3.92 0.83
4,07
0.03
2,93 0.02
2.20 + 0.01
0.1
15.3
13.5
0.2
5.54
0.17
7.50
0.16
11.7
1 0.2
12.5 1 0.2
12.7 1: 0.2
3.09 1 0.15
Clean
Noisy
AE
DAE
RCAE
DCAE
DHAE
IRAE
MCAE
$
A
/
+
+
Figure 10: Human skeleton pose de-noising examples obtained by reconstructing noisy input data (noise level 0.05). Example poses are from the action class "eat meal".
CONCLUSION
In this paper, we have proposed minimum extrinsic curvature principle for manifold regulariza- tion and developed a Minimum Curvature Autoencoder (MCAE), by formulating a coordinate- hence geometrically correct extrinsic curvature measure. Our
invariant repparametrizationninvaaaa experiments show that the minimum curvature regularization can improve manifold learning accu- racy for both noisy and small training datasets. The degree to which the performance is improved depends the datasets, and especially for the grayscale image and human skeleton pose datasets, the MCAE outperforms the existing methods by a significant margin.
Limitations and Future Directions: In the current implementation of MCAE, the manifold's ex- trinsic curvature is minimized globally by using equal weights for all points. However, for manifolds that have locally very different curvatures, it is difficult to find proper weight parameter in (6). Ideally, low and high curvature areas of the manifold need to be regularized with higher and lower weights, respectively. By exploiting local curvature estimation algorithms, e.g., diffusion-basse method (Bhaskar et al. (2022), developing a curvature regularization method with different local weights will be interesting future research direction.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Guillaume Alain and Yoshua Bengio. What regularized auto-encodeers learn from the data-generating distribution. The Journal of Machine Learning Research, 15(1):3563-3593, 2014.
Thomas Bendokat, Ralf Zimmermann, and P-A Absil, A grassmann manifold handbook: Basic geometry and computational aspects. arXiv preprint arXiv:2011.1369) 2020.
Dhananjay Bhaskar, Kincaid MacDonald, Dawson Thomas, Sarah Zhao, Kisung You, Jennifer Paige, Yariv Aizenbud, Bastian Rieck, Ian M Adelstein, and Smita Krishnaswamy, Diffusion- based methods for estimating curvature in data. In ICLR 2022 Workshop Geometrical and Topological Representation Leurning, 2022.
Nutan Chen, Alexej Klushyn, Francesco Ferroni, Justin Bayer, and Patrick Van Der Smagt. Learning flat latent manifolds with vaes. arXiv preprint
2020.
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. arXiv preprint 1XI 2016.
Xingyu Chen, Chunyu Wang, Xuguang Lan, Nanning Zheng, and Wenjun Zeng. Neighborhood ge- ometric structure-preservinn variational autoencoder for smooth and bounded data sources. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Manfredo P Do Carmo. Differential geometry curves and surfaces: revised and updated second edition. Courier Dover Publications, 2016.
Andrés F Duque, Sacha Morin, Guy Wolf, and Kevin R Moon. Extendable and invertible manifold learning with geometry regularized autoencoders. arXiv preprint vrXiv:2077.07144 2020.
James Eells and Luc Lemaire. A report on harmonic maps. Bulletin of the London mathematical society, 10(1): 1978.
James Fells and Luc Lemaire. Another report harmonic maps. Bulletin of the London Mathe- matical Society, 20(5):385-522 1988.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. Proceedings of the International Conference Learning Represen- tations, 2019.
Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Aaaristiss-Simllatioon and Computation, 1803):1059-1076 1989.
Cheongjae Jang, Yung-Kyun Noh, and Frank Chongwoo Park. A riemannian geometric framework for manifold learning of non-euclidean data. Advances in Data Analysis and Classification, pp. 2020.
Diederik P Kingma and Max Welling. 142. 2013.
Auto-encoding variational bayes. arXiv preprint
Alexej Klushyn, Nutan Chen, Richard Kurle, Botond Cseke, and Patrick van der Smagt. Learning hierarchical priors in vaes. arXiv preprint arXiv: /005::/905044888 2019.
Mark A Kramer. Nonlinear principal component analysis using autoassociative neural networks. AIChE journal, 37(2):333-243, 1991.
Wolfgang Kühhel.. Differential geometry, volume 77. American Mathematical Soc., 2015.
Yonghycon Lee, Hyeokjun Kwon, and Frank Park. Neighborhood reconstructing autocncoders. Advances in Neural Information Processing Systems, 34:536-546, 2021.
Yonghyeon LEE, Sangwoong Yoon. MinJun Son, and Frank C. Park. Regularized autoencoders for isometric representation learning. International Conference on Learning Representations, wtte/foormmm tttp://fffrmmmmmmmt.mmmmmm
2022. URL htt.:/tttttttmmmtt
10

---

Under review as a conference paper at ICLR 2023
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, lan Goodfellow, and Brendan Frey, Adversarial autoencoders. arXiv preprint arXiv: 2015.
Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In
International Conference Machine Learning, pp. 7054. PMLR, 2020.
Frank Park and Roger W Brockett. Kinematic dexterity of robotic mechanisms. The International Journal of Robotics Research, 13(1):1-15, 1994.
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto- encoders: Explicit invariance during feature extraction. In Icml, 2011.
Simon Schönenberger, Anastasiia Varava, Vladislav Polianskii, Jen Jen Chung, Roland Siegwart, and Danica Kragic. Witness autoencoder: Shaping the latent space with witness complexes.
Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. Ntu rgb+ d: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1010-1019, 2016.
Tolstikhin, O Bousquet, S Gelly, and B Schölkopf. Wasserstein auto-eneoders. 6th International Conference on Learning Representations (ICLR 2018). OpenReview. net, 2018.
lyy Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto- encoders. arXiv preprint arXiv:1711.01555 2017.
Jakub Tomezak and Max Welling. Vae with a vampprior. In International Conference on Artificial Intelligence and Statistics, pp.
PMLR, 2018.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and
Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. Journal of machine learning research, 1 11())), 2010.
Wei Wang, Yan Huang. Yizhou Wang, and Liang Wang. Generalized autoencoder: A neural network framework for dimensionality reduction. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 490-497, 2014.
A APPENDIX
The appendix is organized as follows: (A.T) Related Works, (A.2) Proof of Proposition I
the Extrinsic Curvature, (A.4) Experiment Details, A.5 Additional Experiment Results, and A.6) Computational Complexity.
A.1 RELATED WORKS: REGULARIZED AUTOENCODERS
The framework of autoencoding together with the recent advances in deep learning techniques used for approximating arbitrary complex functions successfully addresses the manifold learning prob- lem (Kramer 1991). The core idea is learn two mappings an encoder g D + RT and decoder f reconstructs the given data points z; E D,,, i.e., f (a;;) R x;, for 1,... ,N, and that the data points approximately lie on the image of the decoder, which we refer to as the learned manifold.
RD approximated with deep neural networks so that the composition of them
Many existing autoencoder regularization methods have focused on the representation learning per- spective of autoencoders and studied how to regularize the latent space distributions for purposes like sampling, topology and geometry preserving, clustering, or capturing hierarchical structure (Ri- fai et al. 2011 Kingma & Welling 2013 Wang et al.) 2014 Makhzani et al.) 2015): Tolstikhin ct al. [2077] | Schönenberger et al.] Duque al. 2020 Chen al. 2021); since the latent space distributions are entirely determined by the encoders, they mostly focus on regularizing the encoders but not decoders.
J0inn Chen et 20.6| Tomczak Welling 2018 Klushyn et al.) [219 Moor et al. 2020)
As discovered in (Lee et al. 2021). to learn the accurate manifold in the presence of data noise given a small number of training data, regularization of the decoder is indeed more important,

---

Under review as a conference paper at ICLR 2023
because it is the decoder that has information about how the manifold lies in the data space. Based on the intuition that local approximation of the decoder contains local geometric information on the decoded manifold (i.e. learned manifold), e.g., local linear approximation of f spans the tangent space, a priori constructed neighborhood graph is employed to regularize the local approximation of the decoder and hence the decoded manifold. This has shown improved manifold learning accuracy for both noisy and small training dataset cases, however obviously, the performance largely depends on the quality of the graph in many other graph-based methods.
There are graph-free autoencoder regularization methods that regularize not only an encoder but also the decoder. Denoising autoencoder (Vincent al., 2010) trained reconstruct a corrupted input to its clean version with the following loss
Σ | - Mags; + +))1;
for some noise variable e. As a limit case in (Alain & Bengio 2014), the Jacobian of the reconstruc- tion function is minimized where the loss defined as follows:
0fog Or (x))
Σ - F((())))) -((=))())))]
+ a
where a is the regularization coefficient and || || denotes the Frobenius norm. These by construc- tion attempt to learn manifolds robust to noise, but we note that (i) they are designed to be robust to noise during inference after being trained with clean data, but if training data points themselves are noisy, the robust manifold learning performance decreases and (ii) their regularization effects are limited to where data points are available.
Since regularizing the decoder that explicitly parameterizes the manifold is important, one may consider minimizing the norm of decoder's Jacobian as
Of Σ - ((o=((((z))11 + a 0z (g(x,)))
(10)
with the regularization coefficient the norm of decoder's Hessian Σx,/ dz.da; (o(2:))) How- ever, these norms not capture geometric quantities of the learned manifold because they are not coordinate-invarian or reparmetrization- invariant, and thus they do not produce any meaningful regularization effects.
8'
For example, consider coordinate transformation z' lx()) which converts the encoder as y g' h g and decoder as f + I' = I --_ The reconstruction loss is invariant since f fog f'og', and hence the learned manifold is invariant, but the regularization term, the norm of decoder's Jacobian, is different:
0s' Oz (o'( = 2 (g(x:)) dz (o'(z:))) + 02 (o(=:))).
0f
ah-1
0s
(11)
0=
This implies that can minimize the norm of decoder's Jacobian just by increasing the norm of Jacobian of ht without actually changing the learned manifold. A similar argument holds for the Hessian norm.
Recent works (Chen et al. 2020 LEE et al. 2022) have suggested decoder regularization methods for learning isometric representations that preserve geometry of the data space. common goal is to learn a decoder f IR + R² that satisfics
0s (z) cl for all a E ((R as (z) 0z
(12)
az
for some positive scalar c, where is the m x m identity matrix and v(RT) is the support of the latent space data distribution. Such mappings et al. 2022), which are scometry-preserving mappings in the sense that latent space straight lines mapped to the geodesic curves in the learned manifold.
formally defined scaled isometries in (LEE
Regularizing the decoder to be a scaled isometry, beyond finding geometry pwreettiippeeeviiggggg. represen- tations, has implicit manifold regularization effect. According Gauss's Theorema Egregium
12

---

Under review as a conference paper at ICLR 2023
which states that "The Gaussian curvature of surface is invariant under local isometry". for scaled isometries f to exist, the Gaussian curvature of the learned manifold should be the same as that of the Euclidean space (i.e., zero). In other words, it has an implicit intrinsic curvature minimiza- tion effect, which is different from the method proposed in this paper that explicitly minimizes the extrinsic curvature.
A.2 PROOF OF PROPOSITION T
Proof. Let's denote by
(F( @E(Jo),
c00 = Σ [J (J]Jo) Jo) kg
Tr(( dzi
θzj
1,j
Given coordinate transformation 2' = h22) that maps (go. fo) + (g fe.) = (h 9 fo h )
02 some scalar-valued function ((z). We note that, since F.( (JJA) for some arbitrary invertible
the following transformation rules hold: Jo + Jo = Jg 8h -1 and ar
+ 82 02 05
ar ar 8h for
matrix A, the embedding is invariant, i.e., E(Jo') = F(Jo). Let Tas denote the (o, ())mmmmee
of (Jo). then, by using Tr(AB) Ea, Aag and denoting Øh
0= by H.
Olag 0laa
c0, + (0,',, .6) Σ
(J ):3
Σ
0z
0z
0.3
ala8
01 )z 0z
Σ
(J -1
Oz
Ilaß H(H J/ JoHH)-111
dz
0z
0
aLag
0lpa T 0z
Σ a
(Jh Jo)
Oz
dLa8 0l la
(J
Jo)
11
Σ
c(0, 6).
(13)
Oz: 0zj
i,j
a
A.3 ON THE EXTRINSIC CURVATURE MEASURE
In this section, we we will derive the expression of our extrinsic curvature measure
E TT JJJJJ (JJJJ
Oz
Ozi
i.
for a one-dimensional manifold, i.e., curve, embedded in RD. Let RR RD be smooth curve and assume that it is parameterized by arc-length, i.c., becomes
Oi 15
f J = 1. Then, the curvature
DJJJ Tr(( dz
dr Ox
)-)= Tr((
))²)
Oz Oz dz
Tr(( 8x dx
dr 8x. T
)2)
2² 0z
Oz d2
Ix Or
8x 0a
8 x 0r
dr 8x T
dz 02 T
dx 02x T
Tr( 022 dz
2
+
0z Oz
カz 0z
dz 02
dz 0
dz 02
Ox T 8 x Or
8x
0x 0z 8r
T
8r
px T 0r
Oz 02 Oz
02
Oz
Oz 02
0z2
02 Oz 0=
Oz
2( dr T 8x 2
8 Px
(14)
0z 0.2
02
02
13

---

Under review as a conference paper at ICLR 2023
Since 82 0z = O implies that On T d²x
2% 00 = 0, our curvature measure for an arc-length parameterized
curve x22 is simplified to 2| 0²
0.2 that is twice the norm of second derivative. This is equivalent to the classical definition of the curvature of a curve.
A.4 EXPERIMENT DETAILS
The image size is 28 28 and the pixel values are normalized between O and 1. The encoder and decoder are two-layer fully connected neural networks with the ELU activation functions and 512 nodes for each layer. The output layer is linear for the encoder and sigmoid for the decoder. For clean dataset cases, we use the following early stopping criteria in training: we stop the training the mean reconstruction error for the validation dataset increases 10 times in a row; then we use the best model (i.e. the lowest validation errors) for evaluation. For noisy dataset cases, assuming that we don't 't have an access to the clean dataset during training. we do not use the early stopping and trained the model for sufficiently big number of epochs for convergence (the number of epochs is 1000). The number of test data is 60000. For evaluation, we use clean test data for noisy training dataset cases as well. The batch size is 100 and the learning rate is 0.001.
Grayscale Image Data:
The image size is 32 x 32 and the pixel values are normalized between O and 1. For noisy training dataset experiments, we add noises as follows: (i) for Gaussian noise, the standard deviation is 0.1, (ii) for Shot noise, we multiply 0.15 to noise variables sampled from the Poisson distributions where are image pixel values, and (iii) for Impulse noise, with 5% probability we randomly add to each pixel. The encoder and decoder are convolutional and transposed convolutional neural networks with the ReLU activation functions, where, denoting a convolution layer of input channel size Cs output channel size Co. kernel size k. stride 8, and padding p by Conv2d(c,, Cos k 8, p) and transposed convolution layer by Conv ConyTrans2d(c, Co. k. 8, p), the following sequence of layers Conv2d(3, 128, 4, 2)-Conv 2d( 128, 256, 4, 2)-Conv2d(256, 512, 4, 2) 2d 1024, 2. 2)-Conv2d( (1024, 64, 1) is used for encoder and Conv Trans22422264 1024, J.ConnTrans20r 1024, 512, 4, 2, 4CConTTran22(55) 12, 256. 4, 2, ConvTrans2d(512, 3, 1) for decoder. The output layer is linear for the encoder and sigmoid for the decoder. For clean dataset cases,
SVHN& & CIFARIO Image Data:
the following early stopping criteria in training: we stop the training if the mean reconstruction error for the validation dataset increases 10 times in a row; then we use the best model (i.e. the lowest validation errors) for evaluation. For noisy dataset cases, assuming that we don't 't't have an access to the clean dataset during training, we do not use the early stopping and trained the model for a sufficiently big number of epochs for convergence (the number of epochs is 100). The number of test data is 63257 for SVHN and 10000 for CIFAR10. For evaluation, we use clean test data for noisy training dataset cases as well. The batch size and the learning rate is 0.0001.
Human Skeleton Pose Data: From the NTU RGB+D dataset, set of human pose skeleton data that consists of 25 key points extracted and pre-processed to be aligned. Specifically, 10000 poses extracted from each action class (a total of 60 action classes is used), and they are rotated and translated so that the 1-2 key points direction becomes z-axis and 1-13 key points direction becomes the y-axis and the key point number 2 becomes the origin. The encoder and decoder are two-layer fully connected neural networks with the ELU activation functions and 512 nodes for each layer. the
The output layers are linear for both the encoder and decoder. For clean dataset cases, following early stopping criteria in training: we stop the training if the mean reconstruction error for the validation dataset increases 10 times in row; then we use the best model (i.e. the lowest validation errors) for evaluation. For noisy dataset cases, assuming that we don't have an access to the clean dataset during training. we do not use the early stopping and trained the model for a sufficiently big number of epochs for convergence (the number of epochs is 5000). The number of test data is 9000. For evaluation, we use clean test data for noisy training dataset cases as well. The batch size is 100 and the learning rate is 0.0001.
A.S ADDITIONAL EXPERIMENT RESULTS
More Qualitative Results: Figure II
13
15
16 show additional de-noising results for
image data and human skeleton pose data.
14

---

Under review as a conference paper at ICLR 2023
MNIST
FMNIST
KMNIST
Input
8
C
3
3
7
2
4
2
3 3
2
4
7
S
RCAE
2
4
Q
8
2
S
DCAE
3
2
4
Q
2
3
8
7
R
E
DHAE
3
2
4
4
2
3
6
2
5
37
E
IRAE
3
3
2
4
4
3 8 7 C
P
3
MCAE
3
Figure 11: De- De-noising examples of grayscale image data (noise level 0.1).
MNIST
FMNIST
KMNIST
Input
7
7
4
6
7
7
9
G
5
8
5
PIAEE A
DAE
7
4
5
4
6
5
A
RCAE
7
4
4
8
MAAA A
DCAE
L
6
DHAE
7 9
6 4
8
S
IRAE
9 9
6
MCAE
7
9 9
4
Figure 12: De-noising examples of grayscale image data (noise level 0.2).
MNIST
FMNIST
KMNIST
Input
9
7
3
6
7
7
9
7
3
6
7
O
7
3
6
RCAE
O
9
7
3
6 6 6 6
DCAE
6
9
7
3
DHAE
6
6 6
9
7
3
7
IRAE
8
9
7
3
MCCE
9 7
3
?
Figure 13: De-noising examples grayscale image data (noise level 0.3).
Gaussian
Shot
Impulse
Input
119
AE
1919
2
VAE
9393
DAE
9393
RCAE
6393
DCAE
9392
DHAE
19 2
IRAE
19 23
MCAE
Figure 14: De-noising examples of SVHN data.
15

---

Under review as a conference paper at ICLR 2023
Gaussian
Shot
Impulse
Input
AE
VAE
DAE
RCAE
DCAE
DHAE
IRAE
MCAE
Figure 15: De-noising examples of CIFAR1O data.
Noïse level
Clean
Nolsy
AC
VAC
DAE
RCAE
DCAE
DI
IRAC
MCAD
8
0.05
$
0.1
0.05
0,1
0.1
0.1
0.05
0.1
0.05
14444
0.1
0.05
Figure 16: De-noising examples of human pose data.
A.6 COMPUTATIONAL COMPLEXITY
In this section, we provide actual computation time of the curvature measure in and backprop- agation time with two-layer fully connected neural networks used for gray-scaled image data and convolutional and transposed convolutional neural networks used for the SVHN and CIFARIO im- age data. Throughout this study, the NVIDIA GeForce RTX 3090 is used.
16

---

Under review as a conference paper at ICLR 2023
Table 5: Per-batch computation time comparisons. For the FC net with x 28 28 image, the latent space dimension is 16 and the batch size is 100, and for the Conv net with 3 x 32 x 32 image, the latent space dimension 64 and the batch size is 8 (for GPU memory limitation).
net with 1 28 x 28 image
Coav net with 3 x 32 x image
Forward Computation
Back Propugation
Forward Computation
Back Propagation
Reconstruction
0.00037 s
0.00045 s 0,01447
0.00220
0.00114 0.06051
Curvature
Table 6: Per-batch computation times of the intermediate operations in curvature measure with the Conv net with 3 x 32 x 32 image and 64-dimensional latent space.
Jf J
G.
(LR (G,
0.00192s
0.28353
0.00682
0.006477 s
Table 7: Per-batch computation times and percent errors of the approximate matrix inverse G in curvature measure and 64-dimensional latent space.
7 the number of iteration increases with the Conv net with 3 x 32 x 32 image
number of iterations
ground truth
10
100
1000
time pereent error
0,28353
0.000256 99.83 %
0.000507 97.69 %
0.000807
0.005669:
0.053519
0%
71.35 %
0,0109 %
0.0095 %
Table shows the per-batch computation time comparisons between reconstruction loss term and curvature term in (7). Although the curvature computation has become feasible through the stochas- tic trace estimation, compared to the original reconstruction loss term, it still takes much longer time. Especially, looking at the forward computation time for the Conv net case, the curvature computation almost 100 to 150 times slower than the reconstruction term computation.
To see which part in the below curvature measure
Ow E(Jo)) ((E(Jo)w) Go'o]
C(O, o) (ww.-:hhehhwwwW.........w..ww
Oz
Oz
requires major computational cost, compare the computation times of the following operations: (i) the Riemannian metric Go = JT Jos (ii) the inverse Gg. (iii) the Jacobian vector product for v, and (iv) the Jacobian- vector product for ((E(Ja)
80 ((E()))
(G u).
0z
02
Table 6 shows the per- batch computation times of the intermediate operations in curvature measure 7 for the Conv net case. As can be seen, the inverse computation takes up most of the total compu- lation time. To reduce the computation time of the matrix inverse, one consider an approximate inverse computation method. For example, given Gg. let us define a function f Rmxm - R mm such that
f(X) = X Ga. Go.
(15)
To find the root f. we can use the standard Newton-Raphson method:
Xn+1 2X,, Gg,
(16)
which is known the Newton-Schulz iteration method for the matrix inversion. We can get an approximation of Gp 1 by iteratively applying the above, where it gets closer to the true inverse as increase the number of iteration.
Table 7 shows the per-batch computation times and percent errors of the approximate matrix inverse G in curvature measure (7) the number of iteration increases with the Conv net case. The
percent error computed 100 |/rrul G / When the number of iterations is set to be 100, the percent eтror is only 0.01 % while significantly reducing the computation time as 0.28353 + 0.005669 s. highly recommended to use the approximate matrix inverse when the latent space dimension is high.
17


Reviews:
[{"review_id": "xzFzfQKraJ", "reviewer": "Reviewer_xNDN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an extrinsic measure of the curvature of the manifold spanned by a decoder. This measure is then used in an autoencoder as a regularizer. In practice, the curvature measure is approximated to limit computational costs. Empirical results are limited to small models, but results appear promising.", "summary_of_the_review": "The paper has a clearly explained and novel contribution, with a somewhat unclear empirical impact. I vote in favor of acceptance, but the empirical aspect leave me only luke-warm.", "strength_and_weaknesses": "## Strengths\n* I found the paper to be well-motivated and well-written.\n* The proposed measure of curvature seems both novel and well-founded.\n* I found the computational approximation used to numerically evaluate the curvature measure to be interesting and novel. (see also Weakness below)\n\n## Weaknesses\n* The main paper presents the \"theoretical\" way to evaluate the curvature measure, while the \"practical\" implementation is only described in the appendix. I understand that space is limited, but it feels like this aspect of the work is being hidden (does it not work well?). The idea of replacing the inverse metric with a Gauss-Newton approximation is, not entirely novel, but within the context, I find it novel. As such, I really think this part of the work should be in the main paper, and further investigated. I really miss an empirical evaluation of the influence of this approximation. Does it not depend on an initial guess of the inverse metric (i.e. a preconditioner)? The appendix indicates that a single step of Gauss-Newton is sufficient, but is it really? I'd like to see an empirical study here. In general, I think this part of the work is too important to brush aside and not give a more thorough study.\n* I am somewhat skeptical about the idea of using mix-up as a way to evaluate the expectation over the latent representations. I understand the motivation and trust that this is probably a sensible idea. However, this mix-up is also a regularize on its own, which then makes it difficult for me to determine the importance of the curvature regularize as you now have two regularization schemes working in parallel. I really miss some sort of ablation study here showing that the regularization induced by mix-up is not what is driving the empirical results. Without this, I find it difficult to judge the presented empirical results.\n* In general, the empirical results are limited to quite small models. This is understandable for a paper of this type, but it is still a limiting factor. It would be good to demonstrate that the approach can scale to large models.\n\n## Minor things:\n* In the introduction, the term \"graph\" is used. I suppose this refers to the image of the latent space under the decoder, but it would be good to be explicit about what is exactly meant. I can easily imagine confusion caused by some readers thinking about discrete grasp (collections of nodes and edges).\n* In the weaknesses part of the conclusion it is mentioned that it would be good to adapt the regularization strength locally. A recent paper does exactly this, but uses a notion of geometric reach rather than curvature (those are tightly linked topics). https://arxiv.org/abs/2206.01552 It would be good to include this line of work in the related work section.\n* In the last sentence of the first paragraph of section 2.2, you should write $f$ rather than f.\n* The reference to \"LEE et al.\" should be \"Lee et al.\"\n* While I appreciate the opening example of how Jacobian regularizers are sensitive to volume changes, I think this particular example is not entirely convincing as many people, in practice, place Gaussian priors over the latent representations thereby \"fixing\" the volume issue (at least in practice). I get the theoretical argument, but I wonder if another example could be used?\n* In definition 1, does h have to be a smooth function as well (diffeomorphism)?\n* The last sentence of page make it sound like alpha is called the MCAE. I propose to rephrase.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VoLiFZNbwA5", "reviewer": "Reviewer_fxZi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates an unsupervised learning/manifold learning paradigm focusing on the auto-encoder framework and proposes a new regularization formula for the latent space. The authors develop a notion of curvature for the manifold that is extrinsic yet reparameterization invariant which they argue is desirable in contrast to some recent similar prior works in the area. Conceptually the curvature formulation measures the integral of an infinitesimal change in a tangent space at a point on the manifold and utilizes the Grassman manifold structure of linear subspaces. \n\nExperimentally, the authors compare 3 different scenarios, 1 synthetic, image (MNIST, SVHN, CIFAR10) and motion capture dataasets and focus on comparing with only other auto-encoder methods. The results seem to suggest that the proposed curvature minimization in the auto-encoder framework yields a degree of robustness (though not explicitly trained).", "summary_of_the_review": "All in all. I am inclined to weigh in negatively, mainly because I am not fully convinced of the efficacy of the results. Despite that, I did find the paper to be an interesting read and wish the authors can elaborate more on the seemingly novel and interesting aspects of their contribution namely - the desire for extrinsic curvature and reparameterization invariance. ", "strength_and_weaknesses": "Strengths\n\n- The paper reads well and the arguments in the motivation seem persuasive \n- I found the curvature construction interesting, although I am not fully sure how novel it is (see weaknesses) \n- Figure 3 to me was the highlight of this paper, where the proposed method achieves lower reconstruction errors for the same degree of curvature reduction \n\nWeaknesses\n\n- Unfortunately, the practical benefits of this paper seem very limited - even in the context of auto-encoder manifold learning methods. Tables 2, 3 and 4 do not post a convincing picture and when I put that together with Figure 3(a) I find it hard to argue for a substantial practical gain of this method with IRAE since it appears even IRAE minimizes the curvature without really modeling it in the proposed way. \n- I think the authors must provide more historical/literature background for the curvature definition for manifolds. For eg, how is the proposed formulation related to Rici curvature? and why minimizing that is not beneficial? Despite the minor references, a direct formulation of curvature without any such background feels odd. \n- Figure 4 is not clear. Please provide a more clear visualization of where the manifold are badly approximated \n\nQuestions and Suggestions\n\n- Could there be an experimental demonstration of the need for reparametrization invariance? Conceptually it makes sense, but I don’t fully see why it should be a big deal in practice (I suppose overfitting to the discretization of the manifold? but how bad is it really?) The same applies to “extrinsic” measures. It feels a bit too less to only have a diagram in figure 1 and not something concrete in practice. I make this remark because demonstrating such a case could provide more meat to the contribution of this paper, especially in contrast to IRAE\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "B02wG_qNbId", "reviewer": "Reviewer_idbz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Keeping the coordinate-invariant property in mind, the author(s) propose the minimum extrinsic curvature principle for manifold regularization and a Minimum Curvature Autoencoder.   The main focus is to take an appropriate regularization for the decoder function in the autoencoder, as mentioned in the paper as the decoder has information about how the manifold lies in the data space. They introduce a coordinae-invariant extrinsic curvature measure by investigating how smoothly tangent space changes on the manifold and use it as a regularizer.  Under this new regularized, the experiments have shown improved manifold learning accuracy for both noising and small training datasets.", "summary_of_the_review": "Overall the paper has its merit in novelty. It is basically and logically clear with convincing experiments. ", "strength_and_weaknesses": "Strength:\n\n1. Take the view of focusing on regularizing decoder in an autoencoder framework. This is implemented through the newly introduced minimum extrinsic curvature principle for manifold regularization, which is based on the Dirichlet energy for mappings between Riemannian manifolds, while they specially look at the latent space and the Grassmann manifold of the data manifolds in the data space. \n\n2. The new manifold regularizer is introduced into the classic autoencoder objective to form the minimum curvature autoencoders\n\n3. Practical implementation issue has been carefully attained.\n\nWeakness:\n\n1. As the paper mentions, the intrinsic curvature does not capture how the manifold lies in the data space, particularly in dimension higher than 2. Although the new definition of the curvature generalizes classical definition of the curvature of a curve embedded in R^3, it is not clear whether the new measure has better regularization power given that less geometry information is used, for example, torison and/or tensor on manifold.   I know this is not actionable point, but wish to point out.\n\n2. Perhaps an experiments in highdimensional data set.\n\n3. Better readibilty, some details should be added in, see below. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject