Isometric Autoencoders
Authors: Amos Gropp
Matan Atzmon
Yaron Lipman
Venue:  (2021)
Keywords: manifold learning
autoencoders

Abstract: High dimensional data is often assumed to be concentrated on or near a low-dimensional manifold. Autoencoders (AE) is a popular technique to learn representations of such data by pushing it through a neural network with a low dimension bottleneck while minimizing a reconstruction error. Using high capacity AE often leads to a large collection of minimizers, many of which represent a low dimensional manifold that fits the data well but generalizes poorly.

Two sources of bad generalization are: extrinsic, where the learned manifold possesses extraneous parts that are far from the data; and intrinsic, where the encoder and decoder introduce arbitrary distortion in the low dimensional parameterization. An approach taken to alleviate these issues is to add a regularizer that favors a particular solution; common regularizers promote sparsity, small derivatives, or robustness to noise.

In this paper, we advocate an isometry (i.e., local distance preserving) regularizer. Specifically, our regularizer encourages:  (i) the decoder to be an isometry; and (ii) the encoder to be the decoder’s pseudo-inverse, that is, the encoder extends the inverse of the decoder to the ambient space by orthogonal projection.  In a nutshell, (i) and (ii) fix both intrinsic and extrinsic degrees of freedom and provide a non-linear generalization to principal component analysis (PCA). Experimenting with the isometry regularizer on dimensionality reduction tasks produces useful low-dimensional data representations.

Under review as a conference paper at ICLR 2021
ISOMETRIC AUTOENCODERS
Anonymous authors
Paper under double-blind review
ABSTRACT
High dimensional data is often assumed to be concentrated on near low- dimensional manifold. Autoencoders (AE) is a popular technique to learn represen- tations of such data by pushing it through neural network with low dimension bottleneck while minimizing a reconstruction error. Using high capacity AE often leads to a large collection of minimizers, many of which represent a low dimen- sional manifold that fits the data well but generalizes poorly.
Two sources of bad generalization are: extrinsic, where the learned manifold pos- sesses extraneous parts that are far from the data; and intrinsic, where the encoder and decoder introduce arbitrary distortion in the low dimensional parameterization. An approach taken to alleviate these issues is to add a regularizer that favors a particular solution; common regularizers promote sparsity, small derivatives, robustness to noise.
In this paper, we advocate an isometry (i.e., local distance preserving) regularizer. Specifically, our regularizer encourages: (i) the decoder to be an isometry; and (ii) the encoder to be the decoder's pseudo-inverse, that is, the encoder extends the inverse of the decoder to the ambient space by orthogonal projection. In a nutshell, (i) and (ii) fix both intrinsic and extrinsic degrees of freedom and provide a non-linear generalization to principal component analysis (PCA).
Experimenting with the isometry regularizer on dimensionality reduction tasks produces useful low-dimensional data representations.
INTRODUCTION
common assumption is that high dimensional data X C R.D sampled from some distribution p concentrated on, near, some lower d-dimensional submanifold M c RP, where d << D. The task of estimating p can therefore be decomposed into: (i) approximate the manifold M; and (ii) approximate p restricted to, or concentrated near M.
In this paper we focus on task (i), mostly known as manifold learning. A common approach to approximate the d-dimensional manifold M, e.g., in (Tenenbaum et al. 2000 Roweis & Saul 2000 Belkin & Niyogil 2002 Maaten & Hinton| 20088] McQueen et al. [016]] McInnes et al. 2018)) is embed X in 1 This is often done by first constructing graph G where nearby samples in X'
are conngected by edges, and second, optimizing for the locations of the samples in IR striving to minimize edge length distortions in G.
Autoencoders (AE) can also be seen as method to learn low dimensional manifold representation of high dimensional data X' AE are designed reconstruct X' as the image of its low dimensional embedding. When restricting AE to linear encoders and decoders it learns linear subspaces; with mean squared reconstruction loss they reproduce principle component analysis (PCA). Using higher capacity neural networks the encoder and decoder, allows complex manifolds to be approximated. To avoid overfitting, different regularizers are added the AE loss. Popular regularizers include sparsity promoting (Ranzato al.) 2007 2008 Glorot et al., 2011)) contractive penalizing large derivatives (Rifai et al.) [201 la b). and denoising (Vincent et al. 2010 Poolc et al. 20144). Recent AE regularizers directly promote distance preservation of the encoder (Pai al. 2019 Peterfreund et al. 2020).
In this paper we advocate novel AE regularization promoting isometry (i.e., local distance preserva- tion), called Isometric-AE (I-AB). Our key idea is to promote the decoder to be isometric, and the encoder to be its pseudo-inverse. Given an isometric decoder Rd R,P, there is well-defined

---

Under review as a conference paper at ICLR 2021
inverse I2 - Rd; we define the pseudo to be a projection the image of the decoder composed with the inverse of the decoder restricted to its image.
Locally, the I-AE regularization therefore encourages: (i) the differential of the decoder A RDxd to be an isometry, i.c., A" A = Id. where L is the d x d identity matrix; and (ii) the differential of the encoder, B E TR D to be the pseudo-inverns (now in the standard linear algebra sense) of the differential of the decoder A E RDxxd namely, B = A+ In view of (i) this implies B = A". This means that locally our decoder and encoder behave like PCA, where the encoder and decoder are linear transformations satisfying (i) and (ii); That is, the PCA encoder can be seen as a composition of an orthogonal projection on the linear subspace spanned by the decoder, followed by an orthogonal transformation (isometry) to the low dimensional space.
In sense, our method can be seen as a version of denoising/contrrative AEs (DAE/CAE, respectively). DAE and CAE promote a projection from the ambient space onto the data manifold, but can distort distances and be non-injective. Locally, using differentials again, projection on the learned manifold means (AB)² - AB. Indeed, as can be readily checked conditions (i) and (ii) above imply A A)B AB. This means that I-AE also belongs to the same class of DAE/CAE, captur- ing the variations in tangent directions of the data, M,, while ignoring orthogonal variations which often represent noise (Vincent et al.) 2010) Alain & Bengio 2014). The benefit in I-AE is that its projection data manifold is locally an isometry, preserving distances and sampling the learned manifold evenly. That is, I-AE does not shrink or expand the space; locally, it can be imagined as an orthogonal linear transformation. eeperiieeeeeie The inset shows results of a simple experiment comparing contractive AE (CAE-bottom) and isometric AE (I-AE-top). Both AEs are trained on the green data points; the red arrows depict projection of points (in blue) in vicinity of the data onto the learned manifold (in black) as calculated by applying the encoder followed by the decoder. Note that CAE indeed projects on the learned manifold but not evenly, tending to shrink space around data points; in contrast I-AE provides a more even sampling of the learned manifold.
the
Figure 1: Top: I-AE; bot- tom: CAF
Experiments confirm that optimizing the I-AE loss results in close-to- -o-ismet-i encoder/decoder explaining the data. We further demonstrate the efficacy of I-AE for dimensionality reduction of different standard datatsets, showing its benefits over manifold learning and other AE baselines.
2 RELATED WORKS
Manifold learning. Manifold learning generalizes classic dimensionality reduction methods such as PCA (F.R.... 1901) and MDS (Kruskal 1964; Sammonn 1969), by aiming to preserve the local geometry of the data. Tenenbaum et al. (2000) use the nn-graph to approximate the geodesic distances
over the manifold, followed by MDS to preserve it in the lower dimension. Roweis & Saul (2000);
Belkin & Niyogi 2002);; Donoho & Grimes (2003) use spectral methods to minimize different distortion energy functions over the graph matrix. Coifman et al. 2005); Coifman & Lafon (006)
approximate the heat diffusion over the manifold by random walk over the nn-graph, to gain
robust distance measure on the manifold. Stochastic neighboring embedding algorithms (Hinton) & Roweis 2003 Maaten & Hinton 2008) captures the local geometry of the data as a mixture of Gaussians around each data points, and try to find a low dimension mixture model by minimizing the KL-divergence. In relatively recent work, McInnes et al. (2018) use iterative spectral and embedding optimization using fuzzy sets. Several works tried to adapt classic manifold learning ideas to neural networks and autoencoders. Pai et al. (2019) suggest to embed high dimensional points into a low dimension with neural network by constructing metric between pairs of data points and minimizing the metric distortion energy. Kato et al. (2019) suggest to learn an isometric decoder by using noisy latent variables. They prove under certain conditions that it encourages isometric decoder. Peterfreund et al. 2020) suggest autoencoders that promote the isometry of the encoder over the data by approximating its differential gram matrix using sample covariance matrix. Zhha jet al. (2018) encourage distance preserving autoencoders by minimizing metric distortion energy in common feature space.
2

---

Under review as a conference paper at ICLR 2021
Modern autoencoders. There is extensive literature extending autoencoders to generative model (task (ii) in section That is, learning probability distribution in addition to approximating the data manifold M. Variational autoencoder (VAE) Kingma & Welling (2014) and its variants
Makhzani et al. )0 Burda et al. (2016); Sonderby et al. (2016): (iggins et al. (2017); Tolstikhin al. 8); Park et al. (2019); Zhao al, (2019) are examples to such methods. In essence, these methods augment the AE structure with learned probabilistic model in the low dimensional (latent) space Hd that is used approximate the probability P that generated the observed data X. More relevant to our work, are recent works suggesting regularizers for deterministic autoencoders that together with ex-post density estimation in latent space forms generative model. Ghosh al.) (2020) suggested to reduce the decoder degrees of freedom, either by regularizing the norm of the decoder weights or the norm the decoder differential. Other regularizers of the differential of the decoder, aiming towards a deterministic variant of VAE, were recently suggested in Kumar & Poole (2020); Kumar et al. (2020). In contrast to our method, these methods do not regularize the encoder explicitly.
3 ISOMETRIC AUTOENCODERS
We consider high dimensional data points X' {0:) C CR sampled from some probability distribution ((x) in RDD concentrated some d dimensional submanifold M C RD, where d < D.
N
p(x)
Our goal compute isometric autoencoder (I-AE) defined as follows. Let _D_ + IRd denote the encoder, and f = I2 + RD the decoder; M the learned manifold, i.e., the image of the decoder, / f(R"). I-AE is defined by the following requirements:
g(x)
f(z)
R
The data X is close N.
Figure 2: I-AE.
(ii) an isometry.
(iii) is the pseudo-inverse of
Figure is illustration of I- I-AE. Let & denote the parameters of f, and the parameters of g. We
enforce the requirements i in by prescribing loss function L(0, o) and optimize using standard
stochastic gradient descent (SGD). We next break down the loss T. to its different components.
Condition i1 is promoted with the standard reconstruction loss in AE:
L (0, x)
Σ [lf(((z))) - xx11? -x,
(1)
where ||-|| is the 2-norm.
let us first define the notions of isometry and pseudo-inverse. A differentiable mapping between the euclidean spaces Rd and R. is local isometry if it has an orthogonal differential matrix df (z) e Dxd
Before handling conditions 11
ii
((z) Ia.
where Ia E IRd is the identity matrix, and df (z)1; = af (z). ) A local isometry which is also a diffeomorphism a global isometry. Restricting the decoder to isometry is beneficial for several reasons. First, Kuiper Embedding Theorem Nash (1956) asserts that non-expansive maps can be approximated arbitrary well with isometries if D d and hence promoting an isometry does not limit the expressive power of the decoder. Second, the low dimensional representation of the data computed with an isometric encoder preserves the geometric structure of the data. In particular volume, length, angles and probability densities are preserved between the low dimensional representation Rd and the learned manifold A Lastly, for a fixed manifold N there is huge space of possible decoders such that N = (R For isometric f. this space is reduced considerably: Indeed, consider two isometries parameterizing N, i..., f1, f2 Ed + N. Then, since composition of isometries is an isometry we have that fo fi Rd - 1d a dimension dimension-preservinng isometry and hence rigid motion. That is, all decoders of the same manifold are the up rigid motion.
3

---

Under review as a conference paper at ICLR 2021
For the encoder the situation is different. Since D d the encoder g cannot be isometry in the standard sense. Therefore we ask g to be the gheudo-inverss f. For that end we define the projection operator p submanifold N c R as
p(x) arg min | x'1l
æ'EN
Note that the closest point is not generally unique, however the Tubular Neighborhood Theorem (see e.g., Theorem 6.24 Lee (013)) implies uniqueness for points sufficiently close to the manifold .
Definition 1. We say the g is the pscudo-inverse of f can be written g = f o p. where is the projection on N = ((R)).
Consequently, if g is the pseudo-invers of an isometry f then it extends the standard notion of isometry by projecting every point on submanifold N and then applying an isometry between the d-dimensional manifolds N and R. See Figure 2 for an illustration.
First-order characterization. To encourage f.g g satisfy the (local) isometry and the pseudo- inverse properties (resp.) we will first provide a first-order (necessary) characterization using their differentials:
Theorem 1. Let f be a decoder and g an encoder satisfying conditions tials A dz) Dxd B dz) R satisfy
ii1
Then their differen-
AT A Ia
(3)
BB La
(4)
B AT
(5)
The theorem asserts that the differentials the encoder and decoder are orthogonal (rectangular) matrices, and that the encoder is the pseudo-inverse of the differential of the decoder. Before proving this theorem, let us first use to construct the relevant losses for promoting the isometry of f and
pseudo-inverse g. We need promote conditions 3 4.. (5) Since want avoid computing the
full differentials A = (zf).. B = dg(f(z)). we will replace (3) and (4) with stochastic estimations
based on the following lemma: denote the unit d 1-sphere by S {= E R4 ||=|| = =]..
Lemma 1. Let A R Dxd, where d S D. If ||uu| for all u E Sa-1 then is column-
orthogonal, that is A A la-
Therefore, the isometry promoting loss, encouraging 3 is defined by
Tiso E.z.. [Id(())))) (1 -
where (R. and (Ro) is probability measure on Rdd u ~ P(Sd-1), and P(8d-1) is the standard rotation invariant probability measure on the d - 1-sphere 1 The pseudo-invers -ivvr-e promoting loss, encouraging 44) would be
Ipiso (0) EE (1 dg(=)]())] 11)
Ey,,.
where P(M) and P(Sd-1). As usual, the expectation with respect to P(M) is computed empirically using the data samples
(5) might seem challenging to enforce with neural networks, however the orthogonality of A, B can be leveraged to replace this loss with a more tractable loss asking the encoder is merely the inverse of the decoder over its image:
Lastly,
Lemma 2. Let A DDx and B Bddd RdxD, If ATA La BB" and BA Ia then B- AT.
Fortunately, this is already taken care of by the reconstruction loss: since low reconstruction loss in equation [] forces the encoder and the decoder to be the inverse of one another over the data manifold, = it encourages B I and therefore, by Lemma automatically encourages equation 5 Note that invertability also implies bijectivity of the encoder/decodeer restricted to the
i.., ((((2) (2))
data manifold, pushing for global isometries (rather than local). Summing all up, we define loss
for I-AE by
L(O, Φ) Luee (0, Φ) Aiso (Liso (0) + Lpiso (o)):
(8)
where Aiso parameter controlling the conontryneoonnttruction trade-off.
4

---

Under review as a conference paper at ICLR 2021
3.1 DETAILS AND PROOFS.
Let us prove Theorem 1 characterizing the relation of the differentials of isometries and pseudo-
isometries, A (z) RDxd, B dg([(=)) RdxD First, by definition of isometry (equation 2). A A = da We denote by T&N the d-dimensional tangent space to N at x E N.
accordingly. l. T& denotes the normal tangent space.
Lemma 3. The differential dp(x)= DxD
N the the projection operator RR - N is
dp(æ))u { TTN
u TTT
That is, dp(x) is the orthogonal projection on the tangent space of N at x.
Proof. First, consider the squared distance function to N defined by n))) - 2 min || - x' The envelope theorem implies that Vny(x) we get dp(x) = In- (<)).. As proved in Ambrosio & Soner 1994) (Theorem 3.1), (2n(()) is the orthogonal projection on TW
p(x). Differentiating both sides and rearranging
Let f() E W. Since N we have p(x) Condition asserts that g f (p(y))
taking the derivative at y x we get dz(x) = df df'1 (x))d((z)) Lemma 3 implies that dp( ax)
AAT since AA is the orthogonal projection
TaA. Furthermore, df (x) restricted to Im A)
is AT Putting this together we get B = dg(x)) = AT AA" A This implies that BB"
la.
and that B A+ AT This concludes the proof of Theorem
Proof of Lemma 7
Writing the SVD of A = UEVI, where Σ diag(o),........... .a) are the singular
values of A, get that Σ uz 1 for all e Sd-1 Plugging ej. j E [d] (the standard basis) get that all σ; 1 for E [d] and A UVT is orthogonal claimed.
Proof of Lemma U AA e DD(DDdd be a completion of A to an orthogonal matrix in IR DDD. Now, Ia BUU' Ia + BVV BT, and since BVV B > O this means that BV O, that takes to null the orthogonal space the column space of A. A direct computation shows that BU = U which in turn implies B AT A+.
Implementing the losses in equation 6land equation Trequires making choice for the probability densities and approximating the expectations. We take Piso (R4) to be either uniform or gaussian fit to the latent codes g(X); and P(M) is approximated as the uniform distribution on X, as mentioned above. The expectations are estimated using Monte- Carlo sampling. That is, each iteration we draw samples & E X. & ~ Puo (R.)) â ~ P(Sd-1) and use the approximations
Implementation.
(() (1)df()) 1)²
Iep (Ø) - dg( 1)²
The right differential multiplication (f( ()) and left differential multiplication n dg(#) are computed using forward and backward mode automatic differentiation (resp.). Their derivatives with respect to the networks' parameters 0, & are computed by another backward mode automatic differentiation.
EXPERIMENTS
4.1 EVALUATION
We start by evaluating the effectiveness of our suggested I-AE regularizer, addressing the following questions: (i) does our suggested loss L (0, Φ) in equation g drive I-AE training to converge to an isometry? ? (īi) What is the effect of the Lpiso term? In particular, does it encourage better manifold approximations as conjectured? To that end, we examined the I-AE training on data points X' sampled uniformly from 3D surfaces with known global parameterizations. Figure 3 shows qualitative comparison of the learned embeddings for various AE regularization techniques: Vanilla autoencoder (AE); Contractive autoencoder (CAR) (Rifai al. 201 Contractive autoencoder with decoder
5

---

Under review as a conference paper at ICLR 2021
3D Data
I-AE
CAE
TCAE
RAE-GP
DEA
AE
U-MAP
I-SNE
LLE
S
Figure 3: Evaluation of 3D - 2D embeddings.
weights tied the encoder weights (TCAE) (Rifai al. 2011a); Gradient penalty the decoder (RAE-GP) (Ghosh et al.) 2020); and Denoising autoencoder with gaussian noise (DAE) (Vincent et al. 2010). For fairness in evaluation, all methods were trained using the same training hyper-parametees.. See Appendix for the complete experiment details including mathematical formulation of the different AE regularizers. In addition, we compared against popular classic manifold learning techniques: U-MAP (McInnes et al.] 2018), t-SNE (Maaten & Hinton, 2008) and LLE. (Roweis & Saul 2000). The results demonstrate that I-AE able to learn an isometric embedding, showing some of the advantages in our method: sampling density and distances between input points is preserved in the
learned low dimensional space.
In addition, for the AE methods, we quantita- tively evaluate how close is the learnt decoder triangulate to an isometry. For this purpose,
I-AE
CAE
TCAE
RAE-GP
DAE
AE
S Shape
0.03
0.36
0,26
1.22
2.53
1.85
Swiss
a grid of planar points {z:} C IR?. We denote by {eiy} the triangles edges incident to grid points 2; and zj- Then, we measured the edge lengths ratio, ly I (2,)
Open Sphere
0.07
0.21
0,21
0.50
1.09
1.29
Table 1: Std of (4y)
(zy) ||e]] expected to be 1 for all edges Cij isometry. In [ we log the standard deviation (Std) of {46} for I-AE compared to other regularized AEs. For fair comparison, we scaled z, the mean of Lij 1 in all experiments. As can be seen in the table, the distribution of {4i} } for I-AE is significantly more concentrated than the different AE baselines.
Table
Finally, although Liso is already responsible for learning an isometric decoder, the pseudo-inverse encoder (enforced by the loss I helps it converge to simpler solutions. We ran AE training with and without the Liiso term. Figure shows in gray the learnt decoder surface, N,, without Lpiso (left), containing extra (unnatural) surface parts compared to the learnt surface with Liio (right). In both cases we isometry
Figure 4: Decoder surfaces without Lpiso (left) and with (right).
expect (and achieve) decoder approximating that passes through the input data points. Nevertheless, the pseudo-inverse loss restricts some of the degrees of freedom of the encoder which in turn leads to simpler solution.
4.2 DATA VISUALIZATION
In this experiment evaluate method in the task of high dimension data visualization, i.e., reducing high dimensional data into two dimensional space. Usually the data not assumed to lie on manifold with such a low dimension, and is therefore impossible preserve all of its geometric properties. common artifact when squeezing higher dimensional data into the plane is crowding (Maaten & Hinton 2008), that is planar embedded points crowded around the origin.

---

Under review as a conference paper at ICLR 2021
I-AE
CAE
RAE-GP
DAE
AE
U-MAP
t-SNE
Figure 5: Results of data visualization experiment. Different colors indicate different ground turth labels/classes. Top shows MNIST: FC architecture of the encoder/decoder (top row), and CNN (bottom row): Middle shows FMNIST: FC (top row), and CNN (bottom row); Bottom shows COIL20 with CNN architecture, where zoom-ins of 3 classes are shown in the bottom row.
We evaluate method three standard datasets of images: MNIST (LeCun 1998) (60k hand- written digits). Fashion-MNIST (60k Zalando's 's article images) (Xiao et al. (017)) and COIL20 (Nene) al. 1996) (20 different images of object rotated with 72 even rotations). For baselines we take: Vanilla AE; CAE; GP-RAE; DAE; U-MAP and t-SNE.
We the same architecture for all auto-encoder methods each dataset. MNIST and FMNIST we evaluated in two scenarios: (i) Both encoder and decoder are fully-connected (MLP) networks; and (ii) Both encoder and decoder are Convolutional Neural Network (CNN). For COIL20 dataset both encoder and decoder are Convolutional Neural Network. Full implementation details and hyper-parameters values can found in the Appendix.
The results are presented in figure where each embedded point z is colored by its ground-truth class/label. We make several observation. First, in all the datasets our method is more resilient to crowding compared to the baseline AEs, and provide more even spread. U-MAP and t-SNE produce better separated clusters. However, this separation can come at a cost: See the COIL20 result (third expect evenly spaced
row) and blow-ups of three of the classes (bottom row). In this dataset points that correspond to the even rotations the objects in the images. Note (in the blow-ups) that U-MAP maps the three classes top of each other (non-injectivity of the "encoder"), 1-SNE is
7

---

Under review as a conference paper at ICLR 2021
somewhat better but does not preserve well the distance between pairs of data points (we expect them to be more or less equidistant in this datasct). In I-AE the rings are better separated and points are more equidistant; the baseline AEs tend to densify the points near the origin. Lastly, considering the inter and intra-class variations for the MNIST and FMNIST datasets, we are not sure that isometric embeddings are expected to produce strongly separated clusters as in U-MAP and t-SNE (e.g., think about similar digits of different classes and dissimilar digits of the same class with distances measured in euclidean norm).
4.3 DOWNSTREAM CLASSIFICATION
To quantitatively evaluate the unsupervised wiwddimnsiionl embedding computed with the I-AE performed the following experiment: We trained simple classifiers the embedded vectors computed by I-AE and baseline AEs and compared their performance (i.e., accuracy). Note that the process of learning the embedding is unsupervised and completely oblivious to the labels, which used solely for training and testing the classifiers.
We evaluate on the same datasets in Section 4.2 In MNIST and FMNIST we use the standard train-test split, and on COIL20 we split 75%-25% % randomly. As AE baselines we take vanilla AE, CAE, DAE and RAE-GI. E.GP, as described above. We repeat each experiment with 3 different latent dimensions, {16, 64, 256). and use two different simple classification algorithms: linear Support vector machines (SVM) (Cortes & Vapnik 1995) and K-nearest neighbors (K (KNN), with K 5.
Table 2 logs the results, where for both types of classifiers I-AE outperforms the baseline AEs in almost all combinations, where the SVM experiments demonstrate larger margins favor of I-AE. The results of the K-NN indicate that euclidean metric captures similarity in our embedding, and the results of the SVM, especially on the MNIST and COIL20 datasets, indicate that I-AE is able to embed the data in arguably simpler, linearly separable manner. The very high classification rates in COIL20 are probably due to the size and structure of this dataset. Nevertheless with SVM, already in 16 dimensions I-AE provides an accuracy of 95%, with 5% margin from 2nd place.
Dataset
IAE
AE
CAE
DAE
RAE-GP
Dataset
IAE
AB
CAB
DAE
RAB-GP
16
0.9138
0,9044
0.9644
0.9033
D.9016
16
0.9791
0.7784
0.9759
0,9783
0.9778
MNIST
64
0,8905
0,8364
0,8869
0.8373
0.8796
MNIST
64
0.9736
0.9756
0,9761
0.9768
0.9710
0.9140
0.9308
0.9226
256
0.9761
0.9719
0,9734
0.9133
16
0,7910
0,7827
0.7865
0.7845
0,7831
16
0.8790
0,8775
0.7733
0.878
0.8763
FMNIST
64
0.8343
0,8056
0,8350
0.7925
0,8339
FMNIST
64
0.8845
0.8867
0,8855
0,8854
0.8873
256
0.8721
0,8341
0,8623
0.8374
0.8688
256
0.8778
0.8796
0.8763
0,8783
0.8712
16
0.9500
0.8944
0.8917
0.9000
0.8833
16
0.9917
0.9861
0,9889
0,9861
0.9889
COIL20
64
1.0000
0.9944
0,9972
0.9917
0,9833
COIL20
64
0.9917
0,9722
0.9722
256
1.0000
0.9972
1,0000
0.9972
L.0000
256
0.9889
0.9750
0,9806
0.9722
0.9444
SVM
K KNNN
Table 2: Downstream classification experiment. Both tables indicate accuracy in [0, 1]. Left: results with linear SVM classifier; and right: results a K-NN classifier with K=5. The top performance scores are highlighted with colors: first, second and third.
4.4
SENSITIVITY
To evaluate the affect of AiS on the output we compared the visualizations and optimized loss values of MNIST and FMNIST, trained with same CNN architecture as in Section 4.2 with Aiso E {0, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 0.1). Figure 6 shows the different visual- ization results as well as Liees Liso: Lysso as a function of Xiso As can be seen in both datasets the visualizations and losses are stable for Aiso values between 0.01 and 0.5, where significant change to the embedding is noticeable at 0.75. The trends in the loss values are also rather stable; Liso and Lpio start very high in the regular AE, i..., Also O. and quickly stabilize. As for Lye on FMNIST we see a stable increase while in MNIST it also starts with a steady increase until it reaches 0.75 and then it starts to be rockier, which is also noticeable in the visualizations.
8

---

Under review as a conference paper at ICLR 2021
Asso
0.01
0.025
0.05
0.075
0.1
0.25
0.5
0.75
1
Lrec
Liso
piso
Top: visualizations of MNIST (Ist row) and FMNIST (2nd row) datasets trained with different Aiso values. Bottom: plots of the final train losses as function of Aiso: left to right: Lie (linear scale), Liso (log scale), and Liso (log scale).
Figure 6: Sensitivity to
CONCLUSIONS
We have introduced I-AE, regularizer for autoencoders that promotes isometry of the decoder and meeudo-inverss, of the encoder. Our goal was two-fold: (i) producing a favorable low dimensional manifold approximation to high dimensional data, isometrically parameterized for preserving, as much possible, its geometric properties; and (ii) avoiding complex isometric solutions based on the notion of psuedo-inverse. Our regularizers are simple to implement and can be casily incorporated into existing autoencoders architectures. We have tested I-AE on common manifold learning tasks, demonstrating the usefulness of isometric autoencoders.
An interesting future work venue is consider task (ii) from section L namely incorporating I-AE losses in probabilistic model and examine the potential benefits of the isometry prior for generative models. One motivation is the fact that isometries push probability distributions by simple change of coordinates, P(z) (=]())).
REFERENCES
Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data-generating distribution. The Journal of Machine Learning Research, 15(1):3553-3593, 2014.
Luigi Ambrosio and H Mete Soner. Level set approach to mean curvature flow in arbitrary codimen- sion. 1994.
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps and spectral techniques for embedding and clustering. In Advances in neural information processing systems, pp. 585-591, 2002.
Yuri Burda, Roger B. Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. CoRR, 109.1509.00 2016.
Ronald R Coifman and Stéphane Lafon. Diffusion maps. Applied and computational harmonic analysis, 21(1):5-30, 2006.
Ronald R Coifiman, Stephane Lafon, Ann B Lee, Mauro Maggioni, Boaz Nadler, Frederick Warner, and Steven W Zucker. Geometric diffusions a tool for harmonic analysis and structure definition of data: Diffusion maps. Proceedings of the national academy of sciences, 102(21):7426-7444 7431, 2005.

---

Under review as a conference paper at ICLR 2021
Corinna Cortes and Vladimir Vapnik. Support- networks. Machine learning, 20(3):273-299 1995.
David L Donoho and Carrie Grimes. Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data. Proceedings of the National Academy of Sciences, 100(10):5911-559 2003.
Karl Pearson F.R.S. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559-55) 572, 1901. doi: (11111/4446444099627777
Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael Black, and Bernhard Scholkopf. From variational to deterministic autoencoders. International Conference on Learning Representations, 2020. URL
net http://www...eeroommaaaa----... StpEDDD
Xavier Glorot, Antoine Bordes, and Yoshua Bengio.
Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 315-323, 2011.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. Iclr, 2(5):6, 2017.
Geoffrey E Hinton and Sam T Roweis. Stochastic neighbor embedding. In Advances neural information processing systems, pp. 857-864, 2003.
Keizo Kato, Jing Zhou, Tomotake Sasaki, and Akira Nakagawa. Rate-distortion optimization guided autoencoder for isometric embedding in euclidean latent space. arXiv preprint arXiv:1910.04329 2019.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 111iv:1412.6980, 2014.
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. CoRR, abs/1312.61 14, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Joseph B Kruskal. Multidimensional scaling by optimizing goodness of fit to nonmetric hypothesis. Psychometrika, 29(1):1-27, 1964.
Abhishek Kumar and Ben Poole. On implicit regularization in B-vaes. arXiv preprint 000::0002.0041,, 2020.
Abhishek Kumar, Ben Poole, and Kevin Murphy. Regularized autoencoders via relaxed injective probability flow. arXiv preprint arXiv:2002.08997, 2020.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun, com/exdb/mmist// 1998.
John M Lee. Smooth manifolds. In Introduction to Smooth Manifolds, pp. 1-31. Springer, 2013.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015,
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine
learning research, 975v)22579- 2008.
Alireza Makhzani, Jonathon Shlens. Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint
2015.
Leland McInnes. John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 03226, 2018.
10

---

Under review as a conference paper at ICLR 2021
James McQueen, Marina Meila, and Dominique Joncas. Nearly isometric embedding by relaxation. In Advances in Neural Information Processing Systems, pp. 2631-2639, 2016.
John Nash. The imbedding problem for riemannian manifolds. Annals of mathematics, pp. 20-63, 1956.
Sameer A Nene, Shree K Nayar, Hiroshi Murase, et al. Columbia object image library (coil-20). 1996.
Gautam Pai, Ronen Talmon, Alex Bronstein, and Ron Kimmel. Dimal: Deep isometric manifold learning using sparse geodesic sampling. In 2019 IEEE Winter Conference Applications of Computer Vision (WACV), pp. 819-828. IEEE, 2019.
Yookoon Park, Chris Dongjoo Kim, and Gunhee Kim. Variational laplace autoencoders, In ICML, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito,o, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Erez Peterfreund, Ofir Lindenbaum, Felix Dietrich, Tom Bertalan, Matan Gavish, Ioannis G Kevrekidis, and Ronald R Coifiman. Loca: Local conformal autocncoder for standardized data coordinates. arXiv preprint 2000::20004.02334,, 2020.
Ben Poole, Jascha Sohl-Dickstein, and Surya Ganguli. Analyzing noise in autoencoders and deep networks. arXiv preprint 14X6v:1406.14 1831, 2014.
Marc' Aurelio Ranzato, Christopher Poultney, Sumit Chopra, and Yann Cun. Efficient learning of sparse representations with energy-based model. In Advances in neural information processing systems, pp. 1137-1144 2007.
Aurelio Ranzato, Y-Lan Boureau, and Yann L Cun. Sparse feature learning for deep belief' networks. In Advances in neural information processing systems, pp. 1185-1192, 2008
Salah Rifai, Xavier Muller, Xavier Glorot, Grégoire Mesnil, Yoshua Bengio, and Pascal Vincent. Learning invariant features through local space contraction. arXiv preprint arXiv:1104.4152:
2011a.
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto- encoders: Explicit invariance during feature extraction. In ICML, lb.
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embedding. science, 290(5500) 2000.
John W Sammon. A nonlinear mapping for data structure analysis. IEEE Transactions computers, 100(5):401-409 1969.
Casper Kaae Sønderby, Tapani Raiko, Lars Maalde, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In Advances in neural information processing systems, pp. 7738-774., 2016.
Joshua Tenenbaum, Vin De Silva, and John C Langford. global geometric framework for nonlinear dimensionality reduction. science, 29095500) 2000.
Ilya O. Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schölkopf. Wasserstein auto- encoders. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL
https://openrevier net/ http:/ommmmm--11111-00000
Dmitry Ulyanov. Multicore-tsne.
Mul Multicore-TSMe 2016.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations a deep network with local denoising criterion. Journal of machine learning research, 11(Dec)333113400) 2010.
11

---

Under review as a conference paper at ICLR 2021
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion- Fashion-mnist: mnist: novel image dataset for benchmarking machine learning algorithms, 2017.
Yibing Zhan, Jun Yu, Zhou Yu, Rong Zhang, Dacheng Tao, and Qi Tian. Comprehensive distance- preserving autoencoders for cross-modal retrieval. In Proceedings of the 26th ACM International Conference on Multimedia, MM '18, pp. 1137-1145, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450356657. doi: 10. 10.1145/3240508.324000
URL
http://...11112215000000 3240607
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Balancing learning and inference in variational autoencoders. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 2019.
A APPENDIX
A.1 IMPLEMENTATION DETAILS
All experiments were conducted on a Tesla V100 Nvidia GPU using PYTORCH framework Paszke
al. (2017).
A.1.1 NOTATIONS
Table 3 describes the notation for the different network layers.
Description
LIN n FC m CoNY c, k.p.p CoNY B B.cks... CONVT c, S.p.p
Lincar lnyer demotes the ouppur dimension.
CunneCCeee layer with SoftPlus (9 100) non lincur activation. denotes the output dimensicn. consisting Lin n. followed by normalization layer SoftPuu (2
Convolutional layer with kernel of size l x k outpur channels, stride, and p padding.
Block consisting of CONY c, R.
followed batch normalizaioon loyer and SonPlos( 100) mon lincar activation.
Convolational tramppse layer kernel Block consisting CoNvT c, k, s.p. followed by batch nomalizatiin layer and SofiPlust 100) non lincar activation.
cutput
P padding.
CoNvT_ B c, s.p.
Table Layers notation.
A.1.2 EVALUATION
Architecture. We used an autoencoder consisted of 5 FC 256 layers followed by LIN 2 layer for the encoder; similarly, FC 256 layers followed by LIN layer were used for the decoder.
Training details. All methods were trained for a relatively long period of 100K epochs. Training was done with the ADAM optimizer Kingma & Ba (2014), setting fixed learning rate of 0.001 and full batch. I-AE parameter was set to Aioo = 0.01.
Baselines.
The following regularizers were used as baselines: Contractive autoencoder (CAB) Rifai
al. (2011b); Contractive autoencoder with decoder weights tied to the encoder weights (TCAE) Rifai et al. 201 Ia); Gradient penalty on the decoder (RAE-GP) G et al. (2020); Denoising autoencoder
with gaussian noise (DAB) Vincent et al. (2010). For both CAE, and TCAE the regularization term is
[dg(())) (z) For RAE-GP the regularization term is |[df (2))] For U-MAP McInnes al. 2018
McMAnes
set the number of neighbours to 30. For (-SNE Maaten & Hinton 22008) we set perplexity 50.
A.1.3 DATA VISUALIZATION
Architecture. Table 4
lists the complete architecture details of this experiment. Both MNIST and FMNIST were trained with FC-NN and S-CNN, and COIL.20 was trained with L-CNN.
Training details. Training was done using ADAM optimizer Kingma & Ba 2014). The rest of the training details
table 5
12

---

Under review as a conference paper at ICLR 2021
FC-NN
S-CNN
LNNN
Encoder
Decoder
Encoder CoNv_ 32,4,2.1
Docoder 256
Encoder
Decoder
FC_
FC_ 1024
128,4,2,1
CONVT 2048,4.1.0
FC_B 256
PC_B 512
CoNv_ B 64,4,2,1
CoNvT BB 128,4,1,0
CONY_ B 256,4,2,1
CONvT_B 1024,4,2.1
FC_B 512
FC B 256
CoNv B 128,4.2.1
CONvT 64.4.211
CONY_ B 512.4,211
CoNvT_B B 512,4,2.1
FC_B 1024
FC_B 128
Cony _B 256,4.2,0
CONVT 32,4.11
CoNv_B 1024.4.2,1
CoNVT B 256,4,2.1
LIN
LIN 784
LIN
CONvT 1,4,2,3
CoNv_B B 2048,4,2,1
CONvT_B 128,4,2,1
CoNv_ B 4096,4,2,1
CoNvT
CoNY 2,2,2,1
Table 4: High dimensional visualization experiment architectures.
MNIST
FMNIST
COIL20
Architecture
FC-NN
S-NN
FC-NN
S-CNN
L-CNN
Batch Size
128
512
128
512
144
Xiso
0.1
0.075
0.01
0.075
0.1
Epochs
1000
500
1000
500
1000
Table 5: High dimensional visualization training details.
Baselines. The following regularizers were used baselines: Contractive autoencoder (CAE)
Rifai et al. (b); Gradient penalty on the decoder (RAE-GP) Ghosh et al. (2020); Denoising
autoencoder with gaussian noise (DAE) Vincent et al. (2010). For CAE the regularization term
[[dg(>)]]* For RAE-GP the regularization term is [[ We used U-MAP McInnes al. (2018)
official implementation with random_ Sttoottttttttttteeee = 42, and Ulyanov (016) multicore implementation for
t-SNE Maaten & Hinton (2008) with default parameters.
A.2
ADDITIONAL EXPERIMENTS
A.2.1
GENERALIZATION IN HIGH DIMENSIONAL SPACE
Next, evaluate how well our suggested isometric prior induces manifolds that generalizes well to unseen data. We experimented with three different images datasets: MNIST (LeCun) 1998); CIFARIO (Krizhessky et al. 2009); and CclebA (Liu et al. 2015). We quantitatively estimate methods performance by measuring the L2 distance and the Fréchet Inception Distance (FID) Heusel al. (017) held out test set. For each dataset, used the official train-test splits.
For comparison versus baselines we have selected among relevant existing AE based methods the following: Vanilla AE (AE); autoencoder trained with weight decay (AEW); Contractive autoencoder (CAE); autoencoder with spectral weights normalization (RAE-SN); and autoencoder with L2 regularization on decoder weights (RAE-SN). RAE-L2 and RAE-SN were recently successfully applied to this data in (Ghosh et al. 2020). demonstrating tifee-offth-aate performance on this task. In addition, we compare versus the Wasserstein Eico-er (WAE) Tolstikhin et al. (001)), chosen as state-of-the-ee among generative autoencoders.
For evaluation fairness, all methods were trained using the same training aaper-parameters network architecture, optimizer settings, batch size, number of epochs for training and learning rate scheduling. See the appendix for specific out of the training set using 10k samples for the MNIST and CIFAR-10 experiment, whereas for the CelebA experiment we used the official validation set. For cach training epoch, we evaluated the reconstruction L2 loss on the validation set and chose the final network weights to be the one that achieves the minimum reconstruction. We experimented with two variants I-AE regularizers: Lpiso and Liiso + L Table logs the results. Note that I-AE produced competitive results with the current SOTA on this task.
values. In addition, we generated a validation set
Architecture. For all methods, we used an autoencoder with Convolutional and Convolutional transpose layers. Table lists the complete details.
Training details. Training was done with the ADAM optimizer Kingma & Ba 2014), setting a learning rate of 0.0005 and batch size 100. I-AE parameter was set Xiso 0.1.
13

---

Under review as a conference paper at ICLR 2021
MNNST
CIPAR-10
CelbA
Encoder
Deooder
Encoder Comv_ 4.
Decoder TC 16384
Encoder
Decoder
Cowv_l 4.2.1
18384
Cosv_D
FC TC665366
CoNvT
CONv_B 4.2.1
CoNvT_B
CoNv_
Cowv_B 512.
4,
CovyT 128,
CovyT 4,
1024, 5,
CoxvT 4,
LIN 16
CONVT 1, 1.0
LIN 128
CONVI 3, 0.0
LIN
CoNVT S, 1,
Table 6: High dimensional generalization experiment architectures.
Methods
Datasct
Distance
Lyiso
Lpoo
AE
AEW
CAE
RAE-SN
RAE-Ly
WAE
L
0.96
0.99
1.14
1.0
1.15
1.35
1.14
1.64
MNIST
FID
6.09
7.94
4.95
5.59
6.46
10.72
11.41
6.99
I.2 20.19
21.05
20.16
20.33
20.23
21.02
20.2
21.08
CIFAR-10
FID
70.14
56.04
74.79
68.71
71.71
70.79
71.05
74.2
20.38
19.93
20.51
20.46
20.78
20.58
CclebA
FID
34.68
40.73
40.53
40.00
39.52
40.45
38.86
38.98
Table 7: Manifold approximation quality test images. We log the L2 and FID distances (lower is better) from reconstructed images to the input images. The L2 numbers are reported +103, The top performance scores highlighted First, Second.
14

---

Under review as a conference paper at ICLR 2021
input
Liio
Lpiso + Liso
AE
AEW
CAE
RAE-SN
RAE-L2
WAE
Figure 7: CelebA reconstructions.
15

---

Under review as a conference paper at ICLR 2021
input
piso
Ipiso Liso
AE
AEW
CAE
RAE-SN
RAE-L2
WAE
Figure 8: CIFAR-10 reconstructions.
16

---

Under review as a conference paper at ICLR 2021
15709
5
5
4
2
input
/ 15709
7
5
4
2
piso
/ 75700
0
9
5
4
2
Lpio Liso
/ 5
15709
7
5
4
2
AE
/ 5 7
9
5
4
2
AEW
/ 1 0
9
5
4
2
CAE
/
5521
7
9
5
4
2
RAE-SN
/
551 7
0
9
5
4
2
RAE-L2
1570950
4
2
WAE
Figure 9: MNIST reconstructions.
17


Reviews:
[{"review_id": "I8i1y6kgG4F", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new version of the regularized autoencoder where they explicitly regularizes its decoder to be locally isometric and its encoder to be the decoder's pseudo inverse. Through a series of experiments and visualization, the IAE exhibits better manifold structure.\n\nRegarding the motivation and the math, I like the idea of isometric regularizer preserving the geometric properties in the learned manifold. The illustration in figure 1 does clearly point out the advantages of IAE over the contractive autoencoder. The math formulation primarily sticks with a linear version of the autoencoder. It would be great to get some insights for a non-linear counterpart.\n\nRegarding the experiments, indeed the authors successfully show the IAE converges its decoder to be an isometry and the proposed regularizer promotes more favoured manifold. However, the experiments mainly rely on visualization but fail to give some numeric results. For instance, can IAE be useful for semi-supervised learning (Like VAEs)? How can we practically make use of the isometry property in applications other than data visualization?\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9l_1C3NBCX2", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: I appreciate the authors addressing my concerns. I have increased my score accordingly.\n\nOriginal Review:\n\nThis paper describes a new type of regularization for the parameters of an autoencoder - one that forces the decoder to be an isometry. The authors present conditions that need to be satisfied by the encoder and decoder parameters, and show empirically that the regularization terms that they propose ensure that the resulting autoencoder has an isometric decoder. The paper is well written and easy to follow.\n\nWhile the authors assert that forcing the decoder to be an isometry is desirable since isometries preserve distances and angles, it is not clear why that is a desirable property while modeling data on a manifold. Distances between points on a data manifold are not usually measured through L2 distances in a latent dimension, and it is not clear why one should require that L2 distances in the high dimensional space are the same as distances in the latent space. The numerical results on reconstruction error that the authors present in the appendix do not indicate any reason to prefer isometric AEs over other baselines that are considered. In case there is a setting where isometric AEs can be shown to model the data manifold better than regular AEs, that is not highlighted in the current draft.\n\nThe authors claim that isometric autoencoders would \"evenly sample the manifold\" which is a little confusing, since the sampling of the data manifold is separate from the technique used to model the data (regular AEs vs isometric AEs).  \n\nThe experimental results also do not indicate how the embeddings learned using the proposed method perform on downstream classification tasks, for instance. This comparison would be useful to have to compare the usefulness of the embeddings.\n\nA few minor points of confusion:  \n1) the notation f^{-1} is a little misleading since the encoder is not necessarily an invertible function from R^d to R^D. If the encoder mapping is restricted to the range of f then this notation is more appropriate. \n2) The projection operator that is used to define the pseudoinverse of the encoder is not necessarily a function, since there could possibly be many points on the manifold that correspond to the same L2 distance from the point being projected. Are there further assumptions on the structure of the data manifold that prevent this from being the case?\n3) Estimating the L_iso term seems to require a distribution over the latent space R^d, that the authors say is computed using a fit of the latent codes g(x), x \\in \\cal X. Are the latent codes computed using the current estimate of the encoder? If so is there some sort of alternating minimization happening, which holds the current estimate of the encoder fixed while computing the isometric regularization? If not, how are the latent codes computed?", "extra_scores": {}, "extra_text": {}}, {"review_id": "6JVblQOb58z", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength:\n1. This paper provided a novel method to train a local isometric autoencoder, which can preserve the local Euclidean distances well between the original space and the latent space.\n2. The theories are well presented and explained pretty well. Also, Isometry is a very important property in several cases including manifold-learning, etc. \n3. Apart from the typos and several tiny errors, the overall writing is sound and smooth.\n\nWeakness:\n1. I have to say that the argument of global isometry is too ambitious. In the theory and method part, this method only guarantees the local isometry. The authors do mention that \"a local isometry which is also a diffeomorphism is a global isometry\" on page 3 bottom paragraph. However, there's no discussion about the \"diffeomorphism\" in the following sections. \n2. Also, the first experiment (3D $\\rightarrow$ 2D) only supports the local isometry. Due to the fact that the distance is computed only based on the triangular meshes between edges. Is there any possibility that the author can provide one more toy example for the global isometry when the data lie on some manifold shape? This would strongly support the global argument.\n3. From my understanding, the distance in the original space is the Euclidean distance without considering the local geometry of the data. Can the author provide some comments on this? The distance in the original space should be the geodesic distance when arguing the Isometry.\n4. The experiment of the data visualization is somewhat weak. The benefit of the Isometry Autoencoder is not well addressed. The t-SNE is well used for visualization with almost nothing wrong. The only benefit comes when arguing the \"even\" sampling. Can the author provide some comments on this about why we need the \"even\" in the visualization?\n\nSome other minor comments: \n1. There's a typo in the last line of the first page. The encoder should be $R^D\\rightarrow R^d$ with $d<D$. Similar to the inverse $R^d\\rightarrow R^D$. \n2. The differential of the decoder should be the Jacobian matrix right? This would be more clear than just mentioning differential.\n3. Table 1 should be Figure 1. Also, can the author provide more details about this figure? Is this figure for illustration only or this result is actually trained and plotted? The form \"evenly\" is a strong word that needs more explanation of the definition.\n4. The order of Figure 3 and Figure 2 is messed up. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "3jVtnIGgDzm", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper suggests a novel auto-encoder based method for manifold learning, by encouraging the decoder to be an isometry and the encoder to locally be a pseudo-inverse of the decoder.  It is noted that for a linear architecture, this gives PCA, therefore, this can be seen as a nonlinear PCA approach.\n\nIn theorem 1, the authors claim that for the encoder-decoder solution to have the desired properties, certain equalities have to be satisfied by the local differential matrices of the encoder and decoder.  This gives rise to a loss function that is combined of 3 parts:  A reconstruction loss (as usual with autoencoders) plus a combination of a loss penalizing non isometric decoders, plus a loss penalizing an encoder that is not a pseudo-inverse of the decoder.  This loss function is claimed to be the main technical novelty of the paper.\n\nIn the experimental part, the authors compare the merits of this approach on synthetically generated low dimensional manifolds in high dimensional ambient spaces, against other standard manifold learning algorithms, and show that the paper's method outperforms other method using a measure of distortion of triangle edges on a grid.  They also experiment with \"real data\" (e.g. MNIST), show the merits of the proposed algorithm when visualizing the 2 dimensional bottleneck of the autoencoder.  The comparison here is against other algorithms for high dimensional data visualization.\n\nThe overall idea and theory seem interesting.  The experiments are a bit disappointing.  For the synthetic data, I am not sure I understand why they did not chose something of high dimension?  Maybe I am missing something, but would it be impossible to generate, say, a 50 dimensional manifold in 100 dimensions?  Maybe the triangulation part will be challenging, but that is not the only way to compare between the various algorithms.  As for the real data section (e.g. MNIST), I am not sure I see why you compare your algorithm against algorithms that are intended for 2-d visualization (e.g. t-SNE).  Your algorithm does manifold learning.  Why not,for instance, take all the images corresponding to some fixed digit (e.g. \"3\"), which is presumably close to a low (but definitely more than 2....)  dimensional manifold, and see how well your manifold learning algorithm reconstructs them?\n\nThe editorial level of the paper is not very high, due to grammatical English mistakes.  Here are examples (the list is not complete):\np. 1 \"Autoencoder (AE) can also be seen\" => \"Autoencoders can also be seen\" or \"An  autoencoder can also be seen...\"\n\n\"AE is trying to reconstruct X...\"  - The present progressive tense is not suitable here.  Maybe \"AE's try to reconstruct\"?  Or \"AE's are designed to reconstruct...\" or \"An AE reconstructs...\"\n\np. 2 \nManifold learning generalizeS\n\n\np. 4\n\"As-usual \" => As usual\n\np. 5\n\"Does our suggested Loss... drives\" -> \"drive\"\n\np. 6\nWhy is \"Denoising\" capitalized?\n\n\"In addition, we compared versus...\" => \"...compared against...\"\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new formulation for learning low-dimensional manifold representations via autoencoder mappings that are (locally) isometric by design. The key technical ingredient is the use of a particular (theoretically motivated) weight-tied architecture coupled with isometry-promoting loss terms that can be approximated via Monte Carlo sampling. Representative results on simple manifold learning experiments are shown in support of the proposed formulation.

The paper was generally well-received; all reviewers appreciated the theoretical elements as well as the presentation of the ideas.

However, there were a few criticisms. First, the fact that the approach requires Monte Carlo sampling in very high dimensions automatically limits its scope. Second, the experiments seemed somewhat limited to simple (by ICLR standards) datasets. Third and most crucially, the approach lacks a compelling-enough use case. It is not entirely clear what local isometry enables, beyond nice qualitative visualizations (and moreover, what the isometric autoencoder provides over other isometry-preserving manifold learning procedures such as ISOMAP). Some rudimentary results are shown on k-NN classification and linear SVMs, but the gains seem to be in the margins. 

The authors are encouraged to consider the above concerns (and in particular, identifying a unique use case for isometric autoencoders) while preparing a future revision.
Decision: Reject