Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Update: I appreciate the authors addressing my concerns. I have increased my score accordingly.

Original Review:

This paper describes a new type of regularization for the parameters of an autoencoder - one that forces the decoder to be an isometry. The authors present conditions that need to be satisfied by the encoder and decoder parameters, and show empirically that the regularization terms that they propose ensure that the resulting autoencoder has an isometric decoder. The paper is well written and easy to follow.

While the authors assert that forcing the decoder to be an isometry is desirable since isometries preserve distances and angles, it is not clear why that is a desirable property while modeling data on a manifold. Distances between points on a data manifold are not usually measured through L2 distances in a latent dimension, and it is not clear why one should require that L2 distances in the high dimensional space are the same as distances in the latent space. The numerical results on reconstruction error that the authors present in the appendix do not indicate any reason to prefer isometric AEs over other baselines that are considered. In case there is a setting where isometric AEs can be shown to model the data manifold better than regular AEs, that is not highlighted in the current draft.

The authors claim that isometric autoencoders would "evenly sample the manifold" which is a little confusing, since the sampling of the data manifold is separate from the technique used to model the data (regular AEs vs isometric AEs).  

The experimental results also do not indicate how the embeddings learned using the proposed method perform on downstream classification tasks, for instance. This comparison would be useful to have to compare the usefulness of the embeddings.

A few minor points of confusion:  
1) the notation f^{-1} is a little misleading since the encoder is not necessarily an invertible function from R^d to R^D. If the encoder mapping is restricted to the range of f then this notation is more appropriate. 
2) The projection operator that is used to define the pseudoinverse of the encoder is not necessarily a function, since there could possibly be many points on the manifold that correspond to the same L2 distance from the point being projected. Are there further assumptions on the structure of the data manifold that prevent this from being the case?
3) Estimating the L_iso term seems to require a distribution over the latent space R^d, that the authors say is computed using a fit of the latent codes g(x), x \in \cal X. Are the latent codes computed using the current estimate of the encoder? If so is there some sort of alternating minimization happening, which holds the current estimate of the encoder fixed while computing the isometric regularization? If not, how are the latent codes computed?
review_title: Official Blind Review #2

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper suggests a novel auto-encoder based method for manifold learning, by encouraging the decoder to be an isometry and the encoder to locally be a pseudo-inverse of the decoder.  It is noted that for a linear architecture, this gives PCA, therefore, this can be seen as a nonlinear PCA approach.

In theorem 1, the authors claim that for the encoder-decoder solution to have the desired properties, certain equalities have to be satisfied by the local differential matrices of the encoder and decoder.  This gives rise to a loss function that is combined of 3 parts:  A reconstruction loss (as usual with autoencoders) plus a combination of a loss penalizing non isometric decoders, plus a loss penalizing an encoder that is not a pseudo-inverse of the decoder.  This loss function is claimed to be the main technical novelty of the paper.

In the experimental part, the authors compare the merits of this approach on synthetically generated low dimensional manifolds in high dimensional ambient spaces, against other standard manifold learning algorithms, and show that the paper's method outperforms other method using a measure of distortion of triangle edges on a grid.  They also experiment with "real data" (e.g. MNIST), show the merits of the proposed algorithm when visualizing the 2 dimensional bottleneck of the autoencoder.  The comparison here is against other algorithms for high dimensional data visualization.

The overall idea and theory seem interesting.  The experiments are a bit disappointing.  For the synthetic data, I am not sure I understand why they did not chose something of high dimension?  Maybe I am missing something, but would it be impossible to generate, say, a 50 dimensional manifold in 100 dimensions?  Maybe the triangulation part will be challenging, but that is not the only way to compare between the various algorithms.  As for the real data section (e.g. MNIST), I am not sure I see why you compare your algorithm against algorithms that are intended for 2-d visualization (e.g. t-SNE).  Your algorithm does manifold learning.  Why not,for instance, take all the images corresponding to some fixed digit (e.g. "3"), which is presumably close to a low (but definitely more than 2....)  dimensional manifold, and see how well your manifold learning algorithm reconstructs them?

The editorial level of the paper is not very high, due to grammatical English mistakes.  Here are examples (the list is not complete):
p. 1 "Autoencoder (AE) can also be seen" => "Autoencoders can also be seen" or "An  autoencoder can also be seen..."

"AE is trying to reconstruct X..."  - The present progressive tense is not suitable here.  Maybe "AE's try to reconstruct"?  Or "AE's are designed to reconstruct..." or "An AE reconstructs..."

p. 2 
Manifold learning generalizeS


p. 4
"As-usual " => As usual

p. 5
"Does our suggested Loss... drives" -> "drive"

p. 6
Why is "Denoising" capitalized?

"In addition, we compared versus..." => "...compared against..."

review_title: Novel auto-encoder based method for manifold learning

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose a new version of the regularized autoencoder where they explicitly regularizes its decoder to be locally isometric and its encoder to be the decoder's pseudo inverse. Through a series of experiments and visualization, the IAE exhibits better manifold structure.

Regarding the motivation and the math, I like the idea of isometric regularizer preserving the geometric properties in the learned manifold. The illustration in figure 1 does clearly point out the advantages of IAE over the contractive autoencoder. The math formulation primarily sticks with a linear version of the autoencoder. It would be great to get some insights for a non-linear counterpart.

Regarding the experiments, indeed the authors successfully show the IAE converges its decoder to be an isometry and the proposed regularizer promotes more favoured manifold. However, the experiments mainly rely on visualization but fail to give some numeric results. For instance, can IAE be useful for semi-supervised learning (Like VAEs)? How can we practically make use of the isometry property in applications other than data visualization?




review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Strength:
1. This paper provided a novel method to train a local isometric autoencoder, which can preserve the local Euclidean distances well between the original space and the latent space.
2. The theories are well presented and explained pretty well. Also, Isometry is a very important property in several cases including manifold-learning, etc. 
3. Apart from the typos and several tiny errors, the overall writing is sound and smooth.

Weakness:
1. I have to say that the argument of global isometry is too ambitious. In the theory and method part, this method only guarantees the local isometry. The authors do mention that "a local isometry which is also a diffeomorphism is a global isometry" on page 3 bottom paragraph. However, there's no discussion about the "diffeomorphism" in the following sections. 
2. Also, the first experiment (3D $\rightarrow$ 2D) only supports the local isometry. Due to the fact that the distance is computed only based on the triangular meshes between edges. Is there any possibility that the author can provide one more toy example for the global isometry when the data lie on some manifold shape? This would strongly support the global argument.
3. From my understanding, the distance in the original space is the Euclidean distance without considering the local geometry of the data. Can the author provide some comments on this? The distance in the original space should be the geodesic distance when arguing the Isometry.
4. The experiment of the data visualization is somewhat weak. The benefit of the Isometry Autoencoder is not well addressed. The t-SNE is well used for visualization with almost nothing wrong. The only benefit comes when arguing the "even" sampling. Can the author provide some comments on this about why we need the "even" in the visualization?

Some other minor comments: 
1. There's a typo in the last line of the first page. The encoder should be $R^D\rightarrow R^d$ with $d<D$. Similar to the inverse $R^d\rightarrow R^D$. 
2. The differential of the decoder should be the Jacobian matrix right? This would be more clear than just mentioning differential.
3. Table 1 should be Figure 1. Also, can the author provide more details about this figure? Is this figure for illustration only or this result is actually trained and plotted? The form "evenly" is a strong word that needs more explanation of the definition.
4. The order of Figure 3 and Figure 2 is messed up. 
review_title: The theories are nice but the experiment doesn't support the theory enough.

Meta-review: The paper introduces a new formulation for learning low-dimensional manifold representations via autoencoder mappings that are (locally) isometric by design. The key technical ingredient is the use of a particular (theoretically motivated) weight-tied architecture coupled with isometry-promoting loss terms that can be approximated via Monte Carlo sampling. Representative results on simple manifold learning experiments are shown in support of the proposed formulation.

The paper was generally well-received; all reviewers appreciated the theoretical elements as well as the presentation of the ideas.

However, there were a few criticisms. First, the fact that the approach requires Monte Carlo sampling in very high dimensions automatically limits its scope. Second, the experiments seemed somewhat limited to simple (by ICLR standards) datasets. Third and most crucially, the approach lacks a compelling-enough use case. It is not entirely clear what local isometry enables, beyond nice qualitative visualizations (and moreover, what the isometric autoencoder provides over other isometry-preserving manifold learning procedures such as ISOMAP). Some rudimentary results are shown on k-NN classification and linear SVMs, but the gains seem to be in the margins. 

The authors are encouraged to consider the above concerns (and in particular, identifying a unique use case for isometric autoencoders) while preparing a future revision.
Decision: Reject