Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose geometrically regularized autoencoders for non-Euclidean data. The main contribution of the paper is adapting the regularization terms in the denoising autoencoder (DAE) and the reconstruction contractive autoencoder (RCAE) to the non-Euclidean setting. If the manifold from which the data are sampled are known a priori, the proposed autoencoders achieve improved reconstruction performance, and are able to estimate the score of the non-Euclidean data.
strength_and_weaknesses: **Strength**
1. The motivation of the paper is clear. It is also very well-organized. I especially like the different notations for points on the manifold and their coordinates.
2. Theorem 1 suggests that the geometric score can be estimated from the trained auto-encoder (a generalization of the result by Alain & Bengio, 2014 in the Euclidean setting), and the numerical results are convincing.

**Weakness**
1. My biggest concern is the increased computational cost after taking into account the geometric information (exponential maps, geodesics, etc.) This seems to be buried under the rug of the paper. I would like to see come comparison of the computational time in training the vanilla autoencoder and the proposed models.
2. The author seems to be using one single chart for a manifold. However, general manifolds are typically not parametrizable using one chart. What would happen if $r(x)$ and $x$ appear in two different charts?
3. It might be a good idea to elaborate, in the main text, what "$\approx$" means in equation (7).
4. Also, is RCAE actively used in the community? There does not seem to be any reference on that in the paper.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and very easy to read. The idea of adapting the regularization to non-Euclidean data is novel and interesting.
summary_of_the_review: I think this is an interesting paper. I have some concern on the practical implementation and its implication on the computational cost. I am willing to adjust my rating based on the authors' response.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an autoencoder for data residing on a Riemannian manifold alongside a geometric regularize. The reconstruction trivially replaces Euclidean distances with geodesics ones, and the bulk of the paper is concerned with the regularization. I did not fully understand the regulerization scheme (details below), but it seems related to denoising and contractive autoencoders. Empirically, the method is shown to work well on low-dimensional data; it is unclear if the method will scale to high-dimensional manifolds, which arguably is where the dimensionality reduction associated with autoencoders is most valuable.
strength_and_weaknesses: ## Strengths
* The paper approaches an important problem as regularization on manifolds is generally understudied.
* The idea of extending denoising autoencoders to manifold data is neat as it is really simple to work with. I very much appreciate the practicalities of this idea.
* The empirical study is quite broad as several data sets from different manifolds are considered (see also Weaknesses below).

## Weaknesses
* In general, I found the paper somewhat difficult to read. As such, the weaknesses listed here may very well be weaknesses of the reader rather than of the paper.
* The opening motivation for the paper is that standard vector-based regularizers do not work on manifolds. While I generally believe this to be true, I found the demonstration thereof to be somewhat thin. I appreciate the idea of empirically demonstrating the claim, but I think this demonstration should be more elaborate.
* I didn't understand the sentence (page 4), "use the weighted volume element $\rho(x) dx$ instead of $\sqrt{\det G} dx$." Does this mean that you ignore the volume element of the metric? That sounds dodgy.
* I did not understand Eq. 5, in particular the trace-term. Is my reading correct that $\left(\frac{\partial r}{\partial x}\right)^T G \left(\frac{\partial r}{\partial x}\right)$ is a scalar, which can be moved outside the trace-function? I did not understand why the inverse metric appears in this expression. Perhaps it's obvious, but it wasn't to me.
* I also had trouble understanding Eq 8. Why does this use the inverse metric? I would have intuitively guessed that it should be the metric (not its inverse). Can some explanation be provided?
* The experiments generally focus on very low-dimensional manifolds (e.g. 3x3 symmetric positive definite matrices). I would expect that autoencoder-style models have most value on high-dimensional data. I suppose that the method has difficulty scaling to high-dimensional data; what is the main bottleneck?

## Minor issues
* The paper generally talks of Riemannian latent spaces, but, as far as I could tell, mostly Euclidean latent spaces are considered. Perhaps one should here call on one of the many papers which argue that autoencoder-style latent spaces should be viewed as Riemannian (through the pull-back metric). See e.g. https://arxiv.org/abs/1710.11379
clarity,_quality,_novelty_and_reproducibility: ### Clarity:
I found the motivation of the paper to be quite clear, but I struggled with the more mathematical parts of the paper. I would have valued a bit more hand-holding along the way.

### Quality:
The work appears sensible, and I particularly enjoyed the denoising autoencoder extension to manifolds. That's very clean and easy to understand. I have a hard time judging the remaining parts of the paper as I didn't quite understand the math.

### Novelty:
As far as I can tell, the paper is quite novel.

### Reproducibility
I would not be able to reproduce the experiments from the paper alone. I'm happy to see that code will be released, though.


summary_of_the_review: A promising paper, that I did not fully understand. If I understand the work better during the rebuttal phase I expect to have a more positive vote.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a regularization technique for auto-encoders, when the data is non-Euclidean. Conventional regularizers encourage small MSE reconstruction error when input is noised. In contrast, the proposed method replaces MSE reconstruction error with geodesic distances along the data manifold. Empirical study shows that the method is better at estimating geometric score, and could lead to more useful learned features.
strength_and_weaknesses: The paper has some features that can be considered both as strength and weakness.

1. Insights from differential geometry and manifold learning are introduced. This could educate the machine learning community but also has the risk of not being appreciated.

2. The paper shows extensive empirical results. However, most of them aims at showing a more accurate estimate of geometric score. That is, the gradient of log probability distribution function. While I agree it is necessary to show effectiveness using intrinsic measure like this, I think the community may care more about the practical implications.

Another weakness is that the method relies on the metric $G(x)$ at each local coordinate. $G(x)$ may not be known a priori. In fact, $G(x)$ is often unknown for many learning problems. The paper should discuss how to handle it in a separate section. Will it require more training data, compared against the conventional regularizer, to figure out the $G(x)$?
clarity,_quality,_novelty_and_reproducibility: Overall the paper is well written. But as suggested before, language can be made more accessible. For example, I think the notation of Dirichlet energy can be introduced right after Eq. (2). Explain its effect as the regularization term in Eq. (2). Then in section 3.1, manifest that this paper generalizes this conventional Dirichlet energy for manifold case.

The paper seems novel to me.
summary_of_the_review: The paper imports notions from differential geometry and manifold learning. This could have pros and cons.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper initially received two negative (5) and one positive (6) scores. 
The main concerns were mostly exposition issues, and missing discussion of computational costs.  
These were addressed by the authors and during the rebuttal and discussion period the reviewers with initially negative scores increased their scores to positive (6). 
We encourage the authors to discuss the scalability of their method in the final version.
justification_for_why_not_higher_score: The contribution is nice but a somewhat limited (generalizing the reconstruction contractive auto encoder to manifolds by considering the generalized Dirichlet energy on manifolds). Does not justify any special highlight at the conference IMO. 
justification_for_why_not_lower_score: The reviewers unanimously agreed this paper is above the bar.
Decision: Accept: poster