Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces several auto-encoder models that preserve local distances when mapping from the data space to the latent
space. The proposed models use a local distance-preserving loss that is based on the continuous k-nearest neighbours graph which is known to capture topological features at all scales simultaneously. In order to improve training performance, the models formulate learning
as a constraint optimisation problem with local distance preservation as the main objective and reconstruction accuracy as a constraint. The proposed method provides state-ofthe-art or comparable performance across several standard datasets and evaluation metrics.
strength_and_weaknesses: Strength: The paper is well written and easy to follow.
Weaknesses:
1. The novelty of the proposed method is limited, the proposed method just integrates the previous  local distance-preserving loss into auto-encoders;
2. In the experiments, the detailed parameter settings of auto-encoders are not clearly introduced.
clarity,_quality,_novelty_and_reproducibility: The novelty is limited and some details are missed.
summary_of_the_review: Based on the limited novelty and unclear presentation, I am willing to reject this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an extension of auto-encoder via incorporating the local distance preserving. Specifically, the local distance preserving is implemented by: 

(1) constructing two graphs (via CKNN) on the raw features and the learned representations; 
(2) comparing the two graphs via the pairs contained in both two graphs. 
(3) further proposing to optimize a contrained optimization problem for the generative models (e.g., VAE). 

The authors also discuss several cases with different priors.  

Sufficient experiments are also reported in both main paper and appendix. 

strength_and_weaknesses: ## Strength

- The paper is easy to follow. 
- The paper seems technically sound and the authors discuss different situations with diverse prior distributions. 
- The experiments are sufficient for me. 

## Weakness
- My main concern is that the novelty seems not sufficient for me. The constrained optimization for generative models seems well-studied and the idea of local preserving seems widely used. Although this paper may emphasize that the preserving mechanism is built on *distance*, it not a significant contribution for me. Moreover, another key is the utilization of continuous k-nearest neighbors, which may be not a remarkable contribution of this paper. 
- In problem (1), the loss only considers the overlapped links. Should the different links between two graphs be also important for training auto-encoder? Could the author provide some explanations? 
clarity,_quality,_novelty_and_reproducibility: \The constrained optimization for generative models seems well-studied and the idea of local preserving seems widely used. 

Although this paper may emphasize that the preserving mechanism is built on *distance*, it not a significant contribution for me. 

The constrained optimization for generative models is also well-studied. 

Another key is the utilization of continuous k-nearest neighbors, which may be not a remarkable contribution of this paper. 
summary_of_the_review: The main concern is the novelty and some questions about some parts.

I'd like to update my score after the discussion period.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a manifold learning strategy based on autoencoders (with a Euclidean distance reconstruction loss) and continuous k-nearest neighbor graphs (CkNN) (with a topological / local distance preservation loss between input and latent space). It implements a constrained optimization where topological loss is minimized such that the reconstruction loss is upper bounded.
strength_and_weaknesses: 
Pros:
- Paper is well written
- Method is clear and simple
- Experiments on a number of datasets

Cons:
- Novelty is limited
- Choices not motivated or ablated
- Additional choices not explored
- Experiments are weak

The paper is very well written and easy to follow, as a result the method and its evaluation are very clear. The method is simple and found to be effective on a number of datasets.

The novelty is limited, as the method is merely a combination of existing autoencoders, existing priors and an existing nearest neighbor graph definition.

The proposed strategy is presented as a fixed recipe, without motivating why the particular choices are made and without ablating either the algorithmic choices or the hyperparameters. The later must have been tuned but no results are given or discussed. This is important because it is unclear if hyperparameters have been also tuned for competitors. The choice of $k=9$ is given as an arbitrary choice, while for the length scale $\delta$ not even a value is given.

Why is the topological distance minimized while the reconstruction loss posed as a constraint? Why not vice versa? Why not minimize a function of both?

More importantly, why use CkNN and why this way? Using CkNN is claimed as a main contribution ("we are the first to adapt it to..."). Fixing all other choices made, how would the behavior be affected by any other graph definition? For example:
- CkNN uses the geometric mean of the kNN distances of x and y in defining an edge between x and y. Why not experiment with the generalized (power) mean, which subsumes the minimum/maximum, the RMS as well as the harmonic, geometric and arithmetic mean among others? Why not any other symmetric function?
- If the graph definition is so important, why not a weighted graph, which would result in weighted loss terms in eq. (1)? Again, a number of functions could be candidate for the weights.

Experiments are weak. In particular:
- The datasets used are small (in fact, toy) and the metrics are near perfect if I am not mistaken (assuming perfect value 0 for two of the metrics and 1 for the other two). As a result, the differences between methods are tiny.
- Tables use bold black and bold blue, which are not explained in the captions. Bold black must be the best but bold blue is unclear. This is misleading because bold blue is chosen for the proposed method.
- In Table 5 (COIL), SNE is best under 2 of the 4 metrics and in Tables 1-4 (MOCAP) the difference is 0.1%.
- In Fig. 9 (MOCAP), the latent space of UMAP looks excellent, while in most other cases UMAP is missing from the comparison with no explanation.
- On Swiss Roll and CIFAR10, metrics are perfect and there is no benefit over VR.

I would expect experiments on more realistic datasets used for unsupervised metric learning, for example CUB, Cars, SOP and InShop. These experiments should also use larger network architectures and include comparison with modern unsupervised metric learning methods.
clarity,_quality,_novelty_and_reproducibility: As detailed above, the writing is of high quality and the method is very clear. However, the originality is limited as the proposed method essentially combines existing ideas without justifying and supporting the choices made.
summary_of_the_review: This is a simple and interesting manifold learning strategy that makes sense but novelty is limited, choices appear to be arbitrary, additional choices are not explored and experimental evaluation is weak.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Thanks for your submission to ICLR.  The reviewers were in agreement that this paper is not ready for publication.  They noted several weaknesses in the paper.  Most notably, they all felt that the novelty was not high enough, though the reviewers also commented on various concerns about the experimental results as well.  On the positive side, the reviewers all appreciated the clarity of the writing, and some reviewers felt that the experiments were sufficient.
justification_for_why_not_higher_score: All reviewers felt that the paper was not ready for publication, even after the author rebuttal.
Decision: Reject