Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes RTD-AE, a topology-based autoencoder that aims to preserve topological information from the data space in latent embeddings. For this, RTD-AE leverages Representation Topology Divergence (RTD), a technique that allows to measure the similarity of VR complexes between two point clouds, here the data space and latent space.

The paper considers an interesting problem, that may offer some benefits over existing methods. However, there are several small-to-medium red flags that currently let me question the validity of the results and the interpretations followed from them.
strength_and_weaknesses: Strengths:
- Preserving the global structure in non-linear dim. red is important and underexplored
- RTD loss more closely reflects metric properties than e.g. previous TopoAE loss (but is it really a metric, or just identity of indiscernibles?)
- Many baselines considered.


Weaknesses:
- The paper lacks in clarity. Most strikingly this is evident in Fig 2, which should give a clear and simple high-level overview. However, I found it just confusing.
- The required two-step training schedule makes the method less likely to be useful in practice. In general, AE-based dim red / visualization techniques are more challenging to apply for end users out of the box (compared to UMAP, t-SNE etc), so adding an additional layer of complexity like two different training schemes makes it less likely that this method will be widely used by the community.
- Certain experimental results look a bit as if they were optimized to favor the proposed method. For instance, in Fig 1, the proposed method (panel c)) is rotated exactly as the original input data (by chance?), whereas e.g. f) that also looks reasonable is rotated differently making it look worse than it may be. Another reason why certain plots may be unfair to the baselines: You write "All of the representations were generated with default parameters of baseline methods." If this is the case, but the proposed method was tuned in a hyperparameter search, then there is no surprise that on the proposed method shines in several plots. And its also no surprise that certain methods seem to perform poorer than in their original paper (e.g. 2D visualization of Spheres with TopoAE). Another thing raising a minor red flag: The experimental structure / setup follows neatly the TopoAE paper, however certain parameters are chosen differently without explaining why. E.g. 3D visualisation of the Spheres (instead of 2), or 16D latent space instead of 2. Without any explanation, one is tempted to assume that these slightly arbitrary configurations were searched over to find best performing settings.
clarity,_quality,_novelty_and_reproducibility: As indicated above, the paper would benefit from a clearer exposition to the RTD loss, especially Fig 2 is unclear to me.

The method itself looks reasonable, due to the other weaknesses I haven't checked though all the supplementary math in great detail.

In terms of novelty, one may ask of course what the benefit of an additional PH-based dim red method is -- given that there exist solutions already. Improved performance may be an argument, but I am not fully convinced that is the case here, given that we compare against default parameters (if I understand that correctly).



summary_of_the_review: Overall, the paper proposes an interesting idea, but the execution of the experiments, the clarity of the paper as well as the demonstration of the benefits seem insufficient to me at this point.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an approach for topology-preserving representation learning (dimensionality reduction). The topological similarity between data points in original and latent spaces was obtained by minimizing the Representation Topology Divergence (RTD) between original data and latent representations. They demonstrated how to make RTD differentiable and implemented it as an additional loss to the autoencoder, constructing RTD-AE. According to their computational experiments, the proposed RTD-AE better preserves the global structure of the data manifold than popular methods t-SNE and UMAP. Moreover, higher topological similarity than the alternative TopoAE method was achieved.
strength_and_weaknesses: **Strength**

The main idea of the paper as well as the empirical part are very interesting. 

**Weaknesses and Questions**:

1) TDA literature usually includes a theoretical study of the signature's stability when proposing a new topological signature. In terms of noise in the output or weights, how robust is the R-Cross-barcode?


2) For a larger dataset, the number of simplices exponentially increases and so the construction of the VR complex would computationally be inefficient. How does your method deal with this problem? 

3) A barcode is used to compare point clouds, but its superiority over other proposed metrics is unclear. Some statistical measures can also be used to compute a "divergence" or an actual metric distance between point clouds that might perform better. Mahalanobis distance between points, for instance, can be treated as instantiations of random processes. 


4) Can we consider RTD as a metric?


clarity,_quality,_novelty_and_reproducibility: The paper is very well-written and easy to follow. I believe the idea is very interesting, novel and important. 
summary_of_the_review: Overall I liked the idea of the paper and found their contribution very interesting and significant.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new topology-based loss term for dimensionality reduction. The loss term is based on the recently-introduced "Representation Topology Divergence" (RTD) and has beneficial theoretical as well as empirical properties. Next to comparing the loss term with existing losses arising in the context of topological machine learning, the paper presents a very detailed suite of experiments that showcase the empirical utility of the method.

strength_and_weaknesses: The paper makes a strong case for a new topology-based loss term. I appreciate the ingenious use of RTD, which seems to be perfectly apt for the task at hand (permitting the comparison of point clouds of the same size, among other things). Having a strong theoretical foundation with representation quality guarantees thus makes this a very strong contribution to the literature.

I have two minor issues with the manuscript in its current form, though:

1. Some details about RTD and its usage are missing; this should be rectified for a revision (please refer to the "Clarity" section below for more details).
2. The experimental setup needs to be strengthened by adding at least one or two "geometrical" metrics for the representation quality (please refer to the "Quality" section below for more details).

Apart from that, there are several minor issues, which I am confident can be addressed using some additional rewriting and rephrasing.

clarity,_quality,_novelty_and_reproducibility: ## Clarity

- I have some trouble understanding the turn of phrase "features are located in the same places". I think this refers to the sequences and their respective parameters, meaning that two features arise at roughly the same thresholds. If so, a brief explanation would be warranted.

- I would suggest turning the comparison with this "TopoAE" method into a separate paragraph/section instead of already providing details about it in the introduction.

- Since the RTD is being prominently used, Section 3.2 should be extended to provide additional details about the method. The appendix and cited reference are both useful, but the paper should also be understandable on its own without referring to external sources.

- When giving an example in Section 3.2, please use the aforementioned terminology for point clouds, i.e. $X$ and $\widetilde{X}$, instead of introducing a new one.

- The colours in Figure 2 are hard to distinguish for me. I appreciate such a graphical example very much, and would thus urge to update the edge colours a little bit so that the calculation of the cross barcode becomes more apparent.

- I would suggest rephrasing  the use of the term "nullity". As I read the manuscript, I understand the discussion after Proposition 1 to mean that the RTD loss guarantees a kind of metric "identity of indiscernibles", i.e. it is zero if and only if the two barcodes are equal. Maybe it would be easier to rephrase the text slightly here.

- In Section 4.1, $\hat{X}$ should be $\tilde{X}$, I think.

- The discussion of the matrix in Section 4.1 could be simplified. First, I would suggest using a different symbol for the matrix, potentially a bold font; this would make the matrix stand out a little bit from the existing text. Second, the indices seem to follow a different terminology than the one shown in Section 3.1. I prefer $ij$ instead of the $A_i A_j$ of Section 3.1, but this should be homogenised.

- Consider using a symbol to refer to the Wasserstein distance instead of using W.D.

## Quality

The paper is already of high quality. I particularly appreciate the detailed supplements, which serve to provide a further glimpse into the different aspects of the method. Thus, the proposed method has a strong theoretical foundation and clearly improves over the state of the art. However, when *evaluating* the proposed method, there are some issues that need to be rectified:

- Using RTD as its own quality measure strikes me as slightly tautological; I understand that the loss term optimises a very related quantity, and I would therefore suggest to also show additional quality metrics (even post-hoc ones) in the respective tables. For instance, one metric that would be highly relevant to discuss here would be the actual reconstruction loss of the autoencoder. Does the reconstruction quality suffer when using the new method? Moreover, what about metrics that are not directly related to topological quantities of the data, such as a distance correlation? At the moment, the table describing the experiments are slightly favouring topology-based methods. This should be avoided, and other metrics should be considered.

- When comparing to the other topology-based methods such as UMAP and TopoAE, and extension of Section 5.5 would be useful. I understand that the existing methods have some theoretical disadvantages (in particular the discontinuity of the TopoAE), but considering the computational complexity of the proposed approach, are there any trade-offs to existing methods to be discussed? This would only require a few additional lines; to save more space, some of the tables could be relegated to the appendix; personally, I am fine with having a selection of experiments in the main paper, with a more detailed description being provided in the supplements.

## Novelty

- For the delineation to existing research, the paper [*Optimizing persistent homology based functions*](http://proceedings.mlr.press/v139/carriere21a.html) should be more prominently discussed. To the best of my knowledge, this paper provides a general justification and explanation of how to differentiate and optimise topology-based functions. If so, please adjust the statement in the introduction. Similarly, a recent preprint by Wagner et al. on [*Improving Metric Dimensionality Reduction with Distributed Topology*](https://arxiv.org/abs/2106.07613) could be mentioned briefly as it also provides some hints regarding gradient calculations and optimisation in general.

## Reproducibility

The paper is very reproducible; I appreciated that the authors are taking the time to provide the source code of their method as well. Moreover, given the description of the algorithm, a skilled graduate student should be able to re-implement the method.

## Minor issues

- In the introduction: T-SNE --> t-SNE

- The t-SNE paper by 'Van der Maaten & Hinton' is cited twice in the first sentence of the related work section.

- Consider using `\operatorname` or a related LaTeX command for typesetting operators such as the image or the kernel; this improves readability of some equations.

- "Consider R-Cross-Barcode" --> "Consider the R-Cross-Barcode" (there are other places where a definite article could be added; I would suggest another pass over the paper)

- "having similar to $X$ topology" --> "having a topology similar to $X$"

- In certain places, the manuscript uses `\citet` or a plain `\cite` where a `\citep` would be more appropriate. For instance, in the paragraph "Comparison with TopoAE loss", the citation to the TopoAE method should be given in parentheses.

- Below Figure 3: "Let [...] denotes" --> "Let [...] denote"

- Please check the bibliography for consistent spelling and capitalisation. For instance, it should be "Carrière", not "Carriere", and "UMAP" instead of "umap". Moreover, the paper by Moon et al. seems to be cited twice; once as a preprint, the other time as a published version.


summary_of_the_review: This is a high-quality paper with a substantial contribution to the literature, namely a new topology-based loss term that satisfies advantageous theoretical and empirical properties. Apart from minor issues (see above), I am very happy to endorse this paper for publication. I am confident that the proposed changes can be performed within the revision cycle.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new dimensionality reduction method, named RTD-AE, that attempts to preserve the topological structure of the data when they are compressed to a lower-dimensional space. This is done in a standard autoencoder fashion where the loss function consists of a reconstruction loss and a topology-preserving loss. The closest rival to RTD-AE is TopoAE (Moor et al, 2020) but it comes with a discontinuous topology-preserving loss that does not guarantee identity of indiscernibles (d(x,y) = 0 does not mean x = y). Instead, RTD-AE uses the Representation Topological Divergence (RTD) (Barannikov et al, 2022), which is continuous and guarantees identity of indiscernibles. However, RTD is not C1-smooth so the paper's main contribution is RTD differentiation using subgradients.

Experiments on synthetic datasets and practical datasets MNIST, F-MNIST, COIL-20, scRNA mice and scRNA melanoma, where the proposed method, named RTD-AE, is compared against recent methods like UMAP, t-SNE, a vanilla autoencoder (AE), TopoAE, Ivis, PacMAP and PHATE, show favouring results towards RTD-AE.
strength_and_weaknesses: ### Weaknesses

There is not much in terms of theoretical contribution. My impression is that it is mainly about making RTD work with differentiation, since RTD has been shown quite a useful divergence. In contrast, I acknowledge that it can be a very intensive engineering challenge.

The authors provided optimization tricks in Appendix I, including restricting the simplices of interest to just edges (k=1), gradient smoothing via averaging nearby subgradients, and bypassing plateaus when $w_{i,j} > \tilde{w}_{i,j}$. Given that topology-based optimization is typically very compute-intensive, often in cubic time, I was hoping to see some further analysis on how each of those tricks contributed to the learning of RTD-AE, in terms of changes in accuracy and/or speed. However, none was offered.

### Strength

Appendix J pointing out why the TopoAE loss is discontinuous is good to have. It would be good to also give an example why the TopoAE loss does not guarantee identity of indiscernibles.

Appendix M shows the effect of adding the RTD loss on linear correlation, triplet accuracy, and Wasserstein distance is. I find it helpful in understanding RTD.

The results show that RTD-AE outperform TopoAE in almost all settings.


clarity,_quality,_novelty_and_reproducibility: The paper is well-written and clear. I find that key materials supporting the paper are present in the paper. Reproducing of the work is not an issue.

One minor issue is the double usage of symbol k, both as the dimensionality of simplices and as the dimensionality of the compressed data.

It could be interesting to know if RTD can be approximated or upper-bounded by a more easily differentiable loss function and how that can affect the output data of RTD-AE.
summary_of_the_review: This is a good, well-rounded paper. My only concern is that for this conference I would expect the contribution to be more theoretical, although I appreciate the possibly large amount of efforts put in place to make RTD differentiation and RTD-AE work. However, the actual results mostly favouring RTD-AE over TopoAE and more distant methods is encouraging.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes an approach for topology-preserving representation learning (dimensionality reduction). The topological similarity between data points in original and latent spaces was obtained by minimizing the Representation Topology Divergence (RTD) between original data and latent representations. The proposed RTD-AE better preserves the global structure of the data manifold than popular methods t-SNE and UMAP. 

This paper received divergent scores before the authors' rebuttal. And it looks like the authors did a great job addressing reviewers' concerns, and three out of four reviewers have increased their scores to 8. The meta-reviewer agrees with the positive reviewers that this paper provides a general justification and explanation of how to differentiate and optimize topology-based functions, which would be of wide interest to the community. The meta-reviewer recommends acceptance.
Decision: Accept: poster