Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a topological regularization method for incorporating topological prior knowledge for shaping data embeddings. This is achieved by introducing a new family of loss functions based on the characteristics of persistence diagrams. The results of the empirical evaluation suggest that the embeddings produced with the proposed method better capture the topological aspects of input data, by taking into account the prior topological knowledge.
main_review: ## Positive aspects of the paper:
- The family of loss functions introduced is a novel contribution that can lead to practical applications in areas that can benefit from better representation embeddings, provided that knowledge about the data topology is available and that easier ways to translate this knowledge into the loss functions described became available.

## Concerns and points for improvement:
- Certain topics and mathematical concepts considered in the paper are not familiar to a large audience of the machine learning community. Also, the paper does a poor job in explaining those concepts and I believe many readers in this community will have a hard time understanding it in its present form, due to lack of background. The authors acknowledged this by adding Appendix A, but many points are not sufficiently detailed in there either.
- Considering that acquiring knowledge about the data topology as well as translating this knowledge into an adequate loss function can both be difficult tasks, I would like to see a more thorough evaluation of how misleading the obtained can be if any of these steps are not performed adequately. 

summary_of_the_review: In its present form, I believe the paper should be rejected due to the lack of background information necessary for a large part of ICLR's audience to understand the paper and its contributions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper argues for including prior topological information about the structure between data points in some lower dimensional embedding space with the intention of improving the quality of the embeddings produced by tasks such as dimensionality reduction. The paper accomplishes this by adding a novel topological regularisation term to the standard embedding loss. The topological regularisation term incorporates the persistent homology of filtrations of the embedded data points. How this information is manipulated within the term encodes the prior knowledge the investigator wishes to use, such as the embedding should comprise a single cycle. The paper supports its method an array of qualitative and quantitative analyses on synthetic and real datasets. 
main_review: Strengths: The paper tackles an important and challenging problem faced in dimensionality reduction. The proposed method is simple and justified by existing research. The empirical results demonstrate the method improves qualitative results and emphasise that topological optimisation of the embedding space need not enforce any sensible separation of the data, an important consideration. 

Weaknesses: The paper is hampered by a lack of notational clarity in essential sections that makes it difficult to determine the full contributions and impact of the proposed method. The following are questions and suggestions to improve the clarity and content of the paper:
1) The methodology rests on equation (2) which demonstrates how topological information is used by the regularisation term, but the notation is difficult to parse.

a) Do the i and j indices refer to indices of the elements of the ordered persistence diagram? It appears that they do, but this could be emphasised. There appear to be two sets of indices: those that are grouped as essential and those termed regular. The ordering is important for the method, but dropping the inequalities and describing containment would help: k \in D^ess, k \in D^reg.

b) Relatedly, the summations in equation (2) could be reconstructed. The first summation appears unnecessary, as it is effectively neutralised by setting g_ess(.) = 0 for the remainder of the paper. Also, what is an example of this summation that has more than one term? The single connected component the embedding tends to is the only term mentioned. This could be expanded.

c) This is an essential point because the choice of j in the summations appears to be a choice point for the method: how deep into the persistence diagram should we go? Ephemeral components that appear for only one step contribute to the loss, but should they? If these components are treated as appearing and disappearing in the same step, then with the chosen g_reg(.) function, they contribute nothing to the loss and could be excluded from the summation. 

d) The choice of j and choice of g_reg will be crucial to the results, but the default choices are not compared with alternatives either in section (2) or in the results. How sensitive is the method to changes in these selections?

e) What is |D|? How is it determined? This is the number of points in the persistence diagram, but this should be expanded upon. Specifically, the discussion in the caption of Figure 1 should be moved to the main text and the terminology used there of (H0) and (H1) could be used to improve the clarity of subsequent sections (Section 2.2 par 1 line 3 refers to the 0-dimensional persistence diagram).

2) What is the complexity of the method and how does it scale with the number of data points and type of persistence diagram? In an iterated procedure, is it the case that the digram must be updated with each update of the embedding?

3) As conceded by the authors, the prior topological information is vital to the method, but it is unclear how it is discovered. Thus, it would be useful to see how the method behaves under situations where the prior information is misspecified, this could be included with the synthetic data experiments.

Minor points:
1) Motivation par 1 sentence starting ‘In other examples…’: this sentence seems to contradict itself and should be justified.
2) Additional editing could improve clarity and make space for the expanded methodology section. 
summary_of_the_review: As the paper stands, I suggest rejection. The method is interesting and I find the empirical results promising, but the methodology sections need large amounts of reworking to improve clarity and to fully justify and explain the choices made by the authors. The paper would benefit greatly from these changes.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Authors of this paper aim to integrating different prior topological knowledge into low-dimensional embeddings, where new set of topological losses are introduced as the topological regularizations to regulate embedding models. 
main_review: Strengths:

1.	Topological loss is used to regulate low-dimensional embedding method for incorporating prior topological knowledge
2.	Several novel topological loss functions are presented including k-dimensional holes and flares. And the combinations of these basic losses can be applied. 


Weaknesses:
1.	It seems that the loss (1) is an incremental formulation by combining existing low-dimensional embedding methods and the simplified topological loss function (2). From (3), the topological loss of this paper is constructed from the embeddings instead of the input data. However, this may be thought as a straightforward extension. 
2.	Authors did not explain why only the regular part of (2) is used. It looks like only partial power of the topological optimization is considered by this work. Some explanation for choosing only the regular part is needed.
3.	The claim that “a topological model that is naturally present in the data should be represented well by many subsets of the data” is not quite convincing. Taking the clustering problem as an example, this claim might not true if there are many small clusters when uniform sampling is used. Since it Is the key assumption for (4), authors might want to specify the limitation of the loss (4). Even for the experiments using (4), the analysis on the impact of f_s and n_s is not studied. 
4.	Experiments are conducted on both synthetic data and real data, but only the very basic approaches are compared. It is more like the ablation study of the proposed model. Authors need to compare the results obtained by the proposed model with the state-of-the-art methods. At least, the results from the paper using the same real data should be reported as the most relevant baselines. 
5.	From the visualization results, it seems that top. optimization embedding is very similar to the top. regularized embedding on most of the data sets expect Figure 7. Does it mean top. loss is sufficient enough to obtain relatively good results?

summary_of_the_review: Authors proposed to joint optimize low-dimensional embedding and topological loss, but the proposed method might be thought as a straightforward extension of existing work. The experiments lack the relevant methods for comparisons and the proposed method seems not significantly better than one of simple baselines from visual perspectives. Some claims need to be clarified in detail.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper develops a way to construct low-dimensional embeddings that combine an embedding cost (e.g. PCA, t-SNE, UMAP, etc.) with the topological cost, where the latter forces the embedding to have a particular topological structure as represented by the persistent homology. As an example, one can construct a data embedding and force it to be a circle, or a figure-8, or have three clusters.

The paper is based on a series of several 2020-2021 papers, with which I am unfamiliar, so it was difficult for me to judge on the novelty. I found the paper interesting, and the method seems to work well. My biggest criticism is that the goal of the method appears somewhat artificial: low-dimensional embeddings are typically done for data exploration, but one does not want to enforce any particular structure for the purposes of data exploration, and the topological structure is typically not known a priori. That's why I am giving a borderline score.
main_review: The paper develops a way to construct low-dimensional embeddings that combine an embedding cost (e.g. PCA, t-SNE, UMAP, etc.) with the topological cost, where the latter forces the embedding to have a particular topological structure as represented by the persistent homology. As an example, one can construct a data embedding and force it to be a circle, or a figure-8, or have three clusters.

The paper is based on a series of several 2020-2021 papers, with which I am unfamiliar, so it was difficult for me to judge on the novelty. I found the paper interesting, and the method seems to work well. My biggest criticism is that the goal of the method appears somewhat artificial: low-dimensional embeddings are typically done for data exploration, but one does not want to enforce any particular structure for the purposes of data exploration, and the topological structure is typically not known a priori. That's why I am giving a borderline score.

Most of my comments below are relatively minor requests for clarification.

MAJOR ISSUES

* All considered datasets are very small. Add some discussion or benchmarks of how the runtime scales with dataset size.

* Related work subsection in the Introduction should explain what exactly is the novelty in this paper, compared to Gabrielsson2020, Solomon2021, Carriere2021.


MEDIUM/MINOR ISSUES

* I like Figure 1 and appreciate that the authors attempted to explain persistent homology in the main text. However, I think it could be made clearer. The following things could be briefly explained/mentioned in the text:

a) Panel A -- this looks like kNN graph for various values of k (perhaps alpha=k). I think it's not a kNN graph, but mention something about it.

b) Panel A -- what gets filled in green? All triangles?

c) Panel B -- should we see 4 clusters corresponding to the letters I,C,L,R? Where are they on the persistent homology? Maybe highlight them somehow.

d) Panel B -- what are some prominent points on this plot? One triangle at death=200? A cluster of triangles in the upper-right corner? Would it make sense to annotate them?

e) Panel B -- maybe it should be on the log-scale? Currently many points are clumped in the low-left.

* Related: would it make sense to have some notation for k-dimensional persistence diagram? Like D_{(k)} or something. I was very confused, until I finally realized that there are several different D's. At least clearly mention on page 3 that there is one separate D object for each dimension up to the space dimensionality minus 1.

* Page 3, "these parameters are entire point clouds X" -- should it be X or E? I think E. 

* Page 4: it isn't very clear how to set i,j,mu to achieve a desired result. It slowly becomes more clear as one reads further, but it could be great to explain it from the get go. Perhaps even show in Figure 2 what happens for different choices of i? E.g. can one have 4 clusters?

* Particularly unclear is when mu should be 1 and when it should be -1.

* Page 4: If i=j=2, mu=-1 enforces 2 clusters, then why does i=2,j=inf,mu=1 in the example above enforces 1 cluster?

* Page 5, top line: what is n_S? Wasn't defined.

* Section on "flares" -- define "flare".

* Eq (6): very confusing notation in g_E : E -> R : x |-> ||...||

* Figure 3 -- panel (c) looks good but actually the topological loss would be equally happy with any arbitrary splitting of points in two clusters. So the fact that blue points are separated from orange points in panel (c) has nothing to do with the topological loss... this makes this figure and the surrounding text somewhat misleading.

* Page 6: "better captures the circular hole (with L_\bot(W)..." -- but L_\bot has nothing to do with holes, it's the orthogonality term. Should it be L_top? The same confusion appears several times in sections 3.1 and 3.2.

* Section 3.2, 1st dataset -- if it's data on cell differentiation, then why should it be a circle?

* Page 7, Figure 7b: why does Fig 7b look like it does? The topological loss enforces single cluster but here we see multiple clusters. That's confusing.

* Harry Potter dataset should be mentioned in the main text, in Section 3.2.

* Page 9: "we repeated this entire experiment 100 times" -- unclear where the variance comes from.

* Page 9: "we observe that topological regularization consistently..." -- looking at Table 4, the difference between ordinary embedding and topological regularization is much smaller than the standard deviation over repetitions, for three out of four datasets. I think this should be acknowledged in the text.

* Delaunay is consistently spelled wrong.
summary_of_the_review: The paper is based on a series of several 2020-2021 papers, with which I am unfamiliar, so it was difficult for me to judge on the novelty. I found the paper interesting, and the method seems to work well. My biggest criticism is that the goal of the method appears somewhat artificial: low-dimensional embeddings are typically done for data exploration, but one does not want to enforce any particular structure for the purposes of data exploration, and the topological structure is typically not known a priori. That's why I am giving a borderline score.
review_title: null

Meta-review: This paper proposes loss functions to encode topological priors during data embedding, based on persistence diagram constructions from computational topology.  The paper initially had some expositional issues and technical questions, but the authors did an exceptional job of addressing them during the rebuttal period----nearly all reviewers raised their scores (or intended to but didn't update the numbers on their original reviews).

The AC is willing to overlook some of the remaining questions. For example, concerns that topology isn't well known in the ICLR community (8muq) are partially addressed by the improved exposition (and it's OK to have technically sophisticated papers so long as some reviewers were able to evaluate them).  And, future work can address scalability of the algorithm, which indeed does seem to be a challenge here (ey6b).

In the final "camera ready," the authors are encouraged to address any remaining comments and to consider adding experiments/discussion regarding scalability to larger datasets.
Decision: Accept (Poster)