Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper relates tSNE and UMAP, starting from linking noise contrastive estimation and negative sampling. The authors also provided analysis on linking neighbor embedding and self-supervised learning, and it leads to optimizing tSNE with InfoNCE loss. The provided experimental results demonstrates the authors' arguments.
strength_and_weaknesses: The paper extensively covers from neighbor embedding to self-supervised learning, and the authors linked those two concepts, which has the originality. Also, the paper is easy to follow, and the experimental results are promising. 

However, at the same time, the quantitative result seems to be lacking. Are there any other quantitative result that can be shown in the experiment of dimensionality reduction algorithms?

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, and the theoretical soundness is okay. Also, exploring the connection between two major dimensionality reduction algorithms is novel. Finally, the authors also provide their code as supplementary material.

In the meantime, the authors rely on the appendix too much, and interpretation of the experimental results lack in the main body. I suggest authors to add up more informative interpretation in the main body.

summary_of_the_review: While I’m positive on this paper, I’m not very familiar with experiment parts of dimensionality reduction algorithms. Hence, I’m willing to see other reviewer’s comments.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose to unify two popular neighborhood embedding method, t-SNE and UMAP, from the perspective of contrastive learning. It is shown that a single normalization constant $\bar Z$ is controlling the visualization quality, and the same constant interpolates between t-SNE and UMAP. By making the connections, the author argue that the instability of UMAP can be remedied. 
strength_and_weaknesses: **Strengths**: it is quite useful to compare and analyze the two popular data visualization techniques, namely t-SNE and UMAP. I find it plausible that the normalization constant $\bar Z$ is playing a crucial role in controlling the compactness of embedded clusters.

**Weaknesses**: In my opinion, most arguments made in this paper are quite casual and handwaving. Other than the role of $\bar Z$, I do not see a clear argument being made, so I didn't find myself more informed after reading this paper. 

There are also crucial parts of t-SNE and UMAP that are not carefully discussed. For example, it is known that using random initialization or spectral initialization has a great impact on the visualization quality. Also, there are parameters in UMAP that determine how fat the tail is in the loss function, which presumably controls the compactness of embedded clusters. 
clarity,_quality,_novelty_and_reproducibility: I do not find this paper very clearly written. The overall argument made about the normalization constant $\bar Z$ is plausible, but not overwhelmingly convincing. Many statements in Section 3 and 4 may be partly known in the literature, so they may not be novel. I did not check if the results are reproducible or not.
summary_of_the_review: The authors make connections between t-SNE and UMAP from the contrastive learning perspective, notably using a normalization constant $\bar Z$. I find that many statements are quite vague and informal, which unfortunately makes the contribution unclear and obscure. I would not recommend acceptance of this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper reveals the connection between t-SNE and UMAP via a new connection between the contrastive methods NCE and NEG. The paper also builds the connection between contrastive neighborhood embedding and self-supervised contrastive learning. 
strength_and_weaknesses: Pros:

1. The paper provides very novel findings. Its theory (Corollary 2) explains the NEG’s behavior in the minibatch setting. The theory seems to predict the fact that Neg-t-SNE produced more compact clusters than t-SNE. Based on its theory, the paper reveals that UMAP is NEG applied to the t-SNE framework. It explains the empirical phenomenon that UMAP pulls embedding points closer together than t-SNE. 

2. The paper is well-written and well-structured. I am not an expert in the field. I am not able to verify the correctness of the theory. However, following the explanation of the paper. I can grasp the high-level idea and understand the relationship between UMAP and t-SNE. 

Suggestions:

1. It would be better if the author can give further instructions to the practitioner about how to better use UMAP and t-SNE in real practice according to the findings of this paper. As the author says, "Our conceptual relation between UMAP and t-SNE is therefore of practical impact – as is the fact that both are just instances on a spectrum highlighting more continuous or more discrete structures." How should this fact affect our daily usage of UMAP and t-SNE. 
clarity,_quality,_novelty_and_reproducibility: The paper is well-written. The paper has good quality and is very novel, although I do not check the correctness of all the theory. 
summary_of_the_review: Overall, I am very excited about this work. As an ML practitioner, I use t-SNE and UMAP almost everyday. I always wonder what is the connection between them. This paper answers my question. I think this paper's findings are useful to the majority of the ML community.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors discuss noise contrastive estimation (NCE) and negative sampling (NEG) in the context of dimensionality reduction and their connection to t-SNE and UMAP. They propose a variant of t-SNE, called NEG-t-SNE, which relies on NEG to reduce the complexity and produce more compact clusters.
strength_and_weaknesses: ### Strength:
* The authors do a reasonably well job motivating the problem and discussing NCE and NEG in the context of word embedding and dimensionality reduction.

### Weaknesses:
* The current version of the paper fails to distinguish itself from previous work, especially different variants of t-SNE (e.g., Barnes-Hut t-SNE) or its relation to UMAP, discussed in Böhm et al. (2022). It is hard to pinpoint the main contribution of the paper ito using negative samples (already done in UMAP and LargeVis).
* The main shortcoming is the lack of comparison to TriMap, which is the only DR method that truly relies on contrastive learning! Unlike what the authors discuss in Section 2, TriMap is not a variant of UMAP (TriMap stands for Triplet **Mapping**, IIUC, not Manifold Approximation and Projection) and uses NEG sampling (i.e., triplets) to form an embedding. It is well known that TriMap provides a distinct view of the data ito the global structure. I find the lack of comparison and discussion to TriMap as the main shortcoming of the paper.
clarity,_quality,_novelty_and_reproducibility: The write-up is clear, but the main contributions of the paper is not highlighted and distinguished from previous work.
summary_of_the_review: The authors need to expand the connection to contrastive learning and, especially TriMap and provide comparisons. They also need to highlight their main contributions and distinguish the new results from previous approaches such as LargeVis. Also, I recommend providing further evidence that tuning other hyperparameters in t-SNE and UMAP (such as perplexity) fails to provide a similar effect as the one proposed in NEG-t-SNE.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper "From t-SNE to UMAP with contrastive learning" presents a new insight into the connection of two contrastive learning methods: noise contrastive estimation (NCE) and negative sampling (NEG). This insight allows to connect t-SNE and UMAP, and span a spectrum of methods along their connection. Finally, connection to other self-supervised methods (such as SimCLR) are established. Figures of the method show dimensionality reductions on MNIST. The method is accompanied by a package in PyTorch.
strength_and_weaknesses: Strenghts.
- good presentation of a novel insight connecting NCE and NEG
- well and clearly written paper
- new insight about a connection of t-SNE and UMAP, which spans a range of methods

Weaknesses.
- the relation of NCE and NEG could benefit from more empirical support. In particular, Corollary 2 states that the resulting function q is proportional to the density with \hat Z. Fig1(i) shows a monotonic trend, but not a precise relation (as argued on the text). For a simpler model I'd expect that one could be able to establish this relationship, which would empirically harden the statement.
- Equation 10 shows that a Cauchy Kernel in UMAP corresponds to a inverse-square kernel in NEG. However, as far as I understood, you're using a Cauchy kernel in NEG. It'd be illustrative to see the comparison of NEG and explicit inverse-square kernel - which should show a closer resemblance to UMAP. Also, I think this difference in kernels when interpolating between the methods should be emphasized when talking about the interpolation ability.
clarity,_quality,_novelty_and_reproducibility: - the paper is clearly written and of high quality
- to my knowledge the core relationship of NCE & NEG and its relation to UMAP & t-SNE is novel
- a software package is published alongside for reproducability

summary_of_the_review: The paper is well written and offers novel insights into the relation of NCE & NEG. This paves the way to interpolate between UMAP and t-SNE, two highly used dimensionality reduction techniques.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: I would recommend accepting this paper for the following reasons:
- The paper investigates the connection of the two most popular contrastive learning methods between t-SNE and UMAP with interesting insights.
- The paper seems highly relevant to the community, as evidenced by two reviewers strongly championing the paper ("I use t-SNE and UMAP almost everyday. I always wonder what is the connection between them. This paper answers my question.").
- The author seem to adequately address reviewer concerns related to prior work.
justification_for_why_not_higher_score: The paper provides an insight into t-SNE and UMAP that is likely interesting to the community. However, it does not rise to the level of a spotlight paper in terms of novelty and potential impact.
justification_for_why_not_lower_score: The paper seems to provide an interesting insight to the community.
Decision: Accept: poster