The Shape of Attraction in UMAP: Exploring the Embedding Forces in Dimensionality Reduction

## Abstract

Abstract Uniform manifold approximation and projection (UMAP) is among the most popular neighbor embedding methods. The method relies on attractive and repulsive forces among high-dimensional data points to obtain a low-dimensional embedding. In this paper, we analyze the forces to reveal their effects on cluster formations and visualization and compare UMAP to its contemporaries. Repulsion emphasizes differences, controlling cluster boundaries and inter-cluster distance. Attraction is more subtle, as attractive tension between points can manifest simultaneously as attraction and repulsion in the lower-dimensional mapping. This explains the need for learning rate annealing and motivates the different treatments between attractive and repulsive terms. Moreover, by modifying attraction, we improve the consistency of cluster formation under random initialization. Overall, our analysis makes UMAP and similar embedding methods more interpretable, more robust, and more accurate.

## 1 Introduction

The current era is characterized by a deluge of high-dimensional data.
Dimensionality reduction (DR) techniques have emerged as tools for exploratory analysis of such data by visualizing the underlying structure.
The most popular methods, $t$-distributed stochastic neighbor embedding (Maaten and Hinton, 2008) and uniform manifold approximation and projection (UMAP) (McInnes et al., 2018) are grounded in the attraction-repulsion dynamics that bring similar data points closer while pushing dissimilar ones apart.
As unsupervised algorithms, these do not rely on labeled data; instead, they identify and preserve the intrinsic structure of high-dimensional data by leveraging local (attractive) and global (repulsive) relationships (forces). This makes these algorithms particularly well-suited for tasks such as clustering (Becht et al., 2019), exploratory data analysis (Fleischer and Islam, 2020), anomaly detection in semiconductor manufacturing (Fan et al., 2021), visual search (González-Márquez et al., 2024), time series analysis (Altin and Cakir, 2024), studying representation convergence (Huh et al., 2024), and outlier image detection (Islam and Fleischer, 2024), where visualizing hidden patterns in unlabeled data is critical and meaningful. By learning the embeddings in a data-driven, label-free manner, DR exemplifies the power of unsupervised methods to distill complex data into easily interpretable forms.

Building upon the attraction-repulsion principle, newer methods have emerged (Amid and Warmuth, 2019; Agrawal et al., 2021; Wang et al., 2021; Narayan et al., 2021; Yang et al., 2022; Wang et al., 2025; Kury et al., 2025), each designed to emphasize specific aspects of the data.
Despite their relevance in diverse applications, these methods often rely on heuristic practices that may fail to give meaningful interpretations. Moreover, DR introduces distortions that are unavoidable (Chari and Pachter, 2023). Thus, it is imperative to have a deeper understanding of the algorithms so that practitioners can provide better interpretations of the embeddings, avoid spurious structures, and optimize performance. In practice, these algorithms achieve compact clusters using a variety of techniques, including specific initialization, learning rate schedule, and kernel function tuning. However, the underlying dynamics of the attractive and repulsive forces, responsible for cluster formation, have not been thoroughly investigated. Furthermore, the essential tunable parameters are concealed within abstract functional forms, making it harder to explain the algorithms.

In this paper, we decompose the forces into their constituent parts and extract the functional shapes of attraction and repulsion.
We find that the necessity of learning rate annealing, the challenge of providing consistent output under random initialization, and the origin of cluster formation rely on attraction.
Repulsive forces primarily govern inter-cluster distances.
Our specific contributions are:

- 1.
We formulate attraction and repulsion shapes from the attractive and repulsive forces, establish the conditions for contraction and expansion of distance, and provide a fresh perspective for these algorithms (Section [4](https://arxiv.org/html/2503.09101v3#S4)).
- 2.
We show that the attraction shape of UMAP causes the counterintuitive concept of both contraction and expansion of distance. Comparing attraction shapes of different algorithms, we discuss how attraction influences the learning rate annealing scheme (Section [5](https://arxiv.org/html/2503.09101v3#S5)).
- 3.
We modify the attraction shape to improve the consistency of embedding under random initialization. This indicates the encoding of a unique structure (Section [5.1](https://arxiv.org/html/2503.09101v3#S5.SS1)).
- 4.
Analyzing repulsion shapes, we provide a deeper understanding of cluster formation and regulating inter-cluster distance (Section [5.2](https://arxiv.org/html/2503.09101v3#S5.SS2)).
- 5.
We compare attraction and repulsion shapes of UMAP, Parametric UMAP, and NEG-$t$-SNE, unveil the similarities and distinctions among them, and characterize the stability of the algorithms (Section [6](https://arxiv.org/html/2503.09101v3#S6)).

We center the main text on UMAP, now a de facto standard in many fields, and bring in other algorithms where relevant. For further details on different algorithms, see the appendix. The codes used in the research are available at [https://github.com/tariqul-islam/explaining_neighbor_embedding](https://github.com/tariqul-islam/explaining_neighbor_embedding).

## 2 Related Works

The origin of modern iterative graph-based neighbor embedding algorithms can be traced to stochastic neighbor embedding (SNE) (Hinton and Roweis, 2002) and its extension using the t-distribution ($t$-SNE) (Maaten and Hinton, 2008).
Both methods use a dense graph in which each point in a dataset has a pairwise relation with all the others, regardless of whether they are similar to each other or not.
Moreover, the weights of the graphs are normalized to give a notion of probability distribution.
Other concurrent methods, including locally linear embedding (Roweis and Saul, 2000) and Laplacian Eigenmaps (LE) (Belkin and Niyogi, 2002), used a $k$-nearest neighbor ($k$-NN) graph of pairwise interaction.
Known as spectral methods, these algorithms rely on Eigenvalue decomposition.
Subsequent work by Tang et al. (Tang et al., 2016) incorporated the $k$-NN graph in the iterative approach and removed normalization in the lower dimension.
This approach was further extended by McInnes et al. (2018) in UMAP, where the normalization step was removed altogether (both in high and low dimension) and the embedding was obtained using pairwise interactions alone.
The optimization steps use an explicit attractive force to preserve the local neighborhood and a repulsive force to keep dissimilar points apart. Building on these foundations, methods such as PaCMAP (Wang et al., 2021) and NEG-$t$-SNE (Damrich et al., 2023) have been proposed. For a recent survey of methods, see de Bodt et al. (2025).

There has been considerable progress in understanding and explaining the relationship among these algorithms.
An early analysis of SNE found that if the data is well-clustered in the original space, then they are well-clustered in the embedding space (Shaham and Steinerberger, 2017).
A similar analysis for $t$-SNE by Linderman and Steinerberger (2019) showed that the number of clusters in the embedding space is a lower bound on the number of clusters in the original space. This was followed up in further characterization (Arora et al., 2018; Cai and Ma, 2022; Linderman and Steinerberger, 2022).
Since $t$-SNE and UMAP originate from the same underlying framework (but with drastically different visualizations), a major undertaking in the literature has been to find the connection between them (Böhm et al., 2022; Damrich et al., 2023; Draganov et al., 2023).
Böhm et al. (2022) theorized that methods like Laplacian Eigenmap, ForceAtlas2 (Jacomy et al., 2014), UMAP, and $t$-SNE are all samples from the same underlying spectrum.
Indeed, LE and $t$-SNE’s are connected by the early exaggeration phase (Cai and Ma, 2022). The connection between UMAP and $t$-SNE can be related through contrastive estimation (Damrich et al., 2023).
Around the same time, Hu et al. (2023) independently discovered the relation of contrastive learning and SNE. Recent approaches offer a probabilistic perspective (Ravuri et al., 2023; Ravuri and Lawrence, 2024), employ kernel techniques (Draganov and Dohn, 2023), and utilize information geometry (Kolpakov and Rocke, 2024) to explain dimensionality reduction.

## 3 Uniform Manifold Approximation and Projection

UMAP constructs a high-dimensional graph of the dataset $X=\{x_{i}\in R^{n}|i=1,\dots,N\}$ by using a pairwise relation: $p_{i,j}=f_{h}(d(x_{i},x_{j}))$, typically, $\in[0,1]$ (for details, see Appendix [H](https://arxiv.org/html/2503.09101v3#A8)).

The graph of the low-dimensional representation $Y=\{y_{i}\in\mathbb{R}^{d}|i=1,\dots,N\}$ is given by a differentiable function

$$ $\displaystyle q_{ij}=\frac{1}{1+a(||y_{i}-y_{j}||_{2}^{2})^{b}},$ (1) $$

where the parameters $a$ and $b$ determine the density of the mapping and are chosen by fitting $q_{ij}$ to

$$ $\displaystyle\Psi(d_{ij})=\begin{cases}1&\text{if }||y_{i}-y_{j}||_{2}<m_{d}\\ \exp(-(d_{ij}-m_{d}))&\text{otherwise}\end{cases},$ (2) $$

where $m_{d}$ regulates the distance between the two nearest low-dimensional points.
UMAP aims to minimize the following cross-entropy loss function:

$$ $\displaystyle\mathcal{L}=\sum_{i,j}(-p_{ij}\log\left({q_{ij}}\right)-\left(1-p_{ij}\right)\log\left({1-q_{ij}}\right)).$ (3) $$

The first term provides an attractive force and the second term provides a repulsive force. Instead of optimizing every point in each iteration, UMAP takes the negative sampling approach (Mikolov et al., 2013; Tang et al., 2016). For each edge with $p_{ij}>0$, named a positive edge, several edges are sampled randomly, named negative edges.
The attractive force is applied on the positive edge:

$$ $\displaystyle y_{i}^{t+1}$ $\displaystyle=y_{i}^{t}+\lambda\nabla_{y_{i}^{t}}\log(q_{ij}),$ (4) $\displaystyle y_{j}^{t+1}$ $\displaystyle=y_{j}^{t}+\lambda\nabla_{y_{j}^{t}}\log(q_{ij}),$ (5) $$

and the repulsive force is applied on the negative edges:

$$ $\displaystyle y_{i}^{t+1}=y_{i}^{t}+\lambda\nabla_{y_{i}^{t}}\log(1-q_{ij}),$ (6) $$

where $\lambda(>0)$ is the learning rate and $t$ is the step number. Note that $y_{j}$ is not updated for negative edges. For a detailed analysis of UMAP’s loss, see Damrich and Hamprecht (2021).

## 4 Attraction and Repulsion Shapes

The action of the updates ([4](https://arxiv.org/html/2503.09101v3#S3.E4)-[6](https://arxiv.org/html/2503.09101v3#S3.E6)) can be simplified by decomposing the gradients $\nabla_{y_{i}^{t}}\log(q_{ij})$ ($\nabla_{y_{i}^{t}}\log(1-q_{ij})$) into a scalar coefficient dependent on the distance $||y_{i}-y_{j}||_{2}$ acting on the vector $(y_{i}-y_{j})$. We call this scalar coefficient the attraction (repulsion) shape. While we use UMAP as a specific example, this formalism applies to any method that relies on attraction and repulsion.

By writing $\nabla_{y_{i}^{t}}\log(q_{ij})=f_{a}(\zeta^{t})(y_{i}^{t}-y_{j}^{t})$, where $\zeta=||y_{i}-y_{j}||_{2}$, we can update the equations of a positive edge as

$$ $\displaystyle y_{i}^{t+1}$ $\displaystyle=y_{i}^{t}+\lambda f_{a}(\zeta^{t})(y_{i}^{t}-y_{j}^{t}),~$ (7) $\displaystyle y_{j}^{t+1}$ $\displaystyle=y_{j}^{t}-\lambda f_{a}(\zeta^{t})(y_{i}^{t}-y_{j}^{t}).~$ (8) $$

Here, $f_{a}:\mathbb{R}_{\geq 0}\to\mathbb{R}_{\leq 0}$ is the attraction shape, and we use the fact that for Euclidean metric, $\nabla_{y_{i}^{t}}\log(q_{ij})=-\nabla_{y_{j}^{t}}\log(q_{ij})$. Similarly, we reformulate the update equation of a negative edge as

$$ $\displaystyle y_{i}^{t+1}$ $\displaystyle=y_{i}^{t}+\lambda f_{r}(\zeta^{t})(y_{i}^{t}-y_{j}^{t}),~$ (9) $\displaystyle y_{j}^{t+1}$ $\displaystyle=y_{j}^{t},~$ (10) $$

where $f_{r}:\mathbb{R}_{\geq 0}\to\mathbb{R}_{\geq 0}$ is the repulsion shape.

Such decomposition has appeared previously, e.g., in (McInnes et al., 2018; Agrawal et al., 2021; Draganov and Dohn, 2023), but their formulations and utilization vary.
The original UMAP paper (McInnes et al., 2018) used it as a computational trick for fast processing, while Draganov and Dohn (2023) used it for comparing different algorithms from the kernel perspective. In both cases, the primary focus was computing the derivative.
Here, we treat the decompositions as independent functions that can take various forms.
A similar approach was used earlier by Agrawal et al. (2021), but they expressed the decomposition in terms of a scalar coefficient and a unit vector ($(y_{i}-y_{j})/||y_{i}-y_{j}||_{2}$) to emphasize the magnitude of the forces.
The discussion below shows that our shape decomposition is more illuminating than the magnitude alone.

### 4.1 Conditions for Attraction and Repulsion

In this section, we establish the conditions of attraction and repulsion from the update equations ([7](https://arxiv.org/html/2503.09101v3#S4.E7))-([10](https://arxiv.org/html/2503.09101v3#S4.E10)). The following proposition characterizes the contraction of distance between the pair $y_{i}^{t}$ and $y_{j}^{t}$ of a positive edge:

###### Proposition 4.1 .

The update Eqs. ([7](https://arxiv.org/html/2503.09101v3#S4.E7)) and ([8](https://arxiv.org/html/2503.09101v3#S4.E8)) provide a contraction of distance ($||y_{i}^{t+1}-y_{j}^{t+1}||<||y_{i}^{t}-y_{j}^{t}||$) if $-1<\lambda f_{a}<0$.

Here, $\lambda f_{a}$ works as the effective attraction shape. In most cases, however, $f_{a}$ alone is enough to draw meaningful conclusions about the embeddings. For a negative edge, the following proposition characterizes the expansion of the distance between the pair $y_{i}^{t}$ and $y_{j}^{t}$:

###### Proposition 4.2 .

The update Eqs. ([9](https://arxiv.org/html/2503.09101v3#S4.E9)) and ([10](https://arxiv.org/html/2503.09101v3#S4.E10)) provide an expansion of distance ($||y_{i}^{t+1}-y_{j}^{t+1}||>||y_{i}^{t}-y_{j}^{t}||$) if $f_{r}>0$.

Note that the inclusion of a symmetric term $-\lambda f_{r}(\zeta)(y_{i}^{t}-y_{j}^{t})$ in Eq. ([10](https://arxiv.org/html/2503.09101v3#S4.E10)) does not alter the conclusion presented in this proposition. Additionally, these conditions give a per-iteration certificate for guaranteed contraction/expansion, and one should not confuse them with the learning rate tuning/decay mechanism, as we can design the shapes in such a way that the contraction and expansion do not rely on learning rate decay.

Figure: Figure 1: Attraction and repulsion shapes in UMAP. (a) Effect of different values of $f_{a}$ (top) and $f_{r}$ (bottom) on a pair. (b) Attraction shape of UMAP. (c) Effective attraction shape $(\lambda f_{a})$ for various learning rates $\lambda$. (d) Minimum distance for contraction ($\zeta_{-1}$) as $\lambda$ decreases. (e) Repulsion shape of UMAP. (f) Attraction and (g) repulsion shapes of various embedding methods. (h) Trustworthiness and (i) Silhouette Score of various methods as the constant learning rate is varied for attraction ($\lambda_{a}$) and repulsion ($\lambda_{r}$) independently. Default UMAP parameters: $a=1.58$ and $b=0.89$.
Refer to caption: figures_attr_shape/Fig_1_new_ICLR.eps

Figure [1](https://arxiv.org/html/2503.09101v3#S4.F1) (a) shows the effect of different values of $f_{a}<0$ and $f_{r}>0$ on two points. The latter shape is straightforward, as any positive value increases the distance. The former is much more subtle, as $f_{a}$ encodes both attractive and repulsive dynamics. For $f_{a}\in(-1.0,0)$, the distance decreases, with a sign flip at the value $f_{a}=-0.5$ of maximum attraction (coincident points).
Any value lower than $-1.0$ causes the distance to increase. Although these forces act locally, they collectively shape the global structure.

###### Proposition 4.1 .

###### Proposition 4.2 .

## 5 Analysis of UMAP in terms of Attraction and Repulsion Shape

Using the gradient decomposition and the distance form ([1](https://arxiv.org/html/2503.09101v3#S3.E1)), the attraction and repulsion shapes are given by

$$ $\displaystyle f_{a}^{U}(\zeta)=-\frac{2ab\zeta^{2(b-1)}}{1+a\zeta^{2b}}$ (11) $$

and

$$ $\displaystyle f_{r}^{U}(\zeta)=\frac{2b}{\zeta^{2b}(1+a\zeta^{2b})},$ (12) $$

respectively. This section focuses on the default shapes of UMAP, controlled primarily by the parameters $a$ and $b$, and discusses the insights learned by perturbing them. The discussion below is generally valid for $b\leq 1$, where the attraction shape is strictly increasing and thus invertible (For derivation, see Appendix [A](https://arxiv.org/html/2503.09101v3#A1)).

Figure [1](https://arxiv.org/html/2503.09101v3#S4.F1) (b) shows the default attraction shape of UMAP ($a=1.58,b=0.89$).
It becomes unbounded (approaches $\infty$) as $\zeta\to 0$.
As predicted by Proposition [4.1](https://arxiv.org/html/2503.09101v3#S4.Thmtheorem1), the transition from contraction to expansion occurs when $\lambda f_{a}^{U}$ crosses $-1$ as $\zeta$ approaches 0.
Since the attraction shape is invertible, we can identify the distance at this transition, $\zeta_{-1}=(\lambda f_{a}^{U})^{-1}(-1)$, as the minimum distance for contraction due to attractive updates. Effectively, $\zeta>\zeta_{-1}$ causes contraction, and $0<\zeta<\zeta_{-1}$ causes expansion, contradicting the intuition that attractive updates consistently bring points closer together.

If $\zeta_{-1}$ is high, neighboring points oscillate between contraction and expansion, and the clusters appear fuzzy.
For the sharpest boundaries, then, the goal of optimization can be recast as one of achieving the limit $\zeta_{-1}\to 0$.
As a result, UMAP’s learning rate schedule requires annealing to zero (Figs. [1](https://arxiv.org/html/2503.09101v3#S4.F1) (c,d)). On the other hand, the repulsion shape (Eq. [12](https://arxiv.org/html/2503.09101v3#S5.E12)) is always positive and satisfies Proposition [4.2](https://arxiv.org/html/2503.09101v3#S4.Thmtheorem2).
$f_{r}^{U}$ approaches 0 as $\zeta\to\infty$ and approaches $\infty$ as $\zeta\to 0$ (Fig. [1](https://arxiv.org/html/2503.09101v3#S4.F1) (e)).

Attraction shapes for different algorithms show that only UMAP and t-SNE deal with the issue of having $f_{a}<-1$ (and consequently $\zeta_{1}>0$ at $\lambda=1$, Fig. [1](https://arxiv.org/html/2503.09101v3#S4.F1)(f)).
$t$-SNE solves it by weighting the updates with corresponding $p_{ij}$ values, while UMAP relies on learning rate annealing.
Methods like Neg-$t$-SNE and PaCMAP have $f_{a}$ naturally within $[-1,0]$ (and thus, satisfies Proposition [4.1](https://arxiv.org/html/2503.09101v3#S4.Thmtheorem1) for $\lambda\in[0,1]$ with $\zeta_{-1}=0$). Furthermore, PaCMAP’s weighted $f_{a}$ ($<0.5$ always) even prevents any flips during attraction. On the other hand, the repulsion shapes for different algorithms show that only UMAP deals with large values for small distances (Fig. [1](https://arxiv.org/html/2503.09101v3#S4.F1)(g); the shape is unbounded, but during optimization, the values are often clipped). For additional details of these and related methods, see Appendix [G](https://arxiv.org/html/2503.09101v3#A7).

Figure: Figure 2: Effect of random initialization on different attraction shapes for the MNIST dataset. (a) Mapping using PCA. (b-d) Four mappings with the lowest Procrustes distance ($p_{d}$) from the embedding in (a) for (b) UMAP, (c) modified, and (d) composite attraction shapes. (e) Default UMAP and modified attraction shapes. (f-h) Procrustes matrices from 100 runs of (f) UMAP ($0.78\pm 0.13$), (g) modified ($0.49\pm 0.21)$, and (h) composite attraction ($0.50\pm 0.20$) shapes. The diagonal $(i,i)$ entries of the Procrustes matrix are sorted by Procrustes distance ($p_{d}$) from (a), and the off-diagonal, $(i,j)$, entries correspond to $p_{d}$ between $i^{th}$ and $j^{th}$ mapping. The matrices and (mean $p_{d}\pm$std) values show that UMAP’s embeddings are not self-similar, while the modified and composite attraction shapes encourage scale-invariant structure.
Refer to caption: figures_attr_shape/fig_random.eps

We show the effect of these attraction/repulsion choices for MNIST embedding by applying separate constant learning for attraction ($\lambda_{a}$) and repulsion ($\lambda_{r}$), and varying their values (Figs. [1](https://arxiv.org/html/2503.09101v3#S4.F1)(h,i)).
For UMAP, the lower value of $\lambda_{a}$ ($<0.5$ and consequently $\zeta^{-1}\simeq 0$) is preferred for better embedding (trustworthiness (T) (Venna and Kaski, 2001), a measure of structure preservation, in Fig. [1](https://arxiv.org/html/2503.09101v3#S4.F1)(h)) and clustering (silhouette scores (SIL) (Rousseeuw, 1987), a measure of cluster separation, in Fig. [1](https://arxiv.org/html/2503.09101v3#S4.F1)(i)), whereas the whole range of $\lambda_{r}$ ($\in[0,1]$) could be used.
For NEG-$t$-SNE and PaCMAP, for which $\zeta_{-1}=0$, the whole range of parameters is effective.
This confirms that UMAP relies on making $\zeta_{-1}$ close to 0 for satisfactory embedding and clustering, which it achieves in practice by learning rate annealing, whereas $\zeta_{-1}=0$ is the default in newer algorithms.
For additional discussions, results, and illustrations of these methods on different datasets, see Section [6](https://arxiv.org/html/2503.09101v3#S6) and Appendices [B](https://arxiv.org/html/2503.09101v3#A2), [G](https://arxiv.org/html/2503.09101v3#A7), and [I](https://arxiv.org/html/2503.09101v3#A9).

### 5.1 Improving UMAP’s Consistency under Random Initialization

The consistency of UMAP embeddings depends on proper initialization (Kobak and Linderman, 2021; Wang et al., 2021).
Typically, principal component analysis (PCA) of the data or spectral decomposition of the high-dimensional graph initializes the embedding, producing consistent mappings despite various sources of stochasticity.
If randomly initialized, clusters often fail to form or form in a random orientation each time the algorithm executes.
If the initial distance between two points (nearest neighbors in high dimension) is large, the attractive forces become too low to bring them closer.
Known as near-sightedness (Wang et al., 2021), this phenomenon is evident in the attraction shape, where $|f_{a}^{U}|$ diminishes towards zero as the distance increases (i.e., $\lim_{\zeta\to\infty}|f_{a}^{U}(\zeta)|\zeta=0$).

One can induce “far-sightedness” in the mapping by increasing attraction for large distances, facilitating faraway neighbors to come closer. To test this hypothesis, we modify the attraction shape of UMAP to increase the attractive force:

$$ $\displaystyle f_{a}^{M}=f_{a}^{U}-\beta\zeta,$ (13) $$

where $\beta$ is a parameter that regulates the strength of the added term (we used $\beta=0.2$, Fig. [2](https://arxiv.org/html/2503.09101v3#S5.F2) (e)). This addition in the attraction shape translates to adding a regularizer in the attractive term of the loss function (i.e., $\mathcal{L}^{M}=\mathcal{L}+\sum_{i,j}\beta/3~||y_{i}-y_{j}||_{2}^{3}$).
In addition to attracting pairs at faraway distances, this technique enables intermixing of points that help convergence under random initialization (akin to early exaggeration in $t$-SNE).
In ([13](https://arxiv.org/html/2503.09101v3#S5.E13)), we chose the simplest linear correction; other functions, such as $\log{\zeta}$ or $\zeta^{p}$ ($p\in\mathbb{R}_{\geq 0}$), may also be suitable.

We also consider a composite attraction shape:

$$ $\displaystyle f_{a}^{C}=\begin{cases}f_{a}^{M},&\text{epoch}\leq 100\\ f_{a}^{U},&\text{otherwise}\end{cases}.$ (14) $$

The composite shape attempts to remove any distortions introduced by $f_{a}^{M}$ by reverting to the original UMAP. Below, we discuss the effects these modified and composite attraction shapes have on DR from random initialization.

Figure: Figure 3: Control of inter-cluster distances on the MNIST dataset. (a) Computing $a,b$ by varying the low-dimensional distance $m_{d}$ restricts exploration. (b-d) UMAP output by setting $m_{d}$ to $0.1$, $0.01$, and $0.0001$, respectively, shows little improvement in compactness of clusters. (e) Repulsion shapes for different parameters. (f-h) Increasing repulsion by explicitly varying $b$ results in more compact clusters and forms new ones that were absent otherwise. (i) Attraction shapes by varying parameters. (j-l) Increasing repulsion by adding a small positive value ($\varepsilon$) to the repulsion shape increases inter-cluster distance.
Refer to caption: figures_attr_shape/fig_repl.eps

We first created a PCA-initialized embedding of the MNIST dataset Figure [2](https://arxiv.org/html/2503.09101v3#S5.F2) (a). Then, we produced embeddings using random initialization (Gaussian) for each shape and repeated the experiment 100 times.
To quantify the results, we use Procrustes analysis (Gower, 1975) that aligns two point clouds under scaling, translation, rotation, and reflection (for details, see Appendix [D.1](https://arxiv.org/html/2503.09101v3#A4.SS1)).
Here, we align the randomly initialized embeddings to that of the PCA-initialized one and characterize their separation using the Procrustes distance ($p_{d}$).
Figure [2](https://arxiv.org/html/2503.09101v3#S5.F2) (b) shows four embeddings with the lowest $p_{d}$. While the cluster shapes are consistent, their placements are not.
Outputs from the modified and composite attraction shapes (Figs. [2](https://arxiv.org/html/2503.09101v3#S5.F2) (c) and (d), respectively) show improved consistency of cluster placements.

To quantify the placements further, we consider the Procrustes matrix: the diagonal of the matrix is sorted by $p_{d}$ from the PCA-initialized mapping, and the off-diagonal values are $p_{d}$ between two randomly initialized mappings (for details, see Appendix [D.1](https://arxiv.org/html/2503.09101v3#A4.SS1)). This quantification is analogous to the similarity matrix (Foote, 1999).
The embeddings due to the default UMAP attraction shape are not similar to each other (Fig. [2](https://arxiv.org/html/2503.09101v3#S5.F2) (f)), but the modified (Fig. [2](https://arxiv.org/html/2503.09101v3#S5.F2) (g)) and composite (Fig. [2](https://arxiv.org/html/2503.09101v3#S5.F2) (h)) shapes show strong similarity to each other. This indicates that UMAP encodes a unique structure, regardless of the initialization. However, attaining that structure in the low dimension may fall short due to small attraction at longer distances.

### 5.2 Cluster Formation and Compactness

The primary controllable parameter influencing cluster formation in UMAP is the minimum distance parameter $m_{d}$ (through Eq. [2](https://arxiv.org/html/2503.09101v3#S3.E2)).
However, varying $m_{d}$ restricts the exploration of different values of $a$ and $b$ (Fig. [3](https://arxiv.org/html/2503.09101v3#S5.F3) (a)).
Thus, reducing $m_{d}$ often results in embeddings that do not provide additional benefit (Figs. [3](https://arxiv.org/html/2503.09101v3#S5.F3) (b-d)).
The key factor is the limited influence of varying $m_{d}$ on the repulsion shape (Fig. [3](https://arxiv.org/html/2503.09101v3#S5.F3) (e)). Alternatively, we can explicitly vary the values of $a$ and $b$.
Decreasing $a$ increases repulsion, but it decreases attraction at a faster rate (causing a worse case of near-sightedness).
On the other hand, decreasing $b$ gives a better control (Figs. [3](https://arxiv.org/html/2503.09101v3#S5.F3) (e,i)). Figures [3](https://arxiv.org/html/2503.09101v3#S5.F3) (f-h) show increasing inter-cluster distance and breaking up of previous clusters by varying $b$ to $0.6$, $0.5$, and $0.4$, respectively.
This breaking up occurs due to the increasingly heavy-tailed nature of the kernel as $b$ decreases (heavy-tailed kernels result in smaller and distinct clusters (Van der Maaten and Hinton, 2008; Kobak et al., 2019); see Kobak et al. (2019) for how varying $b$ in UMAP modulates the tail).
Although this approach separates all ten MNIST labels into distinct clusters, the relative contributions of attraction and repulsion are hard to disentangle; changing $b$ amplifies repulsion more than changing $m_{d}$, but it also reshapes the attraction profile.

Figure: Figure 4: Embedding of the MNIST dataset with (a) default attraction shape but repulsion shape with $b=0.4$ and (b) default repulsion shape but attraction shape with $b=0.4$. The former shows the same clusters of default UMAP with increased compactness. The latter develops new structures within clusters and forms new clusters.
Refer to caption: figures_attr_shape/cross_shapes.eps

To quantify the effect of repulsion independently, we can keep attraction fixed and vary the other. To achieve this, we modify the repulsion shape by adding a small positive value ($\varepsilon$):

$$ $\displaystyle f_{r}^{M}=f_{r}^{U}+\varepsilon,$ (15) $$

while keeping the values of $a$ and $b$ constant, which effectively adds a regularizer in the repulsive term of the loss function (i.e, $\mathcal{L}^{M}=\mathcal{L}-\sum_{i,j}\varepsilon/2~||y_{i}-y_{j}||_{2}^{2}$). Using this, we can obtain stronger repulsion than previously (Fig. [3](https://arxiv.org/html/2503.09101v3#S5.F3)(e)).
Figs. [3](https://arxiv.org/html/2503.09101v3#S5.F3) (j-l) show that as $\varepsilon$ increases, the inter-cluster distances also increase.
However, the clustering properties show similarity to those obtained by varying $m_{d}$, and we get a loose separation of all the labels as $\epsilon$ increases.
Overall, the parameter $\varepsilon$ keeps the attraction shape unaffected, and varying $m_{d}$ effectively traces similar attraction shapes (Fig. [3](https://arxiv.org/html/2503.09101v3#S5.F3) (i)), suggesting cluster formation is governed predominantly by attraction.

To resolve the mystery of cluster formation, we change either the attraction shape or the repulsion shape individually while leaving the other at the default by simply setting $b$ to $0.4$ (Fig. [4](https://arxiv.org/html/2503.09101v3#S5.F4)).
The default attraction is unable to show new structures or clusters in the embedding, but the increased repulsion ($b=0.4$) gives smaller clusters than the original UMAP (Fig. [4](https://arxiv.org/html/2503.09101v3#S5.F4) (a)).
On the other hand, when the attraction increases by setting $b=0.4$ with the default repulsion, the embedding shows additional structures within each cluster (Fig. [4](https://arxiv.org/html/2503.09101v3#S5.F4) (b)).
Some of the older clusters even separate into smaller ones.
This shows that attraction causes cluster formation, while repulsion makes the clusters more compact (depending on the repulsion shape, it does this either by making smaller clusters or increasing inter-cluster distance). We show in appendix [G.3](https://arxiv.org/html/2503.09101v3#A7.SS3) that LocalMAP embeddings also exploit this interplay of attraction and repulsion to separate the clusters.

## 6 Comparison to NEG- t t -SNE

Setting $a$ and $b$ to $1$ is common for various dimensionality reduction algorithms. This particular setting makes the gradients in many algorithms stable (or bounded) and, in turn, makes optimization easier.
Recently, Damrich et al. (2023) explored this for UMAP and proposed Neg-$t$-SNE as a solution, which, in addition to having a stable gradient, provides a more compact clustering even for a constant learning rate of $1$. On the other hand, Parametric UMAP (Sainburg et al., 2021) initially used the UMAP loss formulation, but later it(^1^11[https://github.com/lmcinnes/umap/pull/856](https://github.com/lmcinnes/umap/pull/856), merged on April 26, 2022) adopted a numerically stable modified cross-entropy loss function (Shi et al., 2023) with a logsigmoid kernel. Analysis of both reveals that the approaches arrive at the same formulation from different starting points, which leads to the following observation:

###### Proposition 6.1 .

Neg-t-SNE is Parametric UMAP with $a=1$ and $b=1$.

With $a=1$ and $b=1$, the attraction and repulsion shapes of UMAP are given by
$f_{a}^{U}=-2/(1+\zeta^{2})$ and
$f_{r}^{U}=2/(\zeta^{2}(1+\zeta^{2}))$,
respectively. The attraction shape becomes bounded within $[-2,0]$, with $f_{a}^{U}(0)=-2$ (Fig. [1](https://arxiv.org/html/2503.09101v3#S4.F1) (f)), while the repulsion shape remains essentially unchanged (i.e., unbounded as $\zeta\to 0$, Fig. [1](https://arxiv.org/html/2503.09101v3#S4.F1) (g)). Since $f_{a}^{U}<-1$ as $\zeta\to 0$, according to Proposition [4.1](https://arxiv.org/html/2503.09101v3#S4.Thmtheorem1) and the discussion provided in Section [5](https://arxiv.org/html/2503.09101v3#S5), this unity case still requires learning rate annealing.

Damrich et al. (2023) compared UMAP’s negative sampling loss function from the perspective of contrastive embedding (CE) and concluded that the effective kernel of UMAP is $1/\zeta^{2}$.
Under the CE framework, the authors introduced NEG-$t$-SNE by changing the kernel to $1/(1+\zeta^{2})$. Reverting to UMAP formalism, this results in the low-dimensional affinity function $q_{ij}^{N}=1/(2+\zeta^{2})$. Consequently, the attraction and repulsion shapes are
$f_{a}^{N}=-2/(2+\zeta^{2})$, and $f_{r}^{N}=2/((1+\zeta^{2})(2+\zeta^{2}))$,
respectively (Figs. [1](https://arxiv.org/html/2503.09101v3#S4.F1) (f,g)).
The attraction shape is bounded within $[-1,0]$ and satisfies Proposition [4.1](https://arxiv.org/html/2503.09101v3#S4.Thmtheorem1).
Any $\lambda\in[0,1]$ would cause contraction and avoid oscillation of expansion and contraction.
Thus, NEG-$t$-SNE is less sensitive to learning rate annealing, and the clusters appear less fuzzy even for constant $\lambda=1.0$.
Moreover, the repulsion shape of NEG-$t$-SNE is also bounded within $[0,1]$ and does not approach infinity as $\zeta\to 0$, which causes fewer points to leave the clusters when sampled randomly.
While Damrich et al. (2023) used only the bounded repulsive forces of NEG-$t$-SNE and unbounded ones of UMAP to explain this disparity, our analysis, in Fig. [1](https://arxiv.org/html/2503.09101v3#S4.F1), shows that the attraction shape is equally responsible for stability and compactness of the clusters. For additional discussion on NEG-$t$-SNE with illustration, see Appendix [F](https://arxiv.org/html/2503.09101v3#A6).
Together with the discussion in Fig. [4](https://arxiv.org/html/2503.09101v3#S5.F4), this suggests that attraction and repulsion shapes from different methods can be mixed, for instance, using the attraction shape of UMAP with the repulsion shape of NEG-$t$-SNE. We show a few examples by experimenting on UMAP, NEG-$t$-SNE, and PaCMAP in Appendix [J](https://arxiv.org/html/2503.09101v3#A10).

###### Proposition 6.1 .

## 7 Discussion and Conclusion

In this work, we studied the relationship of attractive and repulsive forces to cluster formation.
While it is known that attractive forces bring similar points closer together and repulsive forces push dissimilar points further away, the exact mechanisms of such forces have not been well studied.
Here, we demystified much of the dynamics underlying cluster formation. While we focused on UMAP and NEG-$t$-SNE in the main text, the results are general to dimensionality reduction (some other algorithms are discussed in the Appendix [G](https://arxiv.org/html/2503.09101v3#A7)).

We treated the attraction and repulsion coefficients as functional mappings and analyzed them in the context of their respective shapes.
Characterizing these shapes (Propositions [4.1](https://arxiv.org/html/2503.09101v3#S4.Thmtheorem1) and [4.2](https://arxiv.org/html/2503.09101v3#S4.Thmtheorem2)) revealed a counterintuitive result: the attractive forces of UMAP gave repulsion (expansion of distance) for shorter distances (Fig. [1](https://arxiv.org/html/2503.09101v3#S4.F1) (b)), i.e., instead of bringing neighboring points closer together, it pushes them away. We conclude that UMAP’s learning rate schedule alleviates this phenomenon.

We also studied the initialization of the low-dimensional embedding. Similar points starting further apart experience low attraction and never contract well enough.
Our formalism provides a way to influence attraction at larger distances by adding additional terms to the attraction shape.
This modification resulted in outputs that are more consistent under random initialization. This gives us confidence that UMAP encodes a unique structure, with numerical tricks merely helping to achieve it faster.

Analyzing the repulsion shape revealed that higher repulsion causes a larger inter-cluster distance. To this end, we added a small positive value to the shape, giving a higher repulsion that is not achievable by traditional means of varying the minimum distance parameter $m_{d}$.

Finally, we explored the relation between NEG-$t$-SNE and UMAP. For the particular case of $a=1$ and $b=1$, the primary difference is that NEG-$t$-SNE in its default form provides bounded attraction ($\in[-1,0]$) and repulsion ($\in[0,1]$) shapes. Thus, both propositions are satisfied regardless of the learning rate. As a result, the method shows less sensitivity to a fixed learning rate ($\in[0,1]$). Varying $a$ and $b$ in NEG-$t$-SNE introduces the same numerical challenges as UMAP. We also showed that NEG-$t$-SNE and Parametric UMAP are equivalent.

These insights into attraction–repulsion dynamics offer new tools for optimizing dimensionality reduction algorithms. Beyond this, the close connection between dimensionality reduction and contrastive learning (Damrich and Hamprecht, 2021; Hu et al., 2023) suggests that our approach can also enhance representation learning. Taken together, our work aims to make embeddings and their interpretations more principled, consistent, and reliable, and guide future research.

## Acknowledgement

This work was supported by AFOSR grant FA9550-21-1-0317 and the Schmidt DataX Fund at Princeton University, made possible through a major gift from the Schmidt Futures Foundation. Mohammad Tariqul Islam is supported by MIT-Novo Nordisk Artificial Intelligence Fellowship.

## Software and Data

All the data used in this paper are publicly available.
The MNIST dataset is available at [https://yann.lecun.com/exdb/mnist/](https://yann.lecun.com/exdb/mnist/).
Fashion-MNIST is available at [https://github.com/zalandoresearch/fashion-mnist](https://github.com/zalandoresearch/fashion-mnist).
Single-cell transcriptomes data is available at [https://github.com/biolab/tsne-embedding](https://github.com/biolab/tsne-embedding).

## References

- Agrawal et al. (2021)
Akshay Agrawal, Alnur Ali, Stephen Boyd, et al.
Minimum-distortion embedding.
*Foundations and Trends® in Machine Learning*,
14(3):211–378, 2021.
- Altin and Cakir (2024)
Mahsun Altin and Altan Cakir.
Exploring the influence of dimensionality reduction on anomaly
detection performance in multivariate time series.
*IEEE Access*, 2024.
- Amid and Warmuth (2019)
Ehsan Amid and Manfred K Warmuth.
TriMap: Large-scale dimensionality reduction using triplets.
*arXiv preprint arXiv:1910.00204*, 2019.
- Arora et al. (2018)
Sanjeev Arora, Wei Hu, and Pravesh K Kothari.
An analysis of the t-sne algorithm for data visualization.
In *Conference On Learning Theory*, pages 1455–1462. PMLR,
2018.
- Becht et al. (2019)
Etienne Becht, Leland McInnes, John Healy, Charles-Antoine Dutertre,
Immanuel WH Kwok, Lai Guan Ng, Florent Ginhoux, and Evan W Newell.
Dimensionality reduction for visualizing single-cell data using umap.
*Nature biotechnology*, 37(1):38–44, 2019.
- Belkin and Niyogi (2002)
Mikhail Belkin and Partha Niyogi.
Laplacian eigenmaps and spectral techniques for embedding and
clustering.
In *Advances in neural information processing systems*, pages
585–591, 2002.
- Böhm et al. (2022)
Jan Niklas Böhm, Philipp Berens, and Dmitry Kobak.
Attraction-repulsion spectrum in neighbor embeddings.
*The Journal of Machine Learning Research*, 23(1):4118–4149, 2022.
- Borg and Groenen (2007)
Ingwer Borg and Patrick JF Groenen.
*Modern multidimensional scaling: Theory and applications*.
Springer Science & Business Media, 2007.
- Cai and Ma (2022)
T Tony Cai and Rong Ma.
Theoretical foundations of t-sne for visualizing high-dimensional
clustered data.
*Journal of Machine Learning Research*, 23(301):1–54, 2022.
- Chari and Pachter (2023)
Tara Chari and Lior Pachter.
The specious art of single-cell genomics.
*PLOS Computational Biology*, 19(8):e1011288, 2023.
- Damrich and Hamprecht (2021)
Sebastian Damrich and Fred A Hamprecht.
On UMAP’s true loss function.
*Advances in Neural Information Processing Systems*,
34:5798–5809, 2021.
- Damrich et al. (2023)
Sebastian Damrich, Niklas Böhm, Fred A Hamprecht, and Dmitry Kobak.
From $t$-SNE to UMAP with contrastive learning.
In *The Eleventh International Conference on Learning
Representations*, 2023.
- de Bodt et al. (2025)
Cyril de Bodt, Alex Diaz-Papkovich, Michael Bleher, Kerstin Bunte, Corinna
Coupette, Sebastian Damrich, Enrique Fita Sanmartin, Fred A Hamprecht,
Emőke-Ágnes Horvát, Dhruv Kohli, et al.
Low-dimensional embeddings of high-dimensional data.
*arXiv preprint arXiv:2508.15929*, 2025.
- Draganov and Dohn (2023)
Andrew Draganov and Simon Dohn.
Relating tsne and umap to classical dimensionality reduction.
*arXiv e-prints*, pages arXiv–2306, 2023.
- Draganov et al. (2023)
Andrew Draganov, Jakob Jørgensen, Katrine Scheel, Davide Mottin, Ira Assent,
Tyrus Berry, and Cigdem Aslay.
ActUp: Analyzing and consolidating tSNE and UMAP.
In *Proceedings of the Thirty-Second International Joint
Conference on Artificial Intelligence, IJCAI-23*, pages 3651–3658, 8 2023.
- Fan et al. (2021)
Shu-Kai S Fan, Du-Ming Tsai, Chih-Hung Jen, Chia-Yu Hsu, Fei He, and Li-Ting
Juan.
Data visualization of anomaly detection in semiconductor processing
tools.
*IEEE Transactions on Semiconductor Manufacturing*, 35(2):186–197, 2021.
- Fleischer and Islam (2020)
Jason Fleischer and Mohammad Tariqul Islam.
Late breaking abstract-identifying and phenotyping COVID-19
patients using machine learning on chest x-rays.
*European Respiratory Journal*, 2020.
- Foote (1999)
Jonathan Foote.
Visualizing music and audio using self-similarity.
In *Proceedings of the seventh ACM international conference on
Multimedia (Part 1)*, pages 77–80, 1999.
- González-Márquez et al. (2024)
Rita González-Márquez, Luca Schmidt, Benjamin M Schmidt, Philipp
Berens, and Dmitry Kobak.
The landscape of biomedical research.
*Patterns*, 2024.
- Gower (1975)
John C Gower.
Generalized procrustes analysis.
*Psychometrika*, 40:33–51, 1975.
- Gu et al. (2021)
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu,
Tristan Naumann, Jianfeng Gao, and Hoifung Poon.
Domain-specific language model pretraining for biomedical natural
language processing.
*ACM Transactions on Computing for Healthcare (HEALTH)*,
3(1):1–23, 2021.
- Hinton and Roweis (2002)
Geoffrey Hinton and Sam T Roweis.
Stochastic neighbor embedding.
In *Advances in Neural Information Processing Systems*,
volume 15, pages 833–840, 2002.
- Hu et al. (2023)
Tianyang Hu, Zhili Liu, Fengwei Zhou, Wenjia Wang, and Weiran Huang.
Your contrastive learning is secretly doing stochastic neighbor
embedding.
In *The Eleventh International Conference on Learning
Representations*, 2023.
- Huh et al. (2024)
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola.
Position: The platonic representation hypothesis.
In *Proceedings of the 41st International Conference on Machine
Learning*, pages 20617–20642, 2024.
- Islam and Fleischer (2022)
Mohammad Tariqul Islam and Jason W Fleischer.
Manifold-aligned neighbor embedding.
In *ICLR 2022 Workshop on Geometrical and Topological
Representation Learning*, 2022.
- Islam and Fleischer (2024)
Mohammad Tariqul Islam and Jason W Fleischer.
Outlier detection in large radiological datasets using umap.
In *International Workshop on Topology-and Graph-Informed
Imaging Informatics*, pages 111–121. Springer, 2024.
- Jacobs (1988)
Robert A Jacobs.
Increased rates of convergence through learning rate adaptation.
*Neural networks*, 1(4):295–307, 1988.
- Jacomy et al. (2014)
Mathieu Jacomy, Tommaso Venturini, Sebastien Heymann, and Mathieu Bastian.
Forceatlas2, a continuous graph layout algorithm for handy network
visualization designed for the gephi software.
*PloS one*, 9(6):e98679, 2014.
- Kingma and Ba (2015)
Diederik P Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
In *International Conference on Learning Representations*, 2015.
- Kobak and Linderman (2021)
Dmitry Kobak and George C Linderman.
Initialization is critical for preserving global data structure in
both t-sne and umap.
*Nature biotechnology*, 39(2):156–157,
2021.
- Kobak et al. (2019)
Dmitry Kobak, George Linderman, Stefan Steinerberger, Yuval Kluger, and Philipp
Berens.
Heavy-tailed kernels reveal a finer cluster structure in t-sne
visualisations.
In *Joint European Conference on Machine Learning and Knowledge
Discovery in Databases*, pages 124–139. Springer, 2019.
- Kolpakov and Rocke (2024)
Alexander Kolpakov and Aidan Rocke.
The information geometry of umap.
*Le Matematiche*, 79(1):151–164, 2024.
- Kotlov et al. (2024)
Nikita Kotlov, Kirill Shaposhnikov, Cagdas Tazearslan, Madison Chasse, Artur
Baisangurov, Svetlana Podsvirova, Dawn Fernandez, Mary Abdou, Leznath
Kaneunyenye, Kelley Morgan, et al.
Procrustes is a machine-learning approach that removes cross-platform
batch effects from clinical rna sequencing data.
*Communications Biology*, 7(1):392, 2024.
- Kruskal (1964)
Joseph B Kruskal.
Nonmetric multidimensional scaling: a numerical method.
*Psychometrika*, 29(2):115–129, 1964.
- Kury et al. (2025)
Noël Kury, Dmitry Kobak, and Sebastian Damrich.
DREAMS: Preserving both local and global structure in
dimensionality reduction.
*arXiv preprint arXiv:2508.13747*, 2025.
- Lam et al. (2015)
Siu Kwan Lam, Antoine Pitrou, and Stanley Seibert.
Numba: A llvm-based python jit compiler.
In *Proceedings of the Second Workshop on the LLVM Compiler
Infrastructure in HPC*, pages 1–6, 2015.
- Lambert et al. (2022)
Pierre Lambert, Cyril De Bodt, Michel Verleysen, and John A Lee.
Squadmds: a lean stochastic quartet mds improving global structure
preservation in neighbor embedding like t-sne and umap.
*Neurocomputing*, 503:17–27, 2022.
- Linderman and Steinerberger (2019)
George C Linderman and Stefan Steinerberger.
Clustering with t-SNE, provably.
*SIAM Journal on Mathematics of Data Science*, 1(2):313–332, 2019.
- Linderman and Steinerberger (2022)
George C Linderman and Stefan Steinerberger.
Dimensionality reduction via dynamical systems: The case of t-sne.
*SIAM Review*, 64(1):153–178, 2022.
- Maaten and Hinton (2008)
Laurens van der Maaten and Geoffrey Hinton.
Visualizing data using t-sne.
*Journal of machine learning research*, 9(Nov):2579–2605, 2008.
- Macosko et al. (2015)
Evan Z Macosko, Anindita Basu, Rahul Satija, James Nemesh, Karthik Shekhar,
Melissa Goldman, Itay Tirosh, Allison R Bialas, Nolan Kamitaki, Emily M
Martersteck, et al.
Highly parallel genome-wide expression profiling of individual cells
using nanoliter droplets.
*Cell*, 161(5):1202–1214, 2015.
- McInnes et al. (2018)
Leland McInnes, John Healy, and James Melville.
UMAP: Uniform manifold approximation and projection for dimension
reduction.
*arXiv preprint arXiv:1802.03426*, 2018.
- Mikolov et al. (2013)
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.
Distributed representations of words and phrases and their
compositionality.
In *Advances in neural information processing systems*, pages
3111–3119, 2013.
- Narayan et al. (2021)
Ashwin Narayan, Bonnie Berger, and Hyunghoon Cho.
Assessing single-cell transcriptomic variability through
density-preserving data visualization.
*Nature Biotechnology*, 39(6):765–774,
2021.
- Pedregosa et al. (2011)
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel,
M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos,
D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.
Scikit-learn: Machine learning in Python.
*Journal of Machine Learning Research*, 12:2825–2830,
2011.
- Poličar et al. (2024)
Pavlin G. Poličar, Martin Stražar, and Blaž Zupan.
opentsne: A modular python library for t-sne dimensionality reduction
and embedding.
*Journal of Statistical Software*, 109(3):1–30, 2024.
- Ravuri and Lawrence (2024)
Aditya Ravuri and Neil D Lawrence.
Towards one model for classical dimensionality reduction: A
probabilistic perspective on umap and t-sne.
*arXiv preprint arXiv:2405.17412*, 2024.
- Ravuri et al. (2023)
Aditya Ravuri, Francisco Vargas, Vidhi Lalchand, and Neil D Lawrence.
Dimensionality reduction as probabilistic inference.
In *ICML 2023 Workshop on Structured Probabilistic Inference
$\{$$\backslash$&$\}$ Generative Modeling*, 2023.
- Rousseeuw (1987)
Peter J Rousseeuw.
Silhouettes: a graphical aid to the interpretation and validation of
cluster analysis.
*Journal of computational and applied mathematics*, 20:53–65, 1987.
- Roweis and Saul (2000)
Sam T Roweis and Lawrence K Saul.
Nonlinear dimensionality reduction by locally linear embedding.
*science*, 290(5500):2323–2326, 2000.
- Sainburg et al. (2021)
Tim Sainburg, Leland McInnes, and Timothy Q Gentner.
Parametric umap embeddings for representation and semisupervised
learning.
*Neural Computation*, 33(11):2881–2907,
2021.
- Shaham and Steinerberger (2017)
Uri Shaham and Stefan Steinerberger.
Stochastic neighbor embedding separates well-separated clusters.
*arXiv preprint arXiv:1702.02670*, 2017.
- Shi et al. (2023)
Xintong Shi, Wenzhi Cao, and Sebastian Raschka.
Deep neural networks for rank-consistent ordinal regression based on
conditional probabilities.
*Pattern Analysis and Applications*, 26(3):941–955, 2023.
- Tang et al. (2016)
Jian Tang, Jingzhou Liu, Ming Zhang, and Qiaozhu Mei.
Visualizing large-scale and high-dimensional data.
In *Proceedings of the 25th international conference on world
wide web*, pages 287–297, 2016.
- Van der Maaten and Hinton (2008)
Laurens Van der Maaten and Geoffrey Hinton.
Visualizing data using t-sne.
*Journal of machine learning research*, 9(11), 2008.
- Venna and Kaski (2001)
Jarkko Venna and Samuel Kaski.
Neighborhood preservation in nonlinear projection methods: An
experimental study.
In *International Conference on Artificial Neural Networks*,
pages 485–491. Springer, 2001.
- Virtanen et al. (2020)
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy,
David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan
Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod
Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric
Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake
VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen,
E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro,
Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors.
SciPy 1.0: Fundamental Algorithms for Scientific Computing in
Python.
*Nature Methods*, 17:261–272, 2020.
doi: 10.1038/s41592-019-0686-2.
- Wang et al. (2021)
Yingfan Wang, Haiyang Huang, Cynthia Rudin, and Yaron Shaposhnik.
Understanding how dimension reduction tools work: an empirical
approach to deciphering t-SNE, UMAP, TriMAP, and PaCMAP for data
visualization.
*The Journal of Machine Learning Research*, 22(1):9129–9201, 2021.
- Wang et al. (2025)
Yingfan Wang, Yiyang Sun, Haiyang Huang, and Cynthia Rudin.
Dimension reduction with locally adjusted graphs.
In *Thirty-Ninth AAAI Conference on Artificial Intelligence*,
2025.
- Xiao et al. (2017)
Han Xiao, Kashif Rasul, and Roland Vollgraf.
Fashion-MNIST: A novel image dataset for benchmarking machine
learning algorithms.
2017.
- Yang et al. (2022)
Yang Yang, Hongjian Sun, Jialei Gong, Yali Du, and Di Yu.
Interpretable dimensionality reduction by feature preserving manifold
approximation and projection.
*arXiv preprint arXiv:2211.09321*, 2022.
- Zelnik-Manor and Perona (2004)
Lihi Zelnik-Manor and Pietro Perona.
Self-tuning spectral clustering.
*Advances in neural information processing systems*, 17, 2004.
- Zheng et al. (2018)
Jonathan X Zheng, Samraat Pawar, and Dan FM Goodman.
Graph drawing by stochastic gradient descent.
*IEEE transactions on visualization and computer graphics*,
25(9):2738–2748, 2018.

## Appendix

In Appendix [A](https://arxiv.org/html/2503.09101v3#A1), we provide necessary derivations regarding attraction and repulsion shapes of UMAP. Following our discussion in the main text, we explore usage of a constant learning rate again in Appendix [B](https://arxiv.org/html/2503.09101v3#A2). Then, in Appendix [C](https://arxiv.org/html/2503.09101v3#A3), we discuss the implications of improved attraction and repulsion on large language model embeddings. We formally define Procrustes distance and matrix in Appendix [D.1](https://arxiv.org/html/2503.09101v3#A4.SS1). We explore additional datasets for the random initialization experiment in Appendix [E](https://arxiv.org/html/2503.09101v3#A5). We extend the discussion of NEG-$t$-SNE and UMAP in Appendix [F](https://arxiv.org/html/2503.09101v3#A6). After that, in Appendix [G](https://arxiv.org/html/2503.09101v3#A7), we explore alternate dimensionality reduction methods. Then, for completion, we discuss the construction of the high-dimensional graph in these methods in Appendix [H](https://arxiv.org/html/2503.09101v3#A8). In Appendix [I](https://arxiv.org/html/2503.09101v3#A9), we show the detailed result of varying $\lambda_{a}$ and $\lambda_{r}$ (from Fig. [1](https://arxiv.org/html/2503.09101v3#S4.F1)) for UMAP, NEG-$t$-SNE, and PaCMAP on different datasets. We show the effect of mixing and matching the shapes from different algorithms in Appendix [J](https://arxiv.org/html/2503.09101v3#A10). Finally, we provide implementation details in Appendix [K](https://arxiv.org/html/2503.09101v3#A11).

## Appendix A Proofs and Derivations

### A.1 Proof of Proposition 4.1

###### Proof.

We subtract Eq. ([7](https://arxiv.org/html/2503.09101v3#S4.E7)) from Eq. ([8](https://arxiv.org/html/2503.09101v3#S4.E8)) and take a norm:

$$ $\displaystyle||y_{i}^{t+1}-y_{j}^{t+1}||=|1+2\lambda f_{a}|||y_{i}^{t}-y_{j}^{t}||.$ (16) $$

This distance contracts as long as $|1+2\lambda f_{a}|<1$, i.e., provided

$$ $\displaystyle-1<\lambda f_{a}<0,$ (17) $$

∎

###### Proof.

### A.2 Proof of Proposition 4.2

###### Proof.

We subtract Eq. ([9](https://arxiv.org/html/2503.09101v3#S4.E9)) from Eq. ([10](https://arxiv.org/html/2503.09101v3#S4.E10)) and take a norm:

$$ $\displaystyle||y_{i}^{t+1}-y_{j}^{t+1}||=|1+\lambda f_{r}|||y_{i}^{t}-y_{j}^{t}||.$ (18) $$

This distance increases when $|1+\lambda f_{r}|>1$, i.e.,

$$ $\displaystyle\lambda f_{r}<-2\text{~~or~~}f_{r}>0.$ (19) $$

From the definition of $f_{r}$, the latter suffices.
∎

###### Proof.

### A.3 Attraction Shape:

We use a general form of the low-dimensional affinity function, i.e., $q_{ij}=(\gamma+a||y_{i}-y_{j}||_{1}^{2b})^{-1}$, to derive the attraction shape. It reduces to UMAP for $\gamma=1$ and to NEG-$t$-SNE for $\gamma=2$. The attractive force is given by

$$ $\displaystyle\nabla_{y_{i}}\log q_{ij}$ $\displaystyle=-\nabla_{y_{i}}\log{(\gamma+a(||y_{i}-y_{j}||_{2}^{2})^{b})}$ $\displaystyle=-\frac{1}{\gamma+a(||y_{i}-y_{j}||_{2}^{2})^{b}}\nabla_{y_{i}}(\gamma+a(||y_{i}-y_{j}||_{2}^{2})^{b})$ $\displaystyle=-\frac{1}{\gamma+a(||y_{i}-y_{j}||_{2}^{2})^{b}}ab(||y_{i}-y_{j}||_{2}^{2})^{b-1}\nabla_{y_{i}}||y_{i}-y_{j}||_{2}^{2}$ $\displaystyle=-\frac{2ab(||y_{i}-y_{j}||_{2}^{2})^{b-1}}{\gamma+a(||y_{i}-y_{j}||_{2}^{2})^{b}}(y_{i}-y_{j}).$ (20) $$

Defining $\zeta=||y_{i}-y_{j}||_{2}$, the first term gives the attraction shape as:

$$ $\displaystyle f_{a}(\zeta)=-\frac{2ab\zeta^{2(b-1)}}{\gamma+a\zeta^{2b}}$ (21) $$

### A.4 Condition for strictly increasing f a U f_{a}^{U} :

Its behavior with distance can be characterized by computing the derivative of $f_{a}^{U}$:

$$ $\displaystyle\frac{df_{a}^{U}(\zeta)}{d\zeta}=-\frac{2ab\zeta^{2b-3}}{\gamma+a\zeta^{2b}}\left((b-1)-\frac{ab\zeta^{2b}}{\gamma+a\zeta^{2b}}\right).$ (22) $$

Figure: Figure 5: Values of $g(\zeta,b,a)$ for (a) $a$ fixed at $1.576$ and (b) $b$ fixed at $0.89$.
Refer to caption: figures_attr_shape/g_b_a.eps

This leads to a strictly increasing condition ($\frac{df_{a}^{U}}{d\zeta}>0$),

$$ $\displaystyle g(\zeta,b,a)<0,$ (23) $$

where $g(\zeta,b,a)=b-1-\frac{ab\zeta^{2b}}{\gamma+a\zeta^{2b}}$. This inequality is valid as long as $0<b\leq 1$ (using the derivative and asymptotes of $g$). Figure [6](https://arxiv.org/html/2503.09101v3#A1.F6) shows values of $g$ for different $b$ and $a$. As $b$ increases above $1$, the inequality ([23](https://arxiv.org/html/2503.09101v3#A1.E23)) no longer holds.

### A.5 Repulsion Shape:

The repulsive force, using the general form of the low-dimensional affinity, is given by

$$ $\displaystyle\nabla_{y_{i}}\log{(1-q_{ij})}$ $\displaystyle=\nabla_{y_{i}}\log{\left[1-\frac{1}{\gamma+a(||y_{i}-y_{j}||_{2}^{2})^{b}}\right]}$ $\displaystyle=\frac{\gamma+a(||y_{i}-y_{j}||_{2}^{2})^{b}}{(\gamma-1)+a(||y_{i}-y_{j}||_{2}^{2})^{b}}\nabla_{y_{i}}\left[1-\frac{1}{\gamma+a(||y_{i}-y_{j}||_{2}^{2})^{b}}\right]$ $\displaystyle=\frac{1}{\gamma-1+a(||y_{i}-y_{j}||_{2}^{2})^{b}}\frac{ab(||y_{i}-y_{j}||_{2}^{2})^{b-1}}{\gamma+a(||y_{i}-y_{j}||_{2}^{2})^{b}}\nabla_{y_{i}}||y_{i}-y_{j}||_{2}^{2}$ $\displaystyle=\frac{2ab(||y_{i}-y_{j}||_{2}^{2})^{b-1}}{(\gamma-1+a(||y_{i}-y_{j}||_{2}^{2})^{b})(\gamma+a(||y_{i}-y_{j}||_{2}^{2})^{b})}(y_{i}-y_{j}).$ (24) $$

The first term gives the repulsion shape as:

$$ $\displaystyle f_{r}(\zeta=||y_{i}-y_{j}||_{2})=\frac{2ab\zeta^{2(b-1)}}{(\gamma-1+a\zeta^{2b})(\gamma+a\zeta^{2b})}.$ (25) $$

Generally, $f_{r}>0$.

### A.6 Loss functions due to modified attraction and repulsion shape:

The cost function of the attractive term with the modification in Eq. ([13](https://arxiv.org/html/2503.09101v3#S5.E13)) is given by

$$ $\displaystyle-\int(f_{a}^{U}(||y_{i}-y_{j}||_{2})-\beta||y_{i}-y_{j}||_{2})(y_{i}-y_{j})dy_{i}=-\log(q_{ij})+\frac{\beta}{3}||y_{i}-y_{j}||_{2}^{3},$ (26) $$

whereas the repulsive term due to Eq. ([15](https://arxiv.org/html/2503.09101v3#S5.E15)) is given by

$$ $\displaystyle-\int(f_{r}^{U}(||y_{i}-y_{j}||_{2})+\epsilon)(y_{i}-y_{j})dy_{i}=-\log(1-q_{ij})-\frac{\epsilon}{2}||y_{i}-y_{j}||_{2}^{2}.$ (27) $$

In both cases, the additional term acts as a regularizer, simply using norms. However, when we directly add this term to attraction and repulsion shapes, we can easily explain what each term is doing. For the attractive term, it is increasing far-sightedness, whereas for the repulsive term, it adds a constant repulsive coefficient.

## Appendix B More on UMAP’s Learning Rate

As we discussed in Sections [5](https://arxiv.org/html/2503.09101v3#S5) and [6](https://arxiv.org/html/2503.09101v3#S6) of the main text, it is believed that UMAP requires learning rate annealing (Fig [6](https://arxiv.org/html/2503.09101v3#A1.F6)).
To explain this, in Section [5](https://arxiv.org/html/2503.09101v3#S5), we defined the concept of minimum distance for contraction ($\zeta_{-1}$) and established that reducing this value through learning rate annealing results in compact clusters.
Later, in Section [6](https://arxiv.org/html/2503.09101v3#S6), we compared attraction shapes of UMAP (for $a=1.0$ and $b=1.0$) and Neg-$t$-SNE and explained that Neg-$t$-SNE can withstand a constant learning rate of $1.0$ better than UMAP because it’s attraction shape resides within $[-1,0]$ while UMAP’s is within $[-2,0]$.
Following the same logic, we showed that UMAP can withstand a constant learning of $0.5$ by making its attraction shape stay within $[-1,0]$ (Figs. [11](https://arxiv.org/html/2503.09101v3#A6.F11)(f,h)).

However, the embeddings are still better if the learning rate anneals (for a wide range of parameters of $a$ and $b$).
This is because the goal of the algorithm is to eventually reduce $\zeta_{-1}$ to zero and it occurs when the learning rate reduces close to zero.
Otherwise, the embedding becomes diffused (Fig [6](https://arxiv.org/html/2503.09101v3#A1.F6)(a)).
On the other hand, if the learning rate is too low, to begin with, the strength of attraction and repulsion is too low, and thus no clear clusters can form (Fig [6](https://arxiv.org/html/2503.09101v3#A1.F6)(b)).
By analyzing $\zeta_{-1}$ vs $\lambda$ curve (Fig [6](https://arxiv.org/html/2503.09101v3#A1.F6)(c)), we see that a near optimal point is $\lambda=0.1$ where the value of $\zeta_{-1}$ is low with considerable attraction and repulsion strength.
Setting the constant learning rate to $0.1$, we obtain compact clusters with clear boundaries (Fig [6](https://arxiv.org/html/2503.09101v3#A1.F6)(d)).

## Appendix C Large Language Model Embedding

Figure: Figure 7: Visualization of 300K abstracts from the PubMed dataset. (a) Default UMAP ($a=1.576,b=0.89$) shows labels overlapping each other in the mapping, but (b) UMAP with improved repulsion and attraction ($a=1.576,b=0.60$) shows better cluster formation and non-overlapping labels. Both mappings are labeled for easier understanding.
Refer to caption: figures_attr_shape/docu_fig.eps

Increased attraction and repulsion often produce better embeddings.
To show this, we randomly selected 300,000 samples of features using PubMedBERT (Gu et al., 2021) with known labels from the PubMed dataset (González-Márquez et al., 2024).
The default UMAP provides a crowded structure where labels (colors) overlap (Fig. [7](https://arxiv.org/html/2503.09101v3#A3.F7)(a)).
Without labels, one might be perplexed about the embeddings, as explicit clusters are absent.
However, by adjusting $b$ to $0.6$ (as discussed in Section [5.2](https://arxiv.org/html/2503.09101v3#S5.SS2)), we increase repulsion and simultaneously facilitate cluster formation through improved attraction.
The mapping is explorable even without explicit labeling (Fig. [7](https://arxiv.org/html/2503.09101v3#A3.F7)(b)).
The improvement of the embedding can be quantified by the k-NN accuracy ($k=10$).
On a hold-out data of $10,000$ points, k-NN accuracy has increased from $49.2\%$ to $55.48\%$ by reducing $b$ to $0.6$.

## Appendix D Metrics

### D.1 Procrustes Distance and Procrustes Matrix

Figure: Figure 8: Effect of sorting the diagonal of Procrustes matrix on its visualization. (a) Procrustes matrix reproduced from [2](https://arxiv.org/html/2503.09101v3#S5.F2)(f) for the default UMAP attraction shape. The diagonal of the matrix is sorted by $p_{d}$ of a sample embedding with PCA initialization. (b) The same data as in (a), but the diagonal is unsorted (or randomly sorted). (c) Procrustes matrix reproduced from [2](https://arxiv.org/html/2503.09101v3#S5.F2)(g) for modified attraction shape, where the diagonal of the matrix is sorted by $p_{d}$ of a sample embedding with PCA initialization. (d) The same data as in (c), but the diagonal is unsorted.
Refer to caption: figures_attr_shape/MNIST_random_diag.eps

The Procrustes distance (Gower, 1975) measures similarity between two point clouds $\{x\}$ and $\{y\}$ under linear transformations, viz. translation, scaling, and rotation.
Operationally, we hold the former fixed and vary the latter until the two sets are in maximum alignment.
Let $\{y^{\prime}\}$ be the transformation of $\{y\}$ that achieves this objective. Then the Procrustes distance is given by

$$ $\displaystyle p_{d}(\{x\},\{y\})=\sqrt{\sum_{k}(x_{k}-y_{k}^{\prime})}.$ (28) $$

The Procrustes distance is a linear measure that has proven useful in a variety of settings (McInnes et al., 2018; Islam and Fleischer, 2022; Kotlov et al., 2024).

Here, we use the Procrustes distance to measure the consistency of embedding under random initialization. Let $\{x\}_{p}$ be a reference embedding (using PCA initialization in our experiments), and $X_{r}=\{\{x\}_{i}|i=1,2,3,\dots,N\}$ be a set of $N$ embeddings obtained from random initialization. The similarity of the embeddings can be quantified by taking a mean and standard deviation of the strictly lower triangular values of the matrix $P$ (reported as mean$\pm$std in Figs. [2](https://arxiv.org/html/2503.09101v3#S5.F2),  [9](https://arxiv.org/html/2503.09101v3#A5.F9),  and [10](https://arxiv.org/html/2503.09101v3#A5.F10)), with

$$ $\displaystyle\text{mean}=\frac{2}{N(N-1)}\sum_{i,j(i>j)}P_{i,j},\text{~~and~~}\text{std}=\sqrt{\sum_{i,j(i>j)}\frac{2(P_{i,j}-\text{mean})^{2}}{N(N-1)}}$ (29) $$

The indexes of $X_{r}$ can be sorted such that $p_{d}(\{x\}_{i},\{x\}_{p})\leq p_{d}(\{x\}_{i+1},\{x\}_{p})$, so that the diagonal values of the Procrustes matrix are given by

$$ $\displaystyle P_{i,i}=p_{d}(\{x\}_{i},\{x\}_{p})$ (30) $$

and the off-diagonal values are given by

$$ $\displaystyle P_{i,j}=p_{d}(\{x\}_{i},\{x\}_{j}).$ (31) $$

Numerically, the sorting of the diagonal adds little value. Visually, however, the ordering reveals the underlying self-similarity of the embeddings. For example, in Fig. [8](https://arxiv.org/html/2503.09101v3#A4.F8)(a), the sorted diagonal shows that similar embeddings clump in the upper left region of the matrix.
When we use a modified attraction shape, the number of points that are similar to each other increases (as shown by the larger blue region in Fig. [8](https://arxiv.org/html/2503.09101v3#A4.F8)(c)), indicating the presence of a metastable point in the embedding algorithm.
On the other hand, when the diagonal is unsorted, this region disappears, and any sense of similar embeddings is lost (Fig. [8](https://arxiv.org/html/2503.09101v3#A4.F8)(b,d)).

### D.2 Trustworthiness

The trustworthiness metric (Venna and Kaski, 2001) quantifies how well local neighborhoods are preserved after dimensionality reduction:

$$ $\displaystyle T=1-\frac{2}{nk(2n-3k-1)}\sum_{i=1}^{n}\sum_{y_{j}\in\text{KNN}(y_{i},k)}\max(0,r(i,j)-k)$ (32) $$

where $KNN(y_{i},k)$ is the $k$-NN graph in the embedding space and $r(i,j)$ is the rank of $x_{j}$ in the high-dimensional $k$-NN graph. In practice, $k$ is often set to 5, assessing preservation of each point’s five nearest neighbors. For computational efficiency, when we report trustworthiness, we randomly sampled 10,000 indices. When comparing different embeddings, we used the same indices.

### D.3 Silhouette Score

While the silhouette score (Rousseeuw, 1987) aims to assess clustering algorithms, we use it to evaluate label separation in the embeddings, i.e, how well the ground truth labels have been clustered. The idea is that the embedding algorithms naturally produce clusters and should separate the labels as much as possible. For a point $y_{i}$ in a point cloud $\{y\}$, let $a_{i}$ be the mean distance from $y_{i}$ to other points in its own label, and let $b_{i}$ be the minimal mean distance from $y_{i}$ to points in any other label. The pointwise silhouette is thus given by

$$ $\displaystyle s_{i}=\frac{b_{i}-a_{i}}{\max{\{a_{i},b_{i}\}}}$ (33) $$

This value lies within $[-1,1]$. A value close to $1$ means that $y_{i}$ fits within its own label cluster, near 0 suggests a boundary point, and close to $-1$ indicates failed label separation. The overall silhouette score is

$$ $\displaystyle\mathrm{SIL}=\frac{1}{N}\sum s_{i}.$ (34) $$

We computed the silhouette score for the whole embedding (no random sampling) using Euclidean distances.

## Appendix E Effect of Random Initialization in Additional Datasets

In the main text (Section [5.1](https://arxiv.org/html/2503.09101v3#S5.SS1)), we showed results only on the MNIST dataset. Here we perform the same experiment on the Fashion-MNIST (FMNIST) (Xiao et al., 2017) and single-cell transcriptomes (Macosko et al., 2015) data (Fig. [9](https://arxiv.org/html/2503.09101v3#A5.F9) and [10](https://arxiv.org/html/2503.09101v3#A5.F10), respectively).
The main conclusion remains unchanged: modified and composite attraction shapes, such as those that increase attraction at large distances, significantly improve the consistency of reconstruction.

Figure: Figure 9: Effect of random UMAP initialization on different attraction shapes on FMNIST data. (a) Mapping using PCA as a standard. (b-d) Four mappings with the lowest Procrustes distance ($p_{d}$) from the embedding in (a) for (b) default, (c) modified, and (d) composite attraction shapes. (e) Default UMAP and modified attraction shapes. (f-h) Procrustes matrix obtained from 100 runs of (f) default ($0.72\pm 0.15$), (g) modified ($0.38\pm 0.19$), and (h) composite ($0.43\pm 0.18$) attraction shapes.
Refer to caption: figures_attr_shape/fig_random_fmnist.eps

Figure: Figure 10: Effect of random UMAP initialization on different attraction shapes on single-cell transcriptomes data. (a) Mapping using PCA as a standard. (b-d) Four mappings with the lowest Procrustes distance ($p_{d}$) from the embedding in (a) for (b) default, (c) modified, and (d) composite attraction shapes. (e) Default UMAP and modified attraction shapes. (f-h) Procrustes matrix obtained from 100 runs of (f) default ($0.91\pm 0.06$), (g) modified ($0.61\pm 0.13$), and (h) composite ($0.64\pm 0.15$) attraction shapes.
Refer to caption: figures_attr_shape/fig_random_macosko.eps

## Appendix F More on NEG- t t -SNE

### F.1 Comparison to UMAP

Figure: Figure 11: Sensitivity of UMAP and NEG-$t$-SNE to learning rate on the MNIST dataset. (a) Attraction and (b) repulsion shapes for UMAP ($a=1$, $b=1$) and NEG-$t$-SNE. (c,d) UMAP is very sensitive to the learning rate $\lambda$, as $f_{a}^{U}<-1$ as the separation distance $\zeta$ decreases. Thus, without annealing, the clusters become fuzzy. (e,f) NEG-$t$-SNE is less sensitive to $\lambda$ as $f_{a}^{N}\in[-1,0]$ always, and the clusters are thus less fuzzy even when not annealed. (g,h) Confining $f_{a}^{U}$ to $[-1,0]$ by setting $\lambda=0.5$ shows less sensitivity to $\lambda$.
Refer to caption: figures_attr_shape/neg_fig.eps

Figure [11](https://arxiv.org/html/2503.09101v3#A6.F11) shows the shapes of UMAP and NEG-$t$-SNE, along with various MNIST embeddings. When the learning rate is annealed, both UMAP and NEG-$t$-SNE show similar output (Figs. [11](https://arxiv.org/html/2503.09101v3#A6.F11)(c,e)). However, when the learning rate is a constant value of $1$, the UMAP shows a fuzzy structure, while NEG-$t$-SNE shows a structure with much cleaner boundaries (Fig. [11](https://arxiv.org/html/2503.09101v3#A6.F11)(d,f)).
The discussion in Section [6](https://arxiv.org/html/2503.09101v3#S6) suggests that constraining $f_{a}^{U}$ within $[-1,0]$ can potentially result in less fuzzy clusters for fixed $\lambda$. We have seen this previously in Fig. [1](https://arxiv.org/html/2503.09101v3#S4.F1)(h,i) as well, where UMAP provided better embedding and clustering when the learning rate for attraction ($\lambda_{a}$) was $\lesssim 0.5$. A straightforward way to achieve this is to initialize $\lambda$ to $0.5$, which satisfies Proposition [4.1](https://arxiv.org/html/2503.09101v3#S4.Thmtheorem1) for all $\zeta$.
The resulting embeddings, shown in Figs. [11](https://arxiv.org/html/2503.09101v3#A6.F11) (g) and (h), confirm that clusters are similar to those of NEG-$t$-SNE’s, and for a constant $\lambda=0.5$, the clusters are less fuzzy than before as predicted with sharper boundaries (Fig. [11](https://arxiv.org/html/2503.09101v3#A6.F11)(d,h)). There are still a few points outside the clusters due to the characteristics of UMAP’s repulsion shape, which NEG-$t$-SNE solves.

Next, we can introduce the parameters $a$ and $b$ into NEG-$t$-SNE (essentially the formulation of Parametric UMAP; see Proposition [6.1](https://arxiv.org/html/2503.09101v3#S6.Thmtheorem1)). The affinity function becomes $q_{ij}^{N}=1/(2+a\zeta^{2b})$, and the attraction and repulsion shapes become $f_{a}^{N}=-\frac{2ab\zeta^{2(b-1)}}{2+a\zeta^{2b}}$, and
$f_{r}^{N}=\frac{2ab\zeta^{2(b-1)}}{(1+a\zeta^{2b})(2+a\zeta^{2b})}$,
respectively. For $0<b<1$, both shapes become unbounded as $\zeta\to 0$. Thus, NEG-$t$-SNE will face similar numerical challenges to UMAP if $a$ and $b$ vary, and corresponding limitations carry over. One notable distinction is that, compared to UMAP, the attraction shape attains a lower minimum distance ($\zeta_{-1}$) for the attraction. While this may enhance cluster formation, it approaches zero faster (increased near-sightedness as distance increases), potentially diminishing its effectiveness for attraction over longer distances.

### F.2 Comparison to Parametric UMAP

Parametric UMAP was initially trained with the original UMAP objective (Sainburg et al., 2021), but later work adopted a numerically stable, log-sigmoid–based modified cross-entropy loss (Shi et al., 2023). This modification makes Parametric UMAP and Neg-t-SNE Damrich et al. (2023) equivalent (Proposition [6.1](https://arxiv.org/html/2503.09101v3#S6.Thmtheorem1)). We show the equivalence below.

### Proof of Proposition 6.1

###### Proof.

The kernel function under this modification becomes

$$ $\displaystyle q_{ij}^{P}=-\log{(1+a||y_{i}-y_{j}||_{2}^{2b})}.$ (35) $$

The attractive term is

$$ $\displaystyle-\text{logsigmoid}(q_{ij}^{P})$ $\displaystyle=-\log\left(\frac{1}{1+\exp(-q_{ij}^{P})}\right)$ (36) $\displaystyle=-\log\left(\frac{1}{2+a||y_{i}-y_{j}||_{2}^{2b}}\right)$ (37) $\displaystyle=-\log(q_{ij}^{N}).$ (38) $$

And the repulsive term is

$$ $\displaystyle\text{logsidmoid}(q_{ij}^{P})-q_{ij}^{P}$ $\displaystyle=\log\left(\frac{1}{2+a||y_{i}-y_{j}||_{2}^{2b}}\right)+\log\left(1+a||y_{i}-y_{j}||_{2}^{2b}\right)$ (39) $\displaystyle=\log\left(\frac{1+a||y_{i}-y_{j}||_{2}^{2b}}{2+a||y_{i}-y_{j}||_{2}^{2b}}\right)$ (40) $\displaystyle=\log\left(1-\frac{1}{2+a||y_{i}-y_{j}||_{2}^{2b}}\right)$ (41) $\displaystyle=\log(1-q_{ij}^{N}).$ (42) $$

Both these are NEG-t-SNE with explicit parameters $a$ and $b$ (while in Neg-$t$-SNE these are set to $1$).
∎

###### Proof.

## Appendix G Alternate Dimensionality Reduction Algorithms

The alternative algorithms we consider use the same kernel function as UMAP (with $a=1$ and $b=1$ in their low-dimensional weight):

$$ $\displaystyle q_{ij}=\frac{1}{1+||y_{i}-y_{j}||_{2}^{2}}.$ (43) $$

In this section, we first discuss the TriMap (Amid and Warmuth, 2019) algorithm. Even though, this algorithm relies on triplets (and not pairwise interactions), this works as a primer for analyzing attraction and repulsion that are a bit more involved than UMAP. This discussion is followed by Pairwise Controlled Manifold Approximation (PaCMAP) (Wang et al., 2021) and it’s extension Pairwise Controlled Manifold Approximation with Local Adjusted Graph (LocalMAP) (Wang et al., 2025) that modify TriMAP’s loss function that works for pairwise interactions. We then tackle $t$-SNE (Van der Maaten and Hinton, 2008). Then, we provide a short note on SNE (Hinton and Roweis, 2002) that uses an alternate kernel function and finally end the section by briefly discussing multidimensional scaling (Borg and Groenen, 2007).

### G.1 TriMap

Figure: Figure 12: (a) Attraction shapes of TriMap for different $\zeta_{2}$ and (b) repulsion shapes of TriMap for different $\zeta_{1}$.
Refer to caption: figures_attr_shape/atr_rep_trimap.eps

TriMap (Amid and Warmuth, 2019) optimizes low-dimensional using a triplet loss

$$ $\displaystyle\mathcal{L}^{T}=\sum_{(i,j,k)}w_{ijk}\frac{1}{1+\frac{q_{ij}}{q_{ik}}},$ (44) $$

where $w_{ijk}$ is the weight of the triplet $(y_{i},y_{j},y_{k})$, $y_{j}$ is in the $k$-nearest neighbor set of $y_{i}$ in the high dimension, and $y_{k}$ is a far-away point. When minimized, we expect that $y_{i}$ and $y_{j}$ attract each other, while $y_{i}$ and $y_{k}$ repel each other. The update equations are

$$ $\displaystyle y_{i}^{t+1}$ $\displaystyle=y_{i}^{t}+\lambda f_{a}^{T}(\zeta_{1}^{t},\zeta_{2}^{t})(y_{i}^{t}-y_{j}^{t})+\lambda f_{r}^{T}(\zeta_{1},\zeta_{2})(y_{i}^{t}-y_{k}^{t}),$ (45) $\displaystyle y_{j}^{t+1}$ $\displaystyle=y_{j}^{t}-\lambda f_{a}^{T}(\zeta_{1}^{t},\zeta_{2}^{t})(y_{i}^{t}-y_{j}^{t}),$ (46) $\displaystyle y_{k}^{t+1}$ $\displaystyle=y_{k}^{t}-\lambda f_{r}^{t}(\zeta_{1}^{t},\zeta_{2}^{t})(y_{i}^{t}-y_{k}^{t}),$ (47) $$

where $\zeta_{1}=||y_{i}-y_{j}||_{2}$ is the distance between nearest neighbors, $\zeta_{2}=||y_{i}-y_{k}||_{2}$ is the distance from the faraway point, and $f_{a}^{T}$ and $f_{r}^{T}$ are attraction and repulsion shapes of TriMap, respectively. Unlike UMAP, the attractive and repulsive components are non-separable and the shapes depend on two distance measures (making them 2D). The functional form of the attraction shape is

$$ $\displaystyle f_{a}^{T}(\zeta_{1},\zeta_{2})=-\frac{2(1+\zeta_{2}^{2})}{(2+\zeta_{1}^{2}+\zeta_{2}^{2})^{2}},$ (48) $$

and the repulsion shape is

$$ $\displaystyle f_{r}^{T}(\zeta_{1},\zeta_{2})=\frac{2(1+\zeta_{1}^{2})}{(2+\zeta_{1}^{2}+\zeta_{2}^{2})^{2}}.$ (49) $$

The attraction and repulsion shapes (Fig. [13](https://arxiv.org/html/2503.09101v3#A7.F13)) of TriMap shows similar trends of that of UMAP. However, the minium value of repulsion shape is $-0.5$; thus, unlike UMAP there is no position flipping in TriMAP due to attraction alone. However, since Eqs. ([45](https://arxiv.org/html/2503.09101v3#A7.E45)-[47](https://arxiv.org/html/2503.09101v3#A7.E47)) are not decoupled between attractive and repulsive terms, Propositions [4.1](https://arxiv.org/html/2503.09101v3#S4.Thmtheorem1) and [4.2](https://arxiv.org/html/2503.09101v3#S4.Thmtheorem2) do not apply. Focusing on attraction first, we show

###### Proposition G.1 .

Update equations ([45](https://arxiv.org/html/2503.09101v3#A7.E45))-([47](https://arxiv.org/html/2503.09101v3#A7.E47)) provide a contraction if

$$ $\displaystyle h_{a}^{T}(\zeta_{1},\zeta_{2},\theta,\lambda)<1,$ (50) $$

where $h_{a}^{T}(\zeta_{1},\zeta_{2},\theta,\lambda)=(1+2\lambda f_{a}^{T})^{2}+2(1+2\lambda f_{a}^{T})\lambda f_{r}^{t}\frac{\zeta_{2}}{\zeta_{1}}\cos\theta+(\lambda f_{r}^{T})^{2}\frac{\zeta_{2}^{2}}{\zeta_{1}^{2}}$ and $\theta$ is the angle between the vectors $(y_{i}-y_{j})$ and $(y_{i}-y_{k})$, i.e., $cos\theta=\frac{(y_{i}^{t}-y_{j}^{t})^{T}(y_{i}^{t}-y_{k}^{t})}{||y_{i}^{t}-y_{j}^{t}||_{2}||y_{i}^{t}-y_{k}^{t}||_{2}}$.

###### Proof.

We require

$$ $\displaystyle||y_{i}^{t+1}-y_{j}^{t+1}||_{2}^{2}<||y_{i}^{t}-y_{j}^{t}||_{2}^{2}.$ (51) $$

From Eq. ([45](https://arxiv.org/html/2503.09101v3#A7.E45)) and ([46](https://arxiv.org/html/2503.09101v3#A7.E46)): $y_{i}^{t+1}-y_{j}^{t+1}=(1+2\lambda f_{a}^{T})(y_{i}^{t}-y_{j}^{t})+\lambda f_{r}^{T}(y_{i}^{t}-y_{k}^{t})$. Putting this value in Eq. ([51](https://arxiv.org/html/2503.09101v3#A7.E51)), we obtain the desired inequality.
∎

Inequality ([50](https://arxiv.org/html/2503.09101v3#A7.E50)) depends on $\zeta_{1}$, $\zeta_{2}$, $\cos\theta$ and $\lambda$. In particular, the value of $\zeta_{1}=||y_{i}-y_{j}||_{2}$, where we want a contraction, is coupled with additional variables. Figure [13](https://arxiv.org/html/2503.09101v3#A7.F13)(a) shows the attraction behavior for various values of $\zeta_{2}$, while $\cos\theta=1$ and $\lambda=1$. The values below the dotted line indicate attraction (and thus contraction of distance $\zeta_{1}$), whereas the values above indicate repulsion (and therefore expansion of distance $\zeta_{1}$). The value where the dotted line and $h_{a}^{T}$ meet gives the minimum distance for contraction ($\zeta_{1}^{(-1)}$) [analogous to $\zeta_{-1}$ of UMAP], which we define as

$$ $\displaystyle\zeta_{1}^{(-1)}(\zeta_{2},\theta,\lambda)$ $\displaystyle=\arg\min_{\zeta_{1}}|h_{a}^{T}(\zeta_{1},\zeta_{2},\theta,\lambda)-1|,$ (52) $\displaystyle\text{s.t.~~}\zeta_{1}\geq 0.$ (53) $$

$\zeta_{1}^{(-1)}$ has a finite value and is $>0$ for most cases (Fig. [13](https://arxiv.org/html/2503.09101v3#A7.F13)(b,c)). As a result, TriMap can show behavior similar to UMAP and thus require learning rate annealing or a similar approach (in the TriMap implementation, the authors use the delta-bar-delta (Jacobs, 1988) method under appropriate initialization).

###### Proposition G.2 .

Update equations ([45](https://arxiv.org/html/2503.09101v3#A7.E45))-([47](https://arxiv.org/html/2503.09101v3#A7.E47)) provide expansion if

$$ $\displaystyle h_{r}^{T}(\zeta_{1},\zeta_{2},\theta,\lambda)>1,$ (54) $$

where $h_{r}^{T}(\zeta_{1},\zeta_{2},\theta,\lambda)=(1+2\lambda f_{r}^{T})^{2}+2\lambda f_{a}^{T}(1+\lambda f_{r}^{T})\frac{\zeta_{1}}{\zeta_{2}}\cos\theta+(\lambda f_{a}^{T})^{2}\frac{\zeta_{1}^{2}}{\zeta_{2}^{2}}$.

###### Proof.

We require

$$ $\displaystyle||y_{i}^{t+1}-y_{k}^{t+1}||_{2}^{2}>||y_{j}{t}-y_{k}^{t}||_{2}^{2}.$ (55) $$

From Eq. ([45](https://arxiv.org/html/2503.09101v3#A7.E45)) and ([47](https://arxiv.org/html/2503.09101v3#A7.E47)): $y_{j}^{t+1}-y_{k}^{t+1}=(1+2\lambda f_{r}^{T})(y_{i}^{t}-y_{k}^{t})+\lambda f_{a}^{T}(y_{i}^{t}-y_{j}^{t})$. Putting this value in Eq. ([55](https://arxiv.org/html/2503.09101v3#A7.E55)) we obtain inequality ([54](https://arxiv.org/html/2503.09101v3#A7.E54)).
∎

Inequality ([54](https://arxiv.org/html/2503.09101v3#A7.E54)) also depends on the set $\zeta_{1}$, $\zeta_{2}$, $\cos\theta$ and $\lambda$. Here, we are interested in the expansion of $\zeta_{2}=||y_{i}-y_{k}||_{2}$. Figures [13](https://arxiv.org/html/2503.09101v3#A7.F13)(d-f) show the repulsion behavior by varying the other quantities. Any values above the dotted line indicate repulsion (and thus expansion of distance), while the values below indicate attraction. The striking difference compared to UMAP is that repulsion in TriMap can cause contraction instead of expansion. Since this anomaly occurs for small distances, it can be avoided by an appropriate initialization and choice of triplets.

###### Proposition G.1 .

###### Proof.

###### Proposition G.2 .

###### Proof.

### G.2 Pairwise Controlled Manifold Approximation (PaCMAP)

Figure: Figure 14: Attraction and repulsion shapes of PaCMAP. (a,b) Attraction shapes for (a) nearest-neighbor and (b) mid-near points (note that the values are on the order of $10^{-4}$). (c) Repulsion shape for farther pairs. $\lambda=1$ for all figures.
Refer to caption: figures_attr_shape/atr_rep_pacmap.eps

Figure: Figure 15: PaCMAP behavior for different conditions. (a) PaCMAP of MNIST data with PCA initialization. (b,c) Four samples that best match with (a) by (b) original PaCMAP and (c) PaCMAP with modified attraction shape, $f_{a}^{M}(\zeta)=f_{a}^{P}(\zeta)-0.001\zeta$, when randomly initialized. (d) PaCMAP of MNIST with a modified repulsion shape $f_{r}^{M}=f_{r}^{P}+0.00005$. (e,f) Procrustes matrix for (e) original PaCMAP ($0.51\pm 0.22$) and (f) PaCMAP with modified attraction shape ($0.43\pm 0.22$). Similar to UMAP, increased attraction at farther distances show improved consistency, while increased repulsion shows smaller clusters and larger inter-cluster distances.
Refer to caption: figures_attr_shape/pacmap_fig_2.eps

PaCMAP (Wang et al., 2021) optimizes low-dimensional embedding at different scales. The loss function is

$$ $\displaystyle\mathcal{L}^{P}=w_{NB}\sum_{(i,j)\in NN}\frac{1}{1+10q_{ij}}+w_{MN}\sum_{(i,k)\in MN}\frac{1}{1+10000q_{ik}}+w_{FP}\sum_{(i,l)\in FP}\frac{1}{1+\frac{1}{q_{il}}},$ (56) $$

where $w_{NB}$, $w_{MN}$, and $w_{FP}$ are weights of the nearest neighbor (NN) pairs, mid-near (MN) pairs, and further pairs (FP), respectively (details in Appendix [H](https://arxiv.org/html/2503.09101v3#A8)). The first two terms provide attraction, whereas the last term provides repulsion. A closer look at the loss function reveals that the function is a modified form of TriMap’s triplet loss. For the attractive terms, it replaces TriMap’s affinity for distant points with constant terms ($1/10$ for nearest neighbors, $1/10000$ for mid-near pairs, and $1$ for farther points). This loss function is thus separable into three terms and decouples the update equations. The update equations of the nearest neighbor term are

$$ $\displaystyle y_{i}^{t+1}$ $\displaystyle=y_{i}^{t}+\lambda f_{a}^{P}(\zeta_{1}^{t})(y_{i}^{t}-y_{j}^{t}),$ (57) $\displaystyle y_{j}^{y+1}$ $\displaystyle=y_{j}^{t}-\lambda f_{a}^{P}(\zeta_{1}^{t})(y_{i}^{t}-y_{j}^{t}),$ (58) $$

of the mid-near pairs are

$$ $\displaystyle y_{i}^{t+1}$ $\displaystyle=y_{i}^{t}+\lambda f_{m}^{P}(\zeta_{2}^{t})(y_{i}^{t}-y_{k}^{t}),$ (59) $\displaystyle y_{k}^{y+1}$ $\displaystyle=y_{k}^{t}-\lambda f_{m}^{P}(\zeta_{2}^{t})(y_{i}^{t}-y_{k}^{t}),$ (60) $$

and of the farthest pairs are

$$ $\displaystyle y_{i}^{t+1}$ $\displaystyle=y_{i}^{t}+\lambda f_{r}^{P}(\zeta_{3}^{t})(y_{i}^{t}-y_{l}^{t}),$ (61) $\displaystyle y_{l}^{t+1}$ $\displaystyle=y_{l}^{t}-\lambda f_{r}^{P}(\zeta_{3}^{t})(y_{i}^{t}-y_{l}^{t}),$ (62) $$

where $\zeta_{1}=||y_{i}-y_{j}||_{2}$, $\zeta_{2}=||y_{i}-y_{k}||_{2}$, $\zeta_{3}=||y_{i}-y_{l}||_{2}$ are distances, $f_{a}^{P}$ and $f_{m}^{P}$ are attraction shapes for nearest neighbors and mid-near pairs, respectively, and $f_{r}^{P}$ is the repulsion shape for the farthest pairs. Correspondingly, the functional forms of the shapes are

$$ $\displaystyle f_{a}^{P}(\zeta)$ $\displaystyle=-\frac{20}{(11+\zeta^{2})^{2}},$ (63) $\displaystyle f_{m}^{P}(\zeta)$ $\displaystyle=-\frac{20000}{(10001+\zeta^{2})^{2}},$ (64) $\displaystyle f_{r}^{P}(\zeta)$ $\displaystyle=\frac{2}{(2+\zeta^{2})^{2}}.$ (65) $$

$f_{a}^{P}$ and $f_{m}^{P}$ follow Proposition [4.1](https://arxiv.org/html/2503.09101v3#S4.Thmtheorem1), and $f_{r}$ follows Proposition [4.2](https://arxiv.org/html/2503.09101v3#S4.Thmtheorem2) (Fig. [14](https://arxiv.org/html/2503.09101v3#A7.F14)).
The attraction is quite low compared to UMAP, but it is good enough for a wide range of learning rates (modulated by the Adam algorithm (Kingma and Ba, 2015)); with $w_{NB}=3$ the maximum attraction is always below $0.5$ preventing any flips during attractin update.
Typically, PaCMAP initializes the embedding within a small sphere in the low dimension (e.g., in (Wang et al., 2021), the initialization is often on the order of $~10^{-3}$) and relies on repulsion to separate the individual clusters.
Overall, it mostly recovers UMAP’s clustering properties (especially for MNIST) with improved ordering. The consistency under random initialization is better than UMAP (Fig. [15](https://arxiv.org/html/2503.09101v3#A7.F15)(b,e)) which can be improved further using a modified attraction (Fig. [15](https://arxiv.org/html/2503.09101v3#A7.F15)(c,f)).
Increasing repulsion by adding a small value to the repulsion shape increases compactness of the embedding (by increasing inter-cluster distance).

### G.3 Pairwise Controlled Manifold Approximation with Local Adjusted Graph (LocalMAP)

Figure: Figure 16: Behavior of LocalMAP on MNIST data. (a) Default embedding. (b) Attraction-repulsion shape of nearest neighbors. A value below (above) the dotted line indicates attraction (repulsion). Transition from attraction to repulsion occurs as $\zeta$ increases and crosses $\sqrt{C-1}$. Following implementation of LocalMAP, we used $C=10$. (c) When $K$ is large ($=20$), repulsion dominates and clusters become compact. (d) When $K$ is small ($=7$), attraction dominates and clusters break up.
Refer to caption: figures_attr_shape/localmap_fig.eps

LocalMAP (Wang et al., 2025) is an iteration of the PaCMAP algorithm. One of the defining features of LocalMAP is the separation of all $10$ clusters of the MNIST data (Fig. [16](https://arxiv.org/html/2503.09101v3#A7.F16)(a)), with behavior similar to the ones in Figs. [3](https://arxiv.org/html/2503.09101v3#S5.F3)(g,h,j,k) and Figs. [4](https://arxiv.org/html/2503.09101v3#S5.F4)(a,b). Here, we explore the interplay of attractive and repulsive forces on the compactness and connectedness of clusters.

LocalMAP performs PaCMAP and then does additional optimization on the attraction to decouple some clusters. To this end, it minimizes the following loss function

$$ $\displaystyle\mathcal{L}^{L}=\sum_{(i,j)\in NN}\frac{K}{\frac{1}{\sqrt{q_{ij}}}+C\sqrt{q_{ij}}}+\sum_{(i,l)\in FP}\frac{1}{1+\frac{1}{q_{il}}}.$ (66) $$

The first term amalgamates the attractive and repulsive nature of the triplet loss function that works on the same pair.
In one regime, this function causes attraction, while in the other, it causes repulsion.
The second term is identical to the ones in PaCMAP; the only difference is that the algorithm resamples further pairs every few iterations.
Thus, we analyze only the first term involving nearest neighbors. The update equations are

$$ $\displaystyle y_{i}^{t+1}$ $\displaystyle=y_{i}^{t}+\lambda f_{ar}^{L}(\zeta_{1}^{t})(y_{i}^{t}-y_{j}^{t}),$ (67) $\displaystyle y_{j}^{y+1}$ $\displaystyle=y_{j}^{t}-\lambda f_{ar}^{L}(\zeta_{1}^{t})(y_{i}^{t}-y_{j}^{t}),$ (68) $$

where $f_{ar}^{L}$ is the attraction-repulsion shape given by

$$ $\displaystyle f_{ar}^{L}(\zeta)=-\frac{K(C-1-\zeta^{2})}{2\sqrt{1+\zeta^{2}}(1+C+\zeta^{2})^{2}}.$ (69) $$

The update provides contraction as long as $\zeta^{2}<C-1$ and $-1<\lambda f_{ar}^{L}<0$ (from Proposition [4.1](https://arxiv.org/html/2503.09101v3#S4.Thmtheorem1)).
When $\zeta^{2}>C-1$, $f_{ar}>0$, and by Proposition [4.2](https://arxiv.org/html/2503.09101v3#S4.Thmtheorem2) the update equations provide expansion.
The values of $K$ and $C$ determine whether attraction or repulsion dominates the dynamics.
In LocalMAP implementation, both $C$ and $K$ are set to $10$ (Fig. [16](https://arxiv.org/html/2503.09101v3#A7.F16)(b), and the strength of attraction is higher than PaCMAP (Fig. [14](https://arxiv.org/html/2503.09101v3#A7.F14)(a)).
As a result, when $\zeta>3$, the nearest neighbors face repulsion, causing pairs bridging two clusters to separate.
The value of $K$, working as a scaling parameter for the forces, regulates this separation.

Using Proposition [4.1](https://arxiv.org/html/2503.09101v3#S4.Thmtheorem1), $\lambda f_{ar}^{L}(0)\geq-\frac{1}{2}$ gives the maximum values of $K$, and the maximum attraction possible by $f_{ar}^{L}$, without flipping the placements of the pairs. (We would want to avoid flipping the pairs at the LocalMAP optimization to preserve the ordering from PaCMAP; otherwise, it may inhibit cluster separation.)
This simplifies ([69](https://arxiv.org/html/2503.09101v3#A7.E69)) to $K\leq\frac{\lambda(1+C)^{2}}{C-1}$, which for $C=10$ and $\lambda=1$ gives $K\lesssim 13.44$.
Moreover, at a higher value of $K\gtrsim 13.44$, the repulsive forces dominate, and the clusters become more compact, but objective of LocalMAP fails ( the bridge between clusters persists in Fig. [16](https://arxiv.org/html/2503.09101v3#A7.F16)(c) for $K=20$).
On the other hand, as $K$ decreases, attractive forces dominate (because repulsive forces are too low), and the embedding shows the breaking up of existing clusters (Fig. [16](https://arxiv.org/html/2503.09101v3#A7.F16)(d) for $K=7$, which mimics the one in Fig. [4](https://arxiv.org/html/2503.09101v3#S5.F4)(b)).

### G.4 t t -distributed Stochastic Neighbor Embedding ( t t -SNE)

$t$-SNE (Van der Maaten and Hinton, 2008) optimizes pairwise distances. The loss function is

$$ $\displaystyle\mathcal{L}=-\sum_{i,j}w_{i,j}\log{\left(\frac{q_{i,j}}{\sum_{k\neq l}q_{k,l}}\right)},$ (70) $$

where $w_{i,j}$ is the weight of the pair. The original implementation of $t$-SNE considers all the pairs (not just nearest neighbors). This loss function decomposes into attraction and repulsion forces by

$$ $\displaystyle\mathcal{L}=\sum_{i,j}\left[-w_{i,j}\log{\left(q_{i,j}\right)}+w_{i,j}\log{\left(\sum_{k\neq l}q_{k,l}\right)}\right].$ (71) $$

As previously, the first term provides the attractive forces and the second term provides the repulsive forces.
While the attractive term is identical to that of UMAP (with $a=1$ and $b=1$) and is easy to compute, the repulsive term is coupled among every pair and thus, is very costly.
Using the same principles we applied for UMAP, we can write the update equations of t-SNE.
Since the original $t$-SNE didn’t rely on the nearest neighbor graph, the weight $w_{i,j}$, computed for all the pairs, is important in the update equations.
The attractive update equations are

$$ $\displaystyle y_{i}^{t+1}=y_{i}^{t}+\lambda w_{i,j}f_{a}^{t-SNE}(\zeta_{i,j}^{t})(y_{i}^{t}-y_{j}^{t}),$ (72) $\displaystyle y_{j}^{t+1}=y_{i}^{t}-\lambda w_{i,j}f_{a}^{t-SNE}(\zeta_{i,j}^{t})(y_{i}^{t}-y_{j}^{t}),$ (73) $$

where $f_{a}^{t-SNE}$ is the attraction shape of $t$-SNE and $\zeta_{i,j}=||y_{i}-y_{j}||_{2}$. The update equation for the repulsive parts are

$$ $\displaystyle y_{i}^{t+1}$ $\displaystyle=y_{i}^{t}+\lambda\frac{w_{i,j}}{Z}\sum_{k}f_{r}^{t-SNE}(\zeta_{i,k}^{t})(y_{i}^{t}-y_{k}^{t}),$ (74) $\displaystyle y_{j}^{t+1}$ $\displaystyle=y_{j}^{t}-\lambda\frac{w_{i,j}}{Z}\sum_{l}f_{r}^{t-SNE}(\zeta_{l,j}^{t})(y_{l}^{t}-y_{j}^{t}),\forall j,j\neq i,$ (75) $$

where $f_{r}^{t-SNE}$ is the repulsion shape of $t$-SNE and $Z=\sum_{k\neq l}q_{k,l}$. The functional forms of these shapes are

$$ $\displaystyle f_{a}^{t-SNE}(\zeta)$ $\displaystyle=-\frac{2}{1+\zeta^{2}},\text{and}$ (76) $\displaystyle f_{r}^{t-SNE}(\zeta)$ $\displaystyle=\frac{2}{(1+\zeta^{2})^{2}}.$ (77) $$

From the attractive update Eqs. ([72](https://arxiv.org/html/2503.09101v3#A7.E72))-([73](https://arxiv.org/html/2503.09101v3#A7.E73)), $f_{a}^{t-SNE}$ follows Proposition [4.1](https://arxiv.org/html/2503.09101v3#S4.Thmtheorem1) (with $0<\lambda w_{i,j}f_{a}^{t-SNE}<-1$) and gives the minimum distance for contraction, $\zeta_{-1}$). The repulsive update is coupled with all the pairs and thus does not have a simple relation to the repulsion shape. Rather, enabled by our experience from TriMap’s analysis, we can derive the following:

###### Proposition G.3 .

The update Eqs. ([74](https://arxiv.org/html/2503.09101v3#A7.E74))-([75](https://arxiv.org/html/2503.09101v3#A7.E75)) provide an expansion if

$$ $\displaystyle h_{r}^{t-SNE}(\zeta_{i,j},v,\theta,\lambda,w_{i,j})>1,$ (78) $$

where
$h_{r}^{t-SNE}(\zeta_{i,j},v,\theta,\lambda,w_{i,j})=(1+2\lambda\frac{w_{i,j}}{Z}f_{r}^{t-SNE}(\zeta_{i,j}))^{2}+\frac{||v||_{2}^{2}}{\zeta_{i,j}^{2}}+2\lambda\frac{w_{i,j}}{Z}f_{r}^{t-SNE}(\zeta_{i,j})\frac{||v||_{2}}{\zeta_{i,j}}\cos{\theta}$,
$v=\lambda\frac{w_{i,j}}{Z}\left(\sum_{k,k\neq j}f_{r}^{t-SNE}(\zeta_{i,k})(y_{i}-y_{k})+\sum_{l,l\neq i}f_{r}^{t-SNE}(\zeta_{l,j})(y_{l}-y_{j})\right)$,
and $\theta$ is the angle between $(y_{i}-y_{j})$ and $v$.

###### Proof.

We require,

$$ $\displaystyle||y_{i}^{t+1}-y_{j}^{t+1}||_{2}^{2}>||y_{i}^{t}-y_{j}^{t}||_{2}^{2}.$ (79) $$

From Eqs. ([74](https://arxiv.org/html/2503.09101v3#A7.E74)) and ([75](https://arxiv.org/html/2503.09101v3#A7.E75)), $y_{i}-y_{j}=(1+2\lambda\frac{w_{i,j}}{Z}f_{r}^{t-SNE}(\zeta_{i,j}))(y_{i}-y_{j})+v$. Putting this value in Eq. ([79](https://arxiv.org/html/2503.09101v3#A7.E79)), we obtain the desired inequality.
∎

Since $t$-SNE’s condition for repulsion (Eq. [78](https://arxiv.org/html/2503.09101v3#A7.E78)) resembles that of TriMap, the repulsion behavior will be the same. Thus, $t$-SNE’s repulsive forces can give both attraction and repulsion.

Since, $t$-SNE’s forces are scaled (by $w_{i,j}$ for attraction and $\frac{w_{i,j}}{Z}$ for repulsion), the attractive and repulsive forces are typically lower than that of UMAP. As a result, the algorithm generally uses large values for learning rate (e.g., $\approx 10^{3}$ in the Open-$t$-SNE package (Poličar et al., 2024)).
Moreover, most $t$-SNE implementations require an ‘early exaggeration’ step where the attractive forces are multiplied by a constant value for the first few iterations.
This causes points that are supposed to be closer but currently placed far apart to approach each other (inducing far-sightedness).
On the other hand, if some points are very close ($\zeta_{i,j}\approx\zeta_{-1}$) but require separation, this early exaggeration trick achieves that as well.
Thus, ‘early exaggeration’ plays a vital role in finding a consistent embedding in $t$-SNE and is an indispensable feature of the algorithm; especially when initialized randomly.
(This trick, when applied throughout the optimization, makes $t$-SNE embeddings look closer to UMAP embeddings (Böhm et al., 2022).)

###### Proposition G.3 .

###### Proof.

### G.5 Stochastic Neighbor Embedding (SNE)

SNE (Hinton and Roweis, 2002) is one of the oldest algorithms in this class. This follows the same formula of $t$-SNE (Eq. [71](https://arxiv.org/html/2503.09101v3#A7.E71)), but with $q_{i,j}=\exp(-||y_{i}-y_{j}||_{2}^{2})$. This results in the attraction shape

$$ $\displaystyle f_{a}^{SNE}(\zeta)=-2,$ (80) $$

that follows Proposition [4.1](https://arxiv.org/html/2503.09101v3#S4.Thmtheorem1) (with $0<\lambda w_{i,j}f_{a}^{SNE}<-1$) and the repulsion shape

$$ $\displaystyle f_{r}^{SNE}(\zeta)=2\exp(-\zeta^{2}),$ (81) $$

that follows proposition [G.3](https://arxiv.org/html/2503.09101v3#A7.Thmtheorem3) (by replacing the $t$-SNE symbols with SNE counterparts). The attraction shape is ill-posed and thus mainly relies on the values of learning rate ($\lambda$) and the weights ($w_{i,j}$) for contraction resulting in clusters overlapping each other even for small number of samples (called crowding problem (Van der Maaten and Hinton, 2008)), which $t$-SNE and the subsequent algorithms improve by moving away from the Gaussian kernel to a heavy-tailed one, i.e., Eq. ([43](https://arxiv.org/html/2503.09101v3#A7.E43)).

### G.6 Multidimensional Scaling (MDS)

Figure: Figure 17: (a) Attraction-repulsion characteristic of MDS using $f_{ar}$. (b) Comparison of attraction-repulsion characteristic of MDS and the method in (Zheng et al., 2018) for $d=1.0$.
Refer to caption: figures_attr_shape/mds_fig_zz.eps

Multidimensional scaling (MDS) (Borg and Groenen, 2007) typically does not use gradient methods, as they often fail to converge to good mappings; instead, it employs stress majorization. Nevertheless, few works discuss gradient methods (Kruskal, 1964; Zheng et al., 2018). We offer a brief treatment for this below. Particularly, Zheng et al. (2018) formulates a successful gradient descent-based MDS algorithm for graph drawing. We start with the cost function (we can ignore the weight $w_{i,j}$ without loss of generality to their approach, and for discussion, we can lump it into the learning rate):

$$ $\displaystyle\mathcal{L}^{MDS}=\sum_{i,j,i<j}(||y_{i}-y_{j}||-d_{ij})^{2}.$ (82) $$

The loss has a singleton term for each pair and does not have explicit attractive and repulsive terms. The term itself will provide both attraction and repulsion.
Fortunately, the loss is separable into individual $(i,j)$ components, allowing us to analyze a single pair. The derivative is given by

$$ $\displaystyle\frac{\partial}{\partial y_{i}}(||y_{i}-y_{j}||-d_{ij})^{2}$ $\displaystyle=2\frac{||y_{i}-y_{j}||-d_{ij}}{||y_{i}-y_{j}||}(y_{i}-y_{j}).$ (83) $$

Thus, the attraction-repulsion shape of MDS becomes

$$ $\displaystyle f_{ar}^{MDS}(\zeta)=-2\frac{\zeta-d}{\zeta},$ (84) $$

where $\zeta=||y_{i}-y_{j}||_{2}$. Here, the shape is attractive when $\zeta>d$. In this range, the shape is bounded within $[-2,0]$ (Fig. [17](https://arxiv.org/html/2503.09101v3#A7.F17)(a)). However, this is not suitable for convergence as values only within $[-1,0]$ contract, while the others expand. To make things worse, values lower than $-2$ cause the points to flip and expand. To work around this, Zheng et al. (2018) uses an ad-hoc formulation inspired by the force-directed graphs, given by

$$ $\displaystyle f_{ar}^{G}(\zeta)=-\frac{1}{2}\frac{\zeta-d}{\zeta},$ (85) $$

which works for an effective learning rate $\lambda\leq 1$. $f_{ar}^{G}$ is bounded within $[-0.5,0]$ and thus, $\lambda\leq 1$ works. On the other hand, for $\zeta<d$, $f_{ar}^{MDS}(>0)$ shows repulsive behavior. Overall, this attraction-repulsion interaction works best when the initialization is already close to a desired output. If one keeps optimizing for a pair, it will oscillate around the distance ($\zeta=d$) where $f_{ar}^{G}=0$ (and consequently, we can have a notion of distance similar to $\zeta_{-1}$ of UMAP). No choice of learning rate reduces this distance to zero (in fact, this achieves the objective of MDS). Thus, the clusters are often fuzzy compared to methods like UMAP and PaCMAP (for relevant illustrations, see Lambert et al. (2022), de Bodt et al. (2025), and Kury et al. (2025)). Note that Lambert et al. (2022) converts the MDS loss function to a quartet stress (loss involving four samples) and a relative distance (distance divided by the sum of six distances in the quartet), enabling global regularization and reduced computation than the majorization approach.

## Appendix H Construction of the High-Dimensional Graph

Stochastic Neighbor Embedding (SNE) Hinton and Roweis (2002) underpins modern dimensionality reduction algorithms. It constructs a high-dimensional graph of the dataset $X=\{x_{i}\in R^{n}|i=1,\dots,N\}$ by the following system of equations:

$$ $\displaystyle w_{ij}$ $\displaystyle=\frac{p_{j|i}+p_{i|j}}{2N},$ (86) $$

$$ $\displaystyle w_{j|i}$ $\displaystyle=\frac{\exp(-||x_{i}-x_{j}||_{2}^{2}/2\sigma_{i}^{2})}{\sum_{t\neq v}\exp(-||x_{t}-x_{v}||_{2}^{2}/2\sigma_{t}^{2})},$ (87) $$

where $\sigma_{i}^{2}$ is chosen to match a user-defined value perplexity, $P$, defined as

$$ $\displaystyle P$ $\displaystyle=2^{H_{i}}$ (88) $\displaystyle H_{i}$ $\displaystyle=\sum_{j\neq i}w_{j|i}\log_{2}w_{j|i}.$ (89) $$

On the other hand, UMAP constructs its high-dimensional graph by the following system of equations relying on the k-nearest neighbor (k-NN) algorithm:

$$ $\displaystyle p_{i|j}$ $\displaystyle=\begin{cases}\exp{\left(-\frac{d(x_{i},x_{j})-\rho_{i}}{\sigma_{i}}\right)}&\text{if }x_{j}\in\text{KNN}(x_{i},k)\\ 0&\text{otherwise}\end{cases},$ (90) $\displaystyle\rho_{i}$ $\displaystyle=\min_{x_{j}\in\text{KNN}(x_{i},k)}d(x_{i},x_{j}),$ (91) $$

where $\text{KNN}(x_{i},k)$ is the set of $k$-nearest neighbors of the point $x_{i}$ and $\sigma_{i}$ is a scaling parameter such that $\sum_{j}p_{i|j}=\log_{2}(k)$. The graph is then symmetrized by a t-conorm:

$$ $\displaystyle p_{i,j}$ $\displaystyle=p_{i|j}+p_{j|i}-p_{i|j}p_{j|i}.$ (92) $$

PaCMAP uses just the k-NN graph for the affinities (all equal to 1) with a self-tuning distance measure (Zelnik-Manor and Perona, 2004),

$$ $\displaystyle d_{i,j}^{2}=\frac{||x_{i}-x_{j}||^{2}}{\sigma_{i}\sigma_{j}},$ (93) $$

where $\sigma_{i}$ is the average distance between $x_{i}$ and its Euclidean nearest fourth to sixth neighbors. The purpose of this is the same as the corresponding $\sigma_{i}$ parameters in Eq. ([87](https://arxiv.org/html/2503.09101v3#A8.E87)) and ([90](https://arxiv.org/html/2503.09101v3#A8.E90)), despite all three being defined differently.

Regardless of these choices, the optimization, and by our analysis, the attraction and repulsion shapes are the primary drivers of the low-dimensional embedding.

## Appendix I Detailed Results for Varying λ a \lambda_{a} and λ b \lambda_{b}

In this section, we provide detailed results for the experiments in Fig. [1](https://arxiv.org/html/2503.09101v3#S4.F1)((h,i).
We obtained these results, by changing the attraction and repulsion shapes of UMAP to that of the other methods. Figure [18](https://arxiv.org/html/2503.09101v3#A9.F18) reproduces the results given in the main text.
Figures [19](https://arxiv.org/html/2503.09101v3#A9.F19)-[22](https://arxiv.org/html/2503.09101v3#A9.F22) show the individual embeddings for each of the choices of $\lambda_{a}$ and $\lambda_{r}$ for each methods along with their initialization, a reference embedding when learning rate, $\lambda$, is annealed, and when either the $\lambda_{a}$ or $\lambda_{r}$ set to 0.
For PaCMAP, which uses the concept of mid-near pairs, we show additional reference of mid-near pairs included as well.
Figures [23](https://arxiv.org/html/2503.09101v3#A9.F23)- [27](https://arxiv.org/html/2503.09101v3#A9.F27) and  Figures. [28](https://arxiv.org/html/2503.09101v3#A9.F28)- [32](https://arxiv.org/html/2503.09101v3#A9.F32) provide results on FMNIST and single-cell transcriptomes dataset, respectively.

Figure: Figure 18: (a) Trustworthiness and (b) Silhouette score of different methods for the MNIST dataset.
Refer to caption: figures_attr_shape/figure_1_mnist.eps

Figure: Figure 19: Varying $\lambda_{a}$ and $\lambda_{r}$ for the MNIST dataset using UMAP’s attraction and repulsion shapes. (a) Initialization for the embeddings. (b) Baseline when $\lambda$ is annealed from $1$. (c) The embeddings when $\lambda_{a}$ and $\lambda_{r}$ vary (without any annealing). (d) When $\lambda_{r}$ is set to 0 (no repulsion), the attractive force alone cannot produce any cluster. (e) Similarly, when $\lambda_{a}$ is set to 0 (no attraction), the repulsive force alone cannot produce any clusters.
Refer to caption: figures_attr_shape/mnist_k1k2.eps

Figure: Figure 20: Varying $\lambda_{a}$ and $\lambda_{r}$ for the MNIST dataset using UMAP’s attraction and repulsion shapes (by setting $a=1$ and $b=1$). (a) Initialization for the embeddings. (b) Baseline when $\lambda$ is annealed from $1$. (c) The embeddings when $\lambda_{a}$ and $\lambda_{r}$ vary (without any annealing). (d) When $\lambda_{r}$ is set to 0 (no repulsion), the attractive force alone cannot produce any cluster. (e) Similarly, when $\lambda_{a}$ is set to 0 (no attraction), the repulsive force alone cannot produce any clusters.
Refer to caption: figures_attr_shape/umap_ab1_mnist_k1k2.eps

Figure: Figure 21: Varying $\lambda_{a}$ and $\lambda_{r}$ for the MNIST dataset using NEG-$t$-SNE’s attraction and repulsion shapes. (a) Initialization for the embeddings. (b) Baseline when $\lambda$ is annealed from $1$. (c) The embeddings when $\lambda_{a}$ and $\lambda_{r}$ vary (without any annealing). (d) When $\lambda_{r}$ is set to 0 (no repulsion), the attractive force alone cannot produce any cluster. (e) Similarly, when $\lambda_{a}$ is set to 0 (no attraction), the repulsive force alone cannot produce any clusters.
Refer to caption: figures_attr_shape/neg_mnist_k1k2.eps

Figure: Figure 22: Varying $\lambda_{a}$ and $\lambda_{r}$ for the MNIST dataset using PaCMAP’s attraction and repulsion shapes. (a) Initialization for the embeddings. (b) Left: Baseline when $\lambda$ is annealed from $1$ (mid-near points are excluded to observe the interaction of attraction-repulsion alone), right: when mid-near points are considered. (c) The embeddings when $\lambda_{a}$ and $\lambda_{r}$ vary (without any annealing). (d) When $\lambda_{r}$ is set to 0 (no repulsion), the attractive force alone cannot produce any cluster. (e) Similarly, when $\lambda_{a}$ is set to 0 (no attraction), the repulsive force alone cannot produce any clusters.
Refer to caption: figures_attr_shape/pacmap_mnist_k1k2.eps

Figure: Figure 23: (a) Trustworthiness and (b) Silhouette score of different methods for the FMNIST dataset.
Refer to caption: figures_attr_shape/figure_1_fmnist.eps

Figure: Figure 24: Varying $\lambda_{a}$ and $\lambda_{r}$ for the FMNIST dataset using UMAP’s attraction and repulsion shapes. (a) Initialization for the embeddings. (b) Baseline when $\lambda$ is annealed from $1$. (c) The embeddings when $\lambda_{a}$ and $\lambda_{r}$ vary (without any annealing). (d) When $\lambda_{r}$ is set to 0 (no repulsion), the attractive force alone cannot produce any cluster. (e) Similarly, when $\lambda_{a}$ is set to 0 (no attraction), the repulsive force alone cannot produce any clusters.
Refer to caption: figures_attr_shape/fmnist_k1k2.eps

Figure: Figure 25: Varying $\lambda_{a}$ and $\lambda_{r}$ for the FMNIST dataset using UMAP’s attraction and repulsion shapes (by setting $a=1$ and $b=1$). (a) Initialization for the embeddings. (b) Baseline when $\lambda$ is annealed from $1$. (c) The embeddings when $\lambda_{a}$ and $\lambda_{r}$ vary (without any annealing). (d) When $\lambda_{r}$ is set to 0 (no repulsion), the attractive force alone cannot produce any cluster. (e) Similarly, when $\lambda_{a}$ is set to 0 (no attraction), the repulsive force alone cannot produce any clusters.
Refer to caption: figures_attr_shape/umap_ab1_fmnist_k1k2.eps

Figure: Figure 26: Varying $\lambda_{a}$ and $\lambda_{r}$ for the FMNIST dataset using NEG-$t$-SNE’s attraction and repulsion shapes. (a) Initialization for the embeddings. (b) Baseline when $\lambda$ is annealed from $1$. (c) The embeddings when $\lambda_{a}$ and $\lambda_{r}$ vary (without any annealing). (d) When $\lambda_{r}$ is set to 0 (no repulsion), the attractive force alone cannot produce any cluster. (e) Similarly, when $\lambda_{a}$ is set to 0 (no attraction), the repulsive force alone cannot produce any clusters.
Refer to caption: figures_attr_shape/neg_fmnist_k1k2.eps

Figure: Figure 27: Varying $\lambda_{a}$ and $\lambda_{r}$ for the FMNIST dataset using PaCMAP’s attraction and repulsion shapes. (a) Initialization for the embeddings. (b) Left: Baseline when $\lambda$ is annealed from $1$ (mid-near points are excluded to observe the interaction of attraction-repulsion alone), right: when mid-near points are considered. (c) The embeddings when $\lambda_{a}$ and $\lambda_{r}$ vary (without any annealing). (d) When $\lambda_{r}$ is set to 0 (no repulsion), the attractive force alone cannot produce any cluster. (e) Similarly, when $\lambda_{a}$ is set to 0 (no attraction), the repulsive force alone cannot produce any clusters.
Refer to caption: figures_attr_shape/pacmap_fmnist_k1k2.eps

Figure: Figure 28: (a) Trustworthiness and (b) Silhouette score of different methods for the Single-cell transcriptomes dataset.
Refer to caption: figures_attr_shape/figure_1_macosko.eps

Figure: Figure 29: Varying $\lambda_{a}$ and $\lambda_{r}$ for the single-cell transcriptomes dataset using UMAP’s attraction and repulsion shapes. (a) Initialization for the embeddings. (b) Baseline when $\lambda$ is annealed from $1$. (c) The embeddings when $\lambda_{a}$ and $\lambda_{r}$ vary (without any annealing). (d) When $\lambda_{r}$ is set to 0 (no repulsion), the attractive force alone cannot produce any cluster. (e) Similarly, when $\lambda_{a}$ is set to 0 (no attraction), the repulsive force alone cannot produce any clusters.
Refer to caption: figures_attr_shape/macosko_k1k2.eps

Figure: Figure 30: Varying $\lambda_{a}$ and $\lambda_{r}$ for the single-cell transcriptomes dataset using UMAP’s attraction and repulsion shapes (by setting $a=1$ and $b=1$). (a) Initialization for the embeddings. (b) Baseline when $\lambda$ is annealed from $1$. (c) The embeddings when $\lambda_{a}$ and $\lambda_{r}$ vary (without any annealing). (d) When $\lambda_{r}$ is set to 0 (no repulsion), attractive force alone cannot produce any cluster. (e) Similarly, when $\lambda_{a}$ is set to 0 (no attraction), repulsive force alone cannot produce any clusters.
Refer to caption: figures_attr_shape/umap_ab_1_macosko_k1k2.eps

Figure: Figure 31: Varying $\lambda_{a}$ and $\lambda_{r}$ for the single-cell transcriptomes dataset using NEG-$t$-SNE’s attraction and repulsion shapes. (a) Initialization for the embeddings. (b) Baseline when $\lambda$ is annealed from $1$. (c) The embeddings when $\lambda_{a}$ and $\lambda_{r}$ vary (without any annealing). (d) When $\lambda_{r}$ is set to 0 (no repulsion), the attractive force alone cannot produce distinct clusters. (e) Similarly, when $\lambda_{a}$ is set to 0 (no attraction), the repulsive force alone cannot produce any clusters.
Refer to caption: figures_attr_shape/neg_macosko_k1k2.eps

Figure: Figure 32: Varying $\lambda_{a}$ and $\lambda_{r}$ for the single-cell transcriptomes dataset using PaCMAP’s attraction and repulsion shapes. (a) Initialization for the embeddings. (b) Left: Baseline when $\lambda$ is annealed from $1$ (mid-near points are excluded to observe the interaction of attraction-repulsion alone), right: when mid-near points are considered. (c) The embeddings when $\lambda_{a}$ and $\lambda_{r}$ vary (without any annealing). (d) When $\lambda_{r}$ is set to 0 (no repulsion), the attractive force alone cannot produce any cluster. (e) Similarly, when $\lambda_{a}$ is set to 0 (no attraction), the repulsive force alone cannot produce any clusters.
Refer to caption: figures_attr_shape/pacmap_macosko_k1k2.eps

## Appendix J Mix and Match Approach

In our analysis, we considered the attraction and the repulsion shapes to be independent of each other. It thus suggests that we can combine attraction and repulsion shapes derived from different loss functions and principles. Figure [34](https://arxiv.org/html/2503.09101v3#A10.F34) shows a few such examples. For this, we restricted the attraction shapes to $[-1,0]$ (to work with Proposition [4.1](https://arxiv.org/html/2503.09101v3#S4.Thmtheorem1)) so that they become less sensitive to learning rates. We considered finite repulsion shapes, hence mitigating the effect of large repulsive forces for smaller distances. We held the learning rate constant ($\lambda=1$) and used the same optimization (of UMAP) for all of them, so that any changes arise from the shapes rather than learning-rate scheduling or optimization scheme. Overall, UMAP’s attraction shape (by $f_{a}=0.5f_{a}^{U}$) makes better clusters (higher silhouette score), while PaCMAP’s attraction shape provides better local structure (higher trustworthiness). The latter is primarily due to PaCMAP’s attraction shape being within $[-0.5,0]$, which prevents any flips during the updates. By confining UMAP and NEG-$t$-SNE ’s attraction shape within $[-0.5,0]$ (by $f_{a}=0.25f_{a}^{U}$ and $f_{a}=0.5f_{a}^{N}$, respectively), we can improve the local structure at the expense of clustering performance (Fig. [34](https://arxiv.org/html/2503.09101v3#A10.F34)).

Figure: Figure 33: Embedding of MNIST by mixing and matching attraction and repulsion shapes with a constant learning rate ($\lambda=1$). We used attraction shapes that follow Proposition [4.1](https://arxiv.org/html/2503.09101v3#S4.Thmtheorem1) and repulsion shapes that are finite. Top row: UMAP, NEG-$t$-SNE, and PaCMAP’s attraction shapes with NEG-$t$-SNE’s repulsion shape. Bottom row: UMAP, NEG-$t$-SNE, and PaCMAP’s attraction shapes with PaCMAP’s repulsion shape.
Refer to caption: figures_attr_shape/mixnmatch.eps

## Appendix K Implementation Details

For analysis, we implemented our own UMAP algorithm. We used numba (Lam et al., 2015) to compute an exact nearest neighbor graph (instead of an approximate one) with $k=15$ and scikit-learn’s (Pedregosa et al., 2011) implementation of the PCA algorithm for PCA initialization. We used this to produce and quantify the embeddings in Figs. [1](https://arxiv.org/html/2503.09101v3#S4.F1), [2](https://arxiv.org/html/2503.09101v3#S5.F2), [3](https://arxiv.org/html/2503.09101v3#S5.F3), [4](https://arxiv.org/html/2503.09101v3#S5.F4), [6](https://arxiv.org/html/2503.09101v3#A1.F6), [8](https://arxiv.org/html/2503.09101v3#A4.F8), [9](https://arxiv.org/html/2503.09101v3#A5.F9), and [10](https://arxiv.org/html/2503.09101v3#A5.F10). We also used the same implementation when we changed the attraction and the repulsion shapes to those of the alternative methods (Figs. [18](https://arxiv.org/html/2503.09101v3#A9.F18)- [34](https://arxiv.org/html/2503.09101v3#A10.F34), unless otherwise stated). The trustworthiness and silhouette scores were computed using the corresponding function from the scikit-learn package.

The mappings shown in Figs. [2](https://arxiv.org/html/2503.09101v3#S5.F2), [3](https://arxiv.org/html/2503.09101v3#S5.F3), [9](https://arxiv.org/html/2503.09101v3#A5.F9), [10](https://arxiv.org/html/2503.09101v3#A5.F10), and [15](https://arxiv.org/html/2503.09101v3#A7.F15) are rotated to a reference embedding ((a) for each respective Figures). To achieve this, we performed Procrustes alignment of the embeddings by normalizing them (zero mean and unit norm) and then using SciPy’s (Virtanen et al., 2020) orthogonal_procrustes method to extract rotation and scaling parameters.

To compare with Neg-$t$-SNE in Fig. [11](https://arxiv.org/html/2503.09101v3#A6.F11), we used the original implementation of the contrastive embedding framework for both the UMAP and the Neg-$t$-SNE algorithms (available at [https://github.com/berenslab/contrastive-ne](https://github.com/berenslab/contrastive-ne)).

PaCMAP and LocalMAP embeddings in Figs. [15](https://arxiv.org/html/2503.09101v3#A7.F15), [16](https://arxiv.org/html/2503.09101v3#A7.F16), [22](https://arxiv.org/html/2503.09101v3#A9.F22) (b) (right), [27](https://arxiv.org/html/2503.09101v3#A9.F27) (b) (right), and [32](https://arxiv.org/html/2503.09101v3#A9.F32) (b) (right) were obtained using the official PaCMAP package (available at [https://github.com/YingfanWang/PaCMAP](https://github.com/YingfanWang/PaCMAP)).

The codes for displaying the embeddings of the PubMed dataset in Fig. [7](https://arxiv.org/html/2503.09101v3#A3.F7) are adapted from [https://github.com/berenslab/pubmed-landscape](https://github.com/berenslab/pubmed-landscape).

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 2
presentation: 1
questions: - Q1: line 246: "only" UMAP and t-SNE deal with some issue. Only methods among which group?

- Q2: line 252: "only UMAP deals with large [repulsive] values for small distances" Is this a good thing, because non-neighbors get pushed far apart, or a bad thing because it leads to numerical instability and requires gradient clipping?

- Q3: line 294: What is "scale-invariant structure"? Was perhaps "initialization-invariant" structure meant?

- Q4: line 1060: Why does initializing $\lambda$ at $0.5$ satisfies Prop 4.1? It is still the case that $\zeta_{-1} \approx 0.5 > 0$ according to Fig 1d.
soundness: 2
strengths: - S1: The finding about the expansive behavior of the attraction at small scales is interesting, novel and gives relevant pointers for the choice of the learning rate in UMAP. 
- S2: The long-range attraction is an interesting contribution.  
- S3: The extensive appendix extends some of the analysis to many other neighbor embeddings and is a useful resource.  
- S4: Code is submitted, facilitating reproducibility.
summary: This paper discusses various aspects of the attractive and repulsive forces in UMAP and related visualization methods. It uncovers that for nearby points attraction might even increase their distance, explaining why learning rate annealing is crucial for UMAP. The authors also propose adding a long-range attractive force during the first optimization epochs to achieve a more consistent global layout, even from random initializations. Finally, they explore using different similarity kernels for attraction and repulsion in UMAP and compare UMAP's forces with other visualizations methods.
weaknesses: **Major**  

The paper presents many aspects about forces in neighbor embedding methods. While some of them are interesting, many feel underexplored and some are not novel. My suggestion is to expand the sections on expansive and long-range attraction in favor of the other sections. While this would improve the paper, I do not think that the novelty of the paper is sufficient for ICLR.

*W1: Presentation and exploration of UMAP's expansive attraction*:  
While the expansive nature of UMAP's attraction is interesting, the phenomenon should be better explained and explored. The following would help the explanation in my mind: The expansion is a classical case of an overshooting gradient descent. The update size decreases more slowly than the distance, so that the update eventually overshoots. This happen only for $b<1$.
Figure 1 a takes up a lot of space, but conveys very little information, I would omit it, or at least omit the repulsive part and decrease the size for the attractive part. 
Including some figures of UMAP with non-annealed learning rate (maybe even all four corner cases $\lambda_a, \lambda_r \in \\{0.1, 1\\}$) in the main paper and explaining them in more detail would help. Why can we deduce from this experiment that *only* the attraction necessitates annealing (the setting $\lambda_a = 0.1, \lambda_r=1.0$ still produces a "fuzzy" embedding). Based on these four plots, one could also explain the "failure modes" in Fig 1 h (top left and bottom right corner) qualitatively. It may also be useful to state that the distance $\zeta_{-1}$ around which the attraction would oscillate without annealing is large relative to typical UMAP plot diameters. Finally, it would help to state that by default, UMAP starts with $\lambda = 1$ and anneals this learning rate linearly to $0$ over training. Perhaps it might even be nice to show intermediate training stages, when clusters and their relative positions have already formed, but still appear fuzzy as $\lambda$ is still too large. 

Moreover, the phenomenon should be explored more deeply: One could, for instance, let a single pair of points evolve according to UMAP's attraction from various starting distances and plot how its distance evolves (without learning rate annealing). I assume it would eventually oscillate around 1. Similarly, in a UMAP plot without annealed learning rate one could measure the average distance between the embeddings of k-nearest neighbors and check if it is close to 1. Or compare this mean neighbor distance between the annealed and non-annealed case.

*W2: Discussion and exploration of long-range attraction*  
The proposed fix of adding $||y_i - y_j||^3$ to the attractive loss is quite similar to SNE, which uses attraction $|| y_i - y_j||^2$. It would be good to discuss this relation and perhaps even compare quantitatively. A small point is that in line 299 it is not $f_a^U$ going to zero that is problematic, but what is given in brackets ($|f_a^U| \cdot \zeta$ going to zero). But this is not equivalent, so "i.e." is not the correct choice here. 

In general, this chapter should be made more quantitative. While the authors compute Silhouette scores, Trustworthiness, and Procrustes distances, their comparison would be much eased by putting mean values in a Table. Moreover, the relative positioning of clusters might be also captured well by metrics like Spearman correlation between distances (both relative to the high-dimensional and among different UMAP embeddings). A key questions is, of course, how the long-range attraction compares to informative initializations with PCA or LE. So, a results table should also contain metrics for PCA/LE-initialized embeddings. Are they not simply much more stable than UMAP with long-range attraction starting from random noise? Finally, the authors include experiments on other datasets in the appendix, but a tabular summary of these experiments in the main paper would also strengthen the empirical findings. 

*W3: Cluster formation chapter is vague and less novel*  
Overall, the novelty of this chapter seems limited since, e.g., Agrawal et al. (2021) also mix and match different attractive and repulsive forces and Kobak et al. (2019) (which the paper cites in this section) already discuss heavy-tailed neighbor embeddings, including the effect of $b$ on UMAP embeddings. While the present paper refines their analysis to heavy-tailed attraction being more important, claims such as "to resolve the mystery of cluster formation" are way too strong given the limited novelty. In addition, the paper does not investigate if the additional clusters found with heavy-tailed attraction are meaningful or even undesirable fragmentation. Finally, in line 371 the paper states that changing $a$ affects both repulsion and attraction. But a different $a$ really just rescales the final embedding without changing its appearance, since

$$1 / (1+ad\_{ij}^{2b}) = 1 / (1+(a^{1/(2b)}d\_{ij})^{2b}) = 1 /(1+\tilde{d}\_{ij}^{2b})$$

for the rescaled distance $\tilde{d}_{ij} = a^{1/(2b)}d\_{ij}$.

*W4: Discussion of Neg-t-SNE*  
Relating Neg-t-SNE to UMAP in the context of expansive attraction is good and interesting. I particularly appreciate broadening its explanation for why learning rate annealing is required for UMAP to the attractive forces. However, I would strongly suggest to do so in the chapter on the expansive behavior of UMAP's attraction rather than two chapters later. The other statement in the Neg-t-SNE chapter, Prop. 6.2 on Neg-t-SNE's relation to parametric UMAP, is not novel, but can already be found in appendix E of Damrich et al. (2023).


**Minor**  
*W5: Notation* The notation does not get introduced in many places. E.g. at the start of sec 3: $f, h, d, d_{ij}$ are not defined and in section 5 I do not understand why the notation changes from $f_a$ and $f_r$ to $f_a^U$ and $f_r^U$.

*W6: Related work* I would recommend not to include the spectral methods Laplacian Eigenmaps and Locally Linear Embedding in the middle of the paragraph on neighbor embeddings in the related work. 

*W7: Flowery language* The language is in some place a bit to flamboyant for a research paper. E.g. the first sentence uses both "era" and "deluge", and the discussion section "demystifying".

*W8: Font size in many figures too small*



**Typos**

- line 235: approaches $-\infty$ not $\infty$

- line 484: Wrong citation Damrich & Hamprecht (2021) --> Damrich et al. (2023)

- line 1059 $(_a)$? 

- Eq 86 / 87 Switch in notation between $w_{i \mid j}$ and $p_{i \mid j}$

- Abstract: forces among high-dim data point -->  low-dim data points

- Equality sign missing after $\zeta_{-1}$ in axis-label of figure 1d.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - Around line 257 I think there is a typo, you use $\zeta^{-1}$ instead of $\zeta_{-1}$
soundness: 3
strengths: - Very clean exposition and nice focus on the single topic of the role of attraction/repulsion in dimensionality reduction algorithms. 
- Though a lot the ingredients of the paper already appeared in previous work (e.g. empirical attraction/repulsion function graphs), the way they are used and put together creates a novel view of the topic.
- The insights can help further optimize existing methods and provide a solid understanding on how new methods can setup their projection optimization.
- Follows the nice practice of including plenty of projections and plots for different datasets in the supplementary. This increases credibility and also makes the paper a nice reference.
summary: The paper provides an in-depth analysis of the attractive and repulsive forces present in the optimization step of UMAP. The authors first process and simplify the update functions of the point coordinates in the projection space, introducing two functions, $f_a$ and $f_r$ which control contraction and repulsion. Using this tool, they go on and generate the following insights:

- The first function, $f_a$, which is often associated with attraction, can actually act repulsively by flipping points and moving them further apart. $f_a$ and $f_r$ are also plotted for several methods and different learning rates. Lowering the learning rate lowers the distance below which $f_a$ becomes repulsive. Methods where $f_a$ is only attractive by design seem to be more robust to changes of the learning rate.
- The consistency of the projection under random initialization is studied using the Procrustes distance as a measure of projection similarity. It is show that by increasing the attractive force, the projections become more consistent presumably because the force between distant points is large.
- The formation and compactness of clusters is studied. Evidence is provided that the former is caused by attraction while the latter by repulsion.
- Different methods are compared in light of the approach of using $f_a$ and $f_r$. It is shown that Neg-t-SNE are the same in terms of their loss terms.
weaknesses: - Only three datasets where used to support the different arguments, two of them quite similar in nature, MNIST and FMNIST. Though it is enough for visualization and demonstration purposes, the evidence feels a bit more qualitative than quantitative. Usually papers in the domain use almost 10 datasets covering different domains. To give an example of how this could have strengthened the evidence, in the study of cluster formation one could have correlated the attraction shape with the Silhouette score among datasets to support the connection of repulsion with cluster formation. One could also consider such ablations on synthetic datasets.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: (1) The core contribution of this paper is the detailed analysis of UMAP’s gradient update based on the attraction & repulsion shape coefficients. This can be viewed as a stepwise analysis of the local updates. But I wonder whether these shapes can shed light on UMAP's overall convergence property, especially across different settings of a and b? Do the oscillations the authors observe in some conditions cause unstable optimization?
(2) Beyond the cluster formation, the interplay between the attraction and repulsion forces also impacts the preservation of global and local structure. Can the attraction/repulsion shapes help explain some phenomena about this?
soundness: 3
strengths: - Provide a careful theoretical analysis with derivations of several key properties.
- Offer insights that help explain behaviors such as initialization effects and cluster formation.
- Indicate practical implications for DR design, e.g., encouraging consistency.
summary: This paper introduces a theoretical formulation of UMAP’s learning process via a pair of mathematical properties: attraction shape and repulsion shape. With these properties, the authors manage to interpret the general mechanism of how UMAP’s loss function works in two forces under different conditions. The authors further show the practical implications of attraction and repulsion shapes by showing how they can interpret and improve the consistency and cluster formation, and how they explain the connection between Neg-t-SNE and parametric UMAP.
weaknesses: - Ad hoc shape tuning. In the experiments of consistency improvements (Sec. 5.1) and cluster formation (Sec 5.2), the attraction/repulsion curve is adjusted ad hoc. While this lowers Procrustes distance, potential trade-offs with broader DR quality (e.g., Trustworthiness, Continuity) are not evaluated.
- Insufficient parameter analysis. The attractive/repulsive forms hinge on parameters a and b, yet the paper shows only scattered results for fixed choices. A systematic sensitivity study—visualizing how a and b affect the shapes and offering selection guidance—would clarify their roles.
- Limited discussion of alternative consistency methods. Related approaches, such as aligned-UMAP, are not adequately discussed. A comparison and a clear statement of the advantages and costs of the shape-guided method would strengthen the case.
- Poor readability and organization. The writing is dense, with many results and observations. Reorganizing the analysis with clearer signposting and bolded subheadings that highlight key takeaways would improve accessibility.

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 1
questions: My questions are mostly related to my confusion after reading this paper, which I have elaborated in the weaknesses section. Specifically:
- What would you like the reader to walk away after reading this paper? Compared to the default setting of UMAP, shall we increase $a$/$b$ or decrease it, to obtain a more faithful and interpretable result? Shall we increase the far-sightedness in the force gradient? Shall we favor other methods such as Neg-t-SNE/PaCMAP/LocalMAP? Does such choice improve the performance of the algorithm statistically?
- How would you like to separate your work against the existing ones in terms of novelty? Why is the proposed analysis more insightful, if the decomposition method discussed has already been adopted by many?
soundness: 3
strengths: - The writing is fluent, improving the readability of this paper..

- The code release is thorough, including experimental code and plotting scripts. This openness strengthens the paper’s scientific value and facilitates verification.

- The author has provided an extensive list of related works, covering most of the important progresses in this field.
summary: UMAP is a widely-adopted neighbor embedding method, and understanding how its attractive and repulsive forces define the final embedding is core to improving reliability and usability in downstream tasks. This paper conducts numerical analysis on how these forces evolve under different circumstances and discusses the importance of learning-rate annealing and the asymmetry between attraction and repulsion. The authors provide code for both experiments and visualizations, supporting transparency and reproducibility.
weaknesses: - **Uncertain novelty relative to prior research.** 
  - In section 4, the authors assert *a gap* in the literature regarding force dynamics, yet cited works (such as McInnes et al., 2018; Agrawal et al., 2021; Wang et al., 2021; Draganov & Dohn, 2023), as identified by the authors, already examine the evolution of attractive and repulsive updates, hence a distinction must be made to justify the novelty. The authors described their difference as *"Here, we treat the decompositions as independent functions that can take various forms"*, which is quite confusing to me and does not make a good point. In my opinion, the main difference here seems to be the integration of the learning rate into the discussion, so that we can discuss the actual update at each step. The significance of this point is unclear (see below), and it definitely needs more articulation.

  - Section 5.2: It should not be surprising to anyone in the dimensionality reduction field that the cluster formation is dominated by attraction and the repulsion makes the clusters more compact. To list a few examples just from the authors' own citations, similar ideas has been described in Narayan et. al 2020, Bohm et. al 2022, Wang et. al 2025. Besides, description regarding the design of LocalMAP seems to already discussed in Wang et. al 2025 Section 5.

  - Section 6: The authors built a connection between Neg-t-SNE and UMAP, which is a novel point, but the message is also unclear (see below). The authors' other acclaimed contribution, that the attractive and repulsive terms between different DR algorithms are interchangable, has been brought up before, see e.g. Wang et. al, 2021. As long as these terms are following the DR algorithm principles, they can form a good DR algorithm.

- **Lack of a clear overarching message.** While the experimental analysis is detailed, the narrative does not converge on a key takeaway.
  - Section 4 - 5.1: The authors observed that the existence of large updates in UMAP may flip the relative position of certain neighbors. While this observation can be interesting, the reader may wonder how this observation could translate into some real improvements over the dimensionality reduction quality. Comparing Fig 2 (b) and (d), I personally don't feel the authors' proposed change make any difference in the embedding quality, and this observation sort of invalidates the importance of Fig 2 (f-h) comparison. Similar comment could also be made to Fig 1, that for Neg-t-SNE and PaCMAP the default parameters seems to be achieving a good balance between the existing metrics. What **actionable** insight should an algorithm designer or user walk away with? For section 5.1, several conclusions restate design choices already justified in the original UMAP paper—for example, spectral-embedding-based initialization and learning-rate annealing—making the supposed improvements, if any, feel incremental rather than revelatory.

  - Section 6: While the authors built the connection between Neg-t-SNE and UMAP, it's unclear what is the major takeaway one could go from here. Shall we favor methods such as Neg-t-SNE and PaCMAP over the unstable UMAP? Is there statistical evidence to support such a claim? Many questions were left unanswered.


- **Scope feels narrow and presentation should be significantly reorganized**. The analysis focuses in the main text heavily on UMAP and the MNIST dataset. Framing within the broader ecosystem of neighbor-embedding methods could substantially elevate impact to the state required at top venues such as ICLR. It should be noted that, several analyses to other existing works and dataset reside in the appendix, but the main text does not have enough descriptions to these paragraphs. Ultimately, they are not only unlikely to shape reader perception, but also left in an unorganized state as they do not contribute to a coherent message. Integrating those results into the main narrative would considerably improve the overarching message and strengthen the paper’s contribution.


Decision: Reject