SpaceMAP: Visualizing Any Data in 2-dimension by Space Expansion

Abstract: Dimensionality reduction (DR) and visualization of high-dimensional data is of theoretical and practical value in machine learning and related fields. In theory, there exists an intriguing, non-intuitive discrepancy between the geometry of high-dimensional space and low-dimensional space. Based on this discrepancy, we propose a novel DR and visualization method called Space-based Manifold Approximation and Projection (SpaceMAP). Our method establishes a quantitative space transformation to address the ``crowding problem" in DR; with the proposed equivalent extended distance (EED) and function distortion (FD) theory, we are able to match the capacity of high-dimensional and low-dimensional space, in a principled manner. To handle complex high-dimensional data with different manifold properties, SpaceMAP makes distinctions between the near field, middle field, and far field of data distribution in a data-specific, hierarchical manner. We evaluated SpaceMAP on a range of artificial and real datasets with different manifold properties, and demonstrated its excellent performance in comparison with classical and state-of-the-art DR methods. In addition, the concept of space distortion provides a generic framework for understanding nonlinear DR methods such as t-distributed Stochastic Neighbor Embedding (tSNE) and Uniform Manifold Approximation and Projection (UMAP).

Reviews and author responses:

 Thank you for addressing the comments. I think the changes have improved the paper, but I still think the overall verdict from my side is the same. The paper clearly has an important message, but I think it is still too unclear and vague what exactly separates this from the other methodologies such as T-SNE or UMAP. I would suggest a journal venue, where the proposed method can be investigated further both analytically and experimentally. As it is now, I believe it is too compact and difficult to read. Papers such as this need (in my opinion) to be built up piece by piece with thorough examples and a large empirical section. Even though this is a step closer to an analytical solution to dimensionality reduction, variation in data sets and practical use of such methods still requires significant experimental validation. 

 Thank the authors for the response. The revised paper has been improved, but I think many important details still need further clarification. For example, why does the similarity of $x_i$ and $x_j$ defined in (8) use EED from $R^d$ to $R^{d_{local,i}}$? And why does the similarity of $y_i$ and $y_j$ in (10) use EED from $R^{d_{global}}$ to $R^d$? Are $P_{i|j}$ and $Q_{ij}$ in Fig. 2 the same as the ones in (8) and (10)? (And if so, why put section 3.1.2 before section 3.2?) In general I think these issues are fundamental and need to be explained and justified, so I will keep my score unchanged at 5.

 The revision is much better than the original version. But some details are still needed.

* Statistics of the data sets are missing
* Please list n_near, n_middle, d_global for the tested data sets.


Some detailed comments:
* FDScalingFactor is called at two places in Algorithm 1. Within the loop the call looks like Algorithm 3. But below the loop the call has two arguments. Are they calling the same function?
* In Algorithm 4, in the loop over i, is "n-data" a typo?
* I think the output of HierarchicalManifoldP is a matrix P instead of an entry P_ij. Similarly, the input to OptimizeEmbedding is a matrix P, not just the entry P_ij.

 I think the paper has much improved in some regards, but unfortunately many important things (that were criticized by my and also by other reviewers) remained the same, and a lot of work would still be needed to address them. It's a pity, because I would actually really like to see this published. But I cannot recommend it to publication in the current form.

Things that I raised in my review that were not addressed:
* Missing literature. Citations were not added, but it's not only about inserting citations but also about putting this work in the context of prior research and doing some comparisons. For example, comparison of your Q kernel with the heavy-tailed kernel in t-SNE or UMAP's kernel adjusted $b$.
* Proper discussion of ablation experiments. Thanks for adding them, but now many figures are in the Appendix but do not have any captions, any description anywhere, and are not referred to from the main text. This looks simply unfinished. You have to describe the experiments, draw the conclusions, discuss, etc. 
* Simply eyeballing unnumbered/uncaptioned figures on page 14, it seems SpaceMAP on MNIST and FMNIST is pretty close to UMAP with changed $b$ (the procedure for adjusting it wasn't described) and changed n_neighbors. In other words, panel (c) is close to (b) and panel (e) is close to (d). This suggests that SpaceMAP is pretty close to UMAP with two non-default parameter settings. This needs to be discussed right in the main text! It's very important!
* Ablation experiments for the Swiss roll (page 15) are less clear to me, because here P of SpaceMAP works much better. Why is that? What makes it different in this case? Did SpaceMAP for Swiss roll use exact same hyperparameters as for other datasets? What was the number of neighbors used? Bottom of the page 5 suggests that it may have benefitted from a priori knowledge of dimensionality?
* Lots of implementation things are still never described: preprocessing for the Tasic data, implementations/versions used for tSNE etc., the caption of new Figure 1 just breaks in the middle :-( while the panel (e) looks strange and I would like to know how the authors made it, etc. etc.
* I'm still missing a table with estimated dimensionalities and used numbers of neighbors for all datasets from Figure 4. Isn't the whole point of the paper about adjusting for dimensionality change via space expansion? Then I would suggest to show more results relating to that: what was the dimensionality, how it affected the kernels compared to the Cauchy kernel, etc...


 We gratefully acknowledge the reviewer for the expert review and constructive comments. We also thank the reviewer for the encouraging remarks on our method and results. We have made substantial revisions accordingly to improve and complete the work, documented below:

* Concrete algorithm steps:

We have thoroughly rewritten the Method part (Section 3.1 and 3.2) and hopefully it clarifies many of the questions. We have now added explanation/motivation that FD is merely a way to implement EED, to transform 'distance' to 'similarity', as the latter is related to the objective of optimization. The updated Figure 2 provides an illustration of the concept. For computational purposes, we modeled $P$ to realize equivalent EED in near field and middle field, respectively, while keeping $Q$ in a simple exponential form. This makes it easier to perform differentiation during the optimization process. The equations are also rewritten and all parameters now explained in more detail.

* Definition of $d_{global}$ and $d_{local}$:

We have removed the discussion about $d_{global}$ and $d_{local}$ in the Introduction part. The updated Introduction is now more focused on the existing problem to solve (1. lack of analytical form for distance transformation and 2. difficulty to handle different types of manifolds), and provides the underlying intuition for SpaceMAP. In Section 3.2, we provide two ways to compute  $d_{global}$ and $d_{local}$: (1) use the MLE method with the near neighborhood and middle neighborhood preset (20 and 50, respectively), (2) set them directly if there is prior knowledge (such as in the Swill Roll case, both number equal to 2). We also provide additional experiment in Appendix (Figure 10 and 11) showing that like UMAP, these parameters can be fixed with little influence on the final results.


* Construction of $P$:

We have substantially modified this part and simplified the process (see also the answer to the previous question). The SVM part is now removed. To describe the hierarchical manifold, two intrinsic dimensionalities, namely, $d_{local}(i)$ of the near field of point $i$, and $d_{global}$ of the middle and far field of point $i$, can be calculated based on the MLE method with $n_{near}$ and $n_{middle}$ set. (Alternatively, $d_{local}$ or $d_{global}$ can also be set manually with prior knowledge of data, e.g. the Swill Roll. However in this work, in all experiments we used $n_{near}$ and $n_{middle}$ to calculate the two dimensionality.)


* Title:

We have modified the title to be “Visualizing High-dimensional Data by Space Expansion”.

* Ambiguity in dimensionality definition in Introduction:

We have thoroughly rewritten the Introduction and removed the ambiguous definition of dimensionality. The updated Introduction is more focused on the current problem and intuition behind the proposed method.


 We gratefully acknowledge the reviewer for the expert review and constructive comments. We have made substantial revisions accordingly to improve and complete the work, as documented below:


* Better presentation of the Space Expansion idea:

As suggested we have substantially modified the Introduction in the revision, and provided an explicit example of space expansion from 3D to 2D provided in Figure 1. By projecting a simple Gaussian distribution from 3D to 2D, we observe that SNE (which preserves the distance measure) led to the crowding problem, t-SNE resulted in artificial clusters (across a wide range of perplexities), while SpaceMAP is able to preserve the data distribution because of its principled space expansion.


* Real benefit compared to t-SNE or UMAP:

We have emphasized this point in the revision, in both Introduction and Discussion part. In short, the SpaceMAP has two major benefits: (1) SpaceMAP provides an explicit space transformation between LD and HD, while in contrast t-SNE or UMAP make highly implicit transformation, which may not fit the data (Figure 1 as an example). The explicity of space transformation by SpaceMAP, derived from geometrical measure, potentially better preserves the relative distance in the LD map and makes it possible to enforce data priors (i.e. when the intrinsic dimensionality in some cases is known, such as Swiss Roll). (2) The hierarchical formulation of SpaceMAP takes into consideration the local and global structure of the manifold, and is compatible to a wider range of datasets (disjoint or continuous, e.g. both MNIST and Swiss Roll). Furthermore, we have added more description to interpret the experiment results as suggested.

In the Appendix, we have added more ablation studies to benchmark the SpaceMAP method.

* Discrepancy between LD and HD space: 

We agree that this discrepancy has been explored in the statistical and geometry societies, but not as much in the machine learning society. For dimensionality reduction, furthermore, it is an especially important point to focus on. In previous work, the asymmetry between LD and HD space has been visited by t-SNE and UMAP in the form of “heavy tail”, but only implicitly, without an analytical form. SpaceMAP is the first to make use of this asymmetry analytically and quantitatively. 

* Promise of the "Generic framework": 

In this revised version, the promise of a generic framework has been now delivered in Discussion (2nd paragraph).

"We argue that all successful DR methods, including among others t-SNE and UMAP, make use of the rationale of space expansion to enable data visualization in a space of drastically reduced dimension. However, previous methods did such transformations in a highly implicit manner, with the rules concealed in the self-defined similarity measure (e.g. t-distribution in t-SNE or inverse of polynomial in UMAP, with different parameters). Despite their empirical success, we posit that an analytical form of distance transformation is desirable to deal with situations where we would like to take more control of the DR results or impose prior knowledge for DR."

* Clarity in function distortion part (FD): 

We have thoroughly rewritten this part, including the underlying motivation (for computation efficiency and differentiality of the loss function) and detailed steps. More pseudo code is provided in Appendix. We have also made an improved illustration for FD (figure 2).


* Haussdorff measure: 

To avoid confusion we have removed the term.


* MLE:

We have removed the statement that MLE is widely accepted. We agree that this is an open problem warranting further research effort. The exponential term in MLE concords to our EED definition, as a natural result of the exponentially increasing capacity.


* Typo and language:

We have corrected typos and proofread the manuscript.


 We gratefully acknowledge the reviewer for the expert review and constructive comments. We thank the reviewer for pointing us to many interesting, relevant, and inspiring work. We have made substantial revisions accordingly to improve and complete the work, documented below:

* About clarity of algorithm and details:

We have thoroughly rewritten the Method part (Section 3.1 and 3.2) and hopefully it clarifies many of the questions. We have now added explanation/motivation that FD is merely a way to implement EED, to transform 'distance' to 'similarity', as the latter is related to the objective of optimization. Figure 2 provides an illustration of the concept. For computational purposes, we modeled $P$ to realize equivalent EED in near field and middle field, respectively, while keeping $Q$ in a simple exponential form. This makes it easier to perform differentiation during the optimization process. The equations are also rewritten and all parameters now explained in more detail.

* About motivation (what problems to solve):

We have substantially rewritten the Introduction to provide a clear motivation of SpaceMAP. In short, there are two problems we strive to solve: (1) Although existing methods such as tSNE and UMAP successfully address the crowding problem by imposing heavier tail in the similarity measure of LD, the implicity in such a transformation makes it difficult to perform quantitative validation or impose priors. (2) It remains difficult to address different manifolds (disjoint or continuous) in one DR framework.

* About the two modifications and ablation study:

We agree with the reviewer that our work is orientated on two major modifications: (1) An explicit way to transform distance based on estimate of intrinsic dimensionality, (2) A multi-scale way to compute high-dimensional similarity, as induced by the hierarchical structure in data. We also agree that it is insightful to perform ablation studies to demonstrate the benefits of each. In the revised manuscript, we have added more experiments (due to space limitation, we place some of them in Appendix). 

For the first point, Figure 1 gives a simple example showing why explicitly calculated distances in SpaceMAP can be beneficial: it better preserves the distribution given the exact match of space capacity; in contrast, tSNE or UMAP cannot effectively control the extent of distortion and may lead to artificial clusters. Ablation studies are provided in the Appendix by combining different $P$ and $Q$ of SpaceMAP or UMAP for the MNIST, fashion-MNIST, Swill Roll, respectively. 

For the second point, different treatment of different points can be essential to deal with manifolds of different properties. Figure 4 shows examples of both disjoint (MNIST) and continuous (Swill Roll) manifolds, where SpaceMAP yielded overall good results. Besides, we also provided ablation studies in Appendix showing how different choice of $n_{near}$ and $n_{middle}$ (hence different intrinsic dimensions used to calculate the multi-scale $P$) impact the final results.

* About hyper-parameters of SpaceMAP:

The most important hyper-parameters of SpaceMAP is the number of nearest neighors in two fields: $n_{near}$ and $n_{middle}$, which are used to estimate the intrinsic dimensions. It also has a parameter $\eta$ in its definition of $P$. We have provided additional experimental results in Appendix (Figure 10 and 11), showing that the results are not very sensitive to the choice of these hyper-parameters. We have also added the point in Discussion.

* About implementation details in UMAP:

We also noticed the tricky details in UMAP's implementation of SGD. We agree with the authors (https://arxiv.org/pdf/2103.14608.pdf) that the negative sampling (randomly from the dataset instead from the KNN) mainly intends to exert a repulsive force to balance the attraction and repulsion. (Indeed UMAP does not exactly optimize the loss as presented in the original paper.) The approximation of $1-P_{ij}$ with $1$ has little numerical influence though as $P_{ij}$ is very small in the case of negative sampling. 

* About the intrinsic dimension of datasets:

In our calculation with predefined neighborhood, the intrinsic dimension of MNIST is approximately 6, and RNA-seq is approximately 8. However we also noticed that slightly change the number does not significantly change the final results.


* About typo, figure, and text:

We have proofread the text and corrected typos. We have also redrawn many figures to make them better readable.


 We gratefully acknowledge the reviewer for the expert review and constructive comments. We have made substantial revisions accordingly to improve and complete the work, documented below:

* Definition 3.1 (Comment 1):

We agree that HD data normally lie on a complex, highly nonlinear manifold. We did not assume that data manifolds form an N-dimensional hypersphere, instead, Definition 3.1 introduces the space capacity concept that underlies the EED theory. In an N-D space, no matter what manifolds the data lie on, the definition of space capacity remains the same, i.e., the higher the dimensionality, the (exponentially) higher the capacity to express data, independent of the exact manifold and data distribution.

* Definition 3.2 (Comment 2):

Because the volume of a space decreases exponentially with dimensionality reduction, in LD there is a deficiency of expressiveness if it is to display HD data. That is the underlying reason to extend the LD distance to the match the expressiveness of HD space (such that they are not crowded, i.e. to solve the “crowding problem”). We have added more elaboration on this point in the revision.

* Experiment design (Comment 3):

We have added more experiments to validate the EED idea. In introduction we have added an intuitive example (Figure 1) to map data from 3D to 2D while preserving its distribution properties. More experimental results are provided in Appendix (Figure 8, 9, 10, 11) to show the properties of SpaceMAP in comparison to UMAP, and its sensitivity to the choice of hyper-parameters.  

* Unfair comparison and unsupported statements of UMAP/TriMAP (Comment 4):

We have removed the statement about UMAP and TriMAP better preserving global structures, and we agree it is not supported, either theoretically or empirically.
 
* Typo and language (Comment 5): 

We have corrected typos and proofread the manuscript.


 We gratefully acknowledge the reviewer for the expert review and constructive comments. We have made substantial revisions accordingly to improve and complete the work, documented below:

* Clarity in function distortion part (FD) and related equations: 

We have thoroughly rewritten this part in Section 3.1.2 and Section 3.2, including the underlying motivation (for computation efficiency and differentiality of the loss function) and detailed steps. We have also made an improved illustration for FD (Figure 2). The equations were rewritten and all parameters now explained in more detail. The operation in equation 8 amounts to hierarchical space extension in the near field and middle field. The definition is in a large part for the ease of computation, as by modeling $P$ instead of $Q$, we keep the expression of $Q$ as simple as possible for differentiality during optimization.

* Definition of $d_{global}$ and $d_{local}$:

We have removed the discussion about $d_{global}$ and $d_{local}$ in the Introduction part. Thanks for pointing out the confusion. The updated Introduction is now more focused on the existing problem to solve (1. lack of analytical form for distance transformation and 2. difficulty to handle different types of manifolds), and provides the underlying intuition for SpaceMAP. In Section 3.2, we provide two ways to compute  $d_{global}$ and $d_{local}$: (1) use the MLE method with the near neighborhood and middle neighborhood preset (20 and 50, respectively), (2) set them directly if there is prior knowledge (such as in the Swill Roll case, both is 2). We also provide additional experiment in Appendix (Figure 10 and 12) showing that like UMAP, these parameters can be fixed with little influence on the final results.

* MLE:

We have added more formal introduction of the MLE method. It is in principle a method to estimate the intrinsic dimension of a local data neighborhood based on the distribution of neighbor distances. This concept concords to our space expansion theory based on space capacities, and naturally leads to Lemma 3.2, showing that EED can effectively alter the intrinsic dimensionality of a local neighborhood and therefore realize LD mapping.

* Typo and language: 

We have corrected typos and proofread the manuscript.


The paper presents a new solution to dimensionality reduction and data visualization. They define an explicit extension of the pairwise distances in the lower dimensional space, the equivalent extended distance (EED), which is proportional to the reduction in volume when going from a high dimensional space to a lower-dimensional space, This helps to avoid the crowding problem known from T-SNE, UMAP, etc. and could provide a better understanding of dimensionality reduction methods in general. In addition, they define a distortion framework that allows more efficient implementation of the EED. Finally, they introduce a hierarchical scheme for manifold approximation, where instead of splitting the problem into ambient space and data manifold, they add a third layer, the near field. This is similar to UMAP, but they use a Gaussian function instead of a uniform for the near field. Experiments are thorough and the visualizations behave as expected, with the data being more spread out compared to t-sne and UMAP. A quantitative analysis using recently proposed metrics is also provided. This is a reasonably well-written paper, and the idea of modeling dimensionality reduction by directly compensating for the loss in volume is interesting and could provide a deeper understanding of dimensionality reduction methods that are based on solving optimization problems. However, I believe that in its present form, the paper seems unfinished and is not ready for publication. 
A more clear explanation of the intuition behind the novel idea is needed. The authors spend a  great deal of effort to illustrate the counterintuitive effects of high dimensional geometry and state of the art results, but they fail to give a clear picture of how the new approach fits in, other than that they “expand the capacity” of the low dimensional space. An explicit example of the space expansion from 2D and 3D, instead of Figure 1, would definitely help the reader.  

Furthermore, what is the real benefit of using this method compared to t-sne or UMAP? Some figures in the experiment sections gives us a hint that seems promising, such that it behaves better for continuous and disjoint manifolds, and that the geometry is preserved better (swiss roll example), but this is barely mentioned in the text and leaving the interpretation entirely to the reader is not sufficient I believe. The promise of a generic framework for understanding nonlinear dimensionality reduction is not delivered in the paper as it is presented now. I would like to see a concrete example where this is shown. The measures used are not sufficient to give a clear picture of this. A combination of the figures and tables gives us some idea, but, again, they are barely mentioned in the text. E.g. For very large datasets, the increased scale and distance between the data points with this method is expected, but how does this concretely relate to the figures and how can we interpret this together with the metrics in Table 1. How do extended distances behave when the dimension is extremely high?

To conclude, I think that the paper needs a clearer presentation of the main idea as well as a better connection between the results of the experiments and the claims in the introduction. 

General comments: 

Isn’t the discrepancy between very high and low dimensionality already fully explored?

The loss in section 3.1 is not explained and should be properly introduced or a reference to the section where it is should be included.

The paper is well written, but in some parts I find the language a bit oral and verging on being too vague, especially the abstract and introduction.

The Haussdorff measure is perhaps not well known and could be described further. Especially since this is in a ML/CS setting.

Typo in section 3.2.2: based ON the factors

The FD is not explicitly defined as far as I can see, just indirectly via the EED, which is slightly confusing for the reader.
typo: 3.3 avoid the comput?

I do not believe that the MLE method for identifying the intrinsic dimension is necessarily widely accepted. Many others exists and it remains an open problem, so I would avoid leaning too much on this method.

The conclusion contains repetition that should be avoided (first and last paragraph are essentially the same).

 The paper presents a new solution to dimensionality reduction and data visualization. They define an explicit extension of the pairwise distances in the lower dimensional space, the equivalent extended distance (EED), which is proportional to the reduction in volume when going from a high dimensional space to a lower-dimensional space. To efficiently implement the EED, they introduce a distortion-based framework. Finally, they introduce a hierarchical scheme for manifold approximation, where instead of splitting the problem into ambient space and data manifold, they add a third layer, the near field.

This work studies dimensionality reduction and visualization of high-dimensional data. The idea is to construct two graphs in the original high-dimensional space and the low-dimensional space, respectively, based on space expansion and hierarchical manifold approximation. Space expansion is to map the distance in high-dimensional space to an equivalent extended distance (EED) in low-dimensional space such that the volumes of the two (hyper-)spheres in the two spaces are equal. Hierarchical manifold approximation divides the points into near field, middle field, and far field, and assigns EED based similarity for points in the near field. Then a problem minimizing the sum of a KL divergence and a repulsive force term is solved by SGD to determine the projected coordinates. In numerical experiments, the effectiveness of the proposed method is demonstrated by comparing with some baseline methods on synthetic and real-world datasets. Strengths:
1.	The idea of defining EED to preserve the volume after dimension reduction seems novel and well-motivated.
2.	There are not too many tuning parameters in the algorithm, and in some of the experiments the proposed algorithm SpaceMAP is less sensitive to the choice of the number of nearest neighbors $k$ than the comparing methods.
3.	In experiments SpaceMAP is shown to be able to handle both continuous manifold and disjoint manifolds.

Weaknesses: 
1.	One of the key points in SpaceMAP is to define the similarity based on EEDs, as given in (8) and (10). However, it is not explained why the EED used in (8) is inversed. If $\alpha_t$ and $\beta_t$ are factors from the ambient space to the intrinsic space, then the EED in (8) should be $\alpha_tR_{ij}^{\beta_t}$, as given in the paragraph after Example 3.1.
2.	I find the definition of d_global and d_local not clear. In the second to last paragraph in section 1, d_global is the dimension of the manifold, and d_local is the dimension of a local neighborhood. However, for a manifold its dimension is defined as the dimension of an open subset that a neighborhood around each point on the manifold is homeomorphic to, so under such common definition d_global and d_local seem the same.
3.	The MLE of intrinsic dimension in Theorem 3.1 and Lemma 3.2 is not defined or formally introduced. Also, the assumption in Theorem 3.1 is not clear enough. Does $R \leq R_{ij}$ means that $R$ is required to be smaller than $R_{ij}$ for all $j=1, …, k$?
4.	Section 3.2.2 seems very unclear, because it tries to explain distortion of similarity functions without defining the similarity functions. In Figure 2 (a), $f$, $\tilde{f}\_{d\to D^{\prime}}$, and $\tilde{F}\_{D \to d}$ are never defined, and $P_{i|j}$, $Q_{ij}$, and the near field are only defined in the next section. 
5.	The sentence after equation (7) is not finished.
6.	What does the subscript $t$ in equation (8) denote? The so-called FD factors $\sigma_{t,i}$ in (8) and $\sigma$ in (10) are not defined.
 The proposed algorithm has certain novelty and performs well in the experiments. However, several issues need to be addressed to convince the readers of the technical soundness, as pointed out in the main review, so my current recommendation is that this paper is marginally below the acceptance threshold.

This paper proposes Space-based Manifold Approximation and Projection (SpaceMAP) method to solve two existing issues in dimensionality reduction task: (1) capacity matching between high-dimensional data space and low-dimensional visualization space. (2) complexity in various manifold properties. To address these issues, SpaceMAP proposes two ideas: (1) space expansion (Section 3.2): inferring equivalent extended distance (EED) (Definition 3.2) for low-dimensional space based on volume matching and introducing function distortion (FD) (Section 3.2.2) to transform the similarity functions in high-dimensional and low-dimensional space; (2) hierarchical manifold approximation (Section 3.3): making distinctions between the near field, middle field, and far field of data distribution in a data-specific hierarchical manner. SpaceMAP achieves competitive performance on a diversity of datasets. Strengths:
This paper provides an analysis of discrepancy in space volumes in high-dimensional data space and low-dimensional visualization space and the complexity of high-dimensional manifold in the visualization task, and proposes space expansion (Section 3.2) and hierarchical manifold approximation (Section 3.3) solutions. These contributions help further development of nonlinear dimensionality reduction and visualization techniques based on manifold learning.

Weakness:
1. A pattern in a high dimensional data space may lie on a complex manifold with a highly nonlinear structure. The assumption that manifolds form N-dimensional hyperspheres (Definition 3.1) is un-supported. 
2. By definition, a manifold is a lower dimensional surface embedded in the high dimensional data space and the volume increases exponentially as the dimensionality increases. However, no reasons are given as to why to impose the volume-based EED (Definition 3.2) as the a distance measure to justify the correctness of the proposed capacity matching.
3. The experimental design does not validate the effectiveness of the proposed equivalent extended distance (Section 3.2.1) and hierarchical manifold approximation techniques (Section 3.3). Additional ablation study experiments are needed to demonstrate the effectiveness of these techniques.
4. The experimental comparisons are unfair (Table 1). The experimental results do not match the statement “UMAP and TriMAP are reported to better preserve the global structure than T-SNE” mentioned in related work section. The global metric results (Table 1) of UMAP are inferior to those of t-SNE by far. The hyper-parameters of the comparison methods need to be adjusted for different databases, e.g., “perplexity” in t-SNE, “n_neighbors” and “min_dist” in UMAP.
5. There are some typos, e.g., t-distributed Stochastic Neighbor Embedding (tSNE) in Abstract, Local Metrics (M_s and M_σ) in Table 1.
 The foundation of the main idea (capacity matching) is flawed. The experimental method lacks appropriateness, and the results are inadequate to prove the claimed effectiveness of the proposed method.

This paper suggests what it calls a novel dimensionality reduction and data visualization method (SpaceMAP), but perhaps it would be more precise to say that they suggest two modifications to UMAP. One modification is theoretically motivated low-dimensional similarity kernel (that depends on the estimate of the input dimensionality). Another modification is multi-scale high-dimensional similarity kernel. The authors argue that their method performs better than existing methods on a variety of different datasets.

The first part about theoretically grounded low-dimensional similarity kernel (which seems to be the main focus of the paper) I found interesting. However, I feel that combining two very different UMAP modifications together made the paper weaker because it is not clear what effect do they have on their own. I'd much rather see more experiments that really elucidate the "space expansion" idea and effect. Also, the paper is missing a lot of relevant context/citations. Overall I feel that the work has potential but the paper is not really ready yet. I am giving a lower-borderline score. This paper suggests what it calls a novel dimensionality reduction and data visualization method (SpaceMAP), but perhaps it would be more precise to say that they suggest two modifications to UMAP. One modification is theoretically motivated low-dimensional similarity kernel (that depends on the estimate of the input dimensionality). Another modification is multi-scale high-dimensional similarity kernel. The authors argue that their method performs better than existing methods on a variety of different datasets.

The first part about theoretically grounded low-dimensional similarity kernel (which seems to be the main focus of the paper) I found interesting. However, I feel that combining two very different UMAP modifications together made the paper weaker because it is not clear what effect do they have on their own. I'd much rather see more experiments that really elucidate the "space expansion" idea and effect. Also, the paper is missing a lot of relevant context/citations. Overall I feel that the work has potential but the paper is not really ready yet. I am giving a lower-borderline score.


MAJOR ISSUES

* Multiple details of the algorithm are not sufficiently clear: 

a) Is it correct that SpaceMAP is UMAP with exactly two modifications: high-dim similarities are computed in a multi-scale way (Eq 8), and low-dim similarity kernel is replaced with Eq 10 that depends on the estimate of the input dimensionality? Or are there any other more subtle changes as well? This should be stated crystal clear.

b) Appendix A1 -- I did not understand this procedure, it was not clear to me what exactly is being predicted by the SVM. Also, I'd rather see this in the main text on page 5. 

c) Page 5: "n_middle is set to avoid the comput" -- the sentence is cut off and the details are missing.

d) Algorithm 1: gives an equation for n_middle, but it is not motivated and unclear.

e) Page 6: "where alpha_t, beta_t" -- what are subscripts "t"?

f) Same line: what is "sigma_t,i"? How is it an "FD factor" when section 3.2.2 does not mention sigma? 

g) In Eq (10) you use alpha and beta without subscript i. How are these obtained? Also, again, what is sigma?

* It's unclear what effect do the two modifications have on their own. I would like to see some analysis of them separately. 

* Does using multiscale P_ij similarities make any difference compared to standard UMAP (with suitably chosen n_neighbors)? These two papers argue that exact P_ij values in UMAP do not matter at all: https://arxiv.org/abs/2007.08902 and https://arxiv.org/abs/2103.14608.

* The similarity kernel in Eq (10) should be compared to the standard similarity kernels in UMAP, SNE, t-SNE. Moreover, in t-SNE, one can choose t-distribution kernel with various degrees of freedom, see https://link.springer.com/chapter/10.1007/978-3-030-46150-8_8 and https://papers.nips.cc/paper/2009/hash/2291d2ec3b3048d1a6f86c2c4591b7e0-Abstract.html. The former paper shows that it has nontrivial effect on the embeddings. In UMAP, one can adjust the kernel via parameter b. So if you use default UMAP's P_ij and only apply your kernel from Eq (10), what effect does it have? Can one achieve similar effect by adjusting b in standard UMAP?

* What actual values for the intrinsic dimensionality do you obtain for the datasets you analyze? MNIST, RNAseq, COIL, etc? What values of alpha and beta do you then have for Eq (10)? How different is the resulting kernel from standard t-SNE/UMAP?  

* Runtime is never mentioned. The runtime of UMAP heavily depends on n_neighbors. How many non-zero P_i|j values do you end up with on average when following your procedure for your datasets? How does this affect the runtime?


MEDIUM ISSUES

* Sections 1+2 review the the prior related work but do not really emphasize what exactly are the shortcomings of the existing work. So by the time I read Sections 1+2, it was not really clear to me what *problem* the authors are trying to solve in this manuscript.

* The authors could acknowledge and comment on some the debate/discussion about UMAP, e.g. in Section 2. For example https://www.nature.com/articles/s41587-020-00809-z argued that the claims of better global structure preservation than t-SNE are mainly due to initialization. More importantly, https://arxiv.org/abs/2007.08902 and https://arxiv.org/abs/2103.14608 argued that UMAP is not really optimizing its stated loss function (cross-entropy), due to negative sampling. Given that SpaceMAP is based on UMAP, this seems relevant to at least somehow mention.

* Section 3.2.2, first sentence -- unclear.

* Section 3.3 -- this multiscale procedure with "near" and "middle" points reminds me of https://jmlr.org/papers/v22/20-1061.html and also of multi-scale kernels in https://www.nature.com/articles/s41467-019-13056-x. Some context would be good to have.

* Paragraph after Eq 13 -- this is unclear. https://arxiv.org/abs/2007.08902 claims in Discussion that negative sampling used in UMAP is not really similar to SGD. Do you agree? 

* Section 4.1 -- please list all implementations that you used, give exact versions.

* Section 4.1 -- how did you preprocess the Tasic2018 data? Gene selection? Any transformations? PCA reduction prior to t-SNE/UMAP/etc? This absolutely needs to be described. Did you follow some other paper in how this dataset was preprocessed? Then cite. Also describe all these details for all other datasets.

* Section 4.1 -- how did you set hyperparameters of SpaceMAP?

* Section 4.2, last line -- UMAP is not always faster than t-SNE, see e.g. https://www.nature.com/articles/s41467-019-13056-x.

* Most differences in Table 1 are very small, e.g. local metrics are almost the same for t-SNE/UMAP/SpaceMAP. But in fact t-SNE is known to perform much better than UMAP for kNN preservation for small k (like k=10 or k=15), see e.g. Figure 7 in https://arxiv.org/abs/2007.08902. The difference between t-SNE and UMAP there is large. Not sure why this is not seen in your Table 1?


MINOR ISSUES

* The font size in many figure panels is completely unreadable (e.g. Fig 1d, Fig 4, Fig 6). Please make sure everything is readable when printed out.

* page 4, top: "D-dimensional hyper-sphere" --  I think you mean D-dimensional ball, not sphere?

* Eq 11 -- in standard UMAP implementation, the (1-P_ij) term is simply approximated with 1. If you do the same, it's worth a comment.
 The first part about theoretically grounded low-dimensional similarity kernel (which seems to be the main focus of the paper) I found interesting. However, I feel that combining two very different UMAP modifications together made the paper weaker because it is not clear what effect do they have on their own. I'd much rather see more experiments that really elucidate the "space expansion" idea and effect. Also, the paper is missing a lot of relevant context/citations. Overall I feel that the work has potential but the paper is not really ready yet. I am giving a lower-borderline score.

This paper presents a new similarity-based Nonlinear Dimensionality Reduction (NLDR) method for different types of data. The proposed method focuses on more deliberated construction of the input and output similarities.
 The experimental results are impressive. Especially, this is the first method that can achieve both good clustering and unfolding the synthetic swiss-roll data. 

However, the presentation of the methodology is awful. It is almost impossible to reproduce the algorithm and results.

If the following points can be correctly addressed and revised, I am still willing to accept the paper.

The major problem: it is hard to connect the fancy theory to concrete algorithm steps.
* There is no submitted code. The pseudocode is too coarse, with many details missing or hidden.
* It is unclear how to get alpha, beta, and sigma. Without source code, the authors should at least provide the necessary formulas.
* It is unclear how to get gamma in Eq. 8 as well
* The functions FD and FDInverse are not defined
* I don't find the meaning of the subscript t.
* What are the differences (and connections) among sigma, sigma_i, and sigma_{i,t}?
* In the formula \tilde{R}_{ij,D\rightarrow d} = alpha R^{beta}, where is there ij on the left but not on the right?
* In the text below Eq. 7. "n_middle is set to avoid the comput". It seems the sentence is incomplete. Please carefully proofread your paper!

The construction of P could be problematic.
* They tried to automate the choice of n_near by one-class SVM. But one-class SVM itself contains hyperparameters.
* The statistics about n_near(i) over i's should be added
* Besides KNNs (i.e., S_{i, near}), the P construction requires more non-zero entries from S_middle, which could increase the complexity. Especially, n_middle is the average of n_near(i)'s, which can become quite large (when one n_near(i) is very large). Discussion about the time and space complexity should be added.
* It is unclear why P_{j|i} in S_middle takes the form in Eq. 8


Other comments:
* The term "any data" in the title is overselling. There are many types of data not covered in the paper (e.g., graphs). When the similarities and distances between data objects are unavailable, the proposed method doesn't work either. For example, there is no simple distance function between two natural images (consider ImageNet). So removing the data scope actually makes the title non-informative.
* Page 2, the definitions of d_ambient, d_global, and d_local are unclear. What do you mean "the dimension of measurement"? Why do you say d_ambient >> d_global? Why is d_local a single number?
* Are Definitions 3.1 and 3.2 new? If no, it is better to give citations.
* In Algorithm 1, i is already used as subscripts for P_ij. But it is also the index for epochs. Is this a typo? The experimental results are impressive, with certain breakthroughs in NDLR. However, the presentation is poor and needs significant modification.

Meta-review: Reviewers overall found that the paper contains novel and intriguing ideas worth further investigation. There is, however, a consensus that the paper is not ready to be published yet, for several reasons detailed in the reviews pertaining to 1) the fact that several statements should be better supported theoretically or empirically, 2) the technical derivation of the method where several choices made by the authors are surprising and not justified, and 3) the experimental results that do not clearly support the claims of the manuscript. While the authors have improved the manuscript during the discussion phase, there is still too much work to be done in order to address issues remaining. We hope the reviews will be helpful for authors to consider a revision of the paper for a future submission.

Decision: Reject