Generalizable Spectral Embedding with an Application to UMAP
Authors: Nir Ben-Ari
Amitai Yacobi
Uri Shaham
Venue: Accepted by TMLR ()

Abstract: Spectral Embedding (SE) is a popular method for dimensionality reduction, applicable across diverse domains. Nevertheless, its current implementations face three prominent drawbacks which curtail its broader applicability: generalizability (i.e., out-of-sample extension), scalability, and eigenvectors separation. Existing SE implementations often address two of these drawbacks; however, they fall short in addressing the remaining one. In this paper, we introduce $\textit{Sep-SpectralNet}$ (eigenvector-separated SpectralNet), a SE implementation designed to address $\textit{all}$ three limitations. Sep-SpectralNet extends SpectralNet with an efficient post-processing step to achieve eigenvectors separation, while ensuring both generalizability and scalability. This method expands the applicability of SE to a wider range of tasks and can enhance its performance in existing applications. We empirically demonstrate Sep-SpectralNet's ability to consistently approximate and generalize SE, while maintaining SpectralNet's scalability. Additionally, we show how Sep-SpectralNet can be leveraged to enable generalizable UMAP visualization.

Published in Transactions on Machine Learning Research (07/2025)
Generalizable Spectral Embedding with an Application to UMAP
Nir Brn-Ari Deportment of Computer Science Bor-Ilan University
nimniro@@gmail.com com
Amitai Yacobi
amwkaaiyooroiii@mmaal.coo com
Department of Computer Science
Bor-Bln University
Uri Shaham
uri.shaham Obiu. il
Department of Computer Science
Bar-Ilan University
Reviewed on OpenReview: https: // openreview. net/ forum? id- ScuQuzt CKk
Abstract
Spectral Embedding (SE) a popular method for dimensionality reduction, applicable across diverse domains. Nevertheless, its current implementations face three prominent drawbacks which curtail its broader applicability: generalizability (i..,, out-of-sample exten- sion), scalability, and eigenvectors separation. Existing SE implementations often address two of these drawbacks; however, they fall short in addressing the remaining one. In this paper, we introduce Sep-SpeclralNel leigenvector-separaalee SpectralNet), SE implementa: tion designed to address all three limitations. Sep-SpectralNet extends SpectralNet with an efficient post-processing step to achieve cigenvectors soparation, while ensuring both gener- alizability and scalability. This method expands the applicability SE to wider range of lasks and can enhance its performance in existing applications. We empirically demonstrate Sep-SpectralNet's ability to consistently approximate and generalize SE, while maintaining SpectralNet's scalability. Additionally, we show how Sep-SpectralNet can be leveraged to enable generalizable UMAP visualization
1 Introduction
Spectral Embedding (SE) is a popular non-linear dimensionality reduction method (Belkin & Niyogi 2003
Coifman & Lafon 2006b finding extensive utilization across diverse domains in recent literature. Notable
thot--- tho-art visualization mothod), Graph Neural Networks (Zhang el al. 2021 Beaini et al. 2021 Defferrard el al., 2016),, signal propagation on graphs
applications include UMAP (MeInnes et al. 2018 (the current state-of
(Park et al.) 2022 Klaine al. 2017) and analysis of proteins (Campbell et al. 2015 Zhu & Schlick 2021) The core of SE involves projection of the samples into the space spanned by the leading cigenvectors of the Laplacian matrix (i.e., those corresponding to the smallest eigenvalues), derived from the pairwise similarities between the samples. It is an expressive method which is able to preserve the global structure of high-dimensional input data, underpinned by robust mathematical foundations (Belkin & Niyogi 2003
Katz et al. 2019 Lederman Talmon 2018 Ortega et al. 2018
Despite the popularity and significance of SE, current implementations suffer from three main drawbacks: (1) Generalizability the ability to directly embed new set of test points after completing the computation
Seep-ppectralee:: https://github. com/ wtw.shhaaaaaba-EAASS 1ab/GrEASE
NUMAP: https://github. com 1abb Jab/NUMAP

---

Published in Transactions on Machine Learning Research (07/2025)
training set (i.e., out-of-sample extension); (2) Scalability the ability to handle a large number of samples within a reasonable time-frame; (3) Figenwectors separation the ability to output the basis of the leading eigenvectors (U2,..., pk+1), rather than only the space spanned by them. These three properties are crucial for modern applications of SE in machine learning. While most SE implementations address two of these three limitations, they oflen fall short in addressing the remaining one (see Tab. and Sec.
Notably, eigenvector separation has attracted considerable attention in recent years Pfauu et al., 2018 Gemp et al., 2020 Deng et al., 2022 Lim et al., 2022). This work focuses on this property, which is key aspect underlying applications such as Fiedler vector, Diffusion maps, and Diffusion analysis. The latter two involve examining the evolution of random walks on graphs across different time scales. This framework enables a wide range of applications: analyzing molecular transitions (Glielmo et al.) 2021 Chiavazzo et al., 2017).
sampling rare protein transitions (Ghamari et al. 2024), uncovering latent structures in disordered mate-
rials (Hardin et al. 2024), studying the global organization of cortical features in various neuropsychiatrie
conditions (Park et al.) 2021 2022 Dong et al.) 2020), supporting eahancod-sampling techniques (Zheng
et al. 2011 2013); ; and facilitating dimensionality reduction in sell-organizing networks (Klaine et al. 2017
Eigenvector separation is required for those applications to enable scaling each diffusion component.
Shaham et al. 2018) presented SpectralNet, which tackles the scalability and generalizability limitations of Spectral Clustering (SC), a key application of SE. However, we prove that due to a rotation and reflection ambiguity in its loss function, SpectralNet cannot directly be adapted for SE in general, as cannot sep- arate the eigenvectors. In this paper, we present an eigendecomposition based post-processing procedure to resolve the eigenvectors separation issue in SpectralNet, thereby, extending SpectralNet into a scalable and generalizable implementation of SE,, which we call Sep-SpectralNet (eigenvector-seppraaae SpectralNet). Sep-SpectralNet's s ability to separate the eigenvectors, while maintaining the generalizability and scalability of SpectralNet, offers a pathway to enhance numerous existing applications of SE, that require eigenvector separation, and provides a foundation for developing new applications.
One such application is UMAP (Melnnes et al. 2018 a widely used visualization method whose ability to preserve global structure relies heavily on its Spectral Embedding (SE) initialization (Kobak Linderman 2021). Although Parametric UMAP (P. UMAP) was proposed to address UMAP's lack of generalizability (Sainburg et al. 2021), it omits SE, limiting its ability to replicate UMAP's structural fidelity. Despite this
limitaton, P. UMAP has gained traction in various domains (Xu & Zhang 2023 Eckelt el. al. 2023 Leon-
Medina et al. 2021 Xie et al.] 2024 Yoo et al.] 2022), underscoring the need for a generalizable alternative
that retains UMAP's structural strengths. To this end, we introduce an extension of Sep-SpectralNet that integrates SE- based initialization with the UMAP loss, named NUMAP, preserving global structure while enabling generalization. We show that eigenvector separation is required for NUMAP's performance. As shown empirically (Sec. 5), this extension enhances UMAP's applicability to dynamic settings such as online learning and time-series data visualization.
Our key contributions are: (1) We introduce Sep-SpectralNet, a novel approach for approximate Spectral Embedding (SE) that jointly addresses scalability, gencralizability, and cigenvector scparation. (2) We present an application of Sep-SpectralNetlo generalizable UMAP. (3) We propose a new evaluation hod to measure global structure preservation.
2 Related Work
Current SE implementations typically address two out of its three primary limitations: generalizability, scalability, and eigenvector separation (Tab. D. Below, we outline key implementations that tackle each pair of these challenges. Following this, we discuss recent works related to eigenvectors separation and generalizable visualizations techniques.
Scalable with eigenvectors separation. Popular implementations of SE are mostly based on sparse ma- trix decomposition techniques (e.g., ARPACK (Lehoueq et al. 1998), AMG (Brandt et al. 1984), LOBPCG [Benner & Mach 2011)). These methods are relatively scalable, as they are almost linear in the number of samples. Nevertheless, their out-of-sample extension is far from trivial. Usually, is done by out-of-sample extension (OOSE) methods such Nyström Nystrrm 1930) or Geometric Harmonics (Coifman & Lafon
2

---

Published in Transactions on Machine Learning Research (07/2025)
Method
Generalizability
Scalability
Eigenvector Separation
LOBPCG
x
SpectralNct
X
DiffusionNet
X
Sep-SpectrallNet (ours)
Table 1: Sep-SpectralNet the only method to have the three desired properties of SE imple- mentation. Comparison between key SE methods via their ability to generalize to unscen samples, scale to large datasets and separale the eigenvectors.
2006a;
Lafon et al., 2006). However, these methods provide only local extension (i.c., near existing training points); ; and are both computationall and memory restrictive, as they rely on computing the distances between every test point and all training points.
Scalable and generalizable. Several approaches spectral clustering (SC) approximate the space spanned by the first eigenvectors of the Laplacian matrix, which is sufficient for clustering purposes, and also benefit other specific applications. For example, SpectralNet (Shaham et al., 2018 leverages deep neural networks to approximate the first eigenfunctions of the Laplace-Beltrami operator in a scalable man- ner, thus also enabling fast inference of new unseen samples. BASIS (Streicher et al. 2023 achieves these goals using affine registration techniques to align batches. However, these methods' inability to separate the eigenvectors prevents their use in many modern applications.
Generalizable with eigenvectors separation. Another proposed approach to SE is DiffusionNet (Mishne et al.] 2019), a deep-learning framework for generalizable Diffusion Maps embedding (Coifman & Laffo 2006b), which is similar to SE. However, the training procedure of the network is computationally expensive, therefore restricting its usage for large datasels.
In contrast, we introduce Sep-SpectralNet, which addresses all three limitations generalizes the separated eigenvectors to unseen points with single ford-forward operation, while maintaining SpectralNet's scalabil- ity.
Eigenvectors separation. Extensive research has been conducted on the eigenvectors separation problem, both within and beyond the spectral domain (Lim et al. 2022 Ma et al., 2024h. Some rotation criteria such as ICA and VARIMAX are well known, but regarding the spectral domain, they do not yield the natural separation, i.e., the true eigenvectors. Recent spectral approaches remain constrained computationally, both by extensive run-time and memory consumption. For example, Pfau et al. (2018) proposed a solution to this issue by masking the gradient information from the loss function. However, this approach necessitates the computation of full Jacobians each time step, which is highly computationally intensive. Gemp et (2020) employs an iterative method to learn each eigenvector sequentially. Namely, they learn an eigenvector while keeping the others frozen. This process has to be repeated & times (where l is the embedding dimension). which makes this approach also computationally expensive. Deng et al. (2022) proposed improvement of the latter, by parallel training of k NNs. However, as discussed in their paper, this approach becomes costly for large values of k. Furthermore, it necessitates retaining A trained networks in memory, which lcads to significant memory consumption. Chen et al. (2022) proposed a post-processing solution to this problem using the Rayleigh-Ritz method. However, this approach involves the storage and multiplication of very large dense matrices, rendering impractical for large datasets. In contrast, Sep-SpectralNet offers an efficient onc-shot post-processing solution to the cigenvectors scparation problem.
Generalizable visualization. Several works have attempted to develop paramctric approximations for non-pa non-parametric visualization methods, in addition to Parametric UMAP (P. UMAP) (Sainburg et al. 2021
Notable examples include (Van Der Maaten 2009), (Kaaase et al. 2022 and Damrich et al. 2022 which
3

---

Published in Transactions on Machine Learning Research (07/2025)
use NNs to make t-SNE generalizable, and (Schofield & Lensen 2021 which aims to make UMAP more interpretable. However, P. UMAP has demonstrated superior performance. NUMAP presents method to surpass P. UMAP in terms of global structure preservation.
3
Preliminaries
In this section, we begin by providing the fundamental definitions that will be used throughout this work. Additionally, we briefly outline the key components of UMAP and P. UMAP.
3.1 Spectral Embedding
Let X' f#1:...,, ] S Rd denote a collection of unlabeled data points drawn from some unknown distri- bution D. Let W [=xx be positive symmetric graph affinity matrix, with nodes corresponding X, and let. D be the corresponding diagonal degree matrix (i.e. Dj Σド=1 Wty). The Unnormalized Graph the Symmetric Laplacian
Laplacian is defined as L = D W. Other normalized Laplacian versions Leym = D and the Random- Walk (RW) Laplacian Lrw D L. Sop-SpectralNet is applicable to all of these Laplacian versions. The eigenvalues of L can be sorted to satisfy = X S )2 < .... S A with corresponding eigenvectors 01,-.., Un (Von Luxburg, 2007)- is important to note that the first pair (i.e.. v1) is trivial for every Laplacian matrix A 0, and for the unnormalized and RW Laplacians V1 Va namely the constant vector.
-i. I,,
For a given target dimension A the first non-trivial eigenvectors provide a natural non-linear low- dimensional embedding of the graph which known as Speciral Embedding (SE). In practice, we denote by V e ga the matrix containing the first non-trivial & eigenvectors of the Laplacian matrix as its columns (i.c., 02,..., Uk-)). The SE reprosentation of cach sample E Rd is the ith row of V, i.c., Vi (12(1),.... ................................ (e4 (i)).
3.2 SpectralNet
A prominent method for addressing scalability and generalizability Spectral Clustering (SC) using deep neural networks, for example SpectralNet (Shaham al. 2018). SpectralNet follows a common methodology for transferring the problem of matrix decomposition to its smallest cigenvectors to optimization problem, through minimization of the Rayleigh Quotient (RQ).
Definition The Rayleigh quotient (RQ) of Loplacion matriz Raxn is a function Ry paxk + E defined A Bank by
(r.(A) Tr(A
SpectralNet first minimizes the RQ on small batches, while enforcing orthogonality. Namely, it approximates H* which minimizes
1 -/222 Ry (fo(X)) s.t. m L Lspectrainet (0)=
=x(())To(X)) =
(1)
where m is the minibatch size and m contains the d-dimensional data samples as rows. Thereby, learns map f Rdd R (where the input dimension) which approximates the space spanned by the first k eigenfunctions of the -aplace-Beltrami operator on the underlying manifold D (Belkin & Niyogi 2006 Shi 2015). Then, it clusters the representations via k-means. These eigenfunctions generalization of the SE to unseen points, enabling both scalable and generalizable spectral clustering.
natural
3.3 UMAP and Parametric UMAP
UMAP McInnes et al. 2018) widely used visualization method, known for scalability and ability to preserve global structure. It constructs a graph from high-dimensional data and learns fow-dimensional representation by minimizing the KT-divergence between the input graph and the representation graph.
4

---

Published in Transactions on Machine Learning Research (07 (07/202) (025)
Data
UMAP (SE)
UMAP (PCA)
UMAP
NUMAP (ours)
(a)
not generalizable
(b)
generalizable
Figure 1: Spectral Embedding is crucial for global structure preservation. (a) Non-parametric UMAP preserves the global structure when initialized with SE, but fails to do so with PCA initialization note the in-cluster separation in the middle row; (b) A similar effect is observed in generalizable implementations: P. UMAP, which does not involve SE, fails maintain global structure, resulting overlapping clusters (e.g., first and second rows). In contrast, NUMAP (ours) achieves global structure preservation comparable to UMAP while supporting generalizability.
The UMAP method can be divided into three components (summarized in Fig. 3): (1) constructing graph which best captures the global structure of the input data; (2) initializing the representations via SE; (3) Learning the representations, via SGD, which best capture the original graph. This setup does not facilitate generalization, as both steps (2) and (3) lack generalizability.
Additionally, discussed in (Kobak & Linderman 2021). UMAP primarily derives its global preservation abilities, well as its consistency, from initializing the representations using SE. Therefore, the SE initial- ization serves critical step for UMAP to uphold the global strueture (see demonstrations in Fig. [ Global proservation, in this context, refers to the scparation of different classos, and avoiding the scparation of existing classes. We refer the reader to (Kobak & Linderman, 2021) for a more comprehensive discussion about the effects of informative initialization on UMAP's performance.
Recently, generalizable version of UMAP, known as Parametric UMAP (P. UMAP), was introduced (Sain- burg al. 2021). P. UMAP replaces step (3) with the training of a neural network. Importantly, it overlooks step (2), the SE initialization. Consequently, P. UMAP may struggle preserve global structure, particu- larly when dealing with non-linear structures. Fig. D illustrates this phenomenon using several simple yet non-linear structures. These examples are particularly insightful for highlighting the importance of SE in preserving global structure, as the expected outcome of a good visualization is known. Noticeably, P. UMAP fails to preserve global structure (e.g., does not separate different clusters).
4 Method
4.1 Motivation
It well known that. V E R whose columns are the first. k eigenvectors of (i.e., those corresponding to the & smallest eigenvalues), minimizes (A( under orthogonality constraint (i.e., AT A = 1) Li 2015).
However, a rotation and reflection ambiguity of the RQ prohibits trivial adaptation of this concept SE. Basic properties of trace imply that for any orthogonal matrix Q Rkxk the matrix U VQ satisfies
5

---

Published in Transactions on Machine Learning Research (07/2025)
R2(U)= RV). Thus, every such U also minimizes Ry under the orthogonality constraint, and therefore this kind of minimization solely is missing eigenvectors separation, which is crucial for many applications.
In fact, as stated in Lemma the aforementioned form VQ is the only form of a minimizer of Rt under the orthogonality constraint. For conciseness, we provide our proof to the lemma in App. A
Lemma 1. Every minimizer of Rr. under the orthogonabily constrainl, is of the form VQ, where V is the first eigenvectors matriz of L and Q is an arbitrary squared orthogonal matria.
An immodiate result of Lemma iis that SpectralNet's mothod, using deep neural network for RQ minimiza- tion (while enforcing orthogonality), does not lead to the SE. However, it only leads to the space spanned by the constant vector and the leading k eigenvectors of L, with different rotations and reflections for each run. Therefore, each time the RQ is minimized, it results in a different linear combination of the smallest eigenvectors. Although this is sufficient for clustering purposes, we scarch for reproducibility, consistency, and separation of the eigenvectors, the RQ cannot solely provide the SE, necessitating the development of new techniques in Sep-SpectralNet.
4.2 Sep-SpectralNet
Setup. Here present the two key components of Sep-SpectralNet, scalable and generalizable SE method. We consider the following setup: Given a training set X C Rd and larget dimension k we construct an affinity matrix W, and compute an approximation of the leading eigenvectors of its corre- sponding Laplacian. In practice, we first utilize SpectralNet (Shaham et al. 2018) to approximate the space spanned by the first ++1 eigenfunctions of the corresponding haplace-Beltram operator, and then find each of the k leading eigenfunctions within this space (i.e. the SE). Namely, Sep-SpectralNet computes a map Fe : gd + g* which approximates the map f (f fk+1), where f. is the ith eigenfunction of the Laplaco- Laplaco-Beltrami operator on the underlying manifold D.
Eigenspace approximation. As empirically showed in (Sahamm et al. 2018 and motivated from Lemma SpectralNet loss is minimized when Fe = T o (f1...., fx+1), where T : Rk+ k + R is an arbitrary isometry. That is, Fo approximales the space spanned by the first k + 1 eigenfunctions. However, the SE (i.e. each of the leading eigenfunctions) is poorly approximated. Each time the RQ minimized, the cigenfunctions are approximated up to a different isometry T. Fig. [a demonstrates this phenomenon on the toy moon dataset a noisy half circle linearly embedded into 10-dimension input space (see Sec. [.11 Employing SpectralNet approach indeed enables to consistently achieve a perfect approximation of the space (i.e., the errors at the left histograms are accunulated around 0). However, when comparing vector to vector, it becomes apparent that the SE was seldom attained.
SE approximation. To separate the eigenvectors, and thereby consistently get the SE, we present a simple use of Lemma Notice that based on Lemma we can compute a rotated version of the diagonal eigenvalues matrix. Namely,
= (
where A is the diagonal eigenvalues matrix. Due to the uniqueness of eigendecomposition, the eigenvectors and eigenvalues of the small matrix Á E 1+++ are Q" and diag(A), respectively. Hence, by diagonalizing Á we get the eigenvalues and are also able to separate the eigenvectors (i.c., approximate the SE).
In practice, as Q is a property of SpectralNet optimization (manifested by the parameters), we compute the matrix A by averaging over a few random minibatches and diagonalize it. Thereby, making this addition very cheap computationally. The cigenvectors matrix of A is the inverse of the orthogonal matrix Q, and hence by multiplying the output of the learned map Fg by this matrix, the SE is retained. Also, the eigenvalues of À are the eigenvalues of L..
This step is simple and elegant, yet its impact significant. The effect of this intentional rotation is represented in the Fig. [a] Sep-SpectralNet is not only able consistently approximate the space, but also to approximate each eigenvector.
6

---

Published in Transactions on Machine Learning Research (07 (025)
Subspace
1500
100
100
ARPACK
Frequency
LOBPCG
Irain
1200
Sep-SpectralNet (ours)
Time: (xecands)
900
100
Frequency
600
2D moon
0
Distance
Distance
Distance
Distance
SpectralNet
Sep-Spectralnet
20
10f
Number of samples (log scale)
(b)
Figure 2: (a) Sep-SpectralNet separates the eigenvectors. Approximation of the 2-dimensional SE of the moon dataset (a 2D moon linearly embedded into 10D input space) using SpectralNet (in blue) and Sep- SpectralNet (in green) over 100 runs, train (top row) and test (bottom row). Left column: distribution of the Grassmann distance between the output and true subspace. Second to Fourth columns: distribution of the sin" distance between each output and true cigenvector scparately. Evidently, while SpectralNet's 's errors are distributed over large range of values, Sep-Spect errors are small, capturing only the smallest error bin in the figure, indicating on eigenvector separation. (b) Sep-SpectralNet's training is as scalable non-generalizable spectral methods. Running times SE using Sep-SpectralNet vs. other methods on the moon dataset, relative to the number of samples, and with standard deviation confidence intervals. Evidently, Sep-SpectralNet is the fastest asymptotically.
Algorithms Layout. Our end-to-end training approach is summarized in Algorithms 1 and 2 in App. B
We first train Fo to approximate the first eigenfunctions up to isometry (Algorithm] [ (Shaham et al. 2018). then compute Q"" and A separate the eigenvectors and recover the SE and corresponding eigenvalues (Algorithm App. C details additional considerations about the implementation.
Once we have Fo and QT, computing the embeddings of the train set or of new test points (i.c., out-of-sample extension) is straightforward: we simply propagate each test point x; through the network Fg to obtain their embeddings St, and use Q to gel the SE embeddings y4 = J²Q".
Time and Space complexity. As the network iterates over small batches, and the post- pest-processing oper- ation is much cheaper, Sep-SpectralNet's time complexity is approximately linear in the number of samples. This is also demonstrated in Fig. 2b where the continuous red line, representing lincar regression, aligns with our empirical results. App. C provides a discussion about the complexity of Sep-SpectralNet. Sep- SpectralNet is also highly memory-efficient, requiring only minibatch-sized graphs or matrices memory, rather than the full graph.
4.3 NUMAP
Here, we demonstrate Sep-SpectralNet's extension to UMAP, one of many methods that may benefit from generalizable SE. As discussed in Soc. 3.3 the SE initialization is crucial for the global preservation abilities of UMAP. Therefore, we seek a method to incorporate SE into a generalizable version of UMAP. A naive approach would be to fine-tune Sep-SpeetralNet using UMAP loss (Eq. 2 However, during this implementation, we encountered the phenomenon of catastrophic forgetting (sce App.
The core of our idea is illustrated in Fig. 3 Initially, we use Sep-SpectralNet to learn a parametric represen- tation of the k-dimensional SE the imput data. Subsequently, we train an NN to map from the SE to the final E-dimensional embedding space (usually for L 2, 2,3), utilizing UMAP contrastive loss. The objective of the second NN is identify representations that best capture the local structure of the input data graph. SE transforms complex non-linear structures into simpler linear structures, allowing the second NN to preserve
7

---

Published in Transactions on Machine Learning Research (07/202) 2025)
UMAP
FD SE
NUMAP (ours)
Sep- SpectralNet
kD
NN
4
P. UMAP
NN
Figure 3: Incorporating Sep-SpectralNet for generalizable UMAP. UMAP NUMAP P. UMAP overview. NUMAP integrates SE, in UMAP, while enabling generalization. First, it learns parametric representation of the k-dimensional SE. Then, it learns a map from the SE to the final & dimensional space, incorporating residual connections from the first & imput components to the output layer.
both local and global structures effectively. To enhance this capability, we incorporate residual connections in the second NN, Ge, which link the first é components of the input (the SE1.) directly to the output. Thus, the NUMAP embedding is given by Y SE(X) Go(X), that minimizes the UMAP's loss
wh(e) wr(e) + (1 wx(ee) log
1 wh(e) 1 wr(e)
Σ (n( log EEB
(2)
where wx(e), wi(e) are the corresponding weights of the edge in the high-dimensional input (X) graph and the dimennnenia output (Y) graph, respectively. It should be noted that this could not have been possible without Sep-SpectralNet's ability to separate the eigenvectors, as the first components of SpectralNet's output are merely arbitrary linear combinations of the k-dimensional SE, Also, it would not be practical without Sep-SpectralNet's inherent generalizability and scalability. Fig. demonstrates this capability with several simple structures.
4.4 Additional Applications
In this section we soek to highlight Sep-SpectralNet's potential impact on important tasks and applications (besides generalizable UMAP), as it integrates generalizability, scalability and eigenvectors separation. As discussed in Sec. 1 SE is applied across various domains, many of which can benefit generalizability capa- bilities by replacing the current SE implementation with Sep-SpectralNet. We therefore elaborate herein the significance of SE in selected applications, and discuss how Sop-SpectralNet, as generalizable approximation of it, can enhance their effectiveness and applicability.
1975 The Fiedler value, also known as algebraic connectivily, refers the second eigenvalue of the Laplacian matrix, while the Fiedler vector refers to the associated eigenvector. This value quantifies the connectivity of graph, increasing the graph becomes
Fiedler vector and value. special case of SE is the Ficdler voctor and value (Fiedler 1973
connected. Specifically, if graph is not connccted, its Fiedler value is 0. The Fiedler vector and value are main topic of many works (Andersen el al. 2006
Barnard et al.] 1993 Kundu et al. 2004 Shepherd et al., 2007: Cai et al. 2018 Zhu & Schlick 2021 Tam & Dunson 2020). As Sep-SpectralNet is able to distinguish between the eigenvectors and approximate the eigenvalues, has the capability to approximate both the Ficdler vector and value, while also generalizing the vector unseen samples (see Sec. 5.1
Diffusion Maps. A popular method which incorporates SE, alongside the eigenvalues of the Laplacian matrix, is Diffusion Maps (Coifman & Lafon 2006b). Diffusion Maps embeds a graph (or manifold) into
8

---

Published in Transactions on Machine Learning Research (07/2025)
space where the pairwise Euclidean distances are equivalent to the pairwise Diffusion distances on the graph.
This approach is widely used (e.g., (Ghamari et. al. 2024 Hardin et al. 2024 Park et al. 2021))
In practice, for an k-dimensional embedding space and a given t E N, Diffusion Maps maps the points to the leading eigenvectors of the RW-Laplacian matrix of the data as follows:
x - ((1 - 22
(1-Axx (1-A++) = =Y,
where X E RRad is a matrix containing each input point as a row, and Y E Ba*k is a matrix containing each of the representations as a row. As Sep-SpectralNet is able to approximate both the cigenvectors and eigenvalues of the Laplacian matrix, is able to make Diffusion Maps generalizable and efficient (Sec. 5.1
4.5 Evaluating UMAP embedding Grassmann Score
Common evaluation methods for dimensionality reduction, particularly for visualization, are predominantly focused on local structures. For instance, Mclnnes et al. (2018); ; Kawase et al. (2022) use KNN accuracy and Trustworthiness, which only account for the local neighborhoods of cach point while overlooking global structures such as cluster separation. One global evaluation method the Silhouette score, which measures the clustering quality of the classes within the embedding space. However, this score does not capture the preservation of the overall global structure.
To address this gap, we propose a new evaluation method, specifically appropriate for assessing global structure preservation in graph-based dimensionality reduction methods (e.g., UMAP, t-SNE). The leading eigenvectors of the Laplacian matrix are known to encode crucial global information about the graph (Belkin) & Niyogi 2003). Thus, we measure the distance between the global structures of the original and embedding manifolds using the Grassmann distance between the first eigenvectors of their respective Laplacian matrices. We refer to this method as the Grassmann Score (GS).
It is important to note that GS includes a hyper-parameter- - the number of eigenvectors considered. Increas- ing the number of eigenvectors incorporates more local structure into the evaluation. A natural choice for this hyperparameter is 2, which corresponds to comparing the Fiedler vectors (i.c., the second eigenvectors of the Laplacian). The Fiedler vector is well known for encapsulating the global information of graph (Fiedler 1973. 1975). Accordingly, for simplicity and unless specified otherwise, the GS is computed using the first two eigenvectors. Fig. demonstrates GS (alongside Silhouette and kNN scores for comparison) on a few embeddings of two tangent spheres, independently to the embedding methods. Notably, the embedding the right appears to best preserve the global structure, as indicated by the smallest GS value. In contrast, the kNN scores are comparable across all embeddings (e.g., kNN ignores separation of an existing class), and the Silhouette score even favors other embeddings. In App. D we mathematically formalize GS and provide additional examples of embeddings and their corresponding GS. These examples further support the intuition that GS offectively captures global structure preservation better than previous measures.
5 Experiments
5.1
Eigenvectors Separation - Generalizable SE
In this section, we demonstrate Sep-SpectralNetts ability to approximate and generalize the SE using four real-world datasets: CIFAR10 (via their CLIP embedding); Appliances Energy Prediction dataset (Can- danedo 2017); KuZushij-MNISS (KMNIST) dataset (Clanuwat et al.] 2018): Parkinsons Telemonitoring dataset Tsanas & Litule, 2009). Particularly, we compare our results with SpectralNNtt which has been empirically shown to approximate the SE space. However, as our results demonstrate, SpectralNet is insuffi- cient for accurately approximating SE. For additional technical details rogarding the datasets, architectures and training procedures, we refer the reader to Appendix
Evaluation Metrics. To assess the approximation of each eigenvector (i.c., the SE), we compute the sin? of the angle between each predicted and ground truth vector. This can be viewed as the 1-dimensional case of the Grassmann distance, well-known metric for comparing equidimensional linear subspaces (see
9

---

Published in Transactions on Machine Learning Research (07/202) 2025)
GS
0.437
0.686
0.016
0.003
Silhouette KNN
0.603
0.986
0.987
0.984
0.998
Figure 4: Grassmann Score (GS) reflects global structure preservation. A demonstration of the alignment between intuitive expectations and GS on a toy dataset of two 3D tangent spheres. Four possible 2D embeddings of the dataset are shown, alongside their corresponding GS, kNN accuracy, and Silhouette score. The rightmost visualization best preserves the global structure, forming two tangent circles 2D, Notably, kNN fails to distinguish wetween the embeddings, while Silhouette favors the most clustered one, despite its poor reflection of the original structure. In contrast, GS successfully captures the preservation of global structure, assigning the highest score to the most faithful embedding.
CIFARI0
Appliances
KMNIST
Parkinsons
0.6-
Emor
0.6
61
v2
v3 v4
v5
v2
v3
v4
v2
v3
v2
Sep-Spectrallee
SpectralNee
Figure 5: Sep-SpectralNet successfully separates the cigenvectors. A comparison between Sep- SpectralNet and SpectralNet of SE and Fiedler Vector (v2) approximation real-world datasets. The values are the mean and standard deviation of the sin² distance between the predicted and true eigenvector of the test set, over 10 runs. Lower is better. Notably, Sep-SpectralNets errors are significantly smaller than SpectralNet's, supporting Sep-SpectralNet's ability to separate the cigenvectors.
formalization in App. D) Conceruing the eigenvalues approximation evaluation, we measure the Pearson Correlation between the predicted and true eigenvalues (computed via. SVD).
Fig. presents results the real-world datasets. Sop-SpectralNet's output is used directly, while SpectralNet's predicted eigenvectors are resorted to minimize the mean sin? distance. The results clearly show that Sep-SpectralNet consistently produces significantly more accurate SE approximations compared to SpectralNet, due to the improved separation of the cigenvectors.
Additionaly, note the Sep-SpectralNet approximates the eigenvalues as well. When concerning series of Laplacian eigenvalues, the most important property is the relative increase of the eigenvalues (Coifman & Lafon 2006b). Sep-SpoetralNet demonstrates strong ability approximate this property. To this, repeated Sep-SpectralNet's eigenvalue approximation (10 times) and caleulated the Pearson correlation between the predicted and accurate eigenvalues vector. We compared the first 10 eigenvalues. The resulting mean correlation and standard deviation are: Parkinsons Telemonitoring: 0.917 03.881 Appliances Energy
Prediction: 0.839 +..3940
5.2 Scalability
Noteworthy, Sep-SpectralNet not only generalizes effectively but also does more quickly than the most scalable (yet non-generalizable) existing methods. Fig. 2b demonstrates this point the toy moon dataset
10

---

Published in Transactions on Machine Learning Research (07 (07/2025)
2D moon linearly embedded into 10D input space. To evaluate scalability, we measured the computation time required for SE approximation, for an increasing number of samples. We compared the results with the three most popular methods for sparse matrix decomposition, which are currently the fastest implementa- tions: ARPACK (Lchoueq et al. 1998). LOBPCG (Benner & Mach 2011 and AMG (Brandt et al., 1984). For each number of samples, we calculated the Laplacian matrix that is 99% sparse. Each method was executed five times, initialized with different sceds. Notably, for higher numbers of samples, Sep-SpectralNet converges significantly faster. Notably, Sep-SpeetralNetmainlaann the same scalability as SpectralNet, since the separation step incurs negligible overhead compared to the overall training time.
5.3 NUMAP generalizable UMAP
In this section, we demonstrate NUMAP's ability to preserve global structure, while enabling fast inference of test points, and it's ability to enable timo-scrics UMAP visualization. We compare our results with UMAP, with the larget dimensionality set to We refer to App. for various ablations.
We consider four real-world datasets: CIFAR1O (via their CLIP embedding); Appliances Energy Predic-
tion dataset; Wine (Aeberhard & Forina 1992;; Banknote Authentication (Lohwey) 2012). For additional
techmical details regarding the datasets, architectures and training procedures, we refer to App. G
Evaluation Metrics. To evaluate and compare the embeddings, we employed both local and global eval- uation metrics. For local evaluation, we used the well-established accuracy of a kNN classifier on the embeddings (MeInnes et al. 2018 Sainburg et al. 20021 which is applicable only on classed data. For global evaluation, we use GS (see discussion in Sec. 4.5
Local Structure. Tab. 2 presents the KNN results on the real-world datasets. These results are compa- rable across the methods. Namely, NUMAP does not compromise on local structure.
Global Structure. Tab. 3 presents the GS results on these datasets. The results underscore that NUMAP better cap- tures the global structure. Namely, NUMAP enhances global structure preservation while not compromising local structure preservation.
P. UMAP
NUMAP (ours)
In Fig. 1 and Fig. 6 we supplement the empirical results with qualitative examples. Fig. D presents simple non-lincar synthetic 3-dimennioonl structures and their 2-dimensional vi- sualizations using U.MAP (non-parann) poom-parametric. P. UMAP and NUMAP. These examples are particularly insightful, the ex- pected outcome of a good visualization is known. UMAP (using its default configuration, SE initialization) accurately preserves the global structure in its 2-dimensional representations, but lack the ability to generalize to unseen points. Among the gen- eralizable methods (Fig- p) P. UMAP fails to preserve the global structure: in the top two rows, it does not scparate the clusters, while in the bottom row, it introduces undesired color overlaps. In contrast, NUMAP effectively preserves these separations and avoids unnecessary overlapping. Fig. о further demonstrates NUMAP's ability to preserve global structure, as evidenced by the improved class separation in the Banknote dataset.
Figure 6: NUMAP and P. UMAP vi- sualization of the Banknote dataset test set. NUMAP's enhanced ability preserve global structure is evident in the clearer separation between clusters com- pared to P. UMAP
Timc-scries data visualization. Fig. shows a simulation time-scries data, which can be viewed as a simulation of cellular differentiation. Specifically, we may consider disteonntiation of hemalopoielie stem cells (also known as blood stem cells), which are known to differentiate into many types of blood cells, to T-cells. The process involves two kinds of cells (represented by their gene expressions; rod and blue samples in the figure). One represents stem cells, while the other T-cells. A group of cells (colored in pink in the figure) then gradually transitions from stem cells to T-cells. At the top row use UMAP to visualize each time
11

---

Published in Transactions on Machine Learning Research (07/2025)
Metric
Method
Cifar10
Appliances
Wine
Banknote
UMAP
0.908 +0.004
0.953 6...+0.0333
0.927 0.0023
KNN
NUMAP (ours)
0.905++0.004
40.044
0.986+
Table 2: NUMAP local structure preservation is comparable with P. UMAP. kNN accuracy results of P. UMAP and NUMAP on labeled real-world datasets (Appliances dataset, is excluded due to the absence of labels). The values are the mean and standard deviation measures on the test set, over 10 runs.
Metric
Method
Cifar10
Appliances
Wine
Banknote
GS 4
P. UMAP
0.133
0.710 203
0.
0.722=0.079
NUMAP (ours)
0.054
0.261 0.0020
0.429 10.124
0.61840..13
p-value
0.004
0.0002
0.118
0.056
Table 3: NUMAP better preserves global structure. P. UMAP and NUMAP Grassmann Score (GS) results on real-world datasets. The values are the mean and standard deviation measures on the test set, over 10 runs. The best in mcan highlightod. Notably, NUMAP achieves better global structure preservation.
step, while at the bottom we train NUMAP on the first two time-steps and only inference the rest. UMAP is inconsistent over time-steps, which makes it impractical for understanding change and progression. also has train the embeddings at each time-step separately. In contrast, NUMAP only trains on the first two time-steps, and the embeddings of the later time-steps immediate from inference. This also enables consistency over time, and makes the trend and process visible and understandable.
6 Conclusions
We introduced Sep-SpectralNet, deep-learning approach for approximate SE. Sep-SpectraLNet addresses the three primary drawbacks of current SE implementation: generalizability, scalability and cigenvectors separation. By incorporating a post-processing diagonalization step, Sep-SpectralNet enables eigenvectors separation without compromising SpectralNet's generalizability or scalability. Remarkably, this one-shot post-processing operation lays the groundwork for a wide range of new applications of SE, which would not have been possible without its scalable and generalizable implementation. It also presents promising pathway to enhance current applications of SE.
In particular, also presented NUMAP, novel extension of Sep-SpectralNet for generalizable UMAP visualization. We believe the integration of generalizable SE with deep learning can have significant impact on unsupervised learning methods. However, exploring alternative inputs to the UMAP loss network is an interesting direction for future work. Finally, further research should dolve into exploring the applications of SE across various fields.
References
Stefan Aeberhard and M. Forina.
Wine.
UCI
Machine
Learning
Repository, 1992.
DOI:
Reid Andersen, Fan Chung, and Kevin Lang. Local graph partitioning using pagerank vectors. In 2006 47th Annual IEEE Symposium Foundations of Computer Science (FOCS pp. 475-486. IEEE, 2006. doi: D0.1109/FOCS.2006.
Stephen T Barnard, Alex Pothen, and Horst D Simon. A spectral algorithm for envelope reduction of sparse matrices. In Proceedings of the 1993 ACM/IEEE Conference on Supercomputing, pp. 493-502, 1993. doi: 70.1145/169627.1
12

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
UMAP
NUMAP (ours)
timestep
1
3
5
7
9
Training
Inference
Figure 7: Time-series data visualization using NUMAP. A comparison between UMAP and NUMAP visualizations of dynamical system. In the UMAP visualization, the structure and cluster positions vary significantly across timesteps, hindering the interpretability of the underlying process. In contrast, NUMAP produces consistent structures over time, clearly revealing the transition of the pink points between the red and blue regions. Notably, NUMAP requires training only on the first two timesteps, enabling efficient generalization to the rest the sequence.
Dominique Beaini, Saro Passaro, Vincent Létourneau, Will Hamilton, Gabriele Corso, and Pietro Liò. Di- rectional graph networks. In International Conference on Machine Learning, pp. 748 758. PMLR, 2021. doi: https://doi.orr/ 10. 48550/ 111550/rrXiv.200 2011v22000.02663
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation.
Neural computation, 15(6):1373 1396, 2003. doi: 00.116//059970003317888333
Mikhail Belkin and Partha Niyogi. Convergence of laplacian eigenmaps. Advances in neural information processing systems, 19, 2006. doi: http://ddbooogg..0..../////m///s
Peter Benner and Thomas Mach. Locally optimal block preconditioned conjugate gradient method for hierarchical matrices. PAMM, 11(1):741 742, 2011. doi: https://bbl.rrg/100000/////
Achi Brandt, Steve McCormick, and John Ruge. Algebraic multigrid (amg) for sparse matrix equations. Sparsity and its Applications, 257, 1984.
Jiayue Cai, Aiping Liu, Taomian Mi, Saurabh Garg, Wade Trappe, Martin J McKeown, and Z Jane Wang. Dynamic graph theoretical analysis of functional connectivity in parkinson's 's disease: The importance of fiedler value. JEEE journal of biomedical and health informatics, 23(4):1720 1729, 2018. doi: 10.1109/
1BHI.2018.00 28..528754
Kieran Campbell, Chris P Ponting, and Caleb Webber. Laplacian eigenmaps and principal curves for high resolution pseudotemporal ordering of single-cell rna-seq profiles. bioRxiv, pp. 027219, 2015. doi: httt::://doo.rrg/111110////////
Luis Candanedo. Appliances Energy Prediction. UCI Machine Learning Repository, 2017. DOI:
http://do...rg10.24443333
Ziyu Chen, Yingzhou Li, and Xiuyuan Cheng. Specnet2: Orthogonalizationnffee spectral embedding by
neural networks. arXiv preprint arXiv:220.006644 2022. doi: http:///oo.orrg10.485555. Xw..Xii.22606666.
13

---

Published in Transactions on Machine Learning Research (07/2025)
Eliodoro Chiavazzo, Roberto Covino, Ronald R Coifman, C William Gear, Anastasia Georgiou, Gerhard Hummer, and Toamnis G Kevrekidis. Intrinsic map dynamics exploration for uncharted effective free- energy landscapes. Proceedings of the National Academy of Sciences, 114((8):E3494 -5503,, 2017. doi:
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature, 2018.
Ronald R Coifman and Stéphane Lafon. Geometric harmonics: novel tool for multiscale out-of-sample
extension of empirical functions. Applied and Computational Harmonic Analysis, 21(1):31-52, 2006a. doi: https://doi.org org/ 10. a1t.///acca.200..0......
Ronald R Coifman and Stéphane Lafon. Diffusion maps. Applied and computational harmonic analysis, 21 (1):5-30, 30, 2006b. doi: https://doi.org// 10t.//0001000..eee. a0t.100....cca.0000000 200..04.006.0006
Sebastian Damrich, Jan Niklas Böhm, Fred A Hamprecht, and Dmitry Kobak. From t-sne to umap with contrastive learning. arXiv preprint arXiv:2206008116, 2022. doi: https:///do.orrg100488550 /0.4//50//////i/vv/22v 01816.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering- Advances in neural information processing systems, 29, 2016. doi: https://doi.org// 10. 40855/////X..1166..009000
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 2966)1141-142, 2012. doi: N0.1109/MSSP.212..221..
Zhijie Deng, Jiaxin Shi, and Jun Zhu. Neuralef: Deconstructing kernels by deep neural networks. In International Conference on Machine Learning, pp. 1976-4992 PMLR, 2022. doi: https://doi.orr//I1 48550/arXiv.2200 2tt..222500066.5
Debo Dong, Cheng Luo, Xavier Guell, Yulin Wang, Hui He, Mingiun Duan, Simon Eickhoff, and Dezhong Yao. Compression of cerebellar functional gradients in schizophrenia. Schizophrenia bulletin, 46(5):1282- 1295, 2020. doi: (0.1093/schbul/sbae
Klaus Eckelt, Andreas Hinterreiter, Patrick Adelberger, Conny Walchshofer, Vaishali Dhanoa, Christina Humer, Moritz Heckmann, Christian Steinparz, and Mare Strcit. Visual exploration of relationships and structure in low-dinensional embeddings. IEEE Transactions on Visualization and Computer Graphics, 29(7):3312-3333 2023. doi: 10.1109/VVCG.2202.331..
Miroslav Fiedler. Algebraic connectivity of graphs. Czechoslovak mathematical journal, 23(2):298 305, 1973. doi: C221136/CMJJ19730
Miroslav Fiedler. A property of eigenvectors of nonnegative symmetric matrices and its application to graph theory. Czechoslovak mathematical journal, 25(4):619-632) 1975. doi: 10.21136 CMM.1195.10133
Tan Gemp, Brian McWilliams, Claire Vernade, and Thore Graepel. Figengame: Pea as a nash equilibrium. arXiv preprint arXiv:2010.00554 2020. doi:
Danial Ghamari, Roberto Covino, and Pietro Faccioli. Sampling a rare protein transition using quantum anncaling. Journal of Chemical Theory and Computation, 20(8):3322 33323334,, 2024. doi: https://doi.orgg 10.1021/acssjctc.330177
Aristides Gionis, Piotr Indyk, Rajeev Motwani, et al. Similarity search in high dimensions via hashing. In Vldb, volume 99, pp. 518-529, 1999.
Aldo Glielmo, Brooke E. Husic, Alex Rodriguez, Cecilia Clementi, Frank No6, and Alessandro Laio. Unsu- pervised learning methods for molecular simulation data. Chemical Reviews, 121(16):9722-972) 2021. doi: https://dbi.orr/T 10. 10201.1.1///.. /021/acs.chmi www.oseeeeeee.001111.1
14

---

Published in Transactions on Machine Learning Research (07/2025)
Thomas J Hardin, Michael Chandross, Rahul Meena, Spencer Fajardo, Dimitris Giovanis, Ioannis Kevrekidis, Michael I, Falk, and Michael D Shields. Revealing the hidden structure of disordered materials by parameterizing their local structural manifold. Nature communications, 15(1):4424, 2024. doi: https:
Ori Katz, Ronen Talmon, Yu-Lun Lo, and Hau-Tieng Wu. Alternating diffiusion maps for multimodal data fusion. Information Fusion, 45:346-300 360, 2019. doi: https://doi.org/ htt.i/orrr/11111...... 1ttp//10..11/..fff////. www.fus.21......000
Yoshiaki Kawase, Kosuke Mitarai, and Keisuke Fujii. Parametric t-stochastic neighbor embedding with quantum neural network. Physical Review Research, 4(4):043199, 2022. arXiv.2202.04238.
doi: https://doi.org/ 1r1oo/orr1101485500 /10.48550//
Paulo Valente Klaine, Muhammad Ali Imran, Oluwakayode Onireti, and Richard Demo Souza. A survey of machine learning techniques applied to self-organizing cellular networks. TEFE Communications Surveys & Tutorials, 19(4):2392 2431, 2017. doi: 10.1009/COMISTT:011.2220
Dmitry Kobak and George C Linderman. Initialization is critical for proserving global data structure in both (-sne and umap. Nature biotechnology, 39(2):156 157, 2021. doi:
Sibsankar Kundu, Dan C Sorensen, and George N Phillips Jr. Aulomatic domain decomposition of proteins by a gaussian network model. Proteins: Structure, Function, and Bioinformatics, 57(4):725- 2004.
Stephane Lafon, Yosi Keller, and Ronald R Coifman. Data fusion and multicue data matching by diffusion maps. IEEE Transactions on pattern analysis and machine intelligence, 28(11):1784 1797, 2006. doi: 10. M0.109/TPAMM.2000
Roy R Lederman and Ronen Talmon. Learning the geomelry of common latent variables using alternating- diffusion. Applied and Computational Harmonic Analysis, 44(3):509-55) 0365000--33,,-- 2018. doi: https://doi.orgg/ a016/j 2015.09.00.
Richard Lehoueq, Danny C Sorensen, and Chao Yang. ARPACK users' guide: solution of large-scale eigenvalue problems with implicitly restarted Arnoldi methods. SIAM, 1998.
Jersson X Leon-Medina, Núria Parés, Maribel Anaya, Diego A Tibaduiza, and Francese Pozo. Data classifi- cation methodology for electronic noses using uniform manifold approximation and projection and extreme lcarning machine. Mathematics, 10(1):29, 2021. doi: https:/ /doioo//////////////////// wwooor//////0000 10.3390/math1000000 mot0m000100002.
Ren-Cang Li. Rayleigh quotient based optimization methods for eigenvalue problems. In Matrix Functions and Matrix Equations, pp. 76 108. World Scientific, 2015. doi: https:/ /ttp://doooor/100// hMpps///00.rrr/1111112/7777// //511000144/77788146777777 0004.
Derek Lim, Joshua Robinson, Lingxiao Zhao, Tess Smidt, Suvrit Sra, Haggai Maron, and Stefanic Jegelka. Sign and basis invariant networks for spectral graph representation learning. arXiv preprint arXiv:2202.13013 2022. doi:
Leland Zhang Liu. Umap-pytorch: Umap (uniform manifold approximation and projection) in pytorch, 2024.
URL https://github. wwt.///lyyyl///maappttrrrro
Volker Lohweg. Banknote Authentication.
UCI Machine Learning Repository, 2012.
DOL:
https://doi.org/// 10. 24432/C55PP5
George Ma, Yifei Wang, and Yisen Wang. Laplacian canonization: A minimalist approach to sign and basis invariant spectral embedding. Advances in Neural Information Processing Systems, 36, 2024. doi: xttp://orrrrr100000//0axxx.11.1 21.11000.1.......777....
Leland McInnes, John Healy, and Jamos Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 8rX.v:1802.03326,, 2018. doi: https///doi.org/0...55 aiv.. 1802.03426.
15

---

Published in Transactions on Machine Learning Research (07 (225))
Gal Mishne, Uri Shaham, Alexander Cloninger, and Israel Cohen. Diffusion nets. Applied and Computational Harmonic Analysis, 47(2):259 285, 2019. doi: https://hoi.org/100865600 ai......i00.07.....
Evert J Nyström. Über die praktische auflösung von integralgleichungen mit anwendungen auf randwertauf- gaben. 1930. doi: /dopso// 10. 10t110000000/054/777770
Antonio Ortega, Pascal Frossard, Jelena Kovačovíć, José MF Moura, and Pierre Vandergheynst. Graph signal processing: Overview, challenges, and applications. Proceedings of the IEEE, 106(5):808 828, 2018. doi: 10.1109/JJROC.22011222222
Bo-yong Park, Scok-Jun Hong, Sofie I. Valk, Casey Paquola, Oualid Benkarim, Richard AT Bethlehem, Adriana Di Martino, Michael P Milham, Alessandro Gozzi, BT Thomas Yeo, et al. Differences in nabcortico-cortical interactions identified from connectome and microcircuit models in autism. Nature communications, 12(1), 2021. doi: https:/ hoop:://do.rrg.../ 000108//1144---02-2212733
Bo-yong Park, Valeria Kebets, Sara Larivière, Meike D Hettwer, Casey Paquola, Daan van Rooij, Jan Buitclaar, Barbara Franke, Martine Hoogman, Lianne Schmaal, al. Multiscale neural gradients reflect transdiagnostic effects of major psychiatric conditions on cortical morphology. Communications biology, 1008000080000 /s40003-22222 03963-z.
5(1):1024, 2022. doi: https:/ cttp:///ooiorgggg 10. 10.1038/
David Pfau, Stig Petersen, Ashish Agarwal, David GT Barrett, and Kimberly L. Stachenfeld. Spectral inference networks: Unifying deep and spectral learning. arXiv preprint arXiv:1806.02215, 2018. doi: https://doi.org/In wtw.oogg//1........ 205550//XXiv.1066666
Tim Sainburg, Leland McInnes, and Timothy Q Gentner. Parametric umap embeddings for representation and semisupervised learning. Neural Computation, (3(1))28881 -907, 2021. doi: https:/ddii.org//// 10.
20550//2Xiiv2209.12200
Finn Schofield and Andrew Lensen.
Using genetic programming to find functional mappings for umap embeddings. In 2021 IEEE Congress on Evolutionary Computation (CEC), pp. 704 711. IEEE, 2021. doi:
10.1109/CFC45553.22 20219995553.0111994448888
Cri Shaham, Kelly Stanton, Henri Li, Boaz Nadler, Ronen Basri, and Yuval Kluger. Spectralnet: Spectral clustering using deep neural networks. In Proc. ICLR 2018, 2018. doi: https:/ http:///oooooo/0.8855555/ aiv. 1801.01587.
SJ Shepherd, CB Beggs, and Sue Jones. Amino acid partitioning using a fiedler vector model. European Biophysics Journal, 37:105-109 109, 2007. doi: https://doi.org:
Zuoqiang Shi. Convergence of laplacian spectra from random samples. arXiv preprint arXiv:1507.00155.
2015. doi: https://doi.org wtt.:/iiorr/11.455...5.. 10.44050//////// aar.iv.1177.01151......
Or Streicher, Ido Cohen, and Guy Gilboa. Basis: batch aligned spectral embedding space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10396-10400 2023. doi: http:////ooo.oo11..4835330
Edric Tam and David Dunson. Fiedler regularization: Learning neural networks with graph sparsity. In International Conference on Machine Learning, pp. 0346-9355. PMLR, 2020. doi: https://doi.org///
Athanasios Tsanas and Max Little. Parkinsons Telemonitoring. UCI Machine Learning Repository, 2009.
DOI: htttps://doi.org///ie 110024442//577828
Laurens Van Der Maaten. Learning a parametric embedding by preserving local structure. In Artificial intelligence and statistics, pp. 384 391. PMLR, 2009.
Uhike Von Luxburg. A tutorial on spectral clustering. Statistics and computing. 17:39--416, 2007. doi: http:://0...on/11..455000///
16

---

Published in Transactions on Machine Learning Research (07/2025)
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017. doi:
Yuxuan Richard Xie, Daniel C Castro, Stanislav S Rubakhin, Timothy J Trinklein, Jonathan Sweedler, and Fan Lam. Integrative multiscale biochemical mapping of the brain via dee-Learning-cuhanod high-throughput mass spectrometry. Nature Methods, 21:521 530, 2024. doi: htlpp://dooioor/11... -41592-024-02171--
Bingxian Xu and Guangzheng Zhang. Robust parametric umap for the analysis of single-cell data. bioRxiv. pp. 2023 11, 2023. doi: https:/ https://doiorgg////// www.oorg//11111011 http:/22........6.77.. 11....6779.
Eunkyung Yoo, Hyconscop Song, Tachycong Kim, and Chul Lec. Online learning of open-set speaker iden- lification by active user-registration. In INTERSPEECH, pp. 5065 5069, 2022.
Ziwei Zhang, Peng Cui, Jian Pei, Xin Wang, and Wenwu Zhu. Eigen-gun: graph structure preserving plug-in for gnns. IEEE Transactions on Knowledge and Data Engineering, 35(3):2544-25)) 2555, 2021. doi:
https://doi.orr/lun 10/:/10.4880 18550...55//// 2006.04330.
Wenwei Zheng, Bo Qi, Mary A Rohrdanz, Amedeo Caflisch, Aaron R Dinner, and Cecilia Clementi. De- lineation of folding pathways of ß-sheet miniprotein. The Journal of Physical Chemistry B, 115(44): 13065-13074, 2011. doi: https://doi.org 10. 10.1021/jp2066335.
Wenwei Zheng, Mary A Rohrdanz, and Cecilia Clementi. Rapid exploration configuration space with iuffussoon-maapddieeeeeeddd molecular dynamics. The journal physical chemistry B, 117(42):12769 12776, 2013. doi: http:///oooorggg10000011//111111
Qiyao Zhu and Tamar Schlick. A fiedler vector scoring approach for novel ma motif selection. The Journal of Physical Chemistry B, 125(4):1144 1155, 2021. doi: https://doi.orr/ www.oorg/100122.. 1tt.10///a...pcb..00000..
A Proof of Lemma
First, we remind an important property the Rayleigh Quotient.
Remark 1. The Rayleigh Quolient of a posilive semi-defonile matric L E Raxn with eigenvectors U1,...y Un corresponding to the eigenvalues A1 S .... S An, Rr. satisfies arg minl|l|l=1 R2(v)= = o1 and for each i >> 1 arg min mol=/ Rr (v) = v, for I U1,..., 84-1
Li 2015).
Lemma 1 Let L E RRax be an Unnormalized Laplacian matrix and Rr, : O(n, + I its corresponding Ry. Denote V e ga*k as the matrix containing the first & eigenvectors of as its columns, and A the corresponding diagonal eigenvalues matrix. Then, there exists an orthogonal matrix kgkk such that A = VQ.
RQ, and Let A be a minimizer
Proof. As V minimizes Rr, we get that ming RT(U)) Rr.(V) Ek do where 0= A1 S d2 S ... An are the eigenvalues of L. This yields
= TT(A Σ A
A LA is symmetric, and hence orthogonally diagonalizable, which mcans there exists an orthogonal matrix Q k and a diagonal matrix R s.l.
A" Q"
Which can be written as
(AQ")TI(AQ")) D
17

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Denoting by di,...,d, dy the diagonal values of D, the last equation yields
Ed Ro(=R) = Rr = Σ A
Note that based on Remark 1 S de for each i, AQ E k(, Hence, d; Xi, i...,
L(AO)) =
As the eigendecomposition of matrix is unique, this yields AQT V, which means = VQ.
B
Algorithm Layouts
Algorithm 1: SpectralNet training (Shaham et al. 2018)
Input: X' C Rd, number of dimensions k, batch size m
Output: Trained Fş which approximates the first + eigenfunctions up to isometry
1
Randomly initialize the network weights 0
while C(0) not converged do Orthogonalization step:
3
4 Sample a random minibatch X of size m
Forward propagate X and compute inputs to orthogonalization layer Y Compute the QR factorization =
7
Set the weights of the orthogonalization layer to be vmR. Gradient step:
Sample random minibatch T1,..., Zm Compute the m X m affinity matrix W Forward propagate 21,..., Im to get V1.-.., ym
11
12
Compute the loss (Sec. 3.2
13
Use the gradient of (()) to tune Fa weights, except those of the output layer;
Algorithm 2: Eigenvectors separation
Input: X C Rd, batch size m, Trained Fg which approximates the first. k + eigenfunctions
up to isometry
Output: Fe which approximates the leading eigenfunctions
T
[2
2 sample T minibatches X; E Rmxd
3 Forward propogate all X; and obtain Fd outputs Y; E Bm
4 Compute the m x m affinity matrices W,
5 compute all corresponding RW-Laplacians L;
6 + Y.
7 Diagonalize À gel Q" and the leading eigenvalues
8 Sort the leading eigenvalues, and the columns of Q" correspondingly
9 Q" + last & columns of Q
To obtain the representation of new test sample x;, compute yı Fo (xx))
C Implementation's Additional Considerations
C.1 Time and Space Complexity
Specifying the exact complexity of the method is difficult, As this is non-convex optimization problem, However, can discuss the following approximate complexity analysis. Assuming constant input and output
18

---

Published in Transactions on Machine Learning Research (07 (07/2025)
dimensions and given network architecture, can take general view on the complexity of each iteration by the batch size m.. The heaviest computational operations at each iteration are the nearest-neighbors search, the QR decomposition and the loss computation (i.e., computation of the Rayleigh Quotient). For the nearest-ncighbor search, we can use approximation techniques (c.g. LSH Gionis et al. (1999)) which work in almost linear complexity by m. A naive implementation time complexity. The loss computation also takes ((m²) due to the required matrix multiplication. Thereby, the complexity of each iteration is quadratic by the batch size. This is comparable to other approximation Benner techniques such as LOBPCG Benner & Mach (2011) (which also utilizes sparse matrix operations techniques for faster implementation). However, Sep-SpectralNet leverages stochastic training, allowing each iteration to consider only a batch of the data, rather than the entire dataset.
the QR decomposition would lead to an O
Assessing the complexity of each epoch is now straightforward, and results a time complexity of O(nm), where n, the number of samples, satisfies > m. This indicates an almost-linear complexity.
C.2 Graph Construction
To best capture the structure of the input manifold D, given by a finite number of samples X, we use a similar graph construction method used by Gomez al. in UMAP (Melnnes et al. 2018), proven to capture the local topology of the manifold at each point. However, as opposed to the method in (Meinnes et al., 2018), Sep-SpectralNet does not compute the graph of all points, which lead to scalability hurdles and impose significant memory demands. Instead, Sep-SpectralNet either computes small graphs each batch, or can be provided by the user with an affinity matrix W corresponding to X. Our practical construction of the graph affinity matrix W is as follows:
Given a distance mcasure between points, we first compute the k-nearest neighbors of each point z; under 6, ;i and denote
P. =in (x)) or median 21,), و k}
Second, compute the affinity matrix using the Laplace kernel
6(2;,2))
exp
Ij 144 otherwise
Wi
0
Third, symmetries W simply by laking W WOWT
We refer the reader to Mclnnes et al. (2018) for further discussion about the graph construction.
D Grassmann Score
In this section, we provide the formulation for the Grassmann Score (GS) evaluation method, and present simple examples to visualize its meaning.
D.1 Formalization of GS
Grassmann distance (see Def. is a metric function between equidimensional linear subspaces, where each is represented by an orthogonal matrix containing the basis as its columns. In other words, this is a metric which is invariant under multiplication by an orthogonal matrix.
Definition 1. Given two orthogonal matrices A, B the Grassmann Distance between them defined as:
dor (A, B)
Σ sin20,
where 0, arccos or B) the ith principal angle between and B, and o; the ith smallest singular value of AT B.
19

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Assuming are given dataset X' (#1 (#1...., Faf S Rd and corresponding low-dimensional representation y/ = (vn...., Un R We want to evaluate the dissimilarity between the global structures of X' and V. We build graphs from X' and }/ saved as affinity matrices Wx and Wy, respectively. We construct the corresponding Unnormalized Laplacians (see Sec. 3.1 Lx and Ly. We define the matrices Vx, Vy E Raxt so that their columns are the first I eigenvectors of Lxx by,, respectively.
Finally, we define the GS of y (w.r.t X) as follows:
Definition GSx(O) dor V5²)
hyper-parameter of GS. reasonable choice would be to take 2, which is equivalent to measure the Grassmann distance between the Fiedler vectors of the Laplacians. The Fiedler vector is known for its hold of the most important global properties. The larger the more complicated structures taken into consideration in the GS computation (which is not necceray desired).
Note that for the construction of the affinity matrices Wx, Wy we use the same construction scheme detailed in App. C.2 This construction method is similar to the one presented by Mclnnes et al. 2018). and proved to capture the local topology of the underlying manifold.
It is important note that GS might ignore the local structures, while concentrating on the global structures (especially for smaller values of t). The ultimate goal in visualization to find balance between the global and local structure.
D.2 Additional GS examples
GS Silhouette KNN
0.652
0.0
0.154
0.613
0.239
0.999
0.206
1.0
1.0
1.0
0,952
(a)
GS
0.0
0.486
0.0
0.498
Silhouette 0.74
0.369
1.0
0.108
KNN
1.0
1.0
1.0
0.944
(b)
Figure 8: Additional demonsttrations of the alignment between the intuitive expectation and the GS results two toy dataset. Four possible 2-dimensional embeddings of these dataset are provided, along with their corresponding GS, kNN accuracy and Silhouctte score. Unlike kNN and Silhouctte, GS effectively captures the preservation of global structure.
Fig. S depicts two additional demonstrations the alignment between the intuilive expectation and the GS results on two toy dataset. The basic global structure of both of these datasets two distinct clusters. This structure is indeed captured by GS. However, kNN gives perfect score also when the one of the clusters is separated. Silhouette score favourites the 2-points embedding. Namely, trade-offs local structure (i.e., giving lower score for preserving local structure, even when the global properties the same).
20

---

Published in Transactions on Machine Learning Research (07 (025))
E NUMAP Ablation Study
In this section, we start by considering two ablations of NUMAP, which for convenience we name NUMAP- SN and NUMAP-FT. NUMAP-SN is a replication of NUMAP's pipeline with SpectralNet replacing Sep- SpectralNet. That is, we show that eigenvector separation is necessary for NUMAP.
As for NUMAP-FT, this name refers another method to get generalizable version of UMAP by an extension of Sep-SpectralNel by fine-tuning the nelwork with UMAP loss. We tried that idea, but were forced to stop this direction,
stumbled upon the well-known catastrophic forgetting case.
Tab. extends Tab. 2 with the results of NUMAP-SN and NUMAP-FT. Notably, NUMAP-SN local struc- ture preservation are comparable with NUMAP and P. UMAP, as expected, while NUMAP-FT consistently fails in local structure preservation.
Tab. extends Tab. with the results of NUMAP-SN and NUMAP-FT. Notably, NUMAP-SN less consistent than NUMAP in global structure preservation, resulting in worse performance the Cifar10 and Banknote datasets. NUMAP-FT fails in global structure preservation on the Cifar10 and Appliances datasets. Although it achieves better GS the Wine dataset, and comparable global structure preservation on the Banknote dataset, it compromises lot on local structure preservation (see Tab.
Figure D presents an experiment on the simple 2circles dataset. Each row is represented the same experiment, run with a different seed. We trained Sep-SpectralNet to output the 2D SE of the 2circles dataset, as shown in the left column. Then, we initialized a new network, with the same architecture, with the pre-trained weights from Sep-SpectralNet. This network was trained with UMAP loss, as in (Sainburg et al. 2021) We tried different learning-rate for fine-tuning, to best match the desired UMAP embedding (i.e. retaining the local structure), without losing the global structure (e.g., separation of the two clusters). Unfortunately, there was no learning-rate that matched our goals.
Metric
Method
Cifar10
Appliances
Win
Banknote
UMAP
0.90810.004
0.953 go osa
0.927 1:00000000 220
NUMAP (ours)
0.905
0.950
0.986 004
ENN
NUMAP-SN
0.90340.002
0.956+0.022 028
0.963
NUMAP-FT
0.677+0.081
0.36-1+0.00 osa
0,686
Table 4: Local structure preservation ablation. An extension of Tab. with NUMAP-SN and NUMAP- FT.
Metric
Method
Cifar10
Appliances
Wine
Banknote
P. UMAP
0.133 o60
0.710
0.5419
U.722 fo
NUMAP (ours)
0.054 021
0.261 1: 020
0.429 12.444
0.618 181
GS 4
NUMAP-SK
0.281 +0.355
0.262
0.441
0.780
NUMAP-FT
0.680+0.338
0.348 +0. 202
0.002 00000
0.635 +0.114
Table 5: Global structure preservation ablation. An extension of Tab. NUMAP-FT.
with NUMAP-SN and
Another important question is the necessity of the residual connections. For that, we conduct another ablation study on the Cifar10 dataset. The results are shown in Tab. f and Tab. 7 Notably, while the skip
connections does not improve local structure preservation, they are crucial for global structure preservation, indicated by the lower GS.
Metric
Method
Cifar10
NUMAP (no residual connections)
0.907 2..00000
kNN
NUMAP
0.905 2. pos
Table 6: Ablation on residual connections for local structure preservation shows comparable performance with and without their use.
21

---

Published in Transactions on Machine Learning Research (07/2025)
Sep SpectralNet
In 1e-08
1e:07
In 1e-06
1e:05
0.0001
0.001
0.01
Figure 9: The catastrophic forgetting phenomenon when fine-tuning Sep-SpectralNet to match UMAP per- formance on the 2cireles dataset. Each column represents a tne-tuning using diffcrent learning-rate. Each row is a repetition, initialized with different seed.
Metric
Method
Cifar10
GS 4
NUMAP (no residual connections)
10.91
NUMAP
0.05440.00
Table 7: Ablation on residual connections for global structure preservation shows improved performance when they used.
F
Additional results
The full results of Fig. 5 are summarized in Tab. g The visualizations corresponding to Tab. 3 are depicted
in Fig.
10
Fig.
extends Fig. with P. UMAP.
Tab.
extends Tab.
with additional two datasets: MNIST Deng 2012 and FashionMNIST Xiao et al.
2017
Tab. 10
correspondingly extends Tab.
G Technical Details
To compute the ground truth SE the train set and its corresponding eigenvalues, constructed alfinity matrix W from the train set (as detailed in Appendix C.2), with number of neighbors detailed in Table 12 After constructing W, we computed the leading & eigenvectors of its corresponding Unnormalized Laplacian D - W via Python's Numpy SVD or SciPy LOBPCG SVD (depending on the size). To get the generalization ground truth, we constructed an affinity matrix W from the train and test sets combined, computed the leading eigenvectors of its corresponding Unnormalized Laplacian the representations corresponding to the test samples. We used a train-test split of 80-20 for all datasets.
D W, and extracted
For the SE implementation via sparse matrix decomposition techniques, we used Python's sklcarn.mann www.amooo...ssctcaaaaaaaaaaaaaaa
using default configuration (in particular, 10 jobs, 1% neighbors).
The architectures of Sep-SpectralNet's and Nee'' 's networks in all of the experiments were as follows:
size = 256; ReLU, size = 256; ReLU, size = 512; ReLU, size = k + 1; orthonorm. NUMAP's second NN and PUMAP's NN architectures for all datascts was: size 200; ReLU, size 200; ReLU, size 200: ReLU, size The SE dimensions for NUMAP were: Cifar10 10: Appliances 5; Wine 10; Banknote Mnist
10, FashionMnist. 10. For the datasets in Fig. 1 from top to bottom: Circles Cylinders- 11, Line 2.
22

---

Published in Transactions on Machine Learning Research (07 2025))
Table 8: A comparison between Sep-SpectralNet and SpectralNet dimensional SE and Fiedler Vector (FV) approximation on weal-wrrld datascts. The values are the mean and standard deviation of the sin? distance between the predicted and true eigenvector, over 10 runs. Lower is better. Sep-SpectralNet ability to separate the cigenvectors evident.
Dataset
Method
Cifar10
SpeetrpettalNtt
0.016.40.00
0.0522 pos
0.069 084
0.106= 030
SpectralNet
0,449.+0.
0.325
0.399
0,414 0..14
Appliances
Sep-Spectra/Net
0.0633 002
0.09914 007
0.109 001
SpectralNet
0.307+0.047
0.530++..114
0.401+00.10
KMNIST
Sep-SpectralNet
0.0.0444.po.0z
0.101
SpectralNet
0.372 174
0.396
Parkinsons
Sep-SpeetralNet
0.006++0.000
SpectralNet
0.229
Cifar10
Appliances
Wine
Banknote
MNIST
Fashion MNIST
R. P.UMAP
NUMAP-SN
NUMAP (ours))
Figure 10: Test set Visualizations of NUMAP, NUMAP-ST and P. UMAP on the datasets corresponding to Tab. and Tab. 3
The learning rate policy for Sep-SpectralNet. and SpectralNet determined by monitoring the loss on a validation set (a random subset of the training set); once the validation loss did not improve for a specified number of epochs, we divided the learning rale by 10. Training stopped once the learning rate reached 10-7 In particular, used the following approximation to determine the patience epochs, where is the number of samples and m is the batch size: if < 25, we chose the patience to be 10; otherwise, the patience decreases as max (1, 250m (i.e., the number of iterations is the deciding feature).
To run UMAP, we used Python's umap-learn implementation (UMAP's formal implementation) We used the built-in initialization option "spectral" (i.c., SE), and initialized contumely with PCA (implemented via Python's sklearn. www..ddoostiiion...... PCAA) and Sep-SpectralNet. For Parametric UMAP we used the Pytorch implementaion Liu 2024). For all methods used default choice of 10 neighbors.
Metric
Method
Cifar10
Appliances
Wine
Banknote
Muist
FashionMuist
UMAP
0.908 oos
0.953 0 oxa
0.927 o2a
0.801 610
0.71740 oos
NUMAP (ours)
0.90510.00
0.950
0.986
0.758
0.695
KNN
NUMAP-SN
0.903 002
0.956 028
0.963 032
0.750 010
0.695 oos
NUMAP-FT
0.577 1577440000 081
0.364 e5a
0.686 osd
0.329
0.153 eas
Table An extension of Tab. 4 with the MNIST and FashionMNIST datasets. Local preservation results are
corresponding to Tab. 4 Namely, NUMAP, P. UMAP and NUMAP-SN comparable, with NUMAP-FT
failing to preserve local structure.
23

---

Published in Transactions on Machine Learning Research (07/202) 2025)
UMAP
UMAP
NUMAP (ours)
timestep
Training
Inference
Figure 11: Fig.
exlension with P. UMAP.
Metric Method
Cl
Appliances
Wine
Banknote
Mnist
FashionMnist
P. UMAP
0.133
0.710
0.549
0.722
0.31 =
0.029
GS I
NUMAP (ours)
0.004
0.261 +0.000
1..14
0.618 10.18
0..0000042
0.032-0.022
NUMAP-SN
0,281
0.26220.00s
0.1
0.78020.197
0. 0.105
0.036
NUMAP-FT
0.680 20.334
0.348 28.
0.002 20.00
0.635go.141
0.41 10.15
0.272 16
Table 10: An extension of Tab. 5 with the MNIST and FashionMNIST datasets. Global preservation results on MNIST are comparable between UMAP and P. UMAP, while superior to NUMAP-SN and NUMAP-FT. Regarding FashionMNIST,, NUMAP, FashionMNIST, and NUMAP-SN are comparable, with NUMAP-FT having worse preservation.
Table 11: Technical details of the real-world datasets used for Sep-SpectralNet and NUMAP experiments.
Cifar10 60,000
Appliances 19735
KMNIST
Parkinsons 5875
Wine 178
Banknote
MNIST
PashionMNIST
#samples
70,000
1372
60,000
60,000
#features
Table 12: Technical details in the Sep- Sop-SpectralNet experiments for all datasets.
Moon
Cifar10
Appliances
KMNIST
Parkinsons
Batch size
2048
2048
2048
2048
512
neighbors
20
20
20
20
5
Initial LR
10-2
10-2
10-3
10-3
10-2
Optimizer
ADAM
ADAM
ADAM
ADAM
ADAM
24

---

Published in Transactions on Machine Learning Research (07 (07/2025)
As for the evaluation methods, used default choice of 5 neighbors to compute the kNN accuracy. The graph construction for GS is as detailed in App. C.2 using 50 neighbors to ensure connectivity.
For the computation of the p-values in Tab. we used an independent t-test to compare the means of the GS results, that were obtained from 10 runs using 10 different seeds.
Timc-serics simulation. We simulated two complex distributions 10-dimensional space. At each of the time steps, we sample total of 5000 data points, 25% of which belong the dynamic distribution (visualized by the pink dots in Fig. [], while the other two distributions kept the same. The dynamic distribution starts at the first (red) distribution, and linearly transitions into the other (blue). We used
UMAP default parameters settings to visualize each time-step separately. As for NUMAP, we trained only on the first two time-steps, and obtained the others using a simple feed-forward operation.
We ran the experiments using GPU: NVIDIA A100 80GB PCLe; CPU: Int Xeon(R) Gold 6338 CPU 2. 2.00GHz;
25


Reviews:
[{"review_id": "b8QOYuf66f", "reviewer": "Reviewer_TLxe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed Sep-SpectralNet is a conceptually straightforward extension of SpectralNet. I see this as a strength, in the sense that it can be more easily adopted than a more intríncate alternative.\n- Although limited, the experimental results seem convincing.\n\nWeaknesses:\n- It seems that  the proposed SE approximation technique is very close to PCA (or an SVD, to be more specific) applied to the vectors f(x). From Equation (1), f(X)^T f(X) \\approx L. Hence, the SVD of  f(X) approximates the eigendecomposition of L. I would appreciate a detailed discussion of this point.\n- It is unclear why identifying the exact basis helps downstream applications. For clustering and visualization, rotating the input embeddings will not provide meaningful improvements from a practical point of view. Can the authors provide a setting where finding the exact basis is of critical importance?\n- The comparison of Sep-SpectralNet with direct eigendecompositions of L in terms of runtime (Figure 2b) feels a bit unfair for a number of reasons. First, when comparing an approximate method to an exact one (up to numerical precision), one would normally analyze the relationship between error and runtime. Second, the authors should include Nystrom in that comparison, as it offers a significant acceleration over direct eigensolvers. From this perspective, and in combination with my previous point, it would be interesting to see a comparison between Sep-SpectralNet and Nystrom in terms of their errors.\n- Why are skip connections an important part of the architecture? The authors should include an ablation study that shows that skip connections are useful.\n- In a related but different line of work, Random Fourier Features RFFs rely on high-dimensional embeddings to alleviate the computational demands of downstream machine learning methods. These features, through Bochner’s theorem, are such that f(X)^T f(X) \\approx K. Since Problem (1) can be posed as a maximization of A \\^T K A, RFFs should work in theory. Of course, these features increase the dimensionality instead of reducing it, but their use has found a lot of success for many machine learning problems (for clustering, for example). Moreover, it seems like these features could be used as inputs to a NN minimizing the UMAP loss, just as NUMAP.  Of course, this approach would preclude using skip connections, but this may be cirtical or not (see my previous point).\n- From the abstract, I was expecting to find large-scale experiments with millions of points actually shoing the scalability of the approach. There are many (labeled) datasets today at these scales for this evaluation. Since this is not a paper with strong theoretical results, the empirical section should be very strong and thorough.\n- From the abstract, I was expecting a strong evidence of out-of-sample generalization by studying the relationship between the number of samples used for training and the generalization error. That is, an experiment that shows that the test error behaves nicely as function of the number of training samples.\n- The authors train NUMAP in a stage-wise fashion. Since SVDs have been incorporated before in networks, they do not preclude end-to-end training. What would be the effect of this change?\n- Finally, it seems to me that the hyperparameter t in GS has a fundamental effect on its values. For example, if multiple clusters are present in the data, a higher value of t would be needed to correctly recover these structures. I suggest including GS for multiple values of t in each experiment.\n\n- Question: what is GrEASE in Figure 9 of the appendix?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the authors propose a dimensionality reduction method with three key properties: (1) out-of sample generalization, (2) scalability, and (3) edge vector separation (i.e., basis identification). The proposed method is an extension of the SpectralNet, which naturally achieves (1) and (2), by means of a post-processing step that achieves (3). They also propose an application to data visualization.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Critical: PCA discussion\n- Critical: show why finding the exact basis is of critical importance\n- Critical: Ablation study on skip connections in NUMAP\n- Critical: the empirical section should cover scalability generalization thoroughly\n- Critical: Discussion of the hyperparameter t in the main body of the paper and its effect on the metric.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ogyes8lFGZ", "reviewer": "Reviewer_ywS6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well written and easy to understand.\n- The problem is well motivated.\n- The empirical results seem to indicate that the NUMAP algorithm does indeed preserve global structure better than existing approaches.\n\nWeaknesses:\n- The proposed technical contribution of finding the top eigenvectors seem obvious.  It does not seem like this paper adds much on a technical level to SpectralNet.\n- The evaluation of SpectralNet's ability to recover the top eigenvectors seems uninformative because it is not designed to recover these eigenvectors.\n- I'm not sure if the proposed method is required to construct NUMAP.  I would think that using SpectralNet to learn an l-dimensional representation directly and then using that to construct the NUMAP embedding should (in theory) yield the same results as first learning a k-dimensional representation and then taking its top l eigenvectors.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a simple approach to recover approximations of the top k eigenvectors of a graph's laplacian matrix.  The approach performs an eigendecomposition on the output of a trained SpectralNet, evaluated on several minibatches of data, in order to recover the eigenvectors.  The main contribution of this paper is the application of this approach for improving parametric UMAP and constructing a new metric for evaluating umap embeddings.  This new UMAP algorithm preserved the global structure of data better than existing approaches, scales to large datasets, and is easy to implement.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The technical details of the proposed approach (performing an eigendecomposition of the output of SpectralNet) should be added to the abstract or introduction clarify how exactly your approach relates to previous approaches.  I also think that the exposition should be written more from an application standpoint to highlight that this simple method can be used to substantially improve existing algorithms, as opposed to claiming a novel variant of SpectralNet.\n\nMinor changes:\n- Remove the word \"recently\" when referencing Shaham et al 2018 at top of page 2.\n- Explain what A, m and X are in definition 1.\n- Before equation 2 on page 7, write SE(X)_{1:l} to denote that Y uses the top l eigenvectors because SE(X) returns the top k.", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "R1Z549itGw", "reviewer": "Reviewer_JvCb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper addresses a relevant problem and extends existing methods to close a research gap. The proposed method is supported by extensive empirical evaluations.\n\nHowever, the readability and motivation of the paper could be improved to make it more accessible to a broader audience. I have concerns regarding the readability of the paper for readers that are not that deeply rooted in the paper's topic -- people like me.\nI think it would be beneficial if the authors could better explain the meaning and the importance of eigenvector separations, which is their central contribution. The same holds for some other aspects (see \"Requested Changes\" for more detailed comments)\n\nPlease note that I do not have any background in this paper's topic. Maybe things are \"obvious\" to readers having a background in this topic.  I would be willing to step back from my requests if I get convinced that things are already pretty clear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an extension of SpectralNet that is able to separate the eigenvectors while ensuring generalizability and scalability.  In addition, the paper also proposes a novel evaluation score, the *Grassmann score*, for better evaluating how well the global structure is preserved in graph-based dimensionality reduction methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "p.1: \"These three properties are crucial for modern applications of SE in machine learning\"\nWhy is this the case? Can you list easy-to-understand examples where all three properties are important? In particular, the last one (Eigenvector separation) seems to be quite technical, and it might not be clear for the general reader (incl. me) why this is an important property.\nThe examples in the following paragraph are not very detailed and not really convincing to me. Please elaborate more on this and assume less prior knowledge on the readers' side.\n\nThe same holds for \"diffusion maps\" in Section 4.4. I think a short statement on why they are interesting does not hurt.\n\np.2: \"Our key contributions are: ...\" I think the Grassman score is also a contribution, right? If so, why not list it there as well? This new evaluation score seems to be relevant for the community.\n\nSome references have a DOI or a link, while others do not. I recommend being consistent, ideally having a DOI or link for every reference.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)