Random Forest Autoencoders for Guided Representation Learning
Authors: Adrien Aumon
Shuang Ni
Myriam Lizotte
Guy Wolf
Kevin R. Moon
Jake Slater Rhodes
Venue: NeurIPS 2025 poster ()
Keywords: Manifold learning
Random Forest proximities
Regularized autoencoders
Semi-supervised visualization
Out-of-sample extension
Primary area: general_machine_learning
TL;DR: Guided manifold learning and semi-supervised visualization with natural out-of-sample extension based on random forest proximities and diffusion geometry-regularized autoencoder architecture.

Abstract: Extensive research has produced robust methods for unsupervised data visualization. Yet supervised visualization—where expert labels guide representations—remains underexplored, as most supervised approaches prioritize classification over visualization. Recently, RF-PHATE, a diffusion-based manifold learning method leveraging random forests and information geometry, marked significant progress in supervised visualization. However, its lack of an explicit mapping function limits scalability and its application to unseen data, posing challenges for large datasets and label-scarce scenarios. To overcome these limitations, we introduce Random Forest Autoencoders (RF-AE), a neural network-based framework for out-of-sample kernel extension that combines the flexibility of autoencoders with the supervised learning strengths of random forests and the geometry captured by RF-PHATE. RF-AE enables efficient out-of-sample supervised visualization and outperforms existing methods, including RF-PHATE's standard kernel extension, in both accuracy and interpretability.  Additionally, RF-AE is modality-agnostic, demonstrates strong robustness to hyperparameters, supports both classification and regression, and natively accommodates missing feature values. Our code is available at https://github.com/JakeSRhodesLab/RF-AE.

Random Forest Autoencoders for Guided Representation Learning
Adrien Aumon If
Shuang Ni shuang. ni @mila. quebec
adrien. aumon@umontreal.ce
Myriam Lizotte!
Guy Wolf!
Kevin R. Moon
myriam.lizot wew....itztttmmllel. euueeeeeee.....uuueee
guy. www.wo@fmmmtteel.ccmm
kevin. mon.uuuun edu
Jake S. Rhodes
rhodes@stat. edu
Université de Montréal;
Mila Quebec AI Institute
Utah State University
Brigham Young University
Equal contribution
$ *Corresponding author
Abstract
Extensive research has produced robust methods for unsupervised data visualization. Yet supervised visualization where expert labels guide representations- remains underexplored, as most supervised approaches prioritize classification over vi- sualization. Recently, RF-PHATE, diffusion-based manifold learning method leveraging random forests and information geometry, marked significant progress supervised visualization. However, its lack of an explicit mapping function limits scalability and its application to unseen data, posing challenges for large datasets and label-scarce scenarios. To overcome these limitations, we introduce Random Forest Autoencoders (RF-AE), neural network-based framework for out-of-sample kernel extension that combines the flexibility of autoencoders with the supervised learning strengths of random forests and the geometry captured by RF-PHATE. RF-AE enables efficient out-of-sample supervised visualization and outperforms existing methods, including RF-PHATE's 's standard kernel extension, in both accuracy and interpretability. Additionally, RF-AE is modality-annostic demonstrates strong robustness hyperparameters, supports both classification and regression, and natively accommodates missing feature values. Our code is available https: //gi thub. Catuu..comJJaaeeeeee wwwm//aaeSShodssslaaaaaaa..
Introduction
Manifold learning-basse based visualization methods, such as t-SNE [1], UMAP [2], and PHATE [3], are essential for exploring -igh-dimnsiionll data by revealing patterns, clusters, and outliers through low-dimensional embeddings. While these methods excel at uncovering dominant data structures, they often fail to capture task-specific insights when auxiliary labels or metadata are available. Supervised approaches like RF-PHATE [4] bridge this gap by integrating label information into the kernel function through Random Forest-derived proximities [5], generating representations that align with comain--peccife objectives without introducing the exaggerated separations or distortions seen in class-conditional methods [6]. Specifically, RF-PHATE has provided critical insights in biology,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

such as identifying multiple sclerosis subtypes, demonstrating antioxidant effects in lung cancer cells, and aligning COVID-19 antibody profiles with patient outcomes [4].
However, most supervised and unsupervised manifold learning algorithms generate fixed coordinates within a latent space but lack mechanism accommodate new observations. Therefore, to embed previously unseen data, the algorithm must rerun with the new data as part of the training set. One well-known solution to this lack of out-of-sample (OOS) support is the Nyström extension [7] and its variants, such as linear reconstruction [8] or geometric harmonics [9]. While influential, these approaches rely on linear kernel mappings and unconstrained least-squares minimization, making them highly sensitive to the quality of the training set and often inadequate for preserving complex manifold geometry in a way that generalizes effectively to new imputs [10, 11]. Recent neural network- Autoencoders based approaches, such as parametric variants of UMAP [12] or Geonetry-Regularizze
(GRAE) [13, 14), offer promising alternatives for extending embeddings to OOS data points. Yet, most of these methods focus either on unsupervised structure or on predictive performance, without explicitly preserving label-informed geometry needed for interpretable supervised visualization.
In this study, we present Random Forest Autoencoders (RF-AB), an autoencoder (AE) architecture that addresses the underexplored setting of supervised OOS visualization, while taking inspiration from the principles of GRAE [14], which uses a manifold embedding to regularize the bottleneck layer. Instead of reconstructing the original imput vectors, RF-AE incorporates supervised information by reconstructing Random Forest- Geometry- and This framework induces an embedding function that simultancously preserves local structure and class separability, and generalizes naturally to new data without requiring labels at inference. Unlike RF-PHATE [4], which provides supervised embeddings without a parametric mapping, RF-AE yields generalizable and interpretable embedding function. Compared to existing neural network-basee extensions, RF-AE incorporates a strong supervision signal tied to the relational structure between points. To further improve scalability, we also propose a prototype selection strategy that reduces memory and runtime demands during both training and inference.
(RF-GAP) proximities [5].
Through extensive experiments, we show that RF-AE outperforms existing approaches in embedding new data while preserving the local and global structure of the important features for the underlying classification task. We also demonstrate that RF-AE's performance is robust to a wide range of hyperparameter configurations. RF-AF improves the adaptability and scalability of the manifold learning process, allowing for seamless integration of new data points while maintaining the desirable traits of established embedding methods.
2 Related work
2.1 Parametric embedding through multi-task autoencoders
Given high-dimensiona training dataset X = fx; e RD I..,,,,,,N -here X can represent tabular data, images, or other modalities-a manifold learning algorithm can be extended to test data by training a neural network, typically a multi-layer perceptron (MLP) to regress onto precomputed non-paran non-parametric embeddings z, or by means of a cost function underlying the manifold learning algorithm, as in parametric t-SNE [15] and parametric UMAP [12]. However, solely training an MLP for this supervised task often leads to an under-constrained problem, resulting in solutions that memorize the data but fail to capture meaningful patterns or generalize effectively [16, 17]. Beyond implicit regularization techniques such as early stopping [18], dropout [19], or layer-wise pre-training [20], multi-task learning [21] has been shown to improve generalization. Early studies demonstrated that jointly learning tasks reduces the number of required samples [22, 23], while later work introduced trace-norm regularization
the weights of linear, single hidden-layer neural network for a set of tasks [24, 25J. Motivated by Le et al. [26], who empirically showed that training neural networks to predict both target embeddings and inputs (reconstruction) improves generalization compared to encoder-only architectures, we focus on multi-task learning- based regularization in the context of regularized autoencoders.
AE networks are built of two parts: an encoder function f(x;) = 2, E Rd (d < D), which compresses the input data into a latent representation via bottleneck layer [27], and decoder function g(2,) = x;, which maps the low-dimensional embedding back to the original input space. To ensure that (g o fxx)) x; x x,, AEs minimize the average reconstruction loss

---

L(f.g) - N Em= Lrecon(Xe (x) where Crecon (-:-) :) is typically defined as the squared Euclidean distance. AES learn compact data representations that relate meaningfully to the imput data. However, standard AEs often fail to capture the intrinsic geometry of the data and do not produce interpretable embeddings [14]. This led authors to borrow principles of manifold learning to add geometrically motivated constraints to the latent space. Structural Deep Network Embedding (SDNE) [28] preserves both first- and second-order graph neighborhoods for graph-structured data by combining adjacency vector reconstruction with Laplacian Eigenmaps [29] regularization. Local and Global Graph Embed- ding Autoencoders [30] enforce two constraints on the embedding layer: a local constraint to cluster k-nearest neighbors and a global constraint to align data points with their class centers. VAE-SNE [31] integrates parametric t-SNE with variational AEs, enhancing global structure preservation while retaining t-SNE's ' strength in preserving local structure. GRAE [14] explicitly impose geometric con- sistency between the latent space and precomputed manifold embeddings. Other approaches focus on regularizing the decoder. Inspired by Denoising Autoencoders [32], Generalized Autoencoders [33] minimize the weighted mean squared error between the reconstruction & and the k-nearest neighbors of the input x, where weights reflect the predefined similarities between and its neighbors. Centroid Encoders (CE) [34] minimize within-class reconstruction variance to ensure that same-class samples are mapped close to their respective centroids. Self-Supervised Network Projection (SSNP) [35] incorporates neighborhood information by jointly optimizing reconstruction and classification at the output layer, using existing labels or pseudo-labell generated through clustering. Neighborhood Reconstructing Autoencoders [36] extend reconstruction tasks by incorporating the neighbors of x alongside x itself, using local quadratic approximation of the decoder at fxx z to better capture the local geometry of the decoded manifold. Similarly, Geometric Autoencoders [37] introduce a regularization term in the reconstruction loss, leveraging the generalized Jacobian determinant computed at f(x = z to mitigate local contractions and distortions in latent representations. Still, most of these methods are unsupervised or apply supervision via class-conditional constraints, often leading to disrupted inter-class relationships [6, 38).
1.
2.2 Kernel methods for OOS extension
Let k(:s)) be a data-dependent symmetric positive definite kernel function (x,x') + x(x,x') x') 2
0. For simplicity, we consider normalized kernel functions that satisfy the sum-to-one property EN k(x, x,) = 1. Kernel methods for OOS extensions seek an embedding function k + f(k) = 2 E where the input k = kx = [k(x,x1)
k(x,x, xx) ]] is an N-dimensional similarity vector representing pairwise proximities between any instance x and all the points in the training set X. Under the linear assumption fkk) = kW, where W E TI xd is a projection matrix to be determined, we directly define W in the context of a regression task [39, 40] by minimizing the lqaat-s
error
Σ 1 1(k,)
which yields the explicit solution W K Y. where K-1 refers to the pseudo-invers of the train- ing Gram matrix K = [k(x,, x))lis and Y= /2f
2 contains the precomputed training manifold embeddings. In particular, for manifold learning algorithms that directly assign the low-dimensional coordinates from the eigenvectors of K, e.g.. Laplacian Eigenmaps [29], we have the well-known Nyström formula W UA 1 [7, 41, 42], where A 1 are the d largest (or smallest, depending on the method) eigenvalues of K, and U is the matrix whose columns are the corresponding eigenvectors. In Locally Linear Embedding [43] and PHATE [3], the
diag (A - Here, A;
authors suggested default OOS extension through linear reconstruction W = [2f
x
In diffusion-basee methods, this provides an alternative means to compress the diffusion process through the training landmarks and has been shown to outperform a direct application of the Nyström extension to diffusion maps [44].
Unlike parametric extensions discussed in Section 2.1, kernel extensions learn an explicit embedding function using kernel representations rather than the original representations in the feature space. While kernel methods are powerful for handling nigh-dimensional datasets, they require computing pairwise kernels for all points in the training set, which can become computationally expensive for large datasets. In such cases, feature mappings offer greater scalability. However, kernel extensions have been shown to outperform direct parametric methods in unsupervised OOS visualization and classification tasks [40, 45]. Additionally, in supervised visualization, carefully chosen kernel
3

---

mapping can effectively filter out irrelevant features, whereas feature mappings treat all features equally, potentially increasing sensitivity to noisy datasets. In particular, reconstructing RF-based proximities in an -F-plaiTE-reguulriized AE produces OOS embeddings that better align with the ground-truth RE-PHATE embedding computed on the full data [46]. We further validated in Appendix B the ability of RE-GAP inputs to stabilize supervised manifold learning on a toy dataset (Appendix C). Therefore, we align our OOS extension framework with kernel extensions rather than direct parametric methods.
3 Methods
While traditional kernel extensions offer computational benefits over parametric methods, they are limited to linear kernel mappings and are primarily designed for unsupervised data visualization or classification using either unsupervised or class-conditional kernel functions [40, 45]. In this work, we expand the search space of the standard least-squares minimization problem in Eq. 1 include general, potentially nonlinear kernel mapping functions f. We also propose supervised kernel mapping based on Random Forests, specifically tailored for supervised data visualization. Building on the previous Section 2.1, we add a geometrically motivated regularizer to this regression task within a tasti-task autoencoder framework.
In this section, we elaborate on the methodology related to our RF-AE framework to extend any kernel- based dimensionality reduction method with Random Forests and autoencoders for supervised data visualization. Specifically, we explain how we combined RF-GAP proximities and the visualization strengths of RF-PHATE with the flexibility of autoencoders to develop new parametric supervised embedding method, Random Forest Autoencoders (RF-AE). Additionally, we introduce structural importance alignment as metric for evaluating embedding models in supervised settings. Our code is available at https: /tt///gtttt githhb....... com/ SakeeSRhoessaab/RF
3.1 Extended RF-GAP kernel function
The RF-GAP proximity [5] between (possibly unseen) instance x; and training instance x,
1
c, 1 E J(())
p(x,, x,) = S
[M((1
EE
where S; denotes the set of out-of-bag trees for observation X4, Cy (t) is the number of in-bag repetitions for observation x, in tree f, I()) is the indicator function, (t(t) is the set of in-bag points residing in the terminal node of observation x; in tree t, and Mt) is the multiset of in-bag observation indices, including repetitions, co-occurring in a terminal node with x; in tree t. Note that this definition naturally extends to OOS observations x, & X, which can be treated out-of-bag for all trees. However, this definition requires that self-similarity be zero, that is, p( X;, x:) = 0. This is not suitable similarity measure in some applications. Due the scale of the proximities-th the rows sum to one [5], so the proximity values are all near zero for larger datasets-, it is not practical to re-assign self-similarities to one. Otherwise, self-similarity would carry equal weight to the combined significance of all other similarities. Instead, we assign values by. in essence, passing down an identical OOB point to all trees where the given observation is in-bag. That is, we define self-similarig as
Gt) M,(()l)
p(x,,
S
DES;
where S; is the set of trees for which x; is in-bag. Under this formulation, p(x;, x;) is on scale more similar other proximity values, and Proposition A.I (Appendix A) guarantees that, on average, p(x;, xx) > p(x,, xj). Now, we define the row-normalized RF-GAP similarity between a pair of training instances x; and x,
p(x,, xj)
p(x,, x;)
Σ/= p(x,, x,)
We intentionally applied row-normalization to restore the sum-to-one property and refocus on the underlying geometry rather than sample distribution.

---

2 E Rd
Manifold embeddings
A)
L Ligo
X e IPP
P PASCI COCCCC Groan
pi C 10.11
P [
RF-GAP
Prototypes
E IR
9
Lycon
Figure 1: RF-AE architecture with prototype selection and geometric regularization. First, the original feature vectors x; are transformed into one-step transition probability vectors p; derived from RF- GAP proximities (Section 3.1). They are further reduced into lower-dimensional vectors P that represent transition probabilities to N < N selected prototypes (Section 3.3). Meanwhile, manifold embeddings 2 are generated using RE-PHATE from the p;. Finally, P and z serve as input to network within the enclosing box, training an encoder f and decoder by simultaneously minimizing the reconstruction loss Leecon and the geometric loss Lgen defined in Section 3.2.
3.2 RF-AE architecture
To leverage the knowledge gained from RF model, modify the traditional AE architecture to incorporate the RF's learning. The forest-generated proximity measures [5], which indicate similarities between data points relative the supervised task, serve foundation for extending the embedding while integrating the insights acquired through the RF's learning process. In RF-AE, the original input vectors E IR used in the vanilla AE RF-GAP proximity vector between training instance x; and all the other training instances, including N-dimensional vector p;
replaced with the row-normalized
itself. That each input x; used for training is now represented encoding local-to-global supervised neighbourhood information around x,, defined using Eq. 2:
xi)
xxx,,x)) E 10.1]"
Since its elements sum one, p; contains one-step transition probabilities from training observation with index i to its supervised neighbors indexed 1...., , N derived from the RF-GAP proximities. Thus, the encoder fp:) = z, E Rd and decoder gz,) = p. of the unconstrained RF-AE network trained through stochastic gradient descent by minimizing the reconstruction loss L(f.g)
E Lrecon P., P.). Given learned set of cow-dimensional manifold embeddings
{z.
R2 i = 1,,,,,,, N (e.g. obtained from RF-PHATE), we additionally force the RF-AE to learn latent representation z; similar to its precomputed counterpart 2 via an explicit geometric constraint to the bottleneck layer, similar to GRAE [14]. This translates into an added term the loss formulation, which now takes the form:
T(S.9)
1 Σ N
Xrrecon (p., P.) + (1 A) geo 2f)]
The parameter À [0, 1] controls the degree to which the precomputed embedding is used in encoding X4 À = is our vanilla RF-AE model without geometric regularization, while À = reproduces z in the standard kernel mapping formulation. We the standard Euclidean distance for the geometric loss to align with the least-squares formulation in Eq. 1. While one could define the reconstruction loss the squared Euclidean distance between input vectors p; and their reconstructions, this biases learning toward zero-valued entries, which dominate in large datasets but carry little structural meaning. contrast, nonzero entries reflect meaningful links the RF-GAP graph. Although re- ce-weighting the loss to emphasize nonzeros is possible [28]. it introduces extra hyperparameters, Instead, we treat p, and its reconstruction p. (g o (p.) as probability distributions and use the Jensen-Shanma Divergence (JSD) [47] as the reconstruction loss:
Lrecom(p p.) JSD(p; p.),
Lgeo 2,¹)
z
5

---

In this work, we use the JSD reconstruction loss because of its ability to capture both local and global RF- RF-GAP neighborhoods [48]. Other divergences may also be worth exploring. For example, the Kullback-Leibbe (KI.) divergence [49] is natural candidate: RF-GAP already encodes some global structure through its adaptive kernel, and KI. places more emphasis on accurately reconstructing nonzero inputs, which fits well with our setting. That said, the exact choice of divergence is not critical for our main contribution. The key idea is that reconstructing RF-GAP vectors and training the autoencoder to match them leads to improved structural preservation. Exploring alternative divergences is left for future work. We set the latent dimension = to emphasize on visual interpretability. We use RF-PHATE as the geometric constraint due to its effectiveness in supervised data visualization [4, 50], although any dimensionality reduction method can be extended this way. Refer to Fig. for comprehensive illustration of our RF-AE architecture.
3.3 Input dimensionality reduction with class-wise prototype selection
The input dimensionality of our RF-AE architecture scales with the training size N, which may cause memory issues during GPU-optimized training when dealing with large training sets. Thus we further reduce the input dimensionality of p; from N to N < N by selecting N prototypes. The prototypes are selected using uniform class-wise k-medoids [51, 52] on the induced RF-GAP training dissimilarities. First, we max normalize the symmetrized RF-GAP x,) + p(xj,x,)) x,)] 2 to form the symmetric dissimilarity matrix [maxu,r (p'(x (x(x,, x,)} pp' ('x,, x,) E [0, 1|NxN. Then, for dataset with classes, we find No* /q-medoiid for each class using their corresponding RF-GAP dissimilarities as input to FasterPA.M [53, 54]. Let 21 {m1..... my- } denote the resulting set of medoid indices. Then instead of using RF-GAP transition probabilities from any point to every training point j as before, we form RF-GAP transition probabilities from any point to each prototype j Mi as
proximities p' xx)
p(xi, X5) Σ eem p(x;, X²)
p 5 (X,, Xm1)
pd (xi, Xmx- )], pa (x,,
Fig. contextualizes this prototype selection mechanism within our RF-AE architecture. We also note that using prototypes allows for faster OOS projections since we no longer need to compute RF-GAP proximities to all training points.
3.4 Quantifying supervised OOS embedding fit
Beyond standard k--N accuracy [12, 15, 34, 55, 56], which evaluates class separability in the embedding space, it is equally important to assess how well the embedding preserves the structure of informative features. Without this, caas-conditiooall methods that artificially inflate separation may be favored, even if they distort meaningful feature-label relationships. Conversely, purely unsupervised criteria-such as neighbor preservation [12] or global distance correlation [57]-can undervalue supervised models that discard irrelevant features aligned with the classification task.
Inspired by Rhodes et [4], we formalize structural importance alignment, which quantifies the correlation between feature importances for classification and for structure preservation. Given training/test split X Xtrain X test with labels = Yirain Yust, and embeddings femb((X= = femb (X train) U famb (Xuss) = Ztrain U Ztest from trained encoder femb, we define test-train
distance matrices in the original and embedded spaces as:
Drest =
|x] test
train
1|2,
Dembl
[i, = =| test
train l|2,
D Dtest,
Demb E R. Neca X Nataia test
Classification importances are computed using a user-defined classifier Sets : RDD + y trained on Xersin Let acc (X test Ylest) denote its accuracy on the test set. Then, the importance of feature i is:
C acc (Xuest Ytest) acc (8)())
tast> Yuest);
where x(() is the perturbed test set in which feature and its correlated features are permuted across samples (see Algorithm S1 in Appendix D).
Structural importances are computed using unsupervised structure preservation score s(.,) .) that
quantifies how well an embedding preserves pairwise relationships from the original space. Higher scores indicate better preservation of structure. We consider several commonly used definitions of

---

including local scores E {QNX, Trust) and global scores 8 {Spear, Pearson) [12, 57, 58, 59, 60]. Full definitions are provided in Appendix E.I.
Given a test set Deest and its embedding
Demb the importance of feature is then defined as: test
== s(D Deeat) - s(D (1) test D Demb)
S =
)
where D is the perturbed distance matrix obtained by replacing feature in Xtest with noise (Algorithm S1), while holding Xtrain fixed. A larger drop in s indicates that the OOS embedding relies more heavily on the structure induced by feature
To assess whether the embedding structure supports classification ceaafficaatio--ellaavi relevant features, we compute the alignment between structural importances S - {Si,..., Sp} and classification importances C = [C,,....,C C using the Kendall rank correlation coefficient T(C, S) E [-1,1] [61]. Higher values indicate that the embedding prioritizes features most relevant the classification task. Fig. S2 (Appendix E.2) illustrates this Structural Importance Alignment (SIA) framework.
Note that SIA depends on both the choice of classifier feis and structure score For feles we an ensemble with equal-weight majority voting across k-NN, SVM, and MLP classifiers to reduce model-specific bias (see Appendix G for hyperparameters). Each dataset achieves at least 60% accuracy (Appendix E.3). For s, we report four variants of SIA based on the chosen structure score, capturing both local and global structure preservation.
4 Results
4.1 RF-AE balances structural importance alignment and class separability
We assessed the trade-off between SIA and k-NN classification accuracy achieved by RF-AE against several baseline methods across 20 datasets spanning diverse domains. Each dataset contained minimum of 1,000 samples and at least 10 features. Training and OOS embeddings were generated using an 80/20 stratified train/test split, except for Isolet, Landsat Satellite, Optical Digits, USPS, HAR, OrganC MNIST and Blood MNIST, where predefined splits were used. We applied min-max max normalization to the input features prior to training and inference. Detailed descriptions of the datasets are provided in Appendix F.
Table 1 shows average SIA and k-NN accuracy scores across 20 datasets and 10 repetitions. We report separate local (s = QNX, Trust) and global (s Spear, Pearson) SIA scores. Accuracy is averaged over k = 5 to VNerain (in steps of 10) to better reflect global class separability and penalize class fragmentation. We compared RF-AE with 0.01 and N = D.1NNrain to 13 baselines, including the default RF-PHATE linear kernel extension [3] (Section 2.2). vanilla AE, principal component analysis (PCA), supervised PCA, parametric I-SNE (P-TSNE [15, 62]), parametric UMAP (P- UMAP [12, 62)), parametric supervised UMAP (P-SUMAP [12]), pairwise controlled manifold approximation projection (PACMAP [56]), CE [34], CEBRA [63]. SSNP [35] using ground-trut -trut labels, neighborhood component analysis (NCA [55]), and partial least squares discriminant analysis (PLS-DA [64, 65]). All externally sourced methods were run using their default hyperparameter settings, as specified in the original implementations. See Appendix G.1 for full experimental details. The compute resources required for the experiments include a GPU with at least 40 GB of memory and CPU with 128 GB of RAM. further details are provided in Appendix G.2.
Unsurprisingly, unsupervised methods generally rank lower than supervised approaches in terms of k-NN classification accuracy. However, even among high-accuracy models such as RF-PHATE, SSNP and P-SUMAP, we observe a notable drop in local and global SIA scores. This suggests an overemphasis class separability at the expense of preserving the underlying supervised structure. On the other hand, unsupervised methods also struggle with SIA metrics, which is expected given their objective to preserve unsupervised pairwise similarities that may be influenced by irrelevant or noisy features. In contrast, RF-AE achieves the highest k-NN accuracy by a substantial margin, while consistently ranking in the top two across both local and global SIA scores. This demonstrates RF- AE's 's ability to not only ensure class separation but also preserve meaningful supervised relationships, effectively emphasizing the most relevant features for the classification task.
RF-AE maintains strong performance across all evaluation metries when varying X, as shown in Table 2. Additional ablation studies (Appendix H.I) confirm that RF-AE's performance is
7

---

Table 1: Local (s QNX, Trust) and global -s= Spear, Pearson) SIA scores, along with test N-NN accuracies for our RF-AE method and 13 baselines. Scores are shown as mean std across 20 datasets and 10 repetitions. Methods are sorted according to accuracy. Top three values per metric are highlighted in blue, using underlined bold (first) and bold (second). the case of ties, methods are further ranked by their standard deviations. Supervised methods are marked by an asterisk.
LOCAL SIA
GLOBAL SIA
QNX
TRUST
SPEAR
PEARSON
k-NN Acc
RE-AE*
0.809 0.024
0.822 = 0.022
0.782 0.041
0.779 + 0.042
0.861 0.009
RE-PHATE*
0.798 0.025
0.825 = 0.023
0.748 = 0.038
0.750 # 0.040
0.816 0.010
SSNP*
0.760 0.047
0.772 0.045
0.685
0.694 0.080
0.809 + 0.030
SUMAP
0.756 0,028
0.768 = 0.025
0.647 0
0.647 + 0.048
0.797 + 0.011
CE
0.795 0.050
0.818
0.765 0.051
0.763 0.054
0.797 0.043
NCA
0.808 0.027
0.771 0
+ 0.033
0.760 + 0.007
PACMAP
0.749 0.026
0.758 0.025
0.68&=0.022
0.688 # 0.029
0.743
P-TSNE
0.743
0.747
0.028
0.086
0.666 + 0.038
0.712
AE
0.744 0.027
0.751
0.695 0.029
0.655 0.053
0.018
P-UMAP
0.757 0.027
0.024
0.074
1657=00033
0.655
SPCA*
0.767
0.031
0.738 0.032
0.624
PLS-DA#
0,715 0.026
0.708
0.659
0.639 = 0.028
0.592
CEBRA
0.780 0,045
0.775
0.062
0.728 # 0.068
0.042
PCA
0.745 # 0.027
0.742 a 0.026
0.727 + 0.028
0.563
robust to both À and prototype count N However, ablation experiments Appendix H.2 with alternative geometric regularizers show that these methods consistently underperform compared to our proposed RF-PHATE. Still, RF-based variants such as RF-UMAP better align with the RE-GAP reconstruction loss during training, making them interesting candidates for further exploration. Finally, RF-AH's superiority in SIA persists under varying classification importance strategies ( (Appendix H.3), reflecting stronger alignment with the ground-truth feature importance hierarchy.
Table 2: Comparison of SIA scores and k-NN accuracy across different À values for RF-AE. Each score is compared with baseline models in Table I, and highlighted only if it ranks among the top three overall. Top three values per metric are highlighted in blue, using underlined bold (first) and bold (second). RF-AE demonstrates strong robustness across broad range of X values.
LOCAL SIA
GLOBAL SIA
A
QNX
TRUST
SPEAR
PEARSON
I-NN Acc
0
0.812 0.025
0.822 + 0.023
0.751 0.038
0.752
0.836 0.011
0.001
0.809
0.024
0.824 = 0.024
0.763 0.033
0.757+0.00
0.859 = 0.009
0.01
0.809
0.024
0.822 + 0.022
0.782 0.041
0.779 + 0.042
0.861 = 0.009
0.1
0.808 0.023
0.822 = 0.021
# 0.060
0.784 # 0.054
0.864 = 0.009
1
0.804 0.023
0.820 + 0.022
0.689 0.102
0.686 1 0.108
0.864 0.009
4.2 Qualitative comparison through OOS visualizations
We qualitatively assessed the capability of four methods to embed OOS instances on Sign MNIST (A- K) and OrganC MNIST dataset. Each model was trained on the training subset, and we subsequently mapped the test set with the learned encoder. Fig. 2 depicts the resulting visualizations.
From the Sign MNIST (A-K) plot in Fig. 2a, RF-AE retains the overall shape of the RF-PHATE embedding while providing finer within-class detail. In contrast, RF-PHATE's s default kernel ex- tension compresses class clusters, making local relationships harder to discern. RF-AE expands these clusters, revealing within-class patterns, such as the logical transition between variations in shadowing and hand orientation to represent the letter "C"" (top-right cluster). This detail is lost in RF-PHATE's default kernel extension, which over-relies on diffusion and smooths out subtle differences. P-TSNE captures local structure but fragments same-class samples based on irrelevant
8

---

RF-AE (Ours)
RF- -HATE
Parametric E:SNE
Parametric Supervised UMAP
lattrr
Oplaen
0
Tight sodrey
Kicpling
Uierr
elokp voro oive
wall
Figure 2: OOS visualization using four different dimensionality reduction methods. a. Sign MNIST (A-K) dataset (Table S2): Training and test samples are shown with their original images, color-tinted by label. Training samples appear with reduced opacity. RF-AE captures supervised relationships by preserving class-specific variations, such as shadowing and hand orientation, while also highlighting inter-class similarities and maintaining clear class separability. The default RE-PHATE'S kernel extension compresses clusters excessively. Parametric I-SNE and parametric supervised UMAP demonstrate sensitivity to irrelevant features. b. OrganC MNIST dataset: Test points are shown as color-coded circles based on their labels. Training points are omitted for clarity. RF-AE clearly separates similar organ types while preserving their anatomical proximity, capturing both class identity and biological relevance. Other methods tend to merge these classes or distort their relationships, failing to reflect fine- fine-graines grained anatomical distinctions.
background variations-for example, grouping shadowed "G" and "H" instances together (bottom cluster) while separating them from unshadowed counterparts (far-left cluster). RF-AE avoids this issue, distinguishing within-class variations while preserving the class-specific clusters. P-SUMAP is also sensitive to irrelevant features and tends artificially cluster neighboring points of the same class, leading to a sparse and fragmented embedding.
From the OrganC MNIST plot in Fig. 2b, RF-AE yields the most structured and interpretable embedding, forming well-separated clusters while preserving anatomical proximity- between the left and right kidneys, or among the stomach, liver, and pancreas. RF-PHATE captures smooth transitions but merges nearby classes like the left and right kidneys, reducing local separability. P-TSNE shows significant class overlap due the absence of supervision, with only four out of eleven classes clearly separated. While P-SUMAP offers better class separability than P-TSNE, its projections remain difficult to interpret, with elongated structures (e.g., aorta and inferior vena cava) and compact, overlapping anatomical regions near the center that blur class boundaries.
This qualitative analysis of the Sign MNIST and OrganC MNIST dataset underscores the importance of regularization and methodological choices in creating meaningful embeddings for supervised tasks. RF-AE, guided by RF-PHATE regularization, effectively preserves both local and global structure, outperforming existing methods in capturing informative patterns. Additional visualizations on audio data and extended comparisons with the other models are provided in Appendices I.I and I.2.
5 Discussion
The significance of supervised dimensionality reduction lies in its ability to reveal meaningful relationships between features and labels. As shown by Rhodes et al. [4], RF-PHATE stands out as strong solution for supervised data visualization. However, it lacks an embedding function for OOS extension. To address this limitation, we designed Random Forest Autoencoders (RF-AE), an autoencoder-baut based architecture that reconstructs RF-GAP neighborhoods while preserving the supervised geometry captured by precomputed RE-PHATE embeddings. Our experiments confirmed the utility of this extension, demonstrating its ability to embed new data points while retaining the intrinsic supervised manifold structure. We quantitatively showed that RF-AE with RF-PHATE regularization outperforms baseline kernel extensions and other parametric embedding models in

---

generating OOS embeddings that preserve feature importances relevant to the classification task while maintaining class separability. We further showed in Appendix H.1 that RF-AE's performance is robust to the geometric constraint À and the number of selected prototypes N Visually, we observed that RF-AF regularized by RF-PHATE inherits the denoised local-to-global supervised structure of RF-PHATE while increasing resolution for improved within-class visualization. This results in more effective tradeoff between preserving informative structure and achieving class separability than baseline RF- kernel extensions. Other methods either over-emphasize class separability or fail to incorporate sufficient supervision, leading to noisier visualizations, Finally, RF-AE can project unseen data without labels, support semi-supervised training (Appendix J), and handle any data modality without additional preprocessing, making it well suited for semi-supereised tasks across diverse applications.
While effective, RF-AE inherits scalability constraints from the computational cost of the full RE-GAP proximity matrix during training (Appendix K). Our prototype selection strategy already yields substantial speedups over RF-PHATE's default kernel extension at inference, while upcoming vectorized and sparse RF-GAP computation (to be released soon GitHub) will further reduce this bottleneck. Future work includes investigating RF-AE's efficiency in semi-supervised hierarchical clustering and adaptive methods for prototype selection [52]. Building on our ablation studies (Appendix H.2), researchers may also explore other RF-based geometric constraints.
Broader impacts
This paper advances guided data representation learning by integrating expert-derived annotations and enabling out-of-sample extension, thus allowing generalization beyond the training set. Nonetheless, we advise users to interpret supervised 2D visualizations with caution, as label assignments may introduce biases. When labels reflect social or demographic factors, supervised methods are prone to embedding structural biases since they explicitly aim to discriminate between classes. Bias can also arise in highly imbalanced settings: the underlying Random Forest tends to favor majority classes, which can cause minority classes to appear artificially closer to or farther from other groups, as the features that characterize them may not be adequately captured in the RF-GAP proximities. These concerns are not unique to RF-AE but extend to supervised methods in general. That said, RF-AE helps mitigate such issues by avoiding the exaggerated separations often produced by purely class-conditional approaches, making it a comparatively more reliable choice in practice.
Acknowledging these limitations, our method still offers valuable support to decision-maker by providing interpretable visualizations while remaining scalable and applicable in jeupe-isueee tasks. In particular, RF-AE can assist expert- or Al-based disease diagnosis by projecting incoming patient instances into a 2D space, where they can be contextualized relative to existing embeddings. Such visualizations allow practitioners assess whether prediction is consistent with established structures or deviates from them, offering a practical indicator of prediction reliability. Overall, RF-AE has potential societal impact in biomedical research, as well as broader applications for data-driven insights in healthcare, finance, and multimedia.
Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers for their meaningful feedback. which led us to improve our paper. This research was enabled in part by compute resources provided by Mila Quebec AI Institute (mila.quebecc) and the Department of Mathematics and Statistics (Université de Montréal). This research was supported in part by the Ministry of Health and Social Services (Quebec) in collaboration ('ie--''leeeeeee dele-de-Montréall [A.A.]] Canada CIFAR AI Chair [G. W.], NSERC Discovery grant 03267 [G.W.] NIH grant R019M135929 W.J. NSF grant DMS-232721
with the Centre intégré de santé et de services sociaux
[G. (...... NSF grant 221232 [K.M], FRONT Doctoral research scholaship [M.L.] and the IVADO Visiting Scholar program [K.M]. The authors are solely responsible for the content of this work. The views expressed do not necessarily reflect those of the funding agencies. The funders had no involvement the study design, data collection and analysis, decision to publish, or manuscript preparation. The authors declare competing interests.
10

---

References
[1] L. van der Maaten and G. Hinton. Visualizing data using t-SNE. J. Mach. Learn, Res, 9:2579- 2605, 2008.
[2] Leland McInnes, John Healy, Nathamiel Saul, and Lukas Großberger. Umap: Uniform manifold approxi- mation and projection. Journal of Open Source Software, 3(29):661, 2018.
[3] Kevin R. Moon, David van Dijk, Zheng Wang, Scott Gigante, Daniel B. Burkhardt, William S. Chen, Kristina Yim, Antonia van den Elzen, Matthew J. Hirn, Ronald R. Coifman, Natalia B. Ivanova, Guy Wolf, and Smita Krishnaswamy. Visualizing structure and transitions in high-dimensional biological data. Nat. Biotechnol. 37(12):4422-142, Dee 2019.
[4] J. S. Rhodes, A. Aumon, et al. Gaining biological insights through supervised data visualization. bioRxiv, 2024.
[5] Jake S. Rhodes, Adele Cutler, and Kevin R. Moon. Geometry- and accuracy-preserving random forest proximitics. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-13, 2023.
[6] Laureta Hajderanj, Daqing Chen, and Isakh Weheliye. The impact of supervised manifold learning on structure preserving and classification error: A theoretical study. IEEE Access, 9:43909 2021.
[7] Y. Bengio, J. Paiement, et al. Out-of-sample extensions for Ile, isomap, mds, eigenmaps, and spectral clustering. NeurlPS, 16, 2003.
[8] Sam T. Roweis and Lawrence K. Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, (3215500))2233222222 2000.
[9] R. Coifiaan and S. Lafon. Geometric harmonics: novel tool for multiscale out-of-sapple extension of empirical functions. Appl. Comput. Harmon. Anal., 21(1):31 52, 2006.
[10] Alessandro Rudi, Raffaello Camoriano, and Lorenzo Rosasco. Less is more: Nyström computational regularization, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[11] Arturo Mendoza Quispe, Caroline Petitjean, and Laurent Heutte. Extreme learning machine for out-of- sample extension in laplacian eigenmaps. Pattern Recognition Letters, 74:68-73, April 2016.
[12] Tim Sainburg, Leland McInnes, and Timothy Q. Gentner. Parametric umap embeddings for representation and semisupervised learning. Neural Computation, (5(11):28112900)))
2021.
[13] Andrés F. Duque Sacha Morin, Guy Wolf, and Kevin R. Moon. Extendable and invertible manifold learning with geometry regularized autoencoders. 2020 JEEE International Conference on Big Data (Big Data), pages 0027--036,77 2020.
[14] A.F. Duque, S. Morin, G. Wolf, and K.R. Moon. Geometry regularized autoencoders. IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(6):7381-7394, 2022.
[15] Laurens van der Maaten. Leaming parametric embedding by preserving local structure. In David van Dyk and Max Welling, editors, Proceedings the Twelfth International Conference on Artificial Intelligence and Statistics, volume of Proceedings of Machine Learning Research, pages 384-391, Hilton Clearwater Beach Resort, Clcarwater Beach, Florida USA, 16-18 Apr 2009. PMLR.
[16] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. CoRR, abs/1611.003523 2016.
[17] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kan- wal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pages 233-242. PMLR, 2017.
[18] N. Morgan and Bourlard, Generalization and parameter estimation in feedforward nets: Some ex- periments. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 2. Morgan-Kaunn Kaufmann, 1989.
[19] Stefan Wager, Sida Wang, and Percy S Liang. Dropout training as adaptive regularization. In C. J. Burges,
I. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
[20] Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer wise training of deep networks. In B. Schölkopf, Systems, volume 19. MIT Press, 2006.
J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing
11

---

[21] Rich Caruana. Multitask learning. Machine Learning. 28(1):41-75, July 1997.
[22] Jonathan Baxter. Learning internal representations. In Proceedings of the Eighth Annual Conference on Computational Learning Theory, COLT '95, page 311-320, New York, NY, USA, 1995. Association for Computing Machinery.
[23] Jonathan Baxter. A model of inductive bias learning. J. Artif. Int. Res., 12(1):149-198, March 2000.
[24] Andreas Maurer. Bounds for linear multi-task learning. Journal of Machine Learning Research, 7(5):117-139, 2006.
[25] Massimiliano Pontil and Andreas Maurer. Excess risk bounds for multitask learning with trace regularization, In Shai Shalev-Shwarz and Ingo Steinwart, editors, Proceedings of the 26th Annual Conference on Learning Theory, volume 30 of Proceedings of Machine Learning Research, pages 55-76, Princcton, NJ, USA, 12 14 Jun 2013. PMLR.
[26] Lci Le, Andrew Patterson, and Martha White. Supervised autoencoders: Improving generalization performance with unsupervised regularizers. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 31.
Curran Associates, Inc., 2018.
[27] L. Theis, Shi, et al. Lossy image compression with compressive autoencoders. In ICLR, 2022.
[28] Daixin Wang, Peng Cui, and Wenwu Zhu. Structural deep network embedding. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, page 1225-1234, New York, NY, USA, August 2016. Association for Computing Machinery.
[29] M. Belkin and P. Niyogi. Laplacian cigenmaps and spectral techniques for embedding and clustering. In Proceedings of the 14th International Conference on Neural Information Processing Systems: Natural and Synthetic, N1PS'01, page 585-591, Cambridge, MA, USA, 2001. MTT Press.
[30] Rui Li, Xiaodan Wang. Jie Lai, Yafei Song, and Lei Lei. Discriminative auto-encoder with local and global graph embedding. IEEE Access, 8:28614-28623, 2020.
[31] Jacob M. Graving and lain D. Couzin. Vae-sne: deep generative model for simultaneous dimensionality reduction and clustering. bioRxiv, 2020.
[32] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Antore-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th international conference on Machine learning, pages 1096-110,, 2008.
[33] Wei Wang, Yan Huang, Yizhou Wang, and Liang Wang- Generalized autoencoder: neural network framework for dimensionality reduction. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 490-497, 2014.
[34] Tomojit Ghosh and Michael Kirby. Supervised dimensionality reduction and visualization using centroid- encoder. Journal of Machine Learning Research, 23(20): 34, 2022.
[35] Mateus Espadoto, Nina S.T. Hirata, and Alexandru C. Telea Self-supervised dimensionality reduction with neural networks and pseudo-labeling. In Christophe Hurter, Helen Purchase, Jose Braz, and Kadi Bouatouch, editors, IVAPP. VISIGRAPP 2021 Proceedings the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, pages 27-37. SciTePress, 2021.
[36] Yonghyeon LEF, Hycokjun Kwon, and Frank Park. Neighhorhood reconstructing autoencnders. In Advances in Neural Information Processing Systems, volume 34, page 536-546. Curran Associates, Inc., 2021.
[37] Philipp Nazari, Sebastian Damrich, and Fred A Hamprecht. Geometric autoencoders what you see is what you decode. Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 35834-25857 PMLR, 23-29 29 Jul 2023.
[38] Maria-Florina Balcan, Avrim Blum, and Nathan Srebro. A theory of learning with similarity functions. Mach. Learn., ((1):89-112, Aug 2008
[39] Andrej Gisbrecht, Wouter Lueks, Bassam Mokbel, Barbara Hammer, et al. Oft-of-samppl sample kemel extensions for nonparametric dimensionality reduction. In ESANN, volume 2012, pages 531-5366 2012.
12

---

[40] Andrej Gisbrecht, Alexander Schulz, and Barbara Hammer. Parametric nonlinear dimensionality reduction using kernel t-sne. Neurocomputing. 147:71-82, January 2015.
[41] Pablo Arias, Gregory Randall, and Guillermo Sapiro. Connecting the out-of-sample and pre-imase -mmae problems in kernel methods. 2007 IEEE Conference on Computer Vision and Pattern Recognition, page 1-8, June 2007.
[42] George H. Chen, Christian Wachinger, and Polina Golland. Sparse projections of medical images onlo manifolds. Information Processing in Medical Imaging: Proceedings of the Conference, 23:292-303, 2013.
[43] Lawrence K. Saul and Sam T. Roweis. Think globally. fit locally: Unsupervised learning low dimensional manifolds. Journal of Machine Learning Research, 4(Jun): 1 19-155, 2003.
[44] Scott Gigante, Jay S. Stanley, Ngan Vu, David van Dijk. Kevin R. Moon, Guy Wolf, and Smita Kr- ishnaswamy. Compressed diffusion. In 2019 13th International conference on Sampling Theory and Applications (SampTA), pages 1-4, 2019.
[45] Ruisheng Ran, Benchao Li, and Yun Zou, Kumap: Kernel uniform manifold approximation and projection for sample extensions problem, 2024.
[46] Shuang Ni, Adrien Aumon, Guy Wolf, Kevin R. Moon, and Jake Rhodes. Enhancing supervised visualization through autoencoder and random forest proximities for out-of-sample extension. In 2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1-6, 2024.
[47] Jorma Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information Theory. 37(1):145-151, 1991.
[48] Daniel Jiwoong Im, Nakul Verma, and Kristin Branson. Stochastic neighbor embedding under f-divergences. arXiv preprint 181 1.0124,, 2018
[49] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics. 22(1):79-86, 1951.
[50] Jake S. Rhodes, Adele Cutler, Guy Wolf, Kevin R. Moon. Random forest-based diffusion information geometry for supervised visualization and data exploration. 2021 IEEE Statistical Signal Processing Workshop (SSP). pages 331 331-335, 2021.
[51] Ryan Gomes and Andreas Krause. Budgeted nonparametric learning from data streams. In ICML, volume 1, page 3. Citeseer, 2010.
[52] Sarah Tan, Matvey Soloviev, Giles Hooker, and Martin T. Wells. Tree space prototypes: Another look at making tree ensembles interpretable. In Proceedings the 2020 (CM-IMS Foundations of Data Science Conference, page 23-34, Virtual Event USA, Octoher 2020. ACM.
[53] Erich Schubert and Peter Roussceuw, Faster k-medoids clustering: improving the pam, clara, and clarans algorithms. Similarity Search and Applications: 12th International Conference, SISAP 2019, Newark NJ, USA, October 2-4, 2019, Proceedings 12, pages 171-187. Springer, 2019.
[54] Erich Schubert and Peter J Rousseeuw. Fast and eager k-medoids clustering: O (k) runtime improvement of the pam, clara, and clarans algorithms. Information Systems. 101:101804, 2021.
[55] J. Goldberger, S. Roweis, G. Hinton, and R. Salakhutdinov. Neighbourhood components analysis. In Adv. Neural. Inf. Process. Systs., NIPS 04, page 513-520, Cambridge, MA, USA, 2004. MIT Press.
[56] Yingfan Wang, Haiyang Huang, Cynthia Rudin, and Yaron Shaposhnik. Understanding how dimension reduction tools work: an empirical approach to deciphering t-sne, umap, trimap. and pacmap for data visualization. Journal of Machine Learning Research. 22(201):1-73, 2021.
[57] Dmitry Kobak and Philipp Berens. The art using t-sne for single-cell transcriptomics. Nature Communi- cations, 1011)55416, November 2019.
[58] Dmitry Kobak, George Linderman, Stefan Steinerberger, Yuval Kluger, and Philipp Berens. Heavy-tailed kernels reveal finer cluster structure in t-sne visualisations. In Joint European Conference on Machine Learning and Knowledge Discovery in Databuses, pages 124-139. Springer, 2019.
[59] Jarkko Venna and Samuel Kaski, Local multidimensional scaling. Neural Networks, 19(6-7):889 899, 2006.
13

---

[60] Jacob Gildenblat and Jens Pahnke. Preserving clusters and correlations: dimensionality reduction method for exceptionally high global structure preservation. arXiv preprini arXiv:2503.07609, 2025.
[61] M. G. Kendall. A new measure of rank correlation. Biometrika, 30(112):81-93, 1938.
[62] Sebastian Damrich, Jan Niklas Böhm, Fred A Hamprecht, and Dmitry Kobak. From L-SNE to UMAP with contrastive learning. In International Conference on Learning Representations, 2023.
[63] Steffen Schneider, Jin Hwa Lec, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960):360-366 May 2023.
[64] Johan Gottfries, Kaj Blennow, Anders Wallin, and CG Gottfries. Diagnosis of dementias using partial least squares discriminant analysis. Dementia and Geriatric Cognitive Disorders, 6(2):83-88, 1995.
[65] Matthew Barker and William Rayens. Partial Icast squares for discrimination. Journal of Chemometrics: A Journal of the Chemometrics Sociery, 17(3):166-17) 173,, 2003.
[66] Wassily Hoeffding. Probability inequalities for sums of hounded random variahles. Journal of the American Statistical Association, 58(301):13-30, 1963.
[67] Hiromasa Kaneko. Cross-validated permutation feature importance considering correlation between 2022.
features. Analytical Science Advances,
[68] Charles Spearman. "general intelligence," objectively determined and measured. The American Journal of Psychology, 15(2):201-295 1904.
[69] Karl Pearson. Vii. note on regression and inheritance in the case of two parents. proceedings of the royal society of London, 58(347-352):240 2422 1895.
[70] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-leam: Machine learning Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[71] Daniel Massey. Sign language mnist. https:/ http::/wwww..kggae wtwp.kkagll..co//aaaasttttaa wetestttt/ttttattate damanugge/ si gan-langugge-mann
2017. Accessed: 2025-05-14.
[72] Yann LeCun, Corinna Cortes, and C. C. Burges. Mnist handwritten digit database. ATT Labs [Online]. 2010.
Available: hutp://yann. www.aannnwee.....ooo wwwww.com/exx//nneee
[73] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion mmist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 2017.
[74] Bob L. Sturm. The gtzan dataset: Its contents, its faults, their effects evaluation, and its future arXiv preprint arXiv:1306.1 306.11306..4.11 1461, 2013.
[75] JJJ. Hull. A database for handwritten text recognition research. IEEE Transactions Pattern Analysis and Machine Intelligence, 16(5):550-555 554, 1994.
[76] Jiancheng Yang, Rui Shi, and Bingbing Ni. Medmnist classification decathlon: A lightweight automl benchmark medical image analysis. IEEE 18th International Symposiun on Biomedical Imaging (ISBI), pages 191-195, 2021.
[77] Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1))41, 2023.
[78] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[79] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:/807.0372 2018.
[80] Omry Yadan. Hydra a framework for elegantly configuring complex applications. Github, 2019.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction accurately reflect the main contributions and scope of the paper. These claims supported in the main text and appendix through methodological descriptions and empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We explicitly discuss the limitations of our method in the discussion section (Section 5).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification: While the paper includes mathematical formulations of the proposed method and evaluation metrics, it does not present formal theoretical results with assumptions or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
The paper provides sufficient details to reproduce the main experimental results, including a description of the model architecture (Section 3.2), training procedures (Section 4.1), dataset (Appendix F) and evaluation metrics (Section 3.4). Appendix G also contains the full experimental settings to support reproducibility.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibilitty In the case of closed-source models, it may be that access to the model is limited in
16

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We release the full source code, preprocessing scripts, and instructions to reproduce all main experimental results upon publication. An anonymized version of the code and data access links included in single zip file to support reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/ /ewgddee//ooeeuamiiii...
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
gui des/ CodeeSumiiss onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
The paper specifies all relevant experimental details, including data splits, model architecture, training hyperparameters, optimization settings, and evaluation metrics. These details provided in the main text and further expanded in Appendix Section G.2 to ensure full transparency and reproducibility.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports error bars key experimental results, calculated across 10 runs with different random seeds over 20 different datasets, as described in Section 4. and Appendix Section The error bars represent the standard deviation of the metrics and are clearly indicated in the relevant tables.
17

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative етгог rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
The paper provides sufficient information on the compute resources required to reproduce the experiments, including hardware specifications and runtime estimates. These details are documented in Appendix G.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: / //wuunnurpp...// cc/ http/uublcc/Etiee ctw.ihheiineeeneeeeennnnn
Answer: [Yes]
Justification:
This research presented in the paper fully complies with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
18

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses potential positive and negative societal impacts in Ap-
pendix Section 5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The paper does not involve models or datasets that pose high risk misuse dual use. The proposed method is intended for scientific research and does not generate or process sensitive or potentially harmful content.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
19

---

Justification: All external datasets (Appendix F) and models (Section 4.1 used in the paper are properly cited with references their original sources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The paper introduces new models, which are described in detail in Section and appendix. We provide code model architecture, training procedures, and usage instructions to ensure reproducibilltyy
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper uses publicly available single-cell dataset that was collected and shared by third parties. We do not conduct any new research involving human subjects or crowdsourcing, and the data used is ethically sourced.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
20

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
The paper does not involve any direct research with human subjects or par- ticipant interaction. All data used are publicly available and were collected under proper ethical oversight by the original data providers.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: No large language models (LLMs) were used in the development of the core methods presented in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https /tc.arppp.cc/ooofeeeeeeee/////2e for what should or should not be described.
21

---

A Maximality of the RF-GAP self-similarity
Recall the RF-GAP proximity (Section 3.1) between observations x; and xj
ct(t
j=i,
|S
|M(0I
X,)
((O)GEJA 8ti,
S TES.
|M((1)
where:
S is the set of trees in which is out-of-bag (OOB).
S is the set of trees in which X; is in-bag.
(t) is the bootstrap multiplicity of x, in tree
[() is the set of in-bag points that share the terminal node with x; in tree
. M.(() is the multiset of in-bag sample indices in that terminal node (counting multiplicities).
For any fixed tree L, define the random quantities
Bt(t) = [(x is in-bag in tree t), (t)(t) / (t) and x, shares x, leaf in tree t),
(t) ~ Binomial (N, }), i I....,N, N,
Thus, considering all trees in the forest.
(t)
7,00B Σズ Bt)
and per-tree contributions self- and cross-similarity re-written as
(t(t)
G())
a: (1)
(t)
a=y(t):=
(i j).
[MOI
Under the standard Random Forests assumptions, the following holds:
Tree independence. Each tree grown from an independent bootstrap sample and an independent sequence of feature splits, ensuring i.i.d. per-tree contributions (;;(t) and (I)
Bootstrap inclusion probability. An observation is in- in tree t with probability
p:= P[B(() = (t) 2 = (t) = 0j = (1 - }/ N
1 - e Rv 0.632.
Hence,
Bt(t)
Bernoullip))
T.BB
Binomial([T], p)
7,0B
Binomial([7], p)
Co-occurrence probability. Even if x, is very similar to x,, the probability that they end up together in the same leaf and x; OOB strictly less than
4ú == P = Bt = 0 <
(i
Proposition A.1. For every fixed and any j + i, in the limit as the number of trees [T] + co. x(x,,xi) > x(x,,x,)
22

---

Proof. RF-GAP similarities are re-written as random variables:
Oat (1,
if
T
x(),, =
aig agy(t) (t), otherwise.
TOOB
By the Strong Law of Large Numbers and iree-nndependence, as (T)
- oc we have almost surely
T
(t)
[())((t) E
(.)(
[.((1]]
c:(1)
EE
c(( Mc(e)l
[t(t))
1
p
=P
T
Σ
extel (t(() - [t) = (t)(t)
T
(.(A)
(1 p) gij
es(t) |M,00]
Pi((t) = (t) =
$ (1 p) qig p.
The inequality E
B2 = follows from the fact that while ey (1) and (() have identical marginal distributions, the conditional distribution of the shared leaf size [t) is stochastically larger under the event Diş(t) 1, B, = than under (t) = alone. Indeed, conditioning out-of-bag point x, fall in the same leaf, which favors larger leaves with broader decision rules. This increases the expected denominator (f) and thereby reduces the expected normalized multiplicity (t)) Moreover, almost surely,
(,(0
(t)()= 1, - 0 p R
ciit)
[M.00]
[M.(B]
(t)
requires that the in- point x, and the
TBB T
TOOB T
E[B((t)] P.
11 - B.(1)=11-
= 11
Thus,
T
(t)
p(x;, x.)
[
T:B [T]
PM
14
p
p(X;,x;)
77
Σ T
T T.oon
1=1
(t)
E Bag (t)
(1 p) qij
Qiş n.
P
p
Since we assumed gij < 1, it follows that
Qij
lim p(x;
lim x(x;,x;).
1700000
+00
Remark A.2. Finite-[T'] concentration bounds (e.g. Hoeffding's inequality [66]) imply the same inequality holds with overwhelming probability.
Remark A.3. The assumption di < 1 is not necessary for non-strict inequality.
B RF-GAP representations stabilize supervised manifold learning
We designed our RF-AE framework under the premise that encoders operating on (supervised) kernel representations are better suited for supervised settings than those using raw input features. This
23

---

assumption stems from the ability of well-chosen kemee functions to effectively filter out irrelevant features, thereby enhancing the encoder's 's robustness to highly noisy datasets. To empirically validate this, we conducted a toy experiment using the artificial tree dataset described in Appendix C. To simulate noisy input space, we progressively augmented the dataset with additional features sampled from a uniform distribution (0, 1), corresponding to various signal-to-noise ratios (SNR) E {oo, 1, 0.1, 0.01, 0.0013. We then randomly selected 80% of each augmented dataset to train both models to regress onto the precomputed training RF-PHATE embeddings. The two MLP regressors shared the exact same architecture and hyperparameters (A (Appendix G), differing only in their input representations. We evaluated the trained models the remaining 20% test split and visualized their wwo-dimensional embeddings under different SNR conditions, along with the ground truth tree structure and median learning curves over 50 epochs across 10 repetitions, shown in Fig. SI.
The training RF-PHATE embeddings accurately capture the underlying ground-trutt structure, making them strong supervisory signal for manifold learning. Our RF-GAP- encoder proves highly robust to irrelevant features, producing well-structurrd embeddings even under severe noise conditions (e.g., SNR = 0.001). It consistently converges faster and reaches better local minimum without overfitting, as evidenced by its test embeddings (middle row), which closely mirror the ground-truth structure. In contrast, the feature-based MLP is much more sensitive to noise, with increasing training loss and disordered embeddings in both training and test sets. Even under low-noise settings (SNR oo or 1). it fails to achieve comparable performance, highlighting the superior robustness and generalization ability of our kernel-basee encoder.
C Artificial tree construction
We constructed the artificial tree data used in Appendix B following the method described in the original PHATE paper [3]. The first branch of the tree consists of 100 linearly spaced points spanning four dimensions, with all other dimensions set to zero. The second branch starts at the endpoint of the first branch, with its 100 points remaining constant in the first four dimensions while progressing linearly in the next four dimensions, leaving all others at zero. Similarly, the third branch progresses linearly in dimensions 9-12, with subsequent branches following the same pattern but differing in length, resulting in 40 dimensions. Each branch endpoint and branching point includes additional 40 points, and zero-mean Gaussian noise (standard deviation 7) is added to simulate gene expression advancement along the branches. Before visualization, all features are normalized to the range [0, 1].
D Feature correlation-aware data perturbation
In this section, we detail the procedure for generating perturbed datasets using a correlation-aware random sampling strategy [67]. Since ground-trute feature importance are rarely available, this approach is employed to generate pseudo-ground- -ruth feature importances as part of our evaluation scheme (Section 3.4). For each feature instead of permuting feature i's column values-as as in the standard permutation approach-, we reassign them by randomly sampling values from the feature space. Additionally, all other feature column values randomly replaced with a probability proportional to their absolute correlation with i. In other words, column values for features highly correlated with are also replaced by random sampling, while column values for features not correlated with i remain unchanged. This prevents the determination of fallacious feature importances where all correlated features are assigned zero importance. Refer to Algorithm SI for step-by-step description of this feature-wist wise data perturbation procedure.
E
Evaluation metric details for supervised OOS embedding
E. Structure preservation scores
Let Drest, Demb JR Ne-Noon X.aaai denote the test-train distance matrices in the original and embedded
spaces, respectively. We define four structure preservation metrics s(Dtest, Disst) ), which are used to compute multi-view structural alignment scores introduced in Section 3.4. We categorize these metries into local and global types and cite the reference works where they were previously used to assess the quality of embedding methods.
24

---

Algorithm S1: Feature- wise data perturbation with random sampling
Input: Input data X, feature correlation matrix C
Output: Perturbed datasets X for each feature Initialize list X to store perturbed datasets;
2 Generate from X by randomly sampling column values without replacement; foreach feature i do
Generate mask matrix M with elements Mii, j] E {0, 1] sampled from Bernoulli ([Cii [[C];
Build perturbed dataset: X M O X + [I M) @ X;
Store
X [i] X
7 return
Local Structure Preservation Scores
QNX (Quality of Neighborhood eXtrapolation) [12,58)
Nust
QNX D
Demby
Σ
IijENz"
tost
Ntest
K
==1
JEA km)
where Muke (i) are the indices of the smallest entries in row Dtest :] and N (im are those in D embb test :,:]
Trustworthiness [59]]
Nient
2
Trust D test: Demtt Domb)
true K).
Niest (2Ntrain
3K 1)
JEU
where U; = -ggub (1) NN (i), and true is the rank of index j in row D [i,).
Global Structure Preservation Scores
Spearman rank correlation [57]:
Spear( (Desst:
test: Demb) = corTrank (vec(Dtesi)) vec(Demb)
where vec(-) denotes vectorization and corrrank is the Spearman rank correlation [68].
Pearson correlation [60]:
Pearson D test Dembb := corr (voc (Desst) vec Doc(Deum
test
using the Pearson linear correlation [69] between flattened test-train distance vectors.
For robustness, averaged local metrics over different neighborhood sizes, ranging from K to K / /train, in steps of 10.
E.2 Illustration of structural importance alignment framework
Fig. S2 illustrates our SIA framework for evaluating supervised OOS embedding quality using the Sign MNIST (A-K) dataset (Table S2). While both RF-AE and P-TSNE produce locally plausible embeddings, their ability to preserve class-relevant structure differs significantly. RF-AE emphasizes informative regions- such as hand and finger contours while mitigating background effects. In contrast, P-TSNE attributes higher structural importance to background pixels. leading to poorer alignment with dlasiffiationnrelevannt features. This discrepancy is reflected the final local SIA scores: RF-AE achieves much higher alignment (0.85) than P-TSNE (0.55), confirming that RF-AE better preserves the semantic structure needed for accurate classification OOS settings. These findings support our qualitative observations from Section 4.2.
25

---

E.3 Baseline classifiers' hyperparameters and performance
Since ground-truth classification importances are rarely available, our SIA framework (Section 3.4) uses baseline classifier feis derive pseudo-ground-true importances. To ensure these importances are meaningful, fels must achieve reasonably high test accuracy. Table S1 reports per-dataset accuracies for both fels k-NN and an ensemble classifier fels k-NN + SVM MLP combining k-NN, SVM, and MLP predictions via squal-weigh voting. We use & v VNrain for the k- NN classifier. The SVM implemented using scikit-learn's 's LinearSVC [70] with default hyperparameters. The MLP is two-layer feedforward network with hidden dimensions h1 = -3 DJ and h2 DJ, where D is the input dimensionality. Each hidden layer is followed by ReLU activation, dropout (rate 0.2), and layer normalization. Weight normalization is applied to the first two linear layers. The final layer is standard linear projection without activation.
We find that the ensemble consistently improves upon standalone k-NN and achieves above 60% accu- racy on all datasets, making it suitable proxy for generating classification importances. Nonetheless, k--N alone performs reasonably well, falling below 60% accuracy on OrganC' MNIST dataset. For detailed comparison of SIA scores using k-NN instead of the ensemble, see Section H.3.
Table S1: Average test classification accuracy (mean + std) per dataset (see Appendix F), using single k-NN classifier (left column) and an ensemble of k-NN, linear SVM, and MLP classifiers (right column). The ensemble generally outperforms the standalone k-NN, making it robust reference for generating classification feature importances.
DATASET
k-NN
k-NN + SVM + MLP
QSAR BIODEGRADATION
0.835 1 0.032
0,854 0.028
BLOOD MNIST
0.742 1 0.000
0.761 1: 0.049
CARDIOTOCOGRAPHY
0.662 = 0.026
0.681 0.023
CHESS
0.914 0.012
0.942 0.012
DIABETIC RETINOPATHY DEBRECEN
0.657 = 0.043
0.686 - 0.031
FASHION MNIST (TEST)
0.777 = 0.007
0.827 + 0.007
GTZAN (3-SEC)
0.708 + 0.006
0.678 + 0.013
HAR (USING SMARTPHONES)
0.887 + 0.000
0.917 0.009
ISOLET
0.906 + 0.000
0.931 0.004
LANDSAT SATELLITE
0,858 = 0.000
0.829 0.010
MNIST (TEST)
0.894 + 0,008
0.922 + 0.006
OBESITY
0.625 + 0,018
0.661 + 0.026
OPTICAL BURST SWITCHING NETWORK
0.745 = 0.028
0.743 0.023
OPTICAL DIGITS
0,953 = 0.000
0,948 0.004
ORGANC MNIST
0.473 = 0.000
0.627 + 0.004
SIGN MNIST (A-K)
0.908 1 0.007
0.940 1: 0.008
SPAMBASE
0.860 0.008
0.875 0.008
SPORTS ARTICLES
0.809 + 0.019
0.818 0.019
USPS
0.871 = 0.000
0.891 0.002
WAVEFORM
0.848 = 0.012
0.860 + 0.014
F Description of the datasets
Table S2 provides additional details on the datasets used for the quantitative and qualitative com-
parisons between RF-AE and other methods. Sign MNIST (A-K) [71], MNIST (test subset) [72], Fashion MNIST (test subset) [73], GTZAN (3-second version) [74] and USPS [75] were obtained from Kaggle. The Sign MNIST (A-K) dataset subset of the original, containing the first 10 letters (excluding J, which requires motion). Blood MNIST and OrganC MNIST (Med MNIST family [76, 77)) were obtained from Zenodo. other datasets are publicly available from the UCI Machine Learning Repository.
26

---

Table S2: Description of the 20 datasets used in experiments, grouped by data modality.
DATASET
SIZE
TEST %
DIMENSIONS
CLASSES
TABULAR CLINICAL
CARDIOTOCOGRAPHY
2126
0.20
21
10
DIABETIC RETINOPATHY DEBRECEN
1151
OBESITY
2111
0.20
7
QSAR BIODEGRADATION
1055
0.20
41
2
TEXT NLP
SPAMBASE
4601
0.20
2
SPORTS ARTICLES
1000
0.20
59
2
SENSOR / TIME SERIES
HAR (USING SMARTPHONES)
10299
0.29
561
6
ISOLET
7797
0.20
617
26
WAVEFORM
5000
0.20
3
LANDSAT SATELLITE
6435
0.31
36
6
IMAGE (GENERAL)
OPTICAL DIGITS
5620
0.32
64
10
USPS
9298
0.22
256
10
MNIST (TEST)
10000
0,20
784
10
FASHION MNIST (TEST)
10000
0.20
784
10
SIGN MNIST (A-K)
14482
0.20
784
10
IMAGE (B1OMEDICAL)
BLOOD MNIST
15380
0.22
2352
8
ORGANC MNIST
21191
0.39
784
11
AUDIO
GTZAN (3-SEC)
9990
0.20
57
10
NETWORK TRAFFIC
OPTICAL BURST SWITCHING NETWORK
1060
0.20
21
4
GAMES LOGIC
CHESS
3196
0.20
36
2
G Experimental setting
G.1 Model implementations and hyperparameters
Unless otherwise specified, all methods with their default hyperparameters in experi- ments.
RF-AE: Implemented in PyTorc.. The encoder f consisted of three hidden layers with sizes 800, 400, and 100. The bottleneck layer decoder was symmetric with layers of sizes 100, 400, and 800, followed by an output layer matching the input dimensionality. ELU activations were used throughout, except for the bottleneck (identity) and output (softmax) layers. Training was performed using the AdamW optimizer [78] with a learning rate of 10-3 batch size of 512, weight decay of 10-5, and 200 epochs without early stopping. We set the default and N to 0.01 and 0.1 0.1Ntrain respectively.
dimension 2 for visualization. The
SSNP, CE, and vanilla AE: Implemented using the same architecture and activations as RF-AE. For SSNP, we followed the authors' recommendations: sigmoid output activation and a reconstactionncllasifccael activation was the identity function.
loss balance of 0.5. For CE and vanilla AE, the output
Parametric I-SNE and UMAP: Implemented following Damrich et al. [62], using the InfoNCE loss [79]; available at btttp:://gtthii com/ conm//daarcch// wtwcchhhcccceeeeeeeeeeeeeeeeeeee umap.
Parametric supervised UMAP: Official implementation from https: htmps:///itthine tiggthu..... com/
Dacinnes/uuap
27

---

PaCMAP: From https: http:://gttbb.comm
CEBRA: From https:///ithhen com
CEBRA, We used 200 training epochs and batch size of 512, recommended by the authors.
SPCA: From https: http:///ggggt.ccoo////////////// whtp:ppaaacoooooottaaaaaea Analysis.
PCA, NCA, and PLS-DA: Implemented using the scikit-leams library [70].
G.2 Compute resources
Experiments were conducted on a shared computing environment with access to both GPU and CPU resources. For models requiring GPU accelcration, used:
GPU with at least 40 GB of memory (e.g.. NVIDIA A100 40GB, H100 80GB, or equiva- lent),
. CPU with 128 GB of RAM.
For models that do not require GPU acceleration, computations were performed using CPU only, with minimum of 128 GB of RAM.
We conducted experiments across 20 datasets for our RF-AE model and 13 baseline methods, using multiple random seeds to report the mean and standard deviation of performance metrics. All hyperparameters and configurations were managed using Hydra [80]. Code and configuration files will be released to ensure full reproducibility.
The runtime for RF-AE training and the entire evaluation process for individual experiments, where each experiment is defined as running one model on one dataset with single random seed, ranged from to 6 hours depending on the dataset size.
H
Ablation experiments
H.1 Impact of the reconstruction weight and prototype count
We performed ablation experiments on the two main RF-AE hyperparameters: the loss balanc- ing parameter À and the number of selected prototypes N We report local/global SIA scores and k-NN accuracies across combinations (X, N*) E (1,0 0.01, 0.001, 0} x {pN 0.02, 0.05, 0.1, 0.2, 1) in Table S3. Surprisingly, reducing the number of selected prototypes leads to overall improvements in both k-NN accuracy while preserving SIA. We hypothesize that this may be attributed to the reduced input dimensionality of the RF-AE network, which effectively lowers its complexity and introduces additional implicit regularization. Furthermore, selecting only the most representative instances per class may help denoise the training process, thereby enhancing the model's ability to preserve class-relevant features in the embedding space.
=
For the loss balancing hyperparameter, setting À = 1 (i.e., unconstrained RF-AE) yields relatively high accuracy but results in a substantial decline in supervised structure preservation. This is expected, unconstrained autoencoders have been shown poorly capture the underlying data geometry [14, 37]. On the other hand, À == 0, which corresponds to the RF-PHATE kernel-base MLP, leads to both lower accuracy and diminished global SIA, offering no improvement over the standard RF-PHATE extension results reported in Table 1.
petpraaammettrsssppciiiiiiieee À E (0.1, 0.01, 0.001) and N E (p-Ntrai p = 0.02, 0.05, 0.1, 0.2, 1] RF-AE consistently ranks among the top methods across all metrics in Table 1, highlighting its strong robustness to hyperparameter choices. We note that adding a small geometric constraint to RF-AE improves global supervised structure while still preserving local structure. This finding aligns with the observations of Graving et al. [31], who enhanced t-SNE's (unsupervised) global structure by combining it with a VAE.
Across broad range of
Fig. S3 illustrates the impact of varying (A, E (1,0.1, 0.01, 0.001. x {pNurain
0.02, 0.05, 0.1, 1) on Sign MNIST (A-K). smaller number of selected prototypes N led to more clearly separated classes and denoised structure, in line with our quantitative findings. When RF-AE unconstrained (A 1, first column), the resulting embeddings appear more distorted and less structured. This is consistent with recent findings showing that unregularized autoencoders
28

---

Table S3: Local (s QNX, Trust) and global (s Spear, Pearson) SIA scores, and test k-NN accuracy for RF-AE variants across values of X and N (in %Nrrain). Scores are shown as mean std across 20 datasets and 10 repetitions. Each score is compared with baseline models in Table 1, and highlighted only if it ranks among the top three overall. Top three values per metric are highlighted in blue, using underlined bold (first) and bold (second). RF-AE demonstrates strong robustness for E {0.1, 0.01, 0.001} across varying prototype count, consistently ranking among the top 3 methods. Fewer prototypes improve k-NN accuracy while preserving SIA, likely due to implicit regularization and class-level denoising. Extreme values lead to degraded SIA (A = 1) or both SIA and accuracy
(A 0).
LOCAL STA
GLOBAL STA
QNX
TRUST
SPEAR
PEARSON
k-NN Acc
N
2%
0.811 0.026
0.822 0.023
0.752 0.038
0.052
0.839
0.011
5%
0.811 = 0.024
0.821 = 0.022
+ 0.752
0.743
0.838
0.010
10%
0.812 # 0.025
0.822 # 0.023
0.751 + 0.038
0.752
0.836 + 0.011
20%
0.815 # 0.024
0.822 = 0.022
0.751 # 0.037
0.752 0.040
0.833 0.010
100%
0.810
0.025
0.820 0.022
0.750 0.037
0.751 # 0.040
0.829 0.010
À 0.001
2%
0.810 = 0.024
0.824 0.023
0.771 0.037
0.764 0.040
0.857 0.009
5%
0.808 # 0.022
0.823 # 0.022
0.768 = 0.036
0.760 = 0.040
0.859 0.008
10%
0.809 = 0.024
0.824 = 0.024
0.763 = 0.037
0.757 0.039
0.859 0.009
20%
0.812 0.025
0.826 0.023
0.762 + 0.039
0756
0.855 0.009
100%
0,806 + 0.026
0.823 0.024
0.756 + 0.038
0.75240.00 0.041
0.837 0.012
=
2%
0.808 # 0.024
0.822 = 0.022
0.783 + 0.040
0.779 + 0.042
0.859
0.009
5%
0.807 = 0.024
0.822 = 0.021
0.782 + 0.037
0.779 + 0.040
0.863 0.008
10%
0.809 0.024
0.822 0.022
0.782 0.041
0.779 + 0.042
0.861 0.009
20%
0.809 + 0.024
0.023 0.822 0.023
0.778 0.040
0.775 + 0.040
0.860 0.009
100%
0,801 = 0.024
0.819 # 0.023
0.773 à 0.044
0.768 + 0.046
0.843 0.012
À = 0.1
2%
0.808 0.023
0.822 = 0.023
0.023
0.777
0.045
0.780 + 0.043
0.862
0.010
5%
0.807 0.023
0.822 = 0.021
0.021
0.778 0.047
0.781 + 0.047
0.865
0.008
10%
0.808 + 0.023
0.822 = 0.021
0.782 0.060
0.784 + 0.054
0.864
0.009
20%
0.807 = 0.025
0.820 = 0.022
0.780 + 0.049
0.783 + 0.049
0.861 0.010
100%
0.802 # 0.022
0.817 # 0.024
0.780 # 0.059
0.784 + 0.059
0.843 0.012
A=
2%
0.808 0.026
0.822 0.023
0.681 + 0.113
0.681
0.865 0.009
5%
0.806 = 0.024
0.820 0.023
0.673
0.670
0.867 0.009
10%
0,804 = 0.023
0.820 = 0.022
0.68910.100
0.186
0.864 0.009
20%
0.804 # 0.024
0.819 = 0.023
0.697 0.118
0.114
0.860 0.010
100%
0.799 # 0.023
0.812 = 0.024
0.717
0.07
0.808 0.020
often fail produce human-interpretable visualizations that preserve the intrinsic geometry of the data [14, 37]. On the contrary, full geometric constraint (A = O, last column) simply replicates the RF-PHATE embedding, without clear qualitative benefits compared the default linear kernel extension (Fig. 2). To effectively balance reconstruction and geometric losses, the optimal range for À lies approximately between 0.001 -yi-lddg branching structures akin to RF-PHATE but with more pronounced separation-and and 0. which produces more compact, globular embeddings with enhanced class separability. similar qualitative assessment can made for OrganC MNIST in Fig. S4.
29

---

From these results, we draw two practical guidelines to help users select suitable hyperparameters for their specific application:
Loss balancing parameter A: Values of À in the range [0.001,0.] 0.1] yield comparable scores but differ in qualitative behavior. Lower values (e.g., À R 0.001) produce branching structures similar to RF- PHATE, enhancing interpretability of inter-class transitions while mitigating the comp-compe artifacts seen in RF-PHATE. Higher values (e.g. À 0.1) shift the focus toward class separability and expanded within-class structure. We recommend À R 0.001 for capturing smooth transitions or trajectories, and 0.1 for emphasizing distinct class boundaries and detailed internal structure.
Prototype selection N Selecting as few as 2% of training points as prototypes is a good starting point preserve supervised structure while maximizing class separability. If minimizing inference time is essential, users may further reduce the number of selected prototypes accelerate computation.
H.2 Compatibility with other geometric regularizers
Although our RF-PHATE regularizer is the core focus of our paper, we also investigate the per- formance of RF-AE under alternative geometric constraints to guide users toward potential sub- stitutes. Table S4 reports ablation results using RF-PHATE (ours), UMAP, SUMAP, and RF- UMAP (i.e., UMAP applied to RF-GAP dissimilarities). We fixed the default hyperparameters to NA) = (0.01, 0.1 ). Across 20 datasets, RF-AE with RE-PHATE consistently achieved the best overall performance. RF-AE with RF-UMAP ranked second, followed by RF-AE with SUMAP and UMAP constraints. On a per-dataset basis, RF-AE (RF-UMAP) was competitive with RF-AE (RF-PHATE) OrganC MNIST but performed substantially worse Sign MNIST. These results suggest that RF-AE is especially effective when paired with RF-based kernel meth- ods ods-particularly RF-PHATE--WE which already capture the underlying RF geometry. In such cases, the geometric and reconstruction objectives are well aligned, enabling more effective multi-task learning.
Figure S5 shows the OOS visualizations using these four regularizers. On Sign MNIST (Fig. S5a), RF-AE with RE-UMAP still splits same-class clusters, similar to UMAP and SUMAP, though less severely. RF-AE with UMAP or SUMAP attempts to merge same-class fragments, but misalignment with RE-GAP geometry leads to higher class overlap than their parametric counterparts (Fig. S6). On OrganC MNIST (Fig. S5b). which inherently contains less background noise than Sign MNIST, RF- AE with RE-UMAP better highlights anatomical relationships compared to P-UMAP or P-SUMAP, but still shows more noise and overlap than RF-PHATE. This supports the idea that RF-PHATE more effectively captures denoised local and global supervised structure through diffusion, as demonstrated empirically in prior work [4].
In summary, RF-PHATE is strong default regularizer for RF-AE overall, though alternative RF-based regularizers like RE-UMAP may offer valuable refinements in specific scenarios.
H.3 SIA performance comparison under different classification importance strategies
To show that RF-AE's performance is not dependent the choice classification importances C, (Section 3.4), we repeated the quantitative analysis from Section 4.1 using two alternative strategies:
k-NN strategy: We replaced our baseline ensemble classifier with standalone K-NN model.
Aggregate strategy: Instead of deriving feature importances from the ensemble's cs accuracy drop, we computed importances independently using each of the three classifiers-k-NN SVM, and MLP MLP-resulting in the following sets:
NN NN
{C/- i 1, D),
CSVM
1 1 D). {C.VM
MLP
| 1, DJ. LCMLLP
We then averaged these to obtain an aggregated importance set:
1
(Ck-NN + CSVM + CMLP).
3
30

---

Table S5 reports local and global SIA scores for RF-AE and 13 baseline methods using our proposed ensemble classifier (Sections 3.4 and 4.1) as well as the two alternative importance strategies. Overall, RF-AE consistently ranks among the top three methods across all metrics, regardless of the chosen importance strategy. This suggests that RF-AE more effectively preserves the underlying important structure, making it more likely to reflect meaningful feature hierarchies in its embeddings compared to other baselines.
I Extended visualizations and quantitative comparisons
L1 Visualizations on image data
We present OOS visualization plots and quantitative comparison (Table S6) for all models on Sign MNIST (Fig. S6) and OrganC MNIST (Fig. S7) to support our analysis in Section 4.2.
Table $6 shows the local (s = QNX. Trust) and global (s Spear, Pearson) SIA scores, along with test k-NN accuracies for RF-AE and 13 baseline methods on the Sign MNIST and OrganC' MNIST datasets. Our RF-AE method consistently ranks among the top three across all scores on both datasets.
Fig. S6 presents visualizations of all models for the Sign MNIST (A-K) dataset. RF-AE effectively inherits the global structure of the RE-PHATE embeddings while providing greater detail within class clusters. In contrast, RF-PHATE tends to compress representations within each cluster, which are associated with individual classes. Although OOS embeddings are mostly assigned to their correct ground-truth labels, the local arrangement of these samples on the sub-manifold is not easily visualized in RF-PHATE. RF-AE, however, expands the class clusters, revealing within-class patterns that are obscured in the RF-PHATE plot. For example, the top-right cluster in the RF- AE plot illustrates different ways to represent the letter "C"" showing a logical transition between variations based on hand shadowing and orientation. Such nuanced differences are more challenging to detect in RF-PHATE, which compresses these representations into an overly restrictive branch structure. This limitation RE-PHATE may stem from excessive reliance on the diffusion operator, which overemphasizes global smoothing. Since RF-GAP already captures local-to-globe supervised neighborhoods effectively, the additional diffusion applied by RE-PHATE likely diminishes fine- grained local details. Thus, we have demonstrated that RF-AE offers superior balance for visualizing the local-to- to-gbballl supervised structure compared to the basic RF-PHATE kernel extension.
P-TSNE is effective at identifying clusters of similar samples but often splits points from the same class into distinct, distant clusters. This appears to result from variations such as background shadowing, which obstruct the important part of the image. Thus, "G" and "H" instances are closer than expected due to similar shadowing. contrast, RF-AE correctly assigns "G" and "H" instances to their own clusters while dissociating between same-class points with different shadowing, effectively reflecting within-class variations. This demonstrates that P-TSNE is overly sensitive to irrelevant factors, such background differences, which are unrelated to the underlying labels. Similarly, P-UMAP, P-SUMAP and PACMAP exhibit this sensitivity but produces sparser representations. Despite being supervised method, P-SUMAP incorporates class labels in a way that artificially clusters same-class points, potentially oversimplifying their intrinsic relationships. CEBRA yields a circular pattern that offers limited utility for qualitative interpretation. CE and SSNP embeddings are distorted. NCA retains decent local and global relationships, but within-class variations and transitions between classes visually less evident than in regularized RF-AE. Other methods produced noisy embeddings.
For the OrganC MNIST dataset, all models are visualized in Fig. S7. As analyzed in Section 4.2, RF-AE achieves notable improvements over competing methods by enabling finer distinctions between organ types. This is particularly evident in its ability differentiate the left and right kid- neys-whereas other methods tend to merge these classes, RF-AE separates them while maintaining their proximity in the embedding space. This reflects anatomical similarity without losing class identity.
In comparison, RF-PHATE maintains the overall structure but merges certain classes (e.g., left/right kidneys), thereby reducing fine-grained resolution. P-TSNE and P-UMAP recover local structure but yield overlapping clusters due to the lack of supervision, resulting in cluttered embeddings that hinder interpretation. P-SUMAP achieves better class separability than P-TSNE and P-UMAP, but its projections remain difficult to interpret, with elongated structures (c.g., aorta and inferior vena
31

---

cava) and compact, overlapping anatomical regions near the center that obscure class boundaries. NCA, PLS-DA, SPCA, and PCA produce noisy visualizations with weak separation of organ types. reflecting limited class-specific representation. Outliers in the CE and SSNP plots suggest overfitting to the training data. PACMAP exhibits broken structures, where organ clusters are artificially split without clear biological meaning. CEBRA displays an artificial circular pattern, while the AE produces distorted visualizations.
Overall, RF-AE preserves the structural integrity of the data while substantially enhancing class separability, These qualitative findings align with the quantitative results Table S6, where RF-AE outperforms competing methods in both k--N accuracy and local-to- local-to-global SIA.
I.2 Visualizations on audio data
To further demonstrate the modality-agnostic performance of RF-AE, present quantitative and qualitative comparisons on GTZAN (3-sec) in Table S6 (bottom) and Fig. S8, respectively. Although RF-AE shows slightly lower class separability than models such CE and SSNP, these methods suffer from strong structural distortions, reflected in their low Global SIA scores. In contrast, RF-AE achieves robust supervised structure preservation across local and global scales while maintaining competitive class separability.
Visually, the class relationships in RF-AE align well with our general understanding of genre similarities and distinctions. For instance, classical, metal, reggae, and hip-hop appear as more "extreme" genres, while disco, rock, country, and blues cluster near the center, reflecting their less distinctive "sound color" and stronger similarities to one another. The proximity of classical and jazz is intuitive, appear close due to their common reliance on electric guitars and the overlap between subgenres like heavy metal and hard rock. Note that the relatively small sample size and possible biases in label assignment or class-wise sampling may still influence the results. This qualitative analysis is meant to illustrate that RF-AE captures a meaningful and balanced structure in its embeddings, opening the door to further exploration. Future work could examine within- and between-genre variations by coloring points according key acoustic features.
both often rely on acoustic, traditional instruments. Similarly, metal and rock
In contrast, RF-PHATE produces a similar global layout but smooths away important within-class variations, oversimplifying the diversity within each genre. Although CE and SSNP achieve better average class separation, they tend to represent classes as compact, globular clusters. This can be misleading, it may suggest that musical genres share similar internal structure. In addition, these methods often introduce distortions, elongating some structures while compressing others near the center, which hinders effective visual exploration. PACMAP, P-SUMAP, P-UMAP, and P-TSNE tend to fragment into small clusters, even within genres, making it difficult to observe gradual transitions within and across musical styles. CEBRA again produces its characteristic eircular pattern, while the remaining methods yield noisier and less interpretable visualizations.
J Semi-supervised training
While did not experiment with partially labeled data, RF-AE can also be trained in semi- supervised setting. As described in Section 3.1, our extended RF-GAP definition supports computing proximities between training and out-of-sample points. Thus, on the one hand, assuming Nr labeled points and Ny unlabeled points, for total training size of N = Ni + No. we treat unlabeled training samples as -futll--sample" and compute N proximity vectors of size Nz. which are used as input to train the RF-AE network. On the other hand, to generate RF-PHATE embeddings for the full training set, we can rely on the Landmark PHATE algorithm proposed by Moon et al. [3]. First, we construct the RF-GAP kernel matrix of size NL x NL between labeled landmarks and compute their embeddings with PHATE. Then, we project the Nü remaining unlabeled points with the linear landmark extension using their RF-GAP proximities to the labeled points (the landmarks), as in Section 2.2. This provides all the key ingredients to train RF-AE by leveraging both labeled and unlabeled data. We leave this extension for future work.
32

---

K Runtime comparison
We report training and runtimes for each model on Sign MNIST (A-K) and OrganC MNIST in Fig. $9. To assess our scalability improvement from our prototype selection, we include results for RF-AE with different prototype percentages, N E {0.1N N.O2Neraam). We set the geometric weight to its default value À 0.01. During training, RF-AE remains within the same order of magnitude (OoM) as RF-PHATE, P-SUMAP, and NCA, while being one OoM slower than CE and SSNP. At inference, RF-AE is roughly two OoM faster than RF-PHATE and one OoM slower than P-SUMAP. Compared RF-PHATE, these improvements at inference stem from prototype selection, which avoids the costly computation of proximities to all training points. Combined with ongoing vectorized and sparse RF-GAP computation, we expect this strategy to substantially narrow, not eliminate, the runtime gap with other supervised competitors such P-SUMAP.
33

---

a.
b.
Ground truth
Median training curves
RP P--IIATE regresser
Kandi based NLP
Feature-based
Mean squared errer
SNR
C.
Epoch
SNRmaa 00
SNR-1 11
SNR=0.1
SNR=0.001
RF- PHATE (training target
Training
Kernel- MLP
Rest
Taining
Tua
Figure S1: Comparison between the standard feature-based MLP encoder and our proposed RF-GAP kernel-based MLP encoder for regressing onto precomputed RF-PHATE embeddings. (a) Ground- truth tree structure with branch labels (see Appendix C). (b) Log-scaled median training MSE with 25th and t5th enclosing percentiles over 50 epochs across 10 repetitions.
(c) Training RF-PHATE embeddings (top row), followed by training and test embeddings produced by the RF-GAP- based encoder (middle row) and the feature- based encoder (bottom row) after 50 epochs from a single run. The RF-PHATE embeddings closely match the ground-truth structure and provide strong target for supervised regression. Our kernel-based encoder remains effective even under high noise levels (e.g., SNR 0.001), converging more quickly and producing well-structured embeddings with better generalization. contrast, the feature-based MI.P exhibits increasing training loss and disorganized embeddings noise increases, and often fails recover meaningful structure even in low-noise settings (SNR = 00, 1), demonstrating the superior robustness of our kernel-based encoders.
34

---

a.
b.
RF-AE
Parametric E-SNE
Samb
Classification importances
Structural importances
C.
High d.
importance
Importance alignment
e.
T) =
T(C,5) 0.55 0.55
Figure S2: Illustration the structural importance alignment (SIA) score defined in Section 3.4 for evaluating supervised out-of-sample (OOS) embedding fit. Random class samples from the high-dimensional Sign MNIST (A-K) dataset. 2D embeddings of training and test (OOS) points from RF-AE (left) and P-TSNE (right), based are shown with their original images, color-tinted by label; training samples appear with reduced opacity. Pixel-level classification importances from the ensemble baseline classifier (Section 3.4, Appendix E.3), normalized to [0, 1]. d. Pixel-level local structure importances (s Trust) from OOS RF-AE (left) and P-TSNE (right), also normalized. e. Local SIA scores computed as the Kendall correlation between (c) and (d): RF-AE achieves higher alignment (0.85) than P-TSNE (0.55), suppressing background pixels and focusing on class-relevant regions.
stratified 80/20 random split. Samples
35

---

A=1
= 0.1
À 0.001
A 0.001
A
0.002Naiin
N' 0.055Mrain
N = 0.1Naainn
N = 0.2Ntainn
N Musin
Figure S3: RF-AE test embeddings on Sign MNIST (A-K) for various (A, configurations, where À decreases column-wise from 1 (unconstrained RF-AE) 0 (RF-PHATE kernel- based MLP extension), and N* increases row-wise from 2% to 100% the training set size. Samples are shown with their original images, color-tinted by label (see Fig. 2 for the legend).
36

---

A=1
= 0.1
A 0.01
A 0.001
A mm
0..02Mainn
N' 0.055Mrain
N = 0.1inainn
N = 0.2NNaiin
= Muain
Figure S4: RF-AE test embeddings on OrganC MNIST for various (A, .N*) configurations, where decreases column- from (unconstrained RF-AE) 0 (RF-PHATE kernel-based MLP extension), and N increases row-wise from 2% to 100% of the training set size. Points are colored by their ground-truth labels (see Fig. for the legend).
37

---

Table S4: Local (s = QNX, Trust) and global (s = Spear, Pearson) SIA scores, along with test k-NN accuracies for our RF-AE method using four different geometric regularizers: RF-PHATE (ours), RF-UMAP, UMAP and SUMAP. Scores are shown as mean + std across 10 repetitions on Sign MNIST (top), OrganC MNIST (middle), and over 20 datasets (bottom). Refer to Table S2 for summary of the 20 datasets. Each score is compared with baseline models in Tables and S6, and highlighted only if ranks among the top three overall. Top three values per metric are highlighted in blue, using underlined bold (first) and bold (second).
LOCAL SIA
GLOBAL SIA
QNX
TRUST
SPEAR
PEARSON
k-NN Acc
GEO. REG.
SIGN MNIST
RE-PHATE
0.819 0.006
0.848 + 0.006
0.700 0.109
0.681 = 0.135
0.988
0.003
RF-UMAP
0.732 0.008
0.717 0.009
0.624 0.043
0.600
0.936 0.016
UMAP
0.642 0.012
1.54400011
0.319 4 0.048
0.235 0.070
0.745 + 0.021
SUMAP
0.668 0.012
1.594
0,461 # 0.069
0.414
0.863 + 0.013
ORGANC MNIST
RE-PHATE
0.890 = 0.007
0.929 + 0.006
0.901 0.013
0.898 = 0.012
0.766 = 0.004
RE-UMAP
0.889 0.006
0.924 # 0.005
0.936 0.004
0.933 0.005
0.701 0.004
UMAP
0.883 0.007
0.907 # 0.006
0.871 0.006
0.869 = 0.006
0.576 = 0.007
SUMAP
0.875 + 0.008
0.909 + 0.006
0.888 + 0.009
0.875 # 0.008
0.740 + 0.013
20 DATASETS
RE-PHATE
0.809 + 0.024
0.822 + 0.022
0.782 0.041
0.779 + 0.042
0.861 + 0.009
RE-UMAP
0.798 : 0.024
0.806 0.022
0.773 0.031
0.768 0.032
0.832 0.012
UMAP
0.782 0.025
0.762 + 0.029
0.683 0.036
0.674 = 0.038
0.729 = 0.024
SUMAP
0.791 000024
0.788 + 0.024
0.669 + 0.050
0.669 = 0.048
0.817 = 0.017
RF-AE (RF-PHATE)
RF-AE (RF-UMAP)
RF-AE (UMAP)
RF-AE (SUMAP)
Organ
right sodhey
cray
Piophap
Uver
eflior varaa
Figure S5: OOS visualization using RF-AE with four different geometric regularizers: RE-PHATE (ours, far left), RF-UMAP (center left), UMAP (center right) and SUMAP (far right). We set the geometric and reconstruction weights to their default values NX = (0.01, 0.1N a. Sign MNIST (A-K): Samples shown with their original images, color-tinted by label. Training images are shown with reduced opacity. RE-AE with RE-UMAP still fragments same-class points, reflecting the same weaknesses (un)supervised UMAP (Fig. S6). RF-AE with UMAP SUMAP attempts to merge same-class fragments but misalignment with RF-GAP geometry leads to greater class overlap than their parametric baselines. b. OrganC MNIST: Test points are color-coded by label. Training points are omitted for clarity. RF-AE with RE-UMAP performs better, producing a structure closer to RF-AE with RE-PHATE. The reduced artifact level (e.g., less background noise) facilitates clustering of same-class points. Still, RF-UMAP remains slightly noisier than RF-PHATE, with higher class overlap, and RF-AE with UMAP or SUMAP shows no improvement over their parametric counterparts in Fig. S7.
38

---

Table S5: Local (s QNX, Trust) and global (s Spear, Pearson) SIA scores for RF-AE and 13 baseline methods, computed using three strategies: our default ensemble importances suggested in Section 3.4 (top), k-NN-based classification importances (middle) and aggregated importances averaged over k-NN, SVM, and MIP classifiers (bottom). Scores are reported as mean + standard deviation across 20 datasets and 10 repetitions. In general, RF-AE outperforms other models in both local and global SIA, regardless of the importance strategy. Top three values per metric are highlighted in blue, using underlined bold (first) and bold (second). Supervised methods are marked by an asterisk.
LOCAL SIA
GLOBAL SIA
QNX
TRUST
SPEAR
PEARSON
ENSEMBLE IMPORTANCES
RF-AE*
0.809 = 0.024
0.822 0.022
0.782 + 0.041
0.779 + 0.042
RF-PHATE
0.798 = 0.025
0.825 0.023
0.748 + 0.038
0.750 0.040 0.040
SSNP*
0.760 0.047
0.772 #
0.685 # 0.089
0.694 0.080
P-SUMAP*
0.706
0.768 0.025
0.647 0.048
CE
0.818 0.051
0.765
0.763 0.054 0.054
NCA
0.808 = 0.027
0.805 0.025
0.771
0.759 0.033
PACMAP
0.749
0.758
0.688
0.089
P-TSNE
4.022
0.027
0.068 0.084
AE
0.027
0.001
0.65 0.044
P-UMAP
0.757 + 0.027
0.744
0.657 0.674
SPCA*
0.767 = 0.026
0.759 0.030
0.741
0.738 0.032
PLS-DA R
0.026
0.708 0.028
0.639 0.029
CEBRA
0.780 = 0.045
0.775 0.050
0.735
0.728 0.068
PCA
0.745 0.027
0.742 0.026
0.733 # 0.027
0.727 = 0.028
STANDALONE N-NN IMPORTANCES
RF-AE*
0.835 = 0.021
0.832 0.021
0.784 0.036
0.788 # 0.038 0.038
RF-PHATTE
0.834 = 0.024
0.836 0.023 0.023
0.750 # 0.035
0.760 0.040
SSNP*
0.750
0.779 0.046
0.094
0.690
P-SUMAP*
0.780
0.788 + 0.023
0.666400044
0.666 = 0.049
CE
0.829 0.050
0.821 0.048
0.763 0.051
0.760
NCA*
0.826#0.022
0.811
0.774 0.032
0.761 # 0.031
PACMAP
0.771
0.777
0.708 0.027
0.711 0.028
P-TSNE
0.006
0.027
0.000
AE
0.762
0.769
0.046
0.668 0.054
P-UMAP
0.777
0.022
0.695
0.676
SPCA* *
0.785
0.777
0.753
0.026
PLS-DA
0.724
0.002
0.654 0.003
0.025
CEBRA
0.806
0.784
0,736 0.058
0.731
PCA
0.755 0.003
0.752 0.022
0.741 + 0.023
0.736
AGGREGATED
IMPORTANCES
RF-AE*
0.802 = 0.045
0.812 0.044
0.778 # 0.056
0.778 0.056 0.056
RF-PHATE"
0.793
0.820
0.043
0.747
0.752
SSNP*
0.764
0
0.685
0.695
P-SUMAP*
0.047
0.767
0.647 0.065
0.646
CE*
2.98800006
0.819
0.771
0.772
NCA*
0.812 0.045
0.046
0.774 0.048
0.762
PACMAP
0.749 + 0.046
0.044
0.688 0.044
0.690
P-TSEE
0.044
0.747
1684400044
0.667
AE
0.745 + 0.043
0.750
0.695 0.066
0,066
P-UMAP
0.7604
0.044
0.074
0.657
SPCA*
0.770+0.044
0.761 0.047
0.742
0.739
PLS-DA
0.717 0.043
0.710 0.044
0.66440.033
0.0040
CEBRA
0.782 # 0.063
0.778 + 0.068
0.739 # 0.073
0.733
PCA
0.746 + 0.045
0.743 + 0.044
0.002
0.729
39

---

Table S6: Local (s QNX, Trust) and global (s Spear, Pearson) SIA scores, along with test k-NN accuracies for our RF-AE method and 13 baselines. Scores are shown as mean - std across 10 repetitions on Sign MNIST (top). OrganC MNIST (middle) and GTZAN (bottom) (see Table S2 for summary of the datasets). Top three values per metric are highlighted in blue, using underlined bold (first) and bold (second). Supervised methods marked by asterisk.
LOCAL STA
GLOBAL STA
QNX
TRUST
SPEAR
PEARSON
k-NN Acc
SIGN MNIST
RF-AE
0.819 0.006
0.848
0.700 = 0.109
0.131
0.988 + 0.003
RF-PHATE*
0.817 = 0.009
0.854
0.571 0.099
0.434
0.976 0.004
SSNP*
0.139 0.401
0.249
0.174
0.214
0.189 0.258
P-SUMAP*
0.700 =700000011 0.010
0.618 0.009
0.449
0.401
0.967
CE*
0.440
0.427
0.
0.646
0.179
NCA
0.003
0.873 0.012
0.086
0.110
0.984 0.002
PACMAP
0.007
0.006
0.022
0.029
2.930
P-TSNE
0.010
0.025
0.004
0.210
0.806 0.032
AE
0.019
0.046
0.403
0.181
0.524
P-UMAP
0.012
0.001
0.004
0.787 0.064
SPCA
0.005
0.598
0.012
0.019
0.479 0.009
PLS-DA
0.740
0.709
0.011
0.735
0.357 + 0.008
CEBRA
0.742
0.134
0.129
0.564
4.009
PCA
0.660 0.011
0.588 0.015
0.016
0.589 0.012
0.314 0.006
ORGANC MNIST
RF-AE"
0
0.929 0.006
0.901 # 0.013
0.898
0.766 0.004
RE-PHATE*
1.992
0.006
0.898 # 0.009
0.896
0.654 0.008
SSNP*
0.871 0.028
0.019
0
0.636 0.084
P-SUMAP=
+.03000006
0.096
0.006
0.018 0.006
CE*
0.870 0.022
0.887
0.846 0.076
0.570 0.
NCA
0.892 0.006
0.005
0.005
0.868 + 0.005
0.524 # 0.000
PACMAP
0.881 = 0.007
0.007
0.902
0.893 0.007
0.632 + 0.009
P-TSNE
0.867
0.092
0.004
0.871
0.474 0.003
0.006
0.009
0.011
0.834
A.633
P-UMAP
0.008
0.005
0.868 + 0.005
# 0.005
SPCA*
0.916
0.006
0.005
4.886
0.429 # 0.000
PLS-DA
#866000006
0.869#0.005
0.005
0.859
0.358
CEBRA
0.858 0.033
0.881
0.030
0.022
0.358
PCA
0.861 0.005
0.005
0.865 + 0.005
0.861 0.005
0.004
GTZAN (3-SEC)
RF-AE*
0.956 + 0.007
0.946 = 0.005
0.935 0.011
0.914 0.008
0.688 # 0.005
RF-PHATE*
0.954 0.005
0.943 0.008
0.912 0.015
0.907 0.014
0.568 0.0100 0.010
SSNP*
0.940 0.006
0.001
0.788 0.056
0.778
0.786 0.005
CE*
0.951
0.001
0.046
0.049
0.713 0.012
P-SUMAP
4.944000005
0.005
0,010
0.622
0.696 0.007
NCA
0.949 = 0.005
0.005
0.925 0.007
0.788
0.824
0.518 2.008
PACMAP
+.42000000
0.946 0.004
0.004
0.692
0.644 + 0.009
P-TSNE
0.950
0.901
0.017
0.787
0.519 0.007
AE
0.006
0.005
0..47
0.487 + 0.009
P-UMAP
0.949
0.937 0.000
0.023
0.004
0.493 # 0.040
SPCA
0.948 0.003
0.932
0.894
0.900
0.417 0.009
CEBRA
0.839 0.058
0.841 0.055
0.786
0.779
0.309 # 0.020
PLS-DA
0.848 0.015
0.011
0.014
0.706
0.398
PCA
0.943 + 0.006
0.926 0.000
0.887 0.009
0.889
0.404
40

---

RF-AE (Ours)
RF-PHATE
P-TSNE
P-UMAP
P-SUMAP
NCA
PACMAP
CE
SSNP
CEBRA
PLS-DA
SPCA
PCA
AE
Lotter
Figure S6: Visualization of the Sign MNIST (A-K) dataset (Table S2) using 14 dimensionality reduction methods. Training and test samples are shown with their original images, color-tinted by label: training samples appear with reduced opacity. See Appendix I.I for full qualitative analysis.
41

---

RF-AE (Ours)
RF -HATE
PTSNE
P-UMAP
P-SUMAP
NCA
PACMAP
SSNP
CEBRA
PLS-DA
SPCA
PCA
kidney
kidmney
Gall blodder
Eaophaggs
Liver
Stamacch
Inferior
Figure S7: Visualization of the OrganC MNIST dataset (Table S2) using 14 dimensionality reduction methods. Test points shown as color-coded circles based their labels. Training points omitted for clarity. Refer to Appendix 1.1 for full qualitative analysis.
42

---

RF-AE (Ours)
RF RF-PHATE
PTSNE
P-UMAP
P-SUMAP
NCA
PACMAP
CE
SSNP
CEBRA
PLS-DA
SPCA
PCA
country
disco
hiphop
pop
reqqae
rock
Figure S8: Visualization of the GTZAN (3-sec) dataset (Table S2) using 14 dimensionality reduction methods. Training and test points are shown color-coded circles based their labels. Training samples appear with reduced opacity. Refer to Appendix 1.2 for a full qualitative analysis.
43

---

Training and test times per model across datasets
101
Sign MNIST (A-K)
OrganC MNIST
Training time (S)
107
301
10P
107
101
Test time
300
10
RF-PHATE
RF-AE (N" 0.1Nmmm))
RF-AE 0.02Naail)
SSNP
P-SUMAP
CE
NCA
PACMAP
P.TSNE
AE
P-UMAP
SPCA
PLS-DA
CEBRA
PCA
Figure S9: Average training (top) and test (bottom) computation times per model across 10 repetitions on Sign MNIST (blue) and OrganC MNIST (green). Standard deviations are displayed as error bars.
44


Reviews:
[{"review_id": "662wDcQlXF", "reviewer": "Reviewer_RWaH", "summary": "The paper proposes Random Forest Autoencoders (RF-AE), a new framework for supervised representation learning, especially for out-of-sample (OOS) visualization.\nRF-AE integrates the geometry and supervision strengths of RF-PHATE with the flexibility and scalability of autoencoders.\nInstead of reconstructing the raw input, RF-AE reconstructs RF-GAP proximities, a supervised similarity metric derived from random forests, which allows it to learn embeddings aligned with the supervised structure of the data.\nA geometric regularization term ensures alignment between the latent representation and a precomputed RF-PHATE embedding.\nThe authors also introduce a prototype selection mechanism to reduce computational costs and memory usage by selecting representative training points via class-wise k-medoids clustering on RF-GAP dissimilarities.\nRF-AE consistently outperforms baseline parametric and nonparametric methods in terms of both classification accuracy and a novel Structural Importance Alignment (SIA) metric, which measures how well the embedding reflects the classification-relevant structure of the input features.\nRF-AE achieves strong trade-offs between local/global structure preservation and class separability, with robust performance across various hyperparameter settings.", "questions": "1. While the authors acknowledge the computational cost of RF-AE, there is no concrete study and comparison on it, except the running time by different amount of prototypes. There should be training/inference time (and memory consumption) comparison with the other baselines compared across datasets, for a better understanding for practical considerations. \n2. How sensitive is the SIA metric to the choice of classifiers and perturbation schemes? Could the authors provide ablation results or sensitivity analyses with different classifiers or correlation thresholds? The authors may consider reporting SIA variation across: alternative classifiers or non-ensembled versions, different perturbation strategies, and/or datasets with weak label–feature correlation. \n3. Can we utilize unlabeled data also for training RF-AE, e.g., in a semi-supervised way? Unsupervised methods of course can handle all available data without using labels, and methods like UMAP, semi-supervised variants of t-SNE [1], ViVA [2], etc., can learn from both unlabeled and labeled data. In a realistic scenario, learning from labeled data only may reduce data efficiency.\n\n[1] Serna-Serna, Walter, et al. \"Semi-supervised t-SNE with multi-scale neighborhood preservation.\" Neurocomputing 550 (2023): 126496.\n\n[2] An, Sungtae, Shenda Hong, and Jimeng Sun. \"Viva: semi-supervised visualization via variational autoencoders.\" 2020 IEEE International Conference on Data Mining (ICDM). IEEE, 2020.", "limitations": "The authors have done a good job acknowledging technical limitations, particularly about scalability due to the computational cost of RF-GAP proximities.\nHowever, they do not sufficiently address potential negative societal impacts of their method, especially considering its applicability in high-stakes domains like biomedical and healthcare.\nIt would be very helpful if the authors could discuss some potential pitfalls, for example (not must):\n- Risks in biomedical or clinical data visualization, such as over-interpretation of embeddings, model bias from supervised proximities, or misuse of visualizations in decision-making.\n- How the complexity of the RF-GAP kernel and the autoencoder might make the resulting embeddings even harder to interpret, especially by non-technical stakeholders.\n- What if the labels RF-GAP relies on encode social or demographic biases? Could this lead to skewed embeddings?\n\nAddressing those kinds of concerns, even briefly, would demonstrate awareness of broader impact and strengthen the paper’s maturity.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The proposed RF-AE framework combines random forest-derived (RF-GAP) proximities with an autoencoder-based architecture. Reconstructing RF-GAP proximities, rather than original input vectors, within the autoencoder is a meaningful and novel departure from standard AE objectives.\n- The methodological foundation is strong, integrating insights from manifold learning, kernel regression, and autoencoder design in a coherent way. Key innovations include geometric loss regularization based on RF-PHATE embeddings and a prototype selection mechanism to enhance scalability.\n- The geometric loss, guided by RF-PHATE embeddings, ensures alignment between the supervised structure and the learned latent space, helping mitigate overfitting or trivial embedding solutions.\n- The Structural Importance Alignment (SIA) metric fills an important evaluation gap in supervised visualization by quantifying the extent to which embeddings preserve classification-relevant feature structure. This is a valuable and well-motivated contribution.\n- RF-AE ranks among the top across both local/global SIA and classification accuracy benchmarks. Ablation studies further demonstrate robustness to hyperparameter choices such as the geometric constraint weight ($\\lambda$) and number of prototypes.\n\n**Weaknesses**\n- As the authors acknowledge, a key limitation is the computational cost of computing RF-GAP proximities, which scales poorly with dataset size. Although prototype selection via class-wise k-medoids helps mitigate this, it still requires full computation of the RF-GAP proximity matrix which is posing a barrier for large-scale applications.\n- There are no concrete runtime or memory benchmarks demonstrating feasibility on large-scale datasets, although timing for the proposed method on a single dataset with varying prototype counts is reported.\n- The generalizability of the SIA metric may depend on the specific ensemble of classifiers and the perturbation heuristics used, which could introduce variance or bias across different data modalities or domains.", "was_revised": true, "final_justification": "This paper proposes a novel method for supervised dimensionality reduction, especially for data visualization.\nThe paper is technically solid and well-supported by experimental results.\nMost initial concerns on scalability (running time), semi-supervised learning capability, and potential limitations were addressed during the rebuttal period.", "rebuttal": "Thank you for the valuable feedback and overall positive review. Below, we provide a point-by-point\nresponse to the concerns you raised.\n\n**Re Q1.** In the revised version, we will include a training and inference time comparison in the Appendix. While RF-AE incurs additional training time due to the computation of the random forest proximity, we note that it significantly improves out-of-sample supervised embeddings (see quantitative results in Table 1), which we believe makes the additional cost worthwhile in many practical scenarios.\n\nAll experiments were conducted on a shared computing cluster, where compute nodes are allocated dynamically. As a result, runtimes may vary depending on system load and available resources. To give a rough estimate of RF-AE’s runtime relative to other baselines, we report the training and test times on the Sign MNIST (Table R1.1) and OrganC MNIST (Table R1.2) datasets. RF-AE falls within the same order of magnitude (OoM) as RF-PHATE, P-SUMAP, and NCA during training, and one OoM slower than CE and SSNP. At inference time, RF-AE is approximately 2 OoM faster than RF-PHATE, and one OoM slower than P-SUMAP.\nMemory usage did not pose a constraint in any of our experiments. RF-AE scales well across datasets without requiring extensive memory resources, and our design choices, including our prototype selection, help keep computations and memory demand manageable.\n\nFurthermore, we are actively optimizing our RF-GAP implementation by migrating proximity matrix computation to the GPU (in progress) to improve efficiency. These updates will be reflected in our GitHub repository once completed.\n\n**Table R1.1: Average training and test times on Sign MNIST, in seconds, across 10 repetitions.**\n|Model|Training Time|Test Time|\n|--------------------|--------------------------------------------|-----------------------------------------|\n| RF-PHATE| $5.69 \\times 10^2 \\pm 7.74 \\times 10^1$| $2.92 \\times 10^2 \\pm 2.49 \\times 10^2$ |\n|RF-AE ($N^{*}=10\\\\%$)| $6.16 \\times 10^2 \\pm 3.92 \\times 10^1$| $8.45 \\times 10^0 \\pm 4.27 \\times 10^{-1}$ |\n|RF-AE ($N^{*}=2\\\\%$)| $5.88 \\times 10^2 \\pm 6.37 \\times 10^1$| $1.93 \\times 10^0 \\pm 1.72 \\times 10^{-1}$ |\n|PACMAP| $6.58 \\times 10^0 \\pm 6.67 \\times 10^{-1}$ | $5.24 \\times 10^{-1} \\pm 4.60 \\times 10^{-2}$ |\n|P-SUMAP| $3.62 \\times 10^2 \\pm 3.87 \\times 10^1$| $1.09 \\times 10^{-1} \\pm 7.00 \\times 10^{-3}$ |\n|AE| $2.06 \\times 10^1 \\pm 1.25 \\times 10^0$| $1.90 \\times 10^{-2} \\pm 2.00 \\times 10^{-3}$ |\n|P-TSNE| $1.91 \\times 10^1 \\pm 7.97 \\times 10^{-1}$ | $1.80 \\times 10^{-2} \\pm 2.00 \\times 10^{-3}$ |\n|SSNP| $2.95 \\times 10^1 \\pm 1.48 \\times 10^0$    | $1.80 \\times 10^{-2} \\pm 2.00 \\times 10^{-3}$ |\n|P-UMAP| $1.87 \\times 10^1 \\pm 8.03 \\times 10^{-1}$ | $1.80 \\times 10^{-2} \\pm 3.00 \\times 10^{-3}$ |\n|CE| $5.76 \\times 10^1 \\pm 3.86 \\times 10^0$| $1.70 \\times 10^{-2} \\pm 1.00 \\times 10^{-3}$ |\n|CEBRA| $2.59 \\times 10^0 \\pm 3.18 \\times 10^{-1}$ | $1.10 \\times 10^{-2} \\pm 7.00 \\times 10^{-3}$ |\n|PLS-DA| $2.73 \\times 10^{-1} \\pm 7.00 \\times 10^{-2}$ | $1.10 \\times 10^{-2} \\pm 3.00 \\times 10^{-3}$ |\n|PCA| $3.41 \\times 10^{-1} \\pm 6.20 \\times 10^{-2}$ | $7.00 \\times 10^{-3} \\pm 1.00 \\times 10^{-3}$ |\n|NCA| $3.45 \\times 10^2 \\pm 2.10 \\times 10^2$| $4.00 \\times 10^{-3} \\pm 1.00 \\times 10^{-3}$ |\n|SPCA| $5.62 \\times 10^1 \\pm 5.65 \\times 10^0$| $3.00 \\times 10^{-3} \\pm 1.00 \\times 10^{-3}$ |\n\n**Table R1.2: Average training and test times on OrganC MNIST, in seconds, across 10 repetitions.**\n| Model| Training Time|Test Time|\n|-------------------|------------------------------------|------------------------------------|\n| RF-PHATE| $5.87 \\times 10^2 \\pm 7.67 \\times 10^1$ | $5.05 \\times 10^2 \\pm 7.16 \\times 10^1$ |\n| RF-AE ($N^{*}=10\\\\%$)| $7.60 \\times 10^2 \\pm 2.45 \\times 10^1$ | $2.71 \\times 10^1 \\pm 1.24 \\times 10^0$ |\n| RF-AE ($N^{*}=2\\\\%$)| $6.88 \\times 10^2 \\pm 2.39 \\times 10^1$ | $5.89 \\times 10^0 \\pm 6.50 \\times 10^{-2}$ |\n| PACMAP| $7.16 \\times 10^0 \\pm 7.32 \\times 10^{-1}$ | $1.40 \\times 10^0 \\pm 1.72 \\times 10^{-1}$ |\n| P-SUMAP| $2.95 \\times 10^2 \\pm 3.48 \\times 10^1$ | $1.16 \\times 10^{-1} \\pm 4.00 \\times 10^{-3}$ |\n| CE| $5.29 \\times 10^1 \\pm 9.62 \\times 10^0$ | $4.20 \\times 10^{-2} \\pm 4.00 \\times 10^{-3}$ |\n| AE| $1.92 \\times 10^1 \\pm 2.84 \\times 10^0$ | $4.20 \\times 10^{-2} \\pm 4.00 \\times 10^{-3}$ |\n| SSNP| $2.66 \\times 10^1 \\pm 4.82 \\times 10^0$ | $4.00 \\times 10^{-2} \\pm 7.00 \\times 10^{-3}$ |\n| P-TSNE| $1.71 \\times 10^1 \\pm 4.68 \\times 10^0$ | $3.50 \\times 10^{-2} \\pm 7.00 \\times 10^{-3}$ |\n| P-UMAP| $1.70 \\times 10^1 \\pm 4.81 \\times 10^0$ | $3.30 \\times 10^{-2} \\pm 6.00 \\times 10^{-3}$ |\n| PLS-DA| $2.55 \\times 10^{-1} \\pm 5.10 \\times 10^{-2}$ | $3.10 \\times 10^{-2} \\pm 6.00 \\times 10^{-3}$ |\n| CEBRA| $2.11 \\times 10^0 \\pm 5.33 \\times 10^{-1}$ | $2.60 \\times 10^{-2} \\pm 6.00 \\times 10^{-3}$ |\n| PCA| $3.48 \\times 10^{-1} \\pm 6.50 \\times 10^{-2}$ | $2.40 \\times 10^{-2} \\pm 5.00 \\times 10^{-3}$ |\n| NCA| $3.36 \\times 10^2 \\pm 9.77 \\times 10^1$ | $1.10 \\times 10^{-2} \\pm 1.00 \\times 10^{-3}$ |\n| SPCA| $5.60 \\times 10^1 \\pm 1.22 \\times 10^1$ | $7.00 \\times 10^{-3} \\pm 1.00 \\times 10^{-3}$ |\n\n**Re Q2.** We analyzed the SIA performance of all models under different classification importance strategies, as detailed in Appendix I. In addition to our ensemble approach, we derived feature importances from a standalone $k$-NN classifier, as well as from an aggregated strategy where importances were computed separately from each model ($k$-NN, SVM, MLP) and then combined. We observed that RF-AE consistently ranked at the top across all metrics, regardless of the feature importance strategy. This suggests that RF-AE is able to preserve meaningful structure in a more general sense—not just from the perspective of a single classifier. As for a sensitivity analysis of SIA under different perturbation strategies, we are not aware of other correlation-aware schemes beyond our own, which is inspired by Kaneko [1]. That said, we are open to exploring additional strategies in future work. Finally, although not explicitly stated in the manuscript, feature importances were averaged over 10 random sampling procedures to ensure robustness. We will clarify these points in the final paper.\n\n**Re Q3.** Yes: RF-AE supports partially labeled datasets, making it a compelling approach in semi-supervised settings compared to strictly class-conditional methods. As noted in Rhodes \\& Aumon et al. [2]: “ (...) a trained random forest can define similarities to out-of-sample observations, supporting semi-supervised learning (...) ” This is further elaborated in our manuscript (L155): “Note that this definition (of RF-GAP) naturally extends to OOS observations (...) which can be treated as out-of-bag for all trees.” Thus, on the one hand, assuming $N_L$ labeled points and $N_U$ unlabeled points, for a total training size of $N = N_L + N_U$, we treat unlabeled training samples as “out-of-sample” and compute $N$ proximity vectors of size $N_L$, which are used as input to train the RF-AE network. On the other hand, to generate RF-PHATE embeddings for the full training set, we can rely on the Landmark PHATE algorithm proposed by Moon et al. [3]. We first construct the RF-GAP kernel matrix of size $N_{L} \\times N_{L}$ between labeled landmarks and compute their embeddings with PHATE. Then, we project the $N_{U}$ remaining unlabeled points with the linear landmark extension using their RF-GAP proximities to the labeled points (the landmarks), as in L118-L122. This provides all the necessary components to train our regularized RF-AE using information from both labeled and unlabeled data.  We will add this discussion to the final version of the paper.\n\n**Re Limitations:** While technically involved, the core idea of RF-AE—embedding data in 2D—is intuitive. Autoencoders learn low-dimensional representations that capture intrinsic structure. The RF-GAP kernel (Section 3.1) replaces standard Euclidean measures with similarities based on discriminative features, effectively downweighting irrelevant ones. RF-PHATE regularization further denoises the embedding and captures long-range relationships. Combined, these elements let RF-AE organize data based on local and global RF-GAP similarities: nearby points share key features, while distant ones differ meaningfully. We validated this using interpretable metrics (Section 4.1; $k$-NN, SIA metrics) and qualitatively through 2D visualizations showing stronger class-aware structure than competing methods.Thus we do not believe that the results of RF-AE will be substantially more difficult to interpret than other, similar methods.\n\nStill, we agree that supervised 2D visualizations should be interpreted with caution due to potential biases in label assignment. When label assignments reflect social or demographic biases, supervised methods are prone to embedding structural biases, as they aim to discriminate between classes. Another source of label bias arises in highly imbalanced scenarios. In such cases, the Random Forest is biased toward majority classes. As a result, minority classes may appear closer to or farther from other classes than expected, since the features that characterize them may not be adequately captured in the RF-GAP proximities.\nWe note that these concerns are not specific to RF-AE but apply broadly to supervised methods. Additionally, RF-AE helps mitigate these effects by avoiding the exaggerated separations often introduced by purely class-conditional methods. These and the other ethical considerations raised by the reviewer will be discussed in a dedicated section of the manuscript.\n\n[1] Kaneko. Cross-validated permutation feature importance considering correlation between\nfeatures. Analytical Science Advances, 2022\n\n[2] Rhodes, Aumon, et al. Gaining biological insights through supervised data visualization. bioRxiv,\n2024\n\n[3] Moon et al. Visualizing structure and transitions in high-dimensional biological data. Nat.\nBiotechnol., Dec 2019"}, {"review_id": "2JqF0gq6u0", "reviewer": "Reviewer_vZZA", "summary": "The paper proposes to build on top of a recent paper (RF-PHATE) to derive a framework for out of sample kernel based visualization. The paper provides a few empirical validations showing more interpretable visualizations.", "questions": "Please see above.", "limitations": "Please see above.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 1, "strengths_and_weaknesses": "The entire motivation of the paper is unclear to me. Nonlinear visualization of data for out-of-sample has been tackled a long time ago by learning a parametric embedding model of the data and then doing a non-kernel based visualization based on those embeddings (this has been true for decades, even prior Deep Learning). There are also numerous specific papers looking at that problem, such as \n- Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering, Yoshua Bengio, Jean-Francois Paiement, Pascal Vincent Olivier Delalleau, Nicolas Le Roux and Marie Ouimet\n- Supervised Deep Feature Embedding With Handcrafted Feature, Shichao Kan; Yigang Cen; Zhihai He; Zhi Zhang; Linna Zhang; Yanhong Wang\n\nWhile it is possible that I missed the underlying benefits of the method, I believe that the abstract and introduction/background do not provide enough context and motivation which needs to be addressed.", "was_revised": true, "final_justification": "Thank you for your answers, please see my last comment addressed to the authors", "rebuttal": "We appreciate the reviewer highlighting this important line of work. While the out-of-sample extension problem has indeed been extensively studied—including in classical kernel methods such as Nyström and geometric harmonics—our work addresses a distinct and underexplored setting: *supervised* out-of-sample visualization, where the goal is to preserve both the local structure of the data and class-aware separability in the embedding space. The classic work of Bengio et al. [1] laid important groundwork for out-of-sample extensions using linear mappings for LLE, Isomap, MDS, and spectral methods. However, these approaches typically rely on unconstrained least-squares minimization and are restricted to linear or kernel-based mappings, making them sensitive to the quality of the training data and often inadequate for capturing complex manifold structure in a way that generalizes effectively to new inputs [2,3].\n\nThe manifold extension problem remains an active research area, with recent developments such as parametric $t$-SNE [4], parametric UMAP [5], and several geometry-regularized or geometry-aware autoencoders [6--9]. These methods offer more flexible parametric mappings, but most focus either on unsupervised structure or on optimizing embeddings solely for predictive performance, without preserving label-informed geometry that generalizes to new data points.\n\nRF-AE is designed to address this gap by learning an embedding function that preserves both local structure and class separability, based on Random Forest proximities. Unlike supervised methods such as RF-PHATE, which capture label-aware structure but do not offer a learned mapping, RF-AE provides a parametric, interpretable, and generalizable embedding function. Neural methods such as autoencoders can learn mappings, but often lack a strong supervision signal tied to relational structure between points.\n\nAs shown in Table 1 (Section 4), RF-AE consistently ranks highest across 20 datasets on both structure-preserving alignment scores and classification accuracy. These results support our claim that RF-AE improves supervised visualization for out-of-sample data by balancing geometric fidelity and label-based information in a way that prior methods do not fully achieve.\n\nWe will revise the introduction to more clearly distinguish our motivation and contributions, emphasizing that RF-AE targets interpretable, supervised embeddings that generalize effectively to new data points.\n\n**References**\n\n[1] Bengio et al. Out-of-sample extensions for lle, isomap, mds, eigenmaps, and spectral\nclustering. NeurIPS, 16, 2003.\n\n[2] Rudi et al. Less is more: Nyström computational regularization. In Advances in Neural Information Processing Systems, volume 28. Curran Associates,\nInc., 2015.\n\n[3]  Quispe et al. Extreme learning machine for out-of-sample extension in laplacian eigenmaps. Pattern Recognition Letters, 74:68–73, April 2016.\n\n[4] van der Maaten et al. Learning a parametric embedding by preserving local structure. In Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pages 384–391, Hilton Clearwater\nBeach Resort, Clearwater Beach, Florida USA, 16–18 Apr 2009. PMLR.\n\n[5] Sainburg et al. Parametric umap embeddings for representation and semisupervised learning. Neural Computation, 33(11):2881–2907, 10 2021.\n\n[6] A.F. Duque, S. Morin, G. Wolf, and K.R. Moon. Geometry regularized autoencoders. IEEE Transactions\non Pattern Analysis and Machine Intelligence, 45(6):7381–7394, 2022.\n\n[7] Sun et al. Geometry-aware autoencoders for metric learning and generative modeling on data manifolds. ICML 2024 Workshop on Geometry-grounded Representation Learning and Generative Modeling. 2024.\n\n[8] Lee et al. Neighborhood reconstructing autoencoders. In Advances in Neural Information Processing Systems, volume 34, page 536–546. Curran Associates, Inc.,\n2021.\n\n[9] Nazari et al. Geometric autoencoders - what you see is what you decode. In Proceedings of the 40th International Conference on Machine Learning, volume\n202 of Proceedings of Machine Learning Research, pages 25834–25857. PMLR, 23–29 Jul 2023."}, {"review_id": "OAXMGfI8E9", "reviewer": "Reviewer_RUnf", "summary": "This paper introduces a random forest inspired autoencoder (RF-AE), a supervised dimensionality reduction method that combines the \"geometric structure\" derived from Random Forest proximity kernels (RF-GAP) with the parametric capacity of autoencoders. The proposed framework enables out-of-sample embedding via a geometry-regularized encoder trained to preserve the label-informed structure captured by RF-GAP or RF-PHATE. The approach integrates a novel structural importance alignment metric to assess how well learned embeddings retain class-relevant features. A prototype-based approximation is also proposed to reduce input dimensionality and improve scalability.", "questions": "1. The scalability of this framework, as mentioned above.\n\n2. Generalization Beyond RF-PHATE. Have the authors tested RF-AE with other embeddings? Empirical results would substantiate the generality claim.\n\n3. Impact of forest configuration: how sensitive is it to the configuration of RF, such as number of trees, max depth?", "limitations": "Rebuttal content resolves my questions very well.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "S1: The paper is well-motivated to combine RF-derived proximity with AE training to address the shortcoming of out-of-sample mapping problem of RFs. It enables efficient parametric encoding and retain the supervised neighborhood structure.\n\nS2: It shows strong empirical performance in preserving class-relevant information across various datasets.\n\nS3: The paper is clearly written, well-organized, and provides sufficient implementation detail.\n\nW1: Despite the prototype approximation, the full proximity matrix is initially computed using all training points (L177). This may be prohibitively costly for large datasets. It would be better if more statistics and explanations related to scalability and computational cost could be discussed.\n\nW2: Although it is mentioned that this framework is agnostic to the kernel method. However, only results based on RF-PHATE are included, and it would be better to demonstrate that this framework works with other kernels. How does the selection of the kernel affect the quality of the final representation?", "was_revised": true, "final_justification": "This paper, together with the rebuttal results, is well written and sufficiently proves the effectiveness of the proposed method. My concerns about the scalability, computation cost, and extensibility are resolved.", "rebuttal": "We would like to thank you for the positive review and insightful comments. Here are our detailed answers to your  questions.\n\n**Re Q1.** Thank you for this valuable observation. It is true that in our current implementation, the full random forest proximity matrix is computed before selecting prototypes. However, the matrix is not stored in memory permanently. It is held temporarily for processing and then released, so memory usage remains controlled. In practice, memory was not a limiting factor in any of our experiments.\n\nWe agree that for very large datasets, the pairwise proximity computation can still be time-consuming. To improve scalability, we are actively working on accelerating this step through GPU-based implementation. This optimization is currently in progress, and we are confident that our GPU-based implementation will be completed by the time the camera-ready submission is due. Unfortunately, per NeurIPS guidelines, external links (e.g., to updated code) are not allowed during the rebuttal period.\n\nTo provide a clearer view of computational cost, we include training and test time comparisons in Table R3.1 (Sign MNIST) and Table R3.2 (OrganC MNIST), in seconds, across 10 repetitions. RF-AE falls within the same order of magnitude (OoM, i.e. power of ten) as RF-PHATE, P-SUMAP, and NCA during training, and is approximately one OoM slower than CE and SSNP. At inference time, RF-AE is roughly two OoM faster than RF-PHATE’s default kernel extension and about one OoM slower than P-SUMAP.\n\n\n\n**Table R3.1: Average training and test times on Sign MNIST, in seconds, across 10 repetitions.**\n|Model|Training Time|Test Time|\n|--------------------|--------------------------------------------|-----------------------------------------|\n| RF-PHATE| $5.69 \\times 10^2 \\pm 7.74 \\times 10^1$| $2.92 \\times 10^2 \\pm 2.49 \\times 10^2$ |\n|RF-AE ($N^{*}=10\\\\%$)| $6.16 \\times 10^2 \\pm 3.92 \\times 10^1$| $8.45 \\times 10^0 \\pm 4.27 \\times 10^{-1}$ |\n|RF-AE ($N^{*}=2\\\\%$)| $5.88 \\times 10^2 \\pm 6.37 \\times 10^1$| $1.93 \\times 10^0 \\pm 1.72 \\times 10^{-1}$ |\n|PACMAP| $6.58 \\times 10^0 \\pm 6.67 \\times 10^{-1}$ | $5.24 \\times 10^{-1} \\pm 4.60 \\times 10^{-2}$ |\n|P-SUMAP| $3.62 \\times 10^2 \\pm 3.87 \\times 10^1$| $1.09 \\times 10^{-1} \\pm 7.00 \\times 10^{-3}$ |\n|AE| $2.06 \\times 10^1 \\pm 1.25 \\times 10^0$| $1.90 \\times 10^{-2} \\pm 2.00 \\times 10^{-3}$ |\n|P-TSNE| $1.91 \\times 10^1 \\pm 7.97 \\times 10^{-1}$ | $1.80 \\times 10^{-2} \\pm 2.00 \\times 10^{-3}$ |\n|SSNP| $2.95 \\times 10^1 \\pm 1.48 \\times 10^0$    | $1.80 \\times 10^{-2} \\pm 2.00 \\times 10^{-3}$ |\n|P-UMAP| $1.87 \\times 10^1 \\pm 8.03 \\times 10^{-1}$ | $1.80 \\times 10^{-2} \\pm 3.00 \\times 10^{-3}$ |\n|CE| $5.76 \\times 10^1 \\pm 3.86 \\times 10^0$| $1.70 \\times 10^{-2} \\pm 1.00 \\times 10^{-3}$ |\n|CEBRA| $2.59 \\times 10^0 \\pm 3.18 \\times 10^{-1}$ | $1.10 \\times 10^{-2} \\pm 7.00 \\times 10^{-3}$ |\n|PLS-DA| $2.73 \\times 10^{-1} \\pm 7.00 \\times 10^{-2}$ | $1.10 \\times 10^{-2} \\pm 3.00 \\times 10^{-3}$ |\n|PCA| $3.41 \\times 10^{-1} \\pm 6.20 \\times 10^{-2}$ | $7.00 \\times 10^{-3} \\pm 1.00 \\times 10^{-3}$ |\n|NCA| $3.45 \\times 10^2 \\pm 2.10 \\times 10^2$| $4.00 \\times 10^{-3} \\pm 1.00 \\times 10^{-3}$ |\n|SPCA| $5.62 \\times 10^1 \\pm 5.65 \\times 10^0$| $3.00 \\times 10^{-3} \\pm 1.00 \\times 10^{-3}$ |\n\n**Table R3.2: Average training and test times on OrganC MNIST, in seconds, across 10 repetitions.**\n| Model| Training Time|Test Time|\n|-------------------|------------------------------------|------------------------------------|\n| RF-PHATE| $5.87 \\times 10^2 \\pm 7.67 \\times 10^1$ | $5.05 \\times 10^2 \\pm 7.16 \\times 10^1$ |\n| RF-AE ($N^{*}=10\\\\%$)| $7.60 \\times 10^2 \\pm 2.45 \\times 10^1$ | $2.71 \\times 10^1 \\pm 1.24 \\times 10^0$ |\n| RF-AE ($N^{*}=2\\\\%$)| $6.88 \\times 10^2 \\pm 2.39 \\times 10^1$ | $5.89 \\times 10^0 \\pm 6.50 \\times 10^{-2}$ |\n| PACMAP| $7.16 \\times 10^0 \\pm 7.32 \\times 10^{-1}$ | $1.40 \\times 10^0 \\pm 1.72 \\times 10^{-1}$ |\n| P-SUMAP| $2.95 \\times 10^2 \\pm 3.48 \\times 10^1$ | $1.16 \\times 10^{-1} \\pm 4.00 \\times 10^{-3}$ |\n| CE| $5.29 \\times 10^1 \\pm 9.62 \\times 10^0$ | $4.20 \\times 10^{-2} \\pm 4.00 \\times 10^{-3}$ |\n| AE| $1.92 \\times 10^1 \\pm 2.84 \\times 10^0$ | $4.20 \\times 10^{-2} \\pm 4.00 \\times 10^{-3}$ |\n| SSNP| $2.66 \\times 10^1 \\pm 4.82 \\times 10^0$ | $4.00 \\times 10^{-2} \\pm 7.00 \\times 10^{-3}$ |\n| P-TSNE| $1.71 \\times 10^1 \\pm 4.68 \\times 10^0$ | $3.50 \\times 10^{-2} \\pm 7.00 \\times 10^{-3}$ |\n| P-UMAP| $1.70 \\times 10^1 \\pm 4.81 \\times 10^0$ | $3.30 \\times 10^{-2} \\pm 6.00 \\times 10^{-3}$ |\n| PLS-DA| $2.55 \\times 10^{-1} \\pm 5.10 \\times 10^{-2}$ | $3.10 \\times 10^{-2} \\pm 6.00 \\times 10^{-3}$ |\n| CEBRA| $2.11 \\times 10^0 \\pm 5.33 \\times 10^{-1}$ | $2.60 \\times 10^{-2} \\pm 6.00 \\times 10^{-3}$ |\n| PCA| $3.48 \\times 10^{-1} \\pm 6.50 \\times 10^{-2}$ | $2.40 \\times 10^{-2} \\pm 5.00 \\times 10^{-3}$ |\n| NCA| $3.36 \\times 10^2 \\pm 9.77 \\times 10^1$ | $1.10 \\times 10^{-2} \\pm 1.00 \\times 10^{-3}$ |\n| SPCA| $5.60 \\times 10^1 \\pm 1.22 \\times 10^1$ | $7.00 \\times 10^{-3} \\pm 1.00 \\times 10^{-3}$ |\n\n\n\n\n\n**Re Q2.** While RF-AE can incorporate any geometric constraint, RF-PHATE is the core focus of our paper, as it ensures that visualizations are aligned with the underlying supervised data geometry. This choice is motivated by Rhodes \\& Aumon et al. [1], who empirically demonstrated that RF-PHATE outperformed alternatives.\n\nHowever, to demonstrate the generality of  geometry-regularized RF-AE network, we provide in Table R3.3 ablation results on Sign MNIST and OrganC MNIST using three alternative geometric constraints: UMAP, Supervised UMAP (SUMAP), and RF-UMAP (i.e., UMAP applied to RF-GAP proximities). We set our default hyperparameters $\\lambda=0.01,N^*=0.1$. RF-AE (RF-UMAP) achieved the best overall performance, followed by RF-AE with SUMAP and UMAP constraints. While RF-AE (RF-UMAP) was competitive with RF-AE (RF-PHATE) on OrganC MNIST, it largely underperformed on Sign MNIST. These results suggest that RF-AE is especially effective when paired with RF-based kernel methods—particularly RF-PHATE—which already capture the underlying RF geometry. In such cases, the geometric and reconstruction objectives are well-aligned, enabling more effective multi-task learning. Qualitatively, RF-AE (RF-UMAP) still fragments same-class clusters on the Sign MNIST dataset, though less severely than UMAP and SUMAP. This supports the idea that RF-PHATE more effectively captures both local and global supervised structure through diffusion.\n\nIn summary, RF-PHATE is a strong default regularizer for RF-AE overall, though alternative RF-based regularizers like RF-UMAP may offer valuable refinements in specific scenarios. We will include these additional quantitative and qualitative experiments in the Appendix.\n\n**Table R3.3: SIA and kNN accuracies of RF-AE using three different geometric constraints on the Sign MNIST and OrganC MNIST datasets. Scores are averaged across 10 repetitions (mean ± std).**\n\nDataset | Model         | QNX            | Trust          | Spear          | Pearson        | kNN acc.       |\n|---------------|---------------|----------------|----------------|----------------|----------------|----------------|\n| Sign MNIST | RF-AE (RF-UMAP) | 0.756 ± 0.013 | 0.743 ± 0.015 | 0.505 ± 0.217 | 0.478 ± 0.233 | 0.945 ± 0.010 |\n| | RF-AE (SUMAP)   | 0.692 ± 0.010 | 0.586 ± 0.022 | 0.453 ± 0.075 | 0.400 ± 0.101 | 0.845 ± 0.014 |\n| | RF-AE (UMAP)    | 0.669 ± 0.013 | 0.546 ± 0.018 | 0.311 ± 0.039 | 0.225 ± 0.063 | 0.743 ± 0.020 |\n| OrganC MNIST | RF-AE (RF-UMAP)   | 0.893 ± 0.007    | 0.932 ± 0.005    | 0.934 ± 0.007    | 0.931 ± 0.006    | 0.720 ± 0.009    |\n| | RF-AE (SUMAP)     | 0.881 ± 0.008    | 0.905 ± 0.008    | 0.883 ± 0.008    | 0.872 ± 0.007    | 0.691 ± 0.014    |\n| | RF-AE (UMAP)      | 0.887 ± 0.007    | 0.905 ± 0.006    | 0.867 ± 0.006    | 0.865 ± 0.006    | 0.559 ± 0.025    |\n\n\n\n**Re Q3.** Rhodes et al. [2] demonstrated that the geometry captured by RF-GAP proximities remains stable across varying numbers of trees and minimum node sizes—outperforming other RF-derived kernels in robustness. Moreover, RF-PHATE itself is largely insensitive to its primary hyperparameter, the diffusion time $t$, as shown by Moon et al. [3]. Thus, we did not consider it necessary to perform ablation studies on these secondary hyperparameters, but we are open to doing so if requested.\n\n**References**\n\n[1] Rhodes, Aumon, et al. Gaining biological insights through supervised data visualization. bioRxiv,\n2024.\n\n[2] Rhodes et al. Geometry- and accuracy-preserving random forest\nproximities. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–13, 2023.\n\n[3] Moon et al. Visualizing structure and transitions in high-dimensional biological data. Nat. Biotechnol.,\n37(12):1482–1492, Dec 2019."}, {"review_id": "i5X3UbiXP8", "reviewer": "Reviewer_mQiQ", "summary": "## Summary\n\nThe paper presents RF-AE, a novel framework for supervised dimension reduction method that addresses the limitations of the existing RF-PHATE method. It provides an explicit mapping function for out-of-sample extension, which is useful for handling large datasets and new data points. RF-AE combines autoencoders with the strengths of random forests and the geometric insights of RF-PHATE. It provides experiment results from comparing RF-AE with existing dimension reduction methods.", "questions": "1. How is new data defined?\n2. How does RF-AE compare to UMAP in terms of speed and scalability?\n3. How can RF-AE incorporate task-specific insights?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\n\nS1. This paper focuses on developing new algorithms for supervised dimension reduction, which is a timely research question. The proposed method is sound and novel.\n\nS2. The experimental design is comprehensive, using 20 datasets and comparing against popular dimension reduction algorithms such as tSNE and UMAP. The quantitative results look promising.\n\nS3. The paper is well-organized and easy to follow.\n\n## Major Weakness\n\nW1. One motivation for RF-AE is to accommodate new observations in during dimension reduction (L29). For example, the abstract and introduction highlight that supervised approaches like RF-PHATE can capture task-specific insights with auxiliary labels. However, section 4 doesn't seem to utilize any task-specific insights or expert labels.\n\nW2. One major benefit for UMAP is its speed and scalability. It is unclear how RF-AE compares to UMAP in these aspects.\n\n## Minor Weakness\n\nM1. How is new data defined? New data from the same distribution of the existing data, or totally out of distribution data?\n\nM2. The term \"data visualization\" in this paper can lead to confusion. This paper mainly focuses on dimension reduction instead of data visualization. I recommend changing it to dimension reduction.", "was_revised": true, "final_justification": "Most of my questions have been answered. Given that my original rating was already positive, I will be maintaining it.", "rebuttal": "Thank you for your highly relevant remarks and overall positive review.\n\n**Re W1.** The term *expert labels* refers to annotations provided by experts and assigned to each data point in the training set. The training set used in Section 4 is labeled, meaning that each instance is associated with a specific class. Labels are necessary for supervised learning, as they enable us to train a Random Forest and capture task-specific structure in 2D through our RF-AE framework. I.e., the expert labels are used to train the random forest, from which the RF-GAP proximities are extracted. Previous work has extensively shown that the RF-GAP proximities encode the relationships between points, emphasizing the variables that are relevant for the supervised task (e.g. Rhodes et al. [1]). Thus all of the supervised methods in Section 4 were trained using using expert labels, while unsupervised methods operated without labels. For a detailed description of the data sets, including the number of classes in each, please refer to Appendix F.\n\n\n**Re W2.** Thank you for pointing this out. We agree that one of UMAP’s key strengths is its speed and scalability, particularly in unsupervised visualization tasks. However, RF-AE is designed for a different goal: producing class-aware out-of-sample embeddings that preserve supervised geometric structure. This naturally involves additional computation, such as the generation of random forest-based proximity matrices.\n\nSince UMAP does not use label information during training, it addresses a fundamentally different problem. As such, runtime comparisons between UMAP and RF-AE are not meaningful, as the learning objectives and computational pathways are not aligned. A more appropriate comparison should be made with supervised methods such as CE, SSNP, NCA, and supervised kernel extensions of PHATE and UMAP.\n\nTo provide context, we include training and inference time estimates (in seconds) across datasets in the attached tables. Based on average runtime over 10 repetitions on benchmark datasets such as Sign MNIST (Table R4.1) and OrganC MNIST (Table R4.2), RF-AE falls within the same order of magnitude (OoM, i.e. power of ten) as P-SUMAP, NCA, and RF-PHATE during training. It is approximately one OoM slower than CE and SSNP, and about two OoM faster than RF-PHATE’s default kernel extension. At inference time, RF-AE is one OoM slower than P-SUMAP but significantly faster than RF-PHATE.\n\nRF-AE remains scalable across large datasets. Memory usage was not a limiting factor in any of our experiments, and the method benefits from prototype selection and batch-wise processing to keep both computation and memory demand manageable.\n\nIn the current implementation, the most time-consuming step is the computation of the RF-GAP proximity matrix. We are actively working on accelerating this step using GPU, which we expect will substantially reduce training time.\n\n\n**Table R4.1: Average training and test times on Sign MNIST, in seconds, across 10 repetitions.**\n|Model|Training Time|Test Time|\n|--------------------|--------------------------------------------|-----------------------------------------|\n| RF-PHATE| $5.69 \\times 10^2 \\pm 7.74 \\times 10^1$| $2.92 \\times 10^2 \\pm 2.49 \\times 10^2$ |\n|RF-AE ($N^{*}=10\\\\%$)| $6.16 \\times 10^2 \\pm 3.92 \\times 10^1$| $8.45 \\times 10^0 \\pm 4.27 \\times 10^{-1}$ |\n|RF-AE ($N^{*}=2\\\\%$)| $5.88 \\times 10^2 \\pm 6.37 \\times 10^1$| $1.93 \\times 10^0 \\pm 1.72 \\times 10^{-1}$ |\n|PACMAP| $6.58 \\times 10^0 \\pm 6.67 \\times 10^{-1}$ | $5.24 \\times 10^{-1} \\pm 4.60 \\times 10^{-2}$ |\n|P-SUMAP| $3.62 \\times 10^2 \\pm 3.87 \\times 10^1$| $1.09 \\times 10^{-1} \\pm 7.00 \\times 10^{-3}$ |\n|AE| $2.06 \\times 10^1 \\pm 1.25 \\times 10^0$| $1.90 \\times 10^{-2} \\pm 2.00 \\times 10^{-3}$ |\n|P-TSNE| $1.91 \\times 10^1 \\pm 7.97 \\times 10^{-1}$ | $1.80 \\times 10^{-2} \\pm 2.00 \\times 10^{-3}$ |\n|SSNP| $2.95 \\times 10^1 \\pm 1.48 \\times 10^0$    | $1.80 \\times 10^{-2} \\pm 2.00 \\times 10^{-3}$ |\n|P-UMAP| $1.87 \\times 10^1 \\pm 8.03 \\times 10^{-1}$ | $1.80 \\times 10^{-2} \\pm 3.00 \\times 10^{-3}$ |\n|CE| $5.76 \\times 10^1 \\pm 3.86 \\times 10^0$| $1.70 \\times 10^{-2} \\pm 1.00 \\times 10^{-3}$ |\n|CEBRA| $2.59 \\times 10^0 \\pm 3.18 \\times 10^{-1}$ | $1.10 \\times 10^{-2} \\pm 7.00 \\times 10^{-3}$ |\n|PLS-DA| $2.73 \\times 10^{-1} \\pm 7.00 \\times 10^{-2}$ | $1.10 \\times 10^{-2} \\pm 3.00 \\times 10^{-3}$ |\n|PCA| $3.41 \\times 10^{-1} \\pm 6.20 \\times 10^{-2}$ | $7.00 \\times 10^{-3} \\pm 1.00 \\times 10^{-3}$ |\n|NCA| $3.45 \\times 10^2 \\pm 2.10 \\times 10^2$| $4.00 \\times 10^{-3} \\pm 1.00 \\times 10^{-3}$ |\n|SPCA| $5.62 \\times 10^1 \\pm 5.65 \\times 10^0$| $3.00 \\times 10^{-3} \\pm 1.00 \\times 10^{-3}$ |\n\n**Table R4.2: Average training and test times on OrganC MNIST, in seconds, across 10 repetitions.**\n| Model| Training Time|Test Time|\n|-------------------|------------------------------------|------------------------------------|\n| RF-PHATE| $5.87 \\times 10^2 \\pm 7.67 \\times 10^1$ | $5.05 \\times 10^2 \\pm 7.16 \\times 10^1$ |\n| RF-AE ($N^{*}=10\\\\%$)| $7.60 \\times 10^2 \\pm 2.45 \\times 10^1$ | $2.71 \\times 10^1 \\pm 1.24 \\times 10^0$ |\n| RF-AE ($N^{*}=2\\\\%$)| $6.88 \\times 10^2 \\pm 2.39 \\times 10^1$ | $5.89 \\times 10^0 \\pm 6.50 \\times 10^{-2}$ |\n| PACMAP| $7.16 \\times 10^0 \\pm 7.32 \\times 10^{-1}$ | $1.40 \\times 10^0 \\pm 1.72 \\times 10^{-1}$ |\n| P-SUMAP| $2.95 \\times 10^2 \\pm 3.48 \\times 10^1$ | $1.16 \\times 10^{-1} \\pm 4.00 \\times 10^{-3}$ |\n| CE| $5.29 \\times 10^1 \\pm 9.62 \\times 10^0$ | $4.20 \\times 10^{-2} \\pm 4.00 \\times 10^{-3}$ |\n| AE| $1.92 \\times 10^1 \\pm 2.84 \\times 10^0$ | $4.20 \\times 10^{-2} \\pm 4.00 \\times 10^{-3}$ |\n| SSNP| $2.66 \\times 10^1 \\pm 4.82 \\times 10^0$ | $4.00 \\times 10^{-2} \\pm 7.00 \\times 10^{-3}$ |\n| P-TSNE| $1.71 \\times 10^1 \\pm 4.68 \\times 10^0$ | $3.50 \\times 10^{-2} \\pm 7.00 \\times 10^{-3}$ |\n| P-UMAP| $1.70 \\times 10^1 \\pm 4.81 \\times 10^0$ | $3.30 \\times 10^{-2} \\pm 6.00 \\times 10^{-3}$ |\n| PLS-DA| $2.55 \\times 10^{-1} \\pm 5.10 \\times 10^{-2}$ | $3.10 \\times 10^{-2} \\pm 6.00 \\times 10^{-3}$ |\n| CEBRA| $2.11 \\times 10^0 \\pm 5.33 \\times 10^{-1}$ | $2.60 \\times 10^{-2} \\pm 6.00 \\times 10^{-3}$ |\n| PCA| $3.48 \\times 10^{-1} \\pm 6.50 \\times 10^{-2}$ | $2.40 \\times 10^{-2} \\pm 5.00 \\times 10^{-3}$ |\n| NCA| $3.36 \\times 10^2 \\pm 9.77 \\times 10^1$ | $1.10 \\times 10^{-2} \\pm 1.00 \\times 10^{-3}$ |\n| SPCA| $5.60 \\times 10^1 \\pm 1.22 \\times 10^1$ | $7.00 \\times 10^{-3} \\pm 1.00 \\times 10^{-3}$ |\n\n\n\n\n**Re M1.** In our work, we define new data as data points that were not part of the original training set. In practice, these points may come from the same distribution as the training data or from a different one, provided they belong to the same domain. Labels are not required for new data, as all trained models in our comparison—including RF-AE—can generate out-of-sample embeddings without supervision. In our evaluation framework (Section 4), we artificially introduced new data through test splits. We used predefined training/test splits when available; otherwise, we created them using a random 80/20 stratified split to account for class imbalance.\n\n\n\n**Re M2.** While our method belongs to the broader class of dimensionality reduction techniques, our primary focus is on projecting data into 2D or 3D to enable interpretable insights. We believe the term data visualization more accurately reflects this goal, as the purpose of the embedding is to facilitate direct understanding and subsequent annotation of the data. This wording is also typically used in similar papers that perform dimensionality reduction for data visualization.\n\n**References**\n\n[1] Rhodes et al. Geometry- and accuracy-preserving random forest\nproximities. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–13, 2023."}]

Author rebuttal:

Meta-review: All reviews leaned towards acceptance with two accept [RWaH,RUnf] and two borderline accept [vZZA,mQiQ] recommendations.

The reviewers appreciated many parts of the work:

+ Developing new supervised dimension reduction algorithms was considered a timely topic [mQiQ]
+ The paper was considered well motivated [RUnf]
+ The idea of reconstructing RF-GAP proximities was seen as meaningful and novel [RWaH]; similarly, the method was considered sound and novel by another reviewer [mQiQ]
+ The methodological foundation was considered strong [RWaH]
+ Using the geometric loss in alignment to mitigate overfitting and trivial solutions was appreciated [RWaH]
+ The Structural Importance Alignment (SIA) metric was appreciated as a contribution to evaluation of supervised visualisation [RWaH]
+ The experiments were considered comprehensive [mQiQ]
+ The good empirical performance was appreciated [RWaH,RUnf,mQiQ] and robustness in ablation studies was appreciated [RWaH]
+ The paper was considered clearly written [RUnf] and well-organised and easy to follow [mQiQ]

However, weaknesses were also pointed out, and authors responded to them in the rebuttal stage:

- The motivation and novelty were considered unclear compared to existing work on out-of-sample extensions [vZZA]; authors provided discussion. 
- Lack of use of auxiliary/expert labels was criticised [mQiQ]; authors provided discussion arguing their use of labels means expert labels.
- The computational cost was seen as a limitation [RWaH] including especially initial computation of the full prototype matrix [RUnf]; and scalability and computational cost discussion was desired [RUnf]; authors provided some discussion and tables of training and test times.
- Similarly, scalability versus UMAP was a concern [mQiQ]; authors argued UMAP addresses a different problem and that comparisons would not be meaningful; their table of training/test times included one UMAP-related method.
- Showing results only based on RF-PHATE was criticised [RUnf] and showing results with other kernels was desired [RUnf]; authors provided some discussion and results with other geometric constraints.
- Sensitivity to the forest configuration was a concern [RUnf]; authors provided some discussion but not new results.
- Lack of runtime and memory benchmarks for feasibility on large-scale datasets was criticised [RWaH]; authors promised to include a training and inference time comparison and reported some results.
- There was concern the SIA metric might depend on specifics of the classifier ensemble and perturbation heuristics [RWaH]; authors provided some arguments related to this.
- There was a question os using semi-supervised training [RWaH] and authors argued this is possible but did not provide results.
- Addressing negative societal impacts was desired [RWaH]; authors provided some discussion.
- There was concern about the term visualisation versus dimensionality reduction [mQiQ]; authors argued their goal is more accurately described as visualisation.

Ultimately, reviewer [RUnf] considered their concerns addressed, and [RWaH,mQiQ] considered that most of their concerns were addressed; [vZZA] acknowledged the discussion but did not comment in detail whether the concerns were addressed.

Overall, it seems the reviewers appreciate the work, and the majority of concerns were addressed in the discussion. Overall, it seems the work could be ready for presentation at NeurIPS.
Decision: Accept (poster)