Parametric UMAP: learning embeddings with deep neural networks for representation and semi-supervised learning
Authors: Tim Sainburg
Leland McInnes
Timothy Q Gentner
Venue:  (2021)
Keywords: unsupervised learning
representation learning
dimensionality reduction
UMAP
semi-supervised learning

Abstract: We propose Parametric  UMAP, a parametric variation of the UMAP (Uniform Manifold Approximation and Projection) algorithm. UMAP is a non-parametric graph-based dimensionality reduction algorithm using applied Riemannian geometry and algebraic topology to find low-dimensional embeddings of structured data. The UMAP algorithm consists of two steps: (1) Compute a graphical representation of a dataset (fuzzy simplicial complex), and (2) Through stochastic gradient descent, optimize a low-dimensional embedding of the graph. Here, we replace the second step of UMAP with a deep neural network that learns a parametric relationship between data and embedding. We demonstrate that our method performs similarly to its non-parametric counterpart while conferring the benefit of a learned parametric mapping (e.g. fast online embeddings for new data). We then show that UMAP loss can be extended to arbitrary deep learning applications, for example constraining the latent distribution of autoencoders, and improving classifier accuracy for semi-supervised learning by capturing structure in unlabeled data.

Under review as a conference paper at ICLR 2021
LEARNING EMBEDDINGS WITH DEEP NEURAL NETWORKS FOR REPRESENTATION AND LEARNING
PARAMETRIC UMAP:
Anonymous authors Paper under double-bling brindd review
ABSTRACT
We propose Parametric UMAP, parametric variation of the UMAP (Uniform Manifold Approximation and Projection) algorithm. UMAP is non-parametric graph-based dimensionality reduction algorithm using applied Riemannian ge- ometry and algebraic topology to find low-dimensional embeddings of structured data. The UMAP algorithm consists of two steps: (1) Compute a graphical rep- resentation of dataset (fuzzy simplicial complex), and (2) Through stochastic gradient descent, optimize a low-dimensional embedding the graph. Here, we replace the second step of UMAP with deep neural network that learns para- metric relationship between data and embedding. We demonstrate that our method performs similarly to its non-parametric counterpart while conferring the benefit a learned parametric mapping (e.g. fast online embeddings for new data). We then show that UMAP loss be extended to arbitrary deep learning applica- tions, for example constraining the latent distribution of autoencoders, and im- proving classifier accuracy for semi-supervised learning by capturing structure unlabeled data
Current non-linear dimensionality reduction algorithms can divided broadly into non-parann non-parametric algorithms which rely on the efficient computation of probabilistic relationships from neighborhood graphs to extract structure in large datasets (e.g. UMAP (Mcinnes et al., 2018), t-SNE (van der Maaten & Hinton, 2008), Large (Tang et al. 2016)), and parametric algorithms, which, driven by advances in deep-learning, optimize an objective function related to capturing structure a dataset over neural network weights (c.g. Hinton & Salakhutdinoy 2006 Ding et al. [2018] Ding & Regev 2019 Szubert et al. 2019: Kingma & Welling 2013).
The goal of this paper is wed those two classes of methods: learning structured graphical representation of the data and using a deep neural network to embed that graph. Over the past decade several varients of the t-SNE algorithm have proposed parameterized forms of t-SNE (Van) Der Maaten 2009; Gisbrecht et al. 2015; Bunte et al. 2012 Gisbrecht et al. (012).. In partic- ular, Parametric t-SNE (Van Der Maaten 2009) performs exactly that wedding: training deep neural network to minimize loss over a t-SNE graph. However, the t-SNE loss function itself is not well suited to be optimized over deep neural networks using contemporary training schemes. In particular, t-SNE's optimization requires normalization over the entire dataset at each step of optimization, making batch- based optimization and on-line learning of large datasets difficult. In contrast, UMAP is optimized using negative sampling (Mikolov et al. [2013] Tang al. (0166) and requires no normalization step, making it more well-suited to deep learning applications. Our pro- posed method, Parametric UMAP, brings the non parammemttiic graph-based dimensionality reduction algorithm UMAP into a emerging class parametric topologically-inse inspired embedding algorithms (Reviewed in A.5).
In the following section we broadly outline the algorithm underlying UMAP to explain why our proposed algorithm, Parametric UMAP, is particularly well suited to deep learning applications. We contextualize our discussion of UMAP in t-SNE, to outline the advantages that UMAP confers over 1-SNE in the domain of parametric neural-network based embedding. We then perform experiments
Google Colab walkthrough

---

Under review as a conference paper at ICLR 2021
comparing algorithm, Parametric UMAP, to parametric and non- con-parametric algorithms. Finally, we show a novel extension of Parametric UMAP to semisupervised learning.
A
B
Step 1: Compute graphical representation of the dataset
Step (non-parametric)) Learn embedding that preserves the structure of the graph
Step 2 (parametric) Learn set of weights that preserves the structure of the graph
Figure 1: Overview of UMAP (A - B) and Parametric UMAP (A C). (A) The first stage of the UMAP algorithm is to compute a probabilistic graphical representation of the data. (B) The second stage of the UMAP algorithm is to optimize set of embeddings to preserve the structure of the fuzzy simplicial complex. (C) The second stage of UMAP, learning set of embeddings which preserves the structure of the graph, is replaced with neural network which learns set of neural network weights (parameters) that maps the high-dimensional data to an embedding. Both B and C learned through the same loss function.
0.1 PARAMETRIC UMAP
t-SNE and its more recent cousin UMAP are both non-parametric graph-based dimensionality re- duction algorithms that learn embeddings based upon local structure in data. Much prior work has focused on extending t-SNE to learn the mapping between data and embeddings, for example opti- mizing t-SNE's loss over learned neural network weights (Van Der Maaten 2009). However, t-SNE optimizes it's embeddings in manner poorly suited to being translated to neural network opti- mization, requiring significant modification of the learning algorithm to train via stochastic gradient descent. Conversely, UMAP's s loss is optimized in a manner that can be directly translated for neu- ral network optimization. Here, we explain UMAP's compatability with deep learning applications, why this differs for t-SNE, and how take advantage. A full discussion of UMAP and t-SNE explaining this contrast is given in the Appendix A.I
To summarize, both t-SNE and UMAP rely on the construction of a graph, and subsequent embed- ding that preserves the structure of that graph (Fig. [T. UMAP learns an embedding by minimizing cross entropy sampled over positively weighted edges (attraction), and using negative sampling ran- domly over the dataset (repulsion), allowing minimization to occur over sampled batches of the dataset. t-SNE, meanwhile, minimizes KI. divergence loss function normalized over the entire set of embeddings in the dataset using different approximation techniques to compute attractive and repulsive forces.
Because (-SNE optimization requires normalization over the distribution of embedding in projection space, gradient descent can only be performed after computing edge probabilities over the entire dataset. Projecting an entire dataset into a neural network between each gradient descent step would be too computationally expensive optimize however. The trick that Parametric t-SNE proposes to this problem is to split the dataset up into large batches (e.g. 5000 datapoints in the original paper) that are independently normalized over and used constantly throughout training, rather than being randomized. Conversely, UMAP can be trained on batch sizes small as single edge, making suitable for minibatch training needed for datasets well on-line learning.
neural networks trained on large
Given these design features, the UMAP algorithm is better suited to deep neural networks, and more extendable typical neural network training regimes. Parametric UMAP be defined simply by applying the UMAP cost function to a deep neural network over mini batches using negative sampling within minibatches. In implementation, we keep all hyperparameters the
2

---

Under review as a conference paper at ICLR 2021
same the original UMAP implementation, and optimize over the UMAP loss function using the Adam (Kingma & Ba 2014 optimizer
UMAP AS A REGULARIZATION
In machine learning, regularization refers to a modification of the learning algorithm to improve generalization to data. Here, consider both regularizing neural networks with UMAP loss, as well as using additional loss functions to regularize UMAP.
While non-parametric UMAP optimizes UMAP loss directly over embeddings (Fig A), our pro- posed algorithm, Parametric UMAP, applies the same cost function over an encoder network (Fig 2B). Beyond the parameters being optimized, the optimization steps are equivalent. Given this sim- ple extension toward optimizing over a neural network, UMAP can further be extended to arbitrary neural network architectures and loss functions. Specifically, we use UMAP loss over unlabeled data as regularization term for a classifier network trained on labeled data (semi-supervised learning), as well as an autoencoder loss as an additional regularization for UMAP latent projections. In the experiments section we quantitatively explore these networks.
UMAP
B Parametric UMAP
C
UMAP
UMAP with classifier
Rocon
Classif.
X
Z
X
Z
X
X-
C
UMAP
UMAP
UMAP
CUMAP
Z
Figure 2: An outline of the varients of UMAP used in this paper. Solid lines represent neural networks. Dashed lines represent error gradients.
1.1 AUTOENCODING WITH UMAP
AEs are by themselves a powerful dimensionality reduction algorithm (Hinton & Salakhutdinov 2006). Thus, combining them with UMAP may yield additional benefits in capturing latent structure. We used an autoencoder as an additional regularization to Parametric UMAP CC) A UMAP/AE hybrid is simply the combination of the UMAP loss and reconstruction loss and apply both over the network. VAEs have similarly been used in conjunction with Parametric t-SNE for capturing structure animal behavioral data (Graving Couzin, 2020) and combining I-SNE, which similarly emphasizes local structure, with AES aids in capturing more global structure over the dataset (van der Maaten & Hinton, 2008 Graving Couzin 2020).
1.2 SEMISUPERVISED LEARNING
UMAP can also be used to regularize supervised classifier networks, training the network on combination of labeled data with the classifier loss and unlabeled data with the UMAP loss (Fig 2D). supervipe learning refers the use of unlabeled data to jointly learn the structure of dataset while labeled data is used to optimize the supervised objective function, such as classifying images. Here, we explore how UMAP can be jointly trained as an objective function a deep neural network alongside classifier.
In the example Fig 3 we show an intuitive example of semisupervised learning using UMAP over the Moons dataset (Pedregosa et al. 2011). By training Y-shaped network (Fig 2D) both on the classifier loss over labeled datapoints (Fig A red and blue) and the UMAP loss over unlabeled datapoints (Fig 3A, grey) jointly. the shared latent space between between the UMAP and classifier network pulls apart the two moons (Fig 3),, resulting a a decision boundary that divides cleanly between the two distributions in dataspace.
"See code implementations: Experiments (wwwpplaccholder.com) Python package (www.placcholder.com)
3

---

Under review as a conference paper at ICLR 2021
Decision Contour
Embedding
UMAP loss
Classifier ClassifferAcuuraey
Cross Entropy
0.20
Accuracy
0.90
0.15
train
0.10
0.80
valid
10K
10K
Batch
Batch
Figure 3: An example of semi- supervised learning with UMAP on the moons dataset. (A) A de- cision contour learned over the moons dataset with 3 labeled datapoints from each class. Unlabled datapoints are shown in grey and labeled datapoints are shown in red and blue. The decision contour shown in the background using the 'coolwarm' colormap. (B) The learned embeddings in the jointly trained network. (C) UMAP loss over the unlabeled training dataset. (D) Classifier accuracy for the training and validation set.
EXPERIMENTS
Samples (umap-leam)
Amarrite
Came
Milla gla Eorgin oile Fariantil
Figure 4: Example comparison of 2D projections from Parametric UMAP and each baseline. (A) 3D buffalo. (B) Mouse retina single cell transcriptomes. (C) Fashion MNIST. 2D projections of the remaining datasets are given in Fig. 12
Experiments were performed comparing our novel algorithms, Parametric UMAP and UMAP/AE hybrid, to several baselines: nonparametric UMAP, nonparametric t-SNE (FIt-SNE) (Polcar et al. 2019). Parametric t-SNE, an AE, a VAE, and a PCA projections. Many additional relevant com- parisons (i.e. those discussed in A.5) non-parametric UMAP, and our embedding comparisons were performed to confirm similar quality their non-parametric counterparts, not outperform them. We additionally compare second con-aaamettiic UMAP implementation that has the same underlying code as Parametric UMAP, but where optimization is performed over embeddings directly, rather than neural network weights. This comparison is made to provide bridge between the UMAP-learn implementation and parametric UMAP, to control for variation exogenous to differences parametric versus non- -arammetrii embed- ding. Parametric t-SNE, Parametric UMAP, the AE, VAE, and the UMAP/AE hybrid use the same neural network architectures and optimizers within each dataset (described A.2 and A.3). We used the common machine learning benchmark datasets MNIST, FMNIST, and CIFAR 10 alongside two real-world datasets in areas where UMAP has proven a useful tool for dimensionality reduc- tion: single-cell retinal transcriptome dataset (Macosko al. 2015). and bioacoustic dataset of Cassin's vireo song, recorded in the Sierra Nevada mountains Hedley 2016a b).
made, those methods had already been compared to
2.1 EMBEDDINGS
We first confirm that Parametric UMAP produces embeddings that of similar quality parametric UMAP. To quantitatively measure the quality of embeddings we compared embedding algorithms several metrics across datasets. We compared each method/dataset 2D and 641)
4

---

Under review as a conference paper at ICLR 2021
projections Each metric is explained in detail in A.4 The 2D projection of each dataset/method is shown in Fig The results are given in Figs 13 and Tables 6 and summarized below.
Trustworthiness Trustworthiness (Eq. 5 Venna & Kaski 2006) is a measure of how much of the local structure of dataset is preserved in a set of embeddings. In 2D, we observe each of the UMAP algorithms performs similarly in trustworthiness, with t-SNE being slightly more trustworthy in each dataset (Fig 13 Table 2). At 64D, PCA, AE, VAE, and Parametric t-SNE are most trustworthy in comparison to each UMAP implementation, possibly reflecting the more approximate repulsion (negative sampling) used by UMAP.
KNN-Classifier A KNN-classifier is used as a baseline to measure supervised classification ac- curacy based upon local relationships in embeddings. We find KNN-classifier performance largely reflects trustworthiness (Figs [4] [3] Tables 3 n 4 2D, we observe a broadly similar perfor- mance between UMAP and t-SNE varients, each of which is substantially better than the PCA, AE, or VAE projections. At 64 dimensions UMAP projections are similar but in some datasets (FMNIST, CIFARI0) slightly underperform PCA, AE, VAE, and Parametric t-SNE.
Silhouette score Silhouette score measures how clustered a set of embeddings are given ground truth labels. In 2D, across datasets, tend a better silhouette score for UMAP and Parametric UMAP projections than t-SNE and Parametric t-SNE, which are in turn more clustered than PCA in all cases but CIFAR1O, which shows little difference from PCA (Fig 16 5). The clustering of each dataset can also be observed in Fig where t-SNE and Parametric t-SNE are more spread out within cluster than UMAP. 64D projections, we find the silhouette score of Parametric t-SNE is near or below that of PCA, which are lower than UMAP-based methods. We note, however, that the poor performance of Parametric t-SNE may reflect setting the degrees-of-freedonn (a) at d - I which is only one of three parameterization schemes that Van Der Maaten| (2009) suggests. A learned regrees-of-freedon parameter might improve performance for parametric i-SNE at higher dimensions.
Clustering To compare clustering directly across embeddings, we performed k-Means clustering over each latent projection and compared each embedding's 's clustering on the basis of the normalized mutual information (NMI) between clustering schemes (Fig 17. Table 6 In both the 2D and 64D projections, find that NMI corresponds closely the silhouette score. UMAP and t-SNE show comperable clustering in 2D, both well above PCA in most datasets. At 64D, each UMAP approach shows superior performance over t-SNE.
2.2 SPEED
Training speed Optimization in non-paraett UMAP is not influenced by the dimensionality of the original dataset; the dimensionality of the dataset only comes into play in computing the nearest-neighbors graph. In contrast, training speeds for Parametric UMAP are variable based upon the dimensionality of data and the architecture of the neural network used. The dimensionality of the embedding does not have a substantial effect speed. In Fig 5 we show the cross-entropy loss over time for Parametric and conmparamttric UMAP, for the MNIST, Fashion MNIST, and Retina datasets. Across each dataset we find that non-parametric UMAP reaches lower loss more quickly than Parametric UMAP, but that Parametric UMAP reaches a similar cross-entropy within an order of magnitude. Thus, Parametric UMAP can train more slowly than nor-parametric UMAP, but training times remain within a similar range making Paramatric UMAP reasonable alternative to non-parametris UMAP in terms of training time.
Embedding and reconstruction speed A parametric mapping allows embeddings to be inferred directly from data, resulting in a quicker embedding than non- non-parametric methods. The speed of embedding is especially important in signal-processing paradigms where near real-time embedding speeds are necessary. For example in brain-machin interfacing, bioacoustics, and computational
'Where possible. In contrast with UMAP. Parametric UMAP, and Parametric t-SNE, Barnss Huts t-SNE can only embed in two or three dimensions (Van Der Maaten] 2014) and while FIt-SNE can in principle scale to higher dimensions (Landerman al. 2019), emhedding in more than 2 dimensions is unsupported in both
the official implementation (Klugerl aa 2020 and openTSNE: (Policar al. 2019)
5

---

Under review as a conference paper at ICLR 2021
MNIST
Refina
Fashion MNIST
UMAP
P. UMAP
P.UMAP
Closs Entropy
0.4
UMAP (TF)
(TF)
UMAP (TF)
0.3
UMAP (UMAP-(earT)
UMAP (UMAP -leam))
UMAP (UMAP-leATT)
0.2
0.0
10
10
10F
10
102
107
10
10
10
10
10a
10
100
10
10
107
Time (5)
Time
Time
Figure 5: Training times comparison between UMAP and Parametric UMAP. All results were ob- tained with up to 32 threads on machine with 2 AMD EPYC Rome 7252 8-Core CPU running at 3.1 GHz and Quadro RTX 6000.
ethology, fast embedding methods like PCA or deep neural networks are necessary for real-time analysis and manipulations and deep neural networks are increasingly being used (e.g. Pandarinath et al. [2188] Brown & De Bivort 2018 Sainburg et al. (2019). Here, we compare the embedding speed of held-out test sample for each dataset, as well as the speed of reconstruction of the same held out test samples.
Broadly, we observe similar embedding times for the non- non-parametrie t-SNE and UMAP methods, which are several orders of magnitude slower than the parametric methods (Fig Because the same neural networks are used across the different parametric UMAP and i-SNE methods, we show only Parametric UMAP in Fig 12, which is only slightly slower than PCA, making it viable candidate for fast embedding in embedding paradigms where PCA is currently used.
Embodeing speee text
Dimensions
Dimensions
CILLHbLLE
12-1
TME (M0 cam
12-1
PCA
MNIST
FMNIST
Retina
Cassin's
CIFARIO
MN
PMNIST
Retina
Cassie's
CIFAR10
Figure 6: Comparison of embedding speeds using parametric UMAP and other embedding algo- rithms on held-out testing dataset. Embeddings were performed on the same machine as Figure 11. Values shown are the median times over 10 runs.
Similar to the embedding speeds, we compared parametric and non parammettii UMAP reconstruc- tion speeds (Fig We find that the reconstructions of Parametric UMAP are orders of magnitude faster than non-parametric UMAP, and slightly slower, but within the same order of magnitude, as PCA.
2.3 AUTOENCODING
The ability to reconstruct data from embeddings can both aid in understanding the structure non- linear embeddings, as well as allow for manipulation and synthesis of data based on the learned features of the dataset. We compared the reconstruction accuracy across each method which had trressetrannfoom capabilities (i.e. Z X), well as the reconstruction speed across the neu- ral network-based implementations to non-parametric implementations and PCA. In addition, we performed latent space algebra Parametric UMAP embeddings both with and without toencoder regularization, and found that reconstructed data can be linearly manipulated in complex feature space.
Reconstruction accuracy We measured reconstruction accuracy as Mean Squared Error (MSE) across each dataset (Fig Table 7). In two dimensions, we find that Parametric UMAP typically reconstructs better than non panameeree UMAP, which turn performs better than PCA. In addition, the autoencoder regularization slightly improves reconstruction performance. At 64 dimensions, the AE regularized Parametric UMAP generally comparable to the AE and VAE and performs bet- ter than Parametric UMAP without autoencoder regularization. The non-parametric UMAP recon-

---

Under review as a conference paper at ICLR 2021
struction algorithm is not compared at 64 dimensions because it relies estimation of Delaunay triangulation, which does not scale well with higher dimensions.
Dimenalons
L
Relative MSE
UMA AL p. UNA
L
MNST
FNNIST
Retina
Cassin's
CIFARLO
MNIST
FMNIST
Retina
Cassin's
CIFARIO
Figure 7: Reconstruction accuracy measured mean squared error (MSE). MSE shown relative to each dataset (setting mean at 1).
Latent features Previous work shows that parametric embedding algorithms such AEs (e.g. Variational Autoencoders) linearize complex data features in latent-space, for example the presence of pair of sunglasses in pictures of faces (e-g. Radford al. 2013; White 2016 (Sainburg et al. 2018). Here. we performed latent-space algebra and reconstructed manipulations on Parametric UMAP latent-space to explore whether UMAP does the same.
To do so, we use the CelebAMaakkHQ dataset, which contains annotations for 40 different facial features over highly-structure dataset of human faces. We projected the dataset of faces into CNN autoencoder architecture based upon the architecture defined in Huang et al. (2018). We trained the network first using UMAP loss alone (Parametric UMAP: right,, and second using the joint UMAP and AE loss (Fig 8 center). We then fit OLS regression to predict the latent projections of entire dataset using the annotated features (e.g. hair color, presence of beard, smiling, etc). The vectors corresponding treated as feature vectors in latent space, and added and subtracted from projected images, then passed through the decoder observe the resulting image (as in Sainburg al. 2018).
each feature learned by the linear model were then
We find that complex latent features are linearized in latent space, both when the network trained with UMAP loss alone well when the network is trained with AE loss. For example, in the last set of images in Figure 10B, a smile can be added or removed from the projected image by adding subtracting its corresponding latent vector.
A
Orig.
UMAP + AE UMAP
B
UMAP AE
Barys Mo Sungei
UMAP
Bangs
DUrk NaY
Slando hal
Daw halr
Blonde hale
Widh JRCE
Flemoo foce
Wate locn
Figure 8: Reconstruction and interpolation. (A) Parametric UMAP reconstructions of faces from holdout testing dataset. (B) The networks, adding latent vectors corresponding to image features.
2.4 SEMI-SUPERVISER LEARNING
Real-word datasets are often comprised small number labeled data, and a large number of unlabeled data. Semisupervised learning (SSL) aims to use the unlabeled data to learn the structure of the dataset, aiding supervised learning algorithm in making decisions about the data. Current SOTA approaches in many areas of supervised learning such computer vision rely on deep neural networks. Likewise, semisupervised learning approaches modify supervised networks with stucture- learning loss using unlabeled data. Parametrie UMAP, being neural network that learns structure

---

Under review as a conference paper at ICLR 2021
from unlabeled data, is well suited semi-supervised applications. Here, we determine the effi- cacy of UMAP for semisupervised learning by jointly training a Y-shaped neural network (Fig 2D) on classification and UMAP compared to classification alone datasets with varying numbers of labeled data.
We compared datasets ranging from highly-structured (MNIST) unstructured (CIFAR10) in UMAP using a naive distance metric in data space (e.g. Euclidean distance over images). For image datasets, we used deep convolutional neural networks (CNN) which performs with rela- tively high accuracy for CNN classification on the fully supervised networks (see Table X based upon the CNN13 architecture commonly used in SSL (Oliver al. 2018). For the birdsong dataset we used BLSTM network, and for the retina dataset we used a densely connected network.
Naïve UMAP embedding For datasets where structure is learned in UMAP (e.g. MNIST, FM- NIST) we expect that regularizing a classifier network with UMAP loss will aid the network in labeling data by learning the structure of the dataset from unlabeled data. To test this, we com- pared baseline classifier to a network jointly trained on classifier loss and UMAP loss. We first trained the baseline classifier to asymptotic performance on the validation dataset, then us- ing the pretrained-weighe on UMAP over Euclidean distances and a classifier loss over the dataset. We find that for each dataset where categoricallly-reeeann structure is found in latent projections of the datasets (MNIST, FMNIST, birdsong, retina), classifications are improved in the semi-supervised network over the supervised network alone, especially with smaller numbers of training examples (Fig 9 Table R In contrast for CIFARIO, 10, the additional UMAP loss impairs performance in the classifier.
from the baseline classifier, trained a Y-shaped network (Fig 2D) jointly
WNIST
PMNNSTT
Casxin's vireo song
Retina transer stoms
FFRR1
3.00
2.95
6-3
6:90
Buckiburt
5:35
1MA tha
60
Lac 24
14 255 1028
22
04
255 1004 M/
36 C4 255 1004 TTaarin Zoarple.
141
14 255 1004 TTariing Soorpfee
10 04 220 3024
Figure 9: Baseline classifier with an additional UMAP loss with different numbers of labeled train- ing examples. Kon-parametric UMAP projections of the UMAP graph being jointly trained shown in the bottom right of each panel. Error bars show SEM.
Consistency regularization and learned invariance using data augmentation Several current SOTA SSL. approaches employ technique called consistency regularization (Sajjadi et al.) 2016); training a classifier to produce the same predictions with unlabeled data which have been augmented and data that have not been augmented (Sohn et al.] 2020 Berthelot et al. (020). In a similar vein, for each image dataset, train the network to preserve the structure of the UMAP graph when data have been augmented. We computed a UMAP graph over un-augmented data, and using augmented data, trained the network jointly using classifier and UMAP loss, teaching the network learn to optimize the same UMAP graph, invarient to augmentations in the data. We observe a further improvement in network accuracy for MNIST and FMNIST over the baseline, and the augmented baseline (Fig 10 Table 8). For the CIFAR10 dataset, the addition of the UMAP loss, even over augmented data, reduces classification accuracy.
1.50
0.5
0.3
Kcaunay
3.50
0.8
2.85
Eaxeline
0.7
25
faseine
0.4-
Baaclins
Aug.
UMAF Eaclidean
USAP (tucidean)
Aug. UWAP (aadddaa
64
23b
64
25b
Traping
Trainingsaappls
Trahiinghaappss
Figure 10: Comparison of baseline classifier, augmentation, and augmentation with an additional UMAP loss.
8

---

Under review as a conference paper at ICLR 2021
Learning a categorically-rreeaan UMAP metric using supervised network It is unsurpris- ing that UMAP confers no improvement for the CIFAR 10 dataset, as UMAP computed over the pixel-wise Euclidean distance between images in the CIFARIO dataset does not capture very much categorically-releant structure in the dataset. Because no common distance metric over CIFARIO images is likely capture such structure, relevant distance metric for UMAP. We do so by training on UMAP graph computed using dis- tance over latent activations in the classifier network (as in, e.g. Carter et al. 2019), where categorical structure can be seen in UMAP projection (Fig 19). The intuition being that training the network with unlabeled data to capture instributional-structt within the network's 's learned categorically-reie relleaat space will aid in labeling new data.
consider using supervision to learn a categorically-
We find that in all three datasets, without augmentation, the addition of the learned UMAP loss con- fers little no improvement in classification accuracy over the data (Fig 11 T).le When we look at non-parane non-parametric projections of the graph over latent space activations, we see that the learned graph largely conforms to the network's dtready-presenn categorical decision making (e.g. Fig 19 predic- tions vs. ground truth). In contrast, with augmentation, the addition of the UMAP loss improves performance in each dataset, including CIFAR1O. This contrast in improvement demonstrates that training the network to learn a distribution in a antegorically-reeeve to the network does not confer any additional information that the network can use in classification. Training the network to be invariant toward augmentations in the data, however, does aid in regular- izing the classifier. more in-line with directly training the network on consistency in classifications Sajjadi et al. 2016).
space that is already intrinsic
MNIST
FMNIST
CIFARIO
1.00
0.9
0.95
Accrboy
0.90
Baseline
0.8
Iaseins
0.6
Raseline
Asg.
Aus-
Aug.
UMAP (leamad)
UMAFF carned:
UMM Dearzed)
Aag-
UMAP (marned)
0.5
Aug UMAP Onarsed)
0.2
+Aug - US4 (eamme))
fel)
Tratning Examples
Training Exomples
Tramntngtazmotes
Figure 11: SSL using UMAP over the learned latent graph, computed over latent activations in the classifier.
DISCUSSION
In this paper we propose a novel parametric extension to UMAP using arbitrary deep neural net- works. This parametric form of UMAP produces similar embeddings to non-parametric UMAP, with the added benefit of a learned mapping between data space and embedding space. We demon- strated the utility of this learned mapping on several downstream tasks. We showed that UMAP applied to deep neural networks improves inference times for embeddings and reconstructions by orders of magnitude while maintaining similar embedding quality to non- -on-paraettric UMAP. Com- bined with an autoencoder, UMAP improves reconstruction quality and allows for reconstruction of high-dimensional UMAP projections. We also show that parametric UMAP projections linearize complex features in latent space, similar to other latent neural networks such as AES and GANs. Combined with a classifier, UMAP can be used for semi-supervised learning vastly improving train- ing accuracy on datasets where small numbers of training exemplars are available. We showed that UMAP loss applied to classifier improves semi- supervsper learning in real world cases where UMAP projections carry categorically-relee reelevaa information (such as stereotyped birdsongs or single- cell transcriptomes), but not in cases where categorically relevant strueture is present (such as CIFAR10). We devised two downstream approaches based around learned categorically--eelvan distances, and consistency regularization, that show improvements on these more complex datasets. Parametric embedding also makes UMAP feasible in fields where where dimensionality reduction of continuously generated signals plays an important role in real-time analysis and experimental control.
Future work can explore several of these topics such as on-line continuous embeddings of real- world datasets, semi-supervised learning with UMAP with improved nategoricallly-reevan metrics, and applying UMAP loss to novel deep learning applications.

---

Under review as a conference paper at ICLR 2021
REFERENCES
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and aug- mentation anchoring. In International Conference on Learning Representations, 2020. URL
https::/<penreview net http://wwnee//oommmm/////// Forumm21d-HLkkekkPPp
André EX Brown and Benjamin De Bivort. Ethology as physical science. Nature Physics, 14(7): 653-657, 2018.
Kerstin Bunte, Michael Bichl, and Barbara Hammer. A general framework for dimensionality- reducing data visualization mapping. Neural Computation, 24 2012.
Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. Activation atlas. Distill, 4(3):e15, 2019.
Jiarui Ding and Aviv Regev. Deep generative model embedding of single-cell ma-seq profiles on hyperspheres and hyperbolic spaces. BioRxiv, pp. 853457, 2019.
Jiarui Ding, Anne Condon, and Sohrab P Shah. Interpretable dimensionality reduction of single cell transcriptome data with deep generative models. Nature communications, 9(1):1-13, 2018.
Wei Dong, Charikar Moses, and Kai Li. Efficient k-nearest neighbor graph construction for generic similarity measures. In Proceedings of the 20th international conference World wide web, pp. 577-586, 2011.
Douglas Duhaime. Umap zoo.
htttps://githue com www.hchhhaaiineeec wwtp:/cwc.cccccccccaaaaaacaaaaa
zoo 2019.
Andrés Duque, Sacha Morin, Guy Wolf, and Kevin R Moon. Extendable and invertible manifold
learning with geometry regularized autoencoders. arXiv preprint arXi:2200700714. 2020.
Andrej Gisbrecht, Wouter Lueks, Bassam Mokhel, and Barbara Hammer. Out-of-sample kernel extensions for nonparametric dimensionality reduction. In ESANN, 2012.
Andrej Gisbrecht, Alexander Schulz, and Barbara Hammer. Parametric nonlinear dimensionality reduction using kernel t-sne. Neurocomputing, 147:71-82, 2015.
Jacob M Graving and lain D Couzin. Vae-sne: a deep generative model for simultancous dimen- sionality reduction and clustering. BioRxiv, 2020.
Richard W Hedley. Complexity, predictability and time homogeneity of syntax in the songs of cassin's 's vireo (vireo cassinii). PloS one, 11(4)=e010822, 2016a.
Richard W Hedley. Composition and sequential organization of song repertoires in cassin's vireo (vireo cassinii). Journal of Ornithology, 157(1):13 2016b.
Geoffrey E Hinton and Ruslan Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504-507, 2006.
Christoph Hofer, Roland Kwitt, Mare Niethammer, and Mandar Dixit. connectivity-optimzze rep- resentation learning via persistent homology. In International Conference on Machine Learning, pp. 2751-2760. PMLR, 2019.
Xun Huang, Ming-Yu Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 172-189, 2018.
Kui Jia, Lin Sun, Shenghua Gao, Zhan Song, and Bertram Shi. Laplacian auto-encoders: explicit learning of nonlinear data manifold. Neurocomputing, 160:250-260, 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 69800, 2014.
Diederik Kingma and Max Welling, 1312v:1312.61 2013.
Auto-encoding variational bayes.
arXiv preprint
10

---

Under review as a conference paper at ICLR 2021
KlugerLab. Fit-sne. ATT Labs [Online]. Available: http://yam.eecmm..co///ddddmiii 2, 2010.
KlugerLab, Open source survey. http:/aa00e40000000/a/aaeeebbbbb
-SNE, commit
2020.
Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
George C LLndermann Manas Rachh, Jeremy G Hoskins, Stefan Steinerberger, and Yuval Kluger. Efficient algorithms for t-distributed stochastic neighborhood embedding. arXiv preprint arXiv:1712.09005: 2017.
George C Linderman, Manas Rachh, Jeremy Hoskins, Stefan Steinerberger, and Yuval Kluger. Fast interpolatioon-based t-sne for improved visualization of single-cell ma-seq data. Nature meth- ods, 16(3):243-245, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Evan Z Macosko, Anindita Basu, Rahul Satija, James Nemesh, Karthik Shekhar, Melissa Goldman, Itay Tirosh, Allison R Bialas, Nolan Kamitaki, Emily M Martersteck, et al. Highly parallel genome- wide expression profiling of individual cells using nanoliter droplets. Cell, 161(5):1202- 1214, 2015.
Leland McInnes, John Healy, and James Melville. Umap: projection for dimension reduction. arXiv preprint arXiv:1802.03422 2018.
Uniform manifold approximation and
Tomas Mikolov, Ilya Sutskever, Kai Chen. Greg Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp. 3111 19, 2013.
Gal Mishne, Uri Shaham, Alexander Cloninger, and Israel Cohen. Diffusion nets. Applied and Computational Harmonic Analysis, 47(2):259-285 285, 2019.
Kevin R Moon, David van Dijk, Zheng Wang, Scott Gigante, Daniel B Burkhardt, William S Chen, Kristina Yim, Antonia van den Elzen, Matthew Hirn, Ronald Coifman, et al. Visualizing structure and transitions in digh-dimensional biological data. Nature biotechnology. 37(12):1482 1492, 2019.
Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. ICML, 2020.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realis- tic evaluation of deep semi-supervieed learning algorithms. Advances neural information processing systems, pp. 3235-3246, 2018.
Gautam Pai, Ronen Talmon, Alex Bronstein, and Ron Kimmel. Dimal: Deep isometric manifold learning using sparse geodesic sampling. In 2019 IEEE Winter Conference Applications of Computer Vision (WACV), pp. 819-828. IEEE, 2019.
Chethan Pandarinath, Daniel J O'Shea Jasmine Collins, Rafal Jozefowicz, Sergey D Stavisky, Jonathan C Kao, Eric Trautmann, Matthew Kaufman, Stephen Ryu, Leigh R Hochberg, et al. Inferring single-trial neural population dynamics using sequential auto-encoders. Nature methods, 15(10)880-88)) 2018.
Fabian Pedregosa, Gaël Varoquaux. Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Seikit-leam: Machine learning in python. the Journal of machine Learning research, 12:2825-2830. 2011.

---

Under review as a conference paper at ICLR 2021
Pavlin G. Poličar, Martin Stražar, and Blaž Zupan. opentsne: a modular python library for t-sne
dimensionality reduction and embedding. bioRxiv, 2019. doi: 10.1101/731877. URL https:
www.bicrxiv. www. http:///wwr.....ooo/////////////
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep
convolutional generative adversarial networks, arXiv preprint arXiv:/511.06444 2015.
Isaac Robinson. Interpretable visualizations with differentiating embedding networks. arXiv
preprint arXiv:2006. 006400000000640,, 2020.
Peter Rousseeuw, Silhouettes: a graphical aid to the interpretation and validation of cluster analy- sis. Journal of computational and applied mathematics, 20:53-65 -5.3-65,170 1987.
Tim Sainburg, Marvin Thielk, Brad Theilman, Benjamin Migliori, and Timothy Gentner. Gener- ative adversarial interpolative autoencoding: adversarial training on latent space interpolations encourage convex latent distributions. arXiv preprint arXiv:1807.0666 2018.
Tim Sainburg, Marvin Thielk, and Timothy Q Gentner. Latent space visualization, characterization,
and generation of diverse vocal communication signals. bioRviv, 2019. doi: 10.1 1011/01/8 11.
htt.hhttp:://wwwtt wttp:/wwwwwww.w....p/ow/wo http:/wwwwwwww......ooooeeeeeeee
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. In Advances in neural information processing systems, pp. 1163-1171, 2016.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying supervise learning with consistency and confidence. arXiv preprint arXiv:2001.0766 000.22001.07685,, 2020.
Benjamin Szubert, Jennifer E Cole, Claudia Monaco, and Ignat Drozdov. Structure- pencture-preserving visualisation of high dimensional single-cell datasets. Scientific reports, 9(1):1-10, 2019.
Jian Tang, Jingzhou Liu, Ming Zhang, and Qiaozhu Mei. Visualizing large-scale and high- dimensional data. In Proceedings of the 25th international conference world wide web, pp- 287-297, 2016.
Laurens Van Der Maaten, Learning parametric embedding by preserving local structure. In Artifi cial Intelligence and Statistics, pp. 384-391, 2009.
Laurens Van Der Maaten. Accelerating t-sne using tree-based algorithms. The Journal of Machine
Learning Research, 15(1):3221-3245,200
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2605, 2008.
Jarkko Venna and Samuel Kaski. Local multidimensional scaling. Neural Networks, 19(6-7):889 899, 2006.
Tom White. Sampling generative networks. arXiv preprint arXiv:1609.04463 2016.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 1rXivv/708.007477 2017.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 GRAPH CONSTRUCTION AND EMBEDDING
UMAP and t-SNE have the same goal: Given D-dimensional data set X E RD, produce d dimensional embedding Z E IRd such that points that are close together X (e.g. Iy and IT)) are also close together in Z (2; and 2z)-
Both algorithms are comprised of the same two broad steps, first construct a graph of local rela- tionships between datasets (Fig [A]. then optimize an embedding in low dimensional space which preserves the structure of the graph (Fig BB). The parametric approach replaces the second step of this process with an optimization of the parameters of deep neural network (FigTT)
A.1.1 GRAPH CONSTRUCTION
Computing probabilities in X The first step in both UMAP and t-SNE is to compute a distri- bution of probabilities between pairs of points in X based upon the distances between points in data space. Probabilities are initially computed local, one-directional, probabilities between point and its neighbors in data-space, then symmetrized yield final probability representing the relationship between pairs of points.
In t-SNE, these probabilities are treated as conditional probabilities of neighborhood (p: puted using a Gaussian distribution centered at Ti.
SNE) com-
exp (-d(x,, /20)
INE PJ
(I)
Σ Eneei exp (-d(x,, Xx, /20
Where d(x;, x)) represents the distance between J; an 2; (e.g. Euclidean distance) and as is the standard deviation for the Gaussian distribution, set based upon the a perplexity parameter such that one standard deviation of the Gaussian kernel fits a set number nearest-neighbors X.
In UMAP, local, one-directional, probabilities (PUMAP) are computed between a point and its neigh- bors to determine the probability with which an edge (or simplex exists), based upon assumption that data is uniformly distributed across a manifold in a warped dataspace. Under this assumption, a local notion of distance is set by the distance the g.th nearest neighbor and the local probability is scaled by that local notion of distance.
UMAP
(xp(-(d(x,,x)) -p))oo)
Pjjs
Where Pi is local connectivity parameter set to the distance from v; to its nearest neighbor, and o; is local connectivity parameter set to match the local distance around x; upon its k nearest neighbors (where is a hyperparameter).
After computing the one-directional edge probabilities for each datapoint, UMAP computes a global probability as the probability of either of the two local, one-direcii probabilities oc- curring Pij (Pyl: + Palj) - PyjiPilj- In contrast, t-SNE symmetrizes the conditional probabilities
P
as Pij
2N
A.1.2 GRAPH EMBEDDING
After constructing distribution of probahilisticall weighted edges between points in X, UMAP and t-SNE initialize an embedding in Z corresponding to each data point, where probability distri- bution (Q) is computed between points was done with the distribution (P) in the input space. The objective of the UMAP and t-SNE is then to optimize that embedding to minimize the difference between P and Q.
Computing probabilities Z In embedding space, the pairwise probabilities are computed di- rectly without first computing local, one-directional probabilities.
LSNE is computed in In the t-SNE embedding space, the pairwise probability between two points 9 similar manner to pi but where the Gaussian distribution is replaced with the fatter-tailed
13

---

Under review as a conference paper at ICLR 2021
Student's t-distribution (with one degree of freedom), which is used to overcome the 'crowding problem' (van der Maaten & Hinton 2008 in translating volume differences high-dimensional
spaces to low-dimensional spaces:
1 + ||z - -2y11²) -1
ISNE 4i
(3)
Expi
I1+ Iiz= z1l1²)
UMAP's computation of the pairwise probability q.ja between points in the embedding space Z uses different family functions:
UMAP
a1+a 1 2z 23]120)
(4)
Where a and b are hyperparameters set based upon a desired minimum distance between points embedding space. Notably, the UMAP probability distribution in embedding space is not nor- malized, while the t-SNE distribution is normalized across the entire distribution of probabilities, meaning that the entire distribution of probabilities needs be calculated before each optimization step of t-SNE.
Cost function Finally the distribution of embeddings in Z is optimized to minimize the difference betwee Q and P.
In t-SNE, -ulbbeck-Leibler divergence between the two probability distributions is used, and gra- dient descent in t-SNE computed over the embeddings CLSNE Eit Pij log Piş - Pij log qij-
In UMAP, the cost function is cross-entropy, also optimized using gradient descent CUMAP Eig Pij log
Pig
+ (1 Pij) log 1-b4))
Attraction and repulsion Minimizing the cost function over every possible pair of points in the dataset would be computationally expensive. UMAP and more recent varients of t-SNE both use shortcuts to bypass much of that computation. In UMAP, those shortcuts are directly advantageous to batch-wise training neural network.
The primary intuition behind these shortcuts is that the cost function of both t-SNE and UMAP can both be broken out into a mixture of attractive forces between locally connected embeddings and repulsive forces between non-locally connected embeddings.
Attractive forces Both UMAP and t-SNE utilize a similar strategy in minimizing the compu- tational cost over attractive forces: they rely on an approximate nearest neighbors grapt The intuition for this approach is that elements that are further apart in data space have very small edge probabilities, which can be treated as effectively zero. Thus, edge probabilities and attractive forces only need to be computed over the nearest neighbors, non-nearest neighbors can be treated as hav- ing an dige-probability of zero. Because nearest-neighbor graphs are themselves computationally expensive, approximate nearest neighbors (e.g. Dong et al. 2011) produce effectively similar results.
Repulsive forces
Because most data points are not locally connected, we do not need to waste computation on most pairs of embeddings.
UMAP takes a shortcut motivated by the language model word2vec (Mikolov et al. 2013) and performs negative sampling over embeddings. Each training step iterates positive, locally- connected, edges and randomly samples edges from the remainder of the dataset treating their edge probabilities as zero compute cross-entropy. Because most data points are not locally connected and have a very low edge probability, these negative samples are, on average, correct, allowing UMAP to sample only sparsely over edges in the dataset.
'UMP requires substantially fewer nearest neighbors than t-SNE, which generally requires 3 times the perplexity hyperparameter (defaulted 30 here), whereas UMAP computes only 15 neighbors by default, which is computationally less costly.
14

---

Under review as a conference paper at ICLR 2021
In t-SNE, repulsion is derived from the normalization of Q. A few methods for minimizing the amount of computation needed for repulsion have been developed. The first, is the Barnes-Hor tree algorithm Van Der Maaten (2014), which bins the embedding space into cells and where repulsive forces can be computed over cells rather than individual data points within those cells. Similarly, the more recent interpolation- based I-SNE (FIt-SNE; Linderman et al. 2017 2019) divides the em- bedding space up into grid and computes repulsive forces over the grid, rather than the full set of embeddings.
A.2 DATASETS
We performed experiments over several different datasets varying in complexity. The Cassin's 's vireo song dataset (Hedley) 2016b a. Sainburg et al. 2019) consists of spectrograms of 20 of the most frequently sung elements elements of Cassin's vireo song (zero padded to 32 frequency by 31 time bins) produced by several individuals recorded in the Sierra Nevada mountains of California. De- spite heing recorded in the wild, the Cassin's 's vireo song is relatively low noise and vocal elements are highly stereotyped. MNIST is benehmark handwritten digits dataset in 28x28 pixels (Kluger- Lab 2010). Fashion MNIST (FMNIST) is a dataset of fashion items in the same format as Fashion MNIST designed to be a more difficult classification problem than MNIST (Xiao al. 2017). CI- FARIO is a natural image dataset (32x32x3 pixels) with 10 classes (Krizheskk)) 2009). CIFARIO classes are much less structured than the other datasets used. For example, unlike FMNIST, subjects of images are not uniformly centered in the image and can have different background conditions that make neighborhood in pixel-space less likely between members of the same class. The single-cell retina transcriptome dataset consists of PCA projections (50D) of single-cell RNA transcriptome data from mouse retina (Macosko et al., 2015 [Poliar
[Poliaa et al. 2019). The CelebAMask- -Q dataset consists of cropped and aligned photographs of celebrity faces with 40 facial feature annotations (Lee et al. 2020; Liu et al. 2015) and label masks corresponding to face-landmarks. We removed the background of each image to make the task of learning a structured embedding simpler for the neural network. A further description of each dataset is given in Table and a 2D projection of each
dataset using each embedding algorithm is given in Figure 12
Dataset
Dim.
# Siiin//alid/Tess
Citation
Moons
2
IK/N/NA
Pedregosa et al.
2011
Bison
3
50K/NA/NA
Duhamme 22019
Cassin's vireo song
32x31
24.98K/IK/IL
Hedley (2016b
MNIST
28x28
50K/10K/10K
Klugerl.oo Lab 2010
Fashion MNIST
28x28
50K/10K/10K
X1ao 201
CIFAR10
32x32x3
40K/10K/10K
Krizhevsky (2009)
Mouse retina transcriptomes
50
30.33K/4.88 8114//8KKK/KKK
Macosko al. (2015
CelebA-HQ (128)
128x 128x3
28K/IK/IK
Lee et al. 2020 Liu et al. (0155)
Table 1: Datasets used across all of the analyses and visualizations in the paper. NA refers split that was not used in the paper (e.g. no validation or testing set was used for the bison visualization in Fig B
A.3 EMBEDDING ALGORITHMS
Neural network architectures for the Parametric networks differ between datasets. MNIST, FMNIST, and CIFAR1O use convolutional neural networks. The Cassin's 's vireo dataset uses an LSTM encoder (and decoder for the autoencoder). The Retina dataset uses a 3-layer MLP with 100-neurons per layer. Parametric t-SNE and UMAP used the same neural network architectures and optimizer. For UMAP, the distance metric used for Cassin's 's vireo song is dynamic-time warping (DTW) metric. The UMAP Autoencoder (UMAP AE) uses the same architecture of the Parametric UMAP and SNE implementations, combined with corresponding decoder network. The encoder network and decoder network are jointly trained on a reconstruction and UMAP loss function. We additionally trained decoder for the Parametric UMAP network, in which the encoder is trained only the UMAP loss, and is not jointly trained on a reconstruction loss. For both t-SNE and Parametric t-SNE perplexity was left at its default value of 30 across datasets. We also left the degrees of
15

---

Under review as a conference paper at ICLR 2021
freedom for parametric t-SNE - 1, where d is the number of dimensions in the latent projection. Parametric embeddings (UMAP, t-SNE) are initialized with the same random neural network weights. Non-parametric algorithms their corresponding default initializations.
A.4
EMBEDDING METRICS
A.4.1 TRUSTWORTIINESS
Trustworthiness (Venna & Kaski 2006) is a measure of how much of the local structure of a dataset preserved set of embeddings. Trustworthiness is quantified by comparing each datapoint's nearest neighbors in the original space, to its nearest neighbors in the embedding space:
2 T(k) = 1 nk(2n 3k 1)
Σ JENT
max(O, (r(i.)) k))
where l is the number of nearest neighbors trustworthiness being computed over, and for each sample N²th the nearest & neighbors in the original space, and each sample j is i's ri,,j)m nearest neighbor. We compared the Trustworthness using Scikit-learn's 's default k value of 5, Trust- worthiness is scaled between O and 1, with I being more trustworthy.
A.4.2 KNN CLASSIFIER
Related to trustworthiness, a k-Nearest Neighbor's (KNN) classifier is supervised algorithm that classifies each unlabeled point based on its k-nearest labeled datapoints. We applied a KNN classifier with k=1 (Fig and k=5 (Fig 15 to each dataset.
A.4.3 SILHOUETTE SCORE
As opposed to Trustworthiness, which measures the preservation of local structure in a projection, the silhouette score [Rousseeuw] 1987) measures how "clustered" set of embeddings are, given ground truth labels. Silhouette score is computed as the mean silhouette coefficient across em- beddings, where the silhouette coefficient is the distance between each embedding and all other embeddings in the same class, minus the distance to the nearest point in separate class. Silhouette score is scaled between and with 1 being more clustered.
A.4.4 CLUSTERING
To compare clustering directly across embeddings, performed k-Means clustering each latent projection and compared each embeddings clustering on the basis of the normalized mutual used to
information (NMI) between clustering schemes. For each latent projection, k-Means cluster the latent projection with (the number of clusters) varied between - 1 times the true number of categories in the dataset. The clustering was repeated five times per k: The best clustering was then picked on the basis of the silhouette score between the clusters and the projections (i.e. without reference to ground truth).
A.5 RELATED WORK
Beyond Parametric t-SNE and Parametric UMAP, number of recent parametric dimensionality reduction algorithms utilizing pruuture-prreeevviee constraints exist which were not compared here. In this work, we chose not to compare these algorithms to Parametric UMAP because the focus of Parametric UMAP on downstream applications, and not embedding quality, which we confirm in Section 2.1 is similar to provide clarity on the current state of parametric topologically motivated and structure preserving dimensionality reduction algorithms.
non-parametric UMAP. This work is relevant to and is mentioned here
Moor et al., (topological autoencoders; 2020) and Hoffer et al. (Connectivity- connectivity-Opiimizee Repre- sentation Learning; 2019) apply additional topological structure- preeeeriigge loss using persistent homology over bati-batches to the latent space of an autoencoder. Jia et al., (Laplacian Au- toencoders; 2015) similarly defines autoencoder with local structure preserving regularization.
16

---

Under review as a conference paper at ICLR 2021
Mishne al., (Diffusion Nets; 2019) define autoencoder extension based upon diffusion maps that constrains the latent space of the autoencoder. Ding et al., (scvis: 2018) and Graving and Couzin (VAE-SNE; 2020) describe VAE-derived dimensionality reduction algorithms based upon the ELBO objective. Duque et (geometry-regulaaivee autoencoders; 2020) regularize an autoencoder with the PHATE (Potential of Heat-diffusion for Affinit--based Trajectory Embedding) embedding al- gorithm (Moon et al. 2019). Szubert et al (ivis; 2019) and Robinson (Differential Embedding Networks; 2020) both make use of Siamese neural network architectures with structure-preservinng loss functions to learn embeddings. Pai et al., (DIMAL; 2019) similarly uses Siamese networks constrained preserve geodesic distances for dimensionality reduction.
B FIGURES
Data Samples UMP Paramotiic UMAP
UMAP AE
Parametric
VA
A
N
Fats
Anacrincels
Care:
Malarn dla
Caggln
Figure 12: Comparison of projections from multiple datasets using UMAP, UMAP in Tensorflow, Parametric UMAP, Parametric UMAP with an Autoencoder loss, Parametric t-SNE, t-SNE, a VAE, an AE, and PCA. (a) Moons. (B) 3D buffalo. (c) MNIST (d) Cassin's vireo song segments (c) Mouse retina single cell transcriptomes. (f) Fashion MNIST (g) CIFARIO. The Cassin's 's vireo dataset uses dynamic time warping loss and LSTM network for the encoder and decoder for the neural networks. The image datasets use a convnet for the encoder and decoder for the neural networks. The bison examples t-SNE parplexity of 500 and 150 nearest neighbors in UMAP to capture more global structure.
17

---

Under review as a conference paper at ICLR 2021
Dimensions
64 Dimensions
Trusworthiness
USER (URAN borns
UMAR-
USAP At
Use
Trustworthiness scores for five datasets using 2- and 64-dimensional projections using each projection method. 64-dimensional t-SNE is not shown due to limitations in high-dimensional projections with t-SNE. Trustworthiness is computed over 10,000 samples of the training dataset. UMAP (TF) stands for the Tensorflow implementation of non- non-parametree UMAP.
Figure 13:
T5SE
Accuracy
LMAP
MNIIT
FMNIST
Retia
Cassin's
CIFARIO
MIMST
FMNIST
Retina
Cassin's
CIFARID
PCA
Figure 14: Generalization errors of KNN classifiers (k=1) on latent projections.
TSSE
Accuracy
MNNIST
FMNIST
Retina
Cassin's
CIFARIO
MI
FMNIST
Retina
Cassin's
CIFARID
PCA
Figure 15: Generalization errors of KNN classifiers (k=5) on latent projections.
Dimensions
Dimensions
Sithouete Score
7532
IL
(77)
US007
MNIST
FNNIST
Retina
Cassin's
CIFAR10
MNIST
IMNIST
Retina
Cassin's
CIFAR10
Figure 16: Silhouette for five datasets using 2- and 64-dimensional projections using each projection method. 64-dimensional t-SNE is not shown due to limitations high-dimensional pro- jections with t-SNE.
Dimensions
TANS
N811
USAP
At
MMST
FMNIST
Retina
Cassil's
OFAR10
MNIST
FMIIST
Retina
Cassin's
CIPAR1O
Clustering results. Comparisons based upon the Normalized Mutual Information (NMI) between labels and clusters. Rach dataset shows the NMI for the best clustering chosen on the basis of its silhouette score.
Figure 17:
Dimrensions
64 Dimerstons
Ter
Zer
PCA
Figure 18: Reconstruction speed. Reconstructions are performed on the same machine as Fig Reconstructions are not shown for the retina dataset with PCA with 64D latent space because the dataset only 50 dimensions. Because all neural network architectures are held constant, speeds remain equal across each Parametric UMAP and t-SNE inplementation. Values show the median time over 10 runs.
18

---

Under review as a conference paper at ICLR 2021
4/class
16/class
64/class
256/class
1024/class
Full
Ground Truth
MNIST
Predictions
Ground Truth
Fashion MNIST
Predictions
Predictions Ground Truth
20
CIFAR10
Figure 19: Non-parametric UMAP projections of activations in the last layer of trained classifier for MNIST, FMNIST, and CIFARIO. For each dataset, the top row shows the ground truth labels on above, and the model's 's predictions below, in light colormap. On top of each projection, the labeled datapoints used for training are shown in darker colormap.
19

---

Under review as a conference paper at ICLR 2021
B.1
RESULTS TABLES
Dataset
Dim.
t-SNE
P. t-N-
UMAP
P. UMAP
UMAP/AE
AE
VAE
PCA
Cassin's
0.9949
0.9867
0.9758
0.9756
0.9777
0.9488
0.8976
0.8380
64
0.9990
0.9831
0.9840
0.9907
0.9981
0.9949
0.9999
CIFARIO
0.9216
0.7773
0.8310
0.8187
0.8273
0.8564
0.8510
0.8202
64
0,9971
0.9209
0.9140
0.9199
0.9992
0.9913
0.9996
FMNIST
2
0.9906
0.9827
0.9777
0.9733
0.9842
0.9803
0,9751
0.9126
64
0.9991
0.9897
0.9894
0.9913
0.9991
0.9960
0.9995
Retina
0.9702
0.9463
0.9494
0.9435
0.9173
0.8063
0.7533
0.7445
64
0.9918
0.9708
0,9628
0.9542
0.9921
0,9342
1.0000
MNIST
0.9874
0.9655
0.9601
0.9573
0.9675
0.9663
0.9513
0.7434
64
0.9997
0.9895
0.9880
0.9905
0.9997
0.9994
0.9999
Table 2: Trustworthiness score for each method from Fig
Dataset
Dim.
t-SNE
P. t-SNE
UMAP
P. UMAP
UMAP/AE
AE
VAE
PCA
Cassin's
0.9880
0,9860
0.9860
0.9910
0.9890
0.8740
0.7300
0.6260
64
0.9950
0.9850
0.9880
0.9940
0.9950
0.9800
0.9950
CIFARIO
0.2457
0.1675
0.1689
0.1512
0.1592
0.1696
0.1665
0.1436
64
0.3426
0.2375
0.2139
0.2223
0.3790
0.3949
0.3829
FMNIST
0.7825
0.6834
0.7144
0.6941
0.7083
0.6816
0.6646
0.4467
64
0,8300
0.7682
0.7431
0.7772
0.8671
0.8747
0.8398
Retina
0.9717
0,9661
0.9665
0.9643
0,8581
0.9429
0.8545
0.8085
64
0.9772
0.9721
0.9683
0.9576
0.9750
0.9670
0.9759
MNIST
0.9411
0.9118
0.9317
0,9402
0.9403
0.7647
0.7241
0.3765
64
0.9697
0.9449
0.9518
0.9481
0.9748
0.9785
0.9707
Table 3: KNN = 1) scores for each method from Fig 14
Dataset
Dim.
t-SNE
P. t-SNE
UMAP
P. UMAP
UMAP/AE
AE
VAE
PCA
Cassin's
0.9910
0.9930
0.9890
0.9950
0.9930
0.9090
0.7740
0.6910
64
0.9950
0.9860
0.9910
0.9970
0.9930
0.9880
0.9920
CIFAR10
0.2608
0.2017
0.1936
0.1722
0.1833
0.2007
0.1941
0.1503
64
0.3556
0.2694
0.2519
0.2477
0.3728
0.3777
0.3769
FMNIST
2
0.8039
0.7361
0.7608
0.7407
0.7561
0.7339
0.7161
0.5055
64
0.8479
0.8059
0.7878
0.8028
0.8756
0.8830
0.8568
Retina
2
0.9795
0.9766
0.9792
0.9761
0.8933
0.9647
0.8795
0.8429
64
0,9813
0.9801
0.9748
0,9661
0.9817
0.9770
0.9806
MNIST
0.9502
0.9378
0.9544
0.9614
0.9537
0.7926
0.7649
0.4201
64
0.9734
0.9538
0.9680
0.9654
0.9758
0.9791
0.9727
Table 4: KNN (k 5) scores for each method from Fig 14
20

---

Under review as a conference paper at ICLR 2021
Dataset
Dim.
t-SNE
P. t-SNE
UMAP
UMAP
UMAP/AE
AE
VAE
PCA
Cassin's
0.5431
0.7439
0.7749
0.8013
0.7714
0.1125
0.0853
0.0731
64
0.2299
0.8536
0.8173
0.8271
0.2411
0.1583
0.2914
CIFARIO
-0.1216
-0.2757
-0.1340
-0.1359
-0.1320
-0.1436
-0.1114
-0.1142
64
-0.0536
-0.1166
-0.1163
-0.1172
-0.0644
-0.0529
-0.0580
FMNIST
0.1251
0.2013
0.1936
0.2139
0.2060
0.0427
0.1064
-0.0331
64
0.0543
0.2195
0.2315
0.2305
0.0655
0.0376
0.0618
Retina
0.0151
0.2578
0.2800
0.4519
0.3973
0.4394
0.4449
0,4009
64
-0.0214
0.3522
0,4652
0,4662
0.4289
0.3873
0.4188
MNIST
0.3498
0.3710
0.5186
0.5559
0,4637
-0.0258
0.0627
0.0228
64
0.0488
0.5276
0.5571
0.5166
0.0653
0.0431
0.0569
Table 5: Silhouette score for each method from Fig
Dataset
Dim.
t-SNE
P. t-SNE
UMAP
P. UMAP
UMAP/AE
AE
VAE
PCA
Cassin's
0.9628
0.9605
0.9581
0.9686
0.9660
0.5984
0.7198
0.6029
64
0,9434
0.9596
0.9665
0.9662
0.7501
0.7923
0.9019
CIFARIO
0.0688
0.0383
0.0743
0.0719
0.0730
0.0258
0.0560
0.0605
64
0.0570
0.0733
0.0742
0.0746
0.0574
0.0905
0.0599
FMNIST
0.5408
0.6248
0.6603
0.6594
0.6602
0.4818
0.5319
0.4221
64
0.4680
0.6602
0.6618
0.6635
0.5541
0.5639
0.5244
Retina
0.5124
0.5912
0.5510
0.7112
0.5862
0.5695
0.5691
0.5522
64
0.4682
0.7763
0.6356
0.5793
0.5897
0.5908
0.6696
MNIST
0.7704
0.7446
0.8375
0.7824
0.8460
0.4093
0.5467
0.3234
64
0.4977
0.7747
0.7818
0.8701
0.4253
0.5617
0.5010
Table 6: Clustering for each method from Fig 17
Dataset
Dim.
UMAP
P. UMAP
UMAP/AE
AE
VAE
PCA
Cassin's
2
0.0085
0.0028
0.0028
0.0163
0.0125
0.0082
64
0.0034
0.0028
0.0011
0.0013
0.0008
CIFARIO
2
0.0528
0.0369
0.0364
0.0344
0.0217
0.0370
64
0.0300
0.0094
0.0080
0.0084
0.0084
FMNIST
2
0.0347
0.0266
0.0240
0.0244
0.0253
0.0461
64
0.0241
0.0092
0.0054
0.0058
0.0104
Retina 2
0.0003
0,0008
0.0005
0.0005
0.0006
0.0010
64
0.0005
0.0003
0.0001
0.0003
MNIST
2
0.0393
0.0374
0.0360
0.0369
0.0371
0.0557
64
0.0313
0.0027
0.0016
0.0024
0.0090
Table 7: Reconstruction error on held-out testing set for each method.
21

---

Under review as a conference paper at ICLR 2021
64
256
1024
full
MNIST
Baseline
0.814
0.979
0.990
0,994
0.996
Aug.
0.928
0.986
0.990
0.994
0.996
+ UMAP (Euelidean)
0.978
0.986
0.990
0.993
0.996
UMAP (learned)
0.832
0,979
0.990
0,994
0,996
+ Aug. + UMAP (learned)
0.955
0,991
0.994
0.996
0.996
Aug. UMAP (Euclidean)
0.978
0.992
0.993
0.995
0.997
FMNIST
Baseline
0.607
0.835
0.889
0,920
0,943
Aug.
0.692
0.860
0.901
0.932
0.949
+ UMAP (Euclidean)
0.714
0.841
0.885
0,916
0.947
UMAP (learned)
0.629
0,835
0.889
0,920
0.944
+ Aug. + UMAP (learned)
0.747
0.880
0.908
0.932
0.952
Aug. UMAP (Euclidean)
0.737
0.864
0.900
0.930
0.952
CIFARIO
Baseline
0.217
0,499
0.722
0.838
0.905
Aug.
0.281
0,599
0.766
0.867
0.933
UMAP (Euclidean)
0.190
0,450
0.674
0.829
0.913
UMAP (learned)
0.199
0.515
0.748
0.850
0.912
Aug. + UMAP (learned)
0.351
0.674
0.820
0.891
0.932
Aug. UMAP (Euclidean)
0.243
0.560
0.748
0.852
0.932
Cassin's vireo
Baseline
0.952
0.993
0.995
1.000
0.999
+ UMAP (Euclidean)
0.996
0.998
0.997
0.999
0.999
Retina
Baseline
0.888
0.970
0.979
0.978
0.983
UMAP (Euclidean)
0.929
0.973
0.973
0.977
0.979
Table 8: Classification accuracy across each dataset and method for different numbers of labeled training examples.
ACKNOWLEDGMENTS
Work supported by [witheld for anonymity]
22


Reviews:
[{"review_id": "oBKhXEKPsCz", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the manuscript, the authors introduce a parametric version of UMAP, replacing the original embedding optimization step with a deep learning solution detecting a parametric relationship between data and embedding. The novel approach compares favourably with the standard algorithm and, as a major contribution, defines a loss function that can be employed for other important applications such as constraining the latent distribution of autoencoders, and improving classifier accuracy for semi-supervised learning. \n\nThe paper is well written, complete and thoroughly detailed, both in the theoretical and the experimental section. The introduced material represents a significant advancement in the field, becoming a valuable resource for researchers in several areas. \n\nA couple of notes:\n- An application to one or more large real world dataset (e.g. single-cell sequencing, or weather radar data) would strengthen even more the authors’ claims and the paper’s impact, so I would suggest to include it, at least in the Appendix.\n- Fig.3 in the Appendix is extremely useful to graphically explain the algorithm to a broader audience - I understand the page length limit, but I would strongly recommend to fit it in the main text.\n- I would also suggest to include (maybe in the Appendix) a kind of “how-to” fully worked example to help researchers in optimising the use of novel algorithm in a data exploration pipeline\n- I would point out (within the limitation of the anonimity requirement) the availability of the code for the algorithm\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HmUhlp6sKIF", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update following discussion:**\n\nFollowing the revision by the authors and the discussion with them, I am updating my score from 3 (Clear rejection) to 4 (OK, but not good enough - rejection). This reflects in great part the revision the authors made to have the main paper (limited to 8 pages) be self contain and present their main results, while using the appendices for complementary and technical information.\n\nHowever, I still maintain the paper is not ready for publication in its current form. The extension of UMAP to implement the optimization via a neural network applied to input data rather than directly assigning coordinates is rather straightforward. The advantages it provides over UMAP in terms of natural inference on new data without the need for separate (more computationally intensive) out of sample extension method are a direct result of this neural network implementation, and they would be true not only for UMAP, but in fact for any method implemented in a \"parametric\" way via a neural network compared to nonparametric coordinate assignment. Similarly, allowing the addition of reconstruction or classification objectives in training is clearly a direct byproduct of this neural network implementation as well, and not unique to UMAP.\n\nTherefore, an important question has to be asked here for whether the UMAP loss is indeed a good choice for a loss term to impose on networks, for example, to enable visualization or improve various tasks. The authors already look into this to some extent by comparing to parametric tSNE as one alternative approach, but there are many others, as I mention in the initial review, relying on constructions from topological data analysis and manifold learning - most, if not all, of which rely on some graph construction on the data and then ensuring the coordinates provided by a hidden layer in the network match the relations encoded in the graph, similar to the proposed UMAP loss term. How are reconstruction and classification affected by using such other regularizations compared to the UMAP one? Is inference speed the same for these other approaches? How does training speed compare between them? One can clearly expect some tradeoff between such properties and the geometric information encoded by different methods (UMAP and tSNE emphasize clusters, while other methods may emphasize other patterns), but this should be discussed and demonstrated clearly rather than just ignoring the vast amount of related work on parametric approaches to capturing intrinsic geometry in data.\n\nNow, beyond the described lack of relevant comparisons for autoencoding and semi-supervised classification, even simply as a parametric implementation of UMAP (which would be a rather narrow scope, which is not very enticing as a motivation on its own),  I am not sure this work is sufficient to establish the presented approach. First, for the inference or embedding speed - this is essentially and out of sample extension task. As such, even if one insists on only comparing to UMAP-based methods, there are multiple OOS methods that can be used, such as Nystrom, geometric harmonics, etc. Some analysis of the tradeoff between extension quality and speed seems warranted here, but as I said previously - I think a comparison should also be provided to other parametric embedding methods beyond just OOS of UMAP (and tSNE for that matter). Second, as the authors clarified in discussion - their approach relies on the suitability of the UMAP loss to be incorporated directly in the network optimization, essentially comparing activations to the UMAP graph. However, an alternative approach presented in related work is to provide a loss term between activations and a UMAP embedding. This second approach is more general, since other embeddings can also be considered there, but also probably has some disadvantages (for example, the a priori fixed dimensionality, as the authors suggest). The differences between these two approaches should be addressed better in the manuscript, and importantly, since previous work exists already on the embedding loss approach, the authors should present a comparison establishing the benefits of the graph-based loss one, in addition to discussion regarding them. \n\nTo conclude, the idea behind this work seems reasonable, albeit rather straightforward since it's a reimplementation of the UMAP optimization. However, as it currently stands, I find it is not mature enough for publication and would need nonnegligible amount of work to properly position the contribution provided by this work compared to previous and related ones. I would like to encourage the authors to invest the time in adding such comparisons and clarifying not only how they are different from other methods, but also how they are better, and why choose UMAP to begin with as the basis for their proposed loss terms (compared to various other approaches - not just tSNE).\n\n---\n\n**Initial review:**\nBefore getting into the details of this work, I note that in my opinion it should have been desk rejected for violating the page limit base on the way it is written. The main 8 pages of the paper are far from being self contained, and regularly reference materials from the appendix as integral parts that are crucial for the presentation and understanding here. These include not only methodological illustrations, but also all results establishing the method. In fact, the main paper here does not show ANY result - it only describes the setting for getting them. ALL the figures and tables showing results appear solely in the appendix. If we are to ignore the appendices and only judge the paper based on these main eight pages, then there is no support, no results, and very little in the way of presenting the method here. If, on the other hand, we include the result figures as integral parts of the main paper (as they should be), then it clearly has significantly more than eight pages. Considering most papers submitted to this conference do  try to provide a coherent and relatively self-contained presentation of their work within the page limit, according to the guidelines of the conference, while only leaving technical and supporting details to the appendix, I believe it would be inappropriate to consider this work as meeting the conference page limit.\n\nAs for the work itself, this paper presents a rather naïve attempt to combine together the UMAP visualization with deep learning. It essentially proposes to consider the coordinates optimized by UMAP as resulting from a neural network applied to input data. Then, instead of adjusting directly these coordinates via the UMAP optimization, the method here continues to backpropagate the coordinate optimization through the network to provide a parametric model, via a feed forward neural network, that embeds the data in low dimensions while preserving the local neighborhood structure in the same sense that UMAP, tSNE and LargeVis do with their nonparametric approach. This neural network formulation can also naturally be extended to consider other loss terms, such as reconstruction loss of autoencoders or any predictive loss (classification, regression, etc.) enabling supervised visualization. \n\nFrom a methodological perspective, this is a pretty straight-forward extension of the UMAP optimization, and does not indicate a clear advantage over it for the main task of unsupervised visualization or dimensionality reduction, neither in embedding quality or scalability. The authors show some interesting results (albeit only in the appendix and not in the main paper) on supervised visualization and out of sample extension speed, but these are not compared to relevant baselines that directly aim to address these tasks. Moreover, there is significant related work that is either ignored by the authors, or just mentioned in passing in the appendix without providing proper discussion and comparison with the proposed method. For example, in A.4, the authors mention topological autoencoders, connectivity-optimized representation learning, SCVIS, VAE-SNE, geometry regularized autoencoders, IVIS, and Differential Embedding Networks, but they do not compare their work to any of these, even though such comparison seems highly relevant here. More work that is completely ignored here includes, for example, Diffusion Nets (Mishne et al., 2015), Laplacian Autoencoders (Jia et al., 2015), DIMAL (Pai et al., 2019). Finally, briefly looking at Duque et al. (2020) cited here, while the main method there uses PHATE coordinates to regularize autoencoders, it seem they have also proposed the incorporation of UMAP loss terms in autoencoders, albeit only mentioned as somewhat of a sidenote together with tSNE regularization in their appendix. A discussion about the difference between these two approaches should be added to the main paper here, and it seems some comparison between them should also be presented to establish the advantages of the proposed approach here. Hence, even without the page limit argument, it does not seem the work presented here reaches the ICLR acceptance threshold without major revision to its presentation, discussion, and results. I must therefore recommend its rejection at this stage.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DjM0EYniFXZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a parametric version of UMAP by replacing sampling embeddings in the optimization of UMAP with directly learning weights of a neural network. The paper is very well and clearly written, but I have several significant concerns:\n\n1. I don't see significant methodological novelty. Replacing embeddings with neural networks learning seems to be quite basic and straightforward. It is certainly a cherry on top of original UMAP, but I am not sure it could be counted as a separate method. The simplicity of methodology could be neglected, if the authors demonstrated significant improvement in their experiments, especially on downstream tasks.\n2. A large part of the experiments is devoted to the comparison with tSNE, however it is not very clear why there is a lack of comparison with other parametric methods, such as Topological Autoencoders. Also not very clear why the authors mention these very relevant methods only in Appendix and not in the introduction in the beginning. \n3. The performance of parametric UMAP achieves similar results to non-parametric UMAP, which is certainly nice, but also quite expected. Therefore, I would consider applications of parametric UMAP to other downstream tasks as a more significant and interesting contribution. However, experiments on this part are not convincing at all (especially on CIFAR10 dataset). Would be interesting to see the performance on some other datasets. Also, it would be very interesting to see confidence intervals for Figures 15, 16, 18.\n4. In terms of speed I also don't see an improvement compared to non-parametric UMAP (TF). I see clear improvement compared to UMAP-learn version, but this as far as I understood due to a different implementation of the original UMAP and not in particular novelty of this paper. \n\nAfter authors' response to revisions, I reconsidered my evaluation and updated the score.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3mEHJgWiScp", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Response to authors:\n\nThe authors have largely responded well to my original concerns. However, after reading through the discussions with other reviewers, I agree with reviewer 2 that more work is required to make this publishable. In particular, this should include comparisons to the other methods suggested and justification of the use of the UMAP loss function. Given this, I have downgraded my score accordingly.\n\nOriginal Review:\nThis paper presents a parametric approach for UMAP, a dimensionality reduction method. This area is of interest to the community as dimensionality reduction can be useful in a lot of different tasks such as visualization, semisupervised learning, etc. If my comments below can be addressed, I would be willing to increase my score.\n\nPros\nThe parametric version presented here appears to work well in the experiments given. The incorporation of the UMAP loss directly in a neural network as a regularization is also interesting.\n\nCons\nSome of the results don't appear to have a corresponding figure or table, e.g. \"Reconstruction accuracy\" in section 3.3. These should be included.\n\nUMAP tends to inherit some of the weaknesses of t-SNE as it tends to overemphasize local structure at the expense of global structure. In particular, it's been shown [R1] that UMAP and t-SNE are basically equivalent when using the same initialization. Could similar results be obtained by using the same initialization as UMAP instead of including the regularization term? \n\nUMAP is traditionally used for visualization. Some of the applications presented (e.g. SSL) require/would benefit from higher dimensions than 2 or 3. t-SNE is known to be considerably slower for higher dimensions. Does UMAP inherit this problem? If so, that should be mentioned as a potential drawback.\n\nAll of the figures are given in the appendix. While this allows for more results, I think it would be a better paper if some of the figures were included in the main paper and some results were moved to the appendix.\n\nThe authors should verify that their references are as up to date as possible. For example, the PHATE paper should be updated to the Nature Biotechnology version (not bioRxiv).\n\n[R1] Kobak and Linderman, https://www.biorxiv.org/content/10.1101/2019.12.19.877522v1\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: We thank the authors (and reviewers) for engaging in a detailed and constructive discussion, and providing a revised version of the paper after the initial round of reviews.

Regarding quality, the work is technically correct and the amount of experiments significant. However, as highlighted by reviewers 2 and 3, some important questions remain unanswered, in particular 1) more empirical evidence to support the claim that the UMAP loss is a relevant for neural networks, and 2) more comparison with existing approaches (beyond t-SNE).

Regarding clarity, the paper is overall clear and pleasant to read. However, after the revision round, all details about the proposed methods have been moved to the annex. While the initial version was criticized for the opposite reason (all experiments were in a annex), the balance may not be found yet; e.g., the equation for the UMAP loss, which is at the core of the paper, would certainly find its place in the main part of the manuscript for an ICLR paper.

The originality is the weakest aspect of the paper (besides the lack of comparison with related work). As mentioned by several reviewers, plugging the UMAP loss to a differentiable model is nowadays an idea that lacks originality. What would be important to justify that such a "straightforward" idea makes it to ICLR would be to demonstrate convincingly that it outperforms existing alternative approaches.

Finally, regarding the significance of the work, it is limited by the lack of thorough comparison with existing method. On the other hand, if the method is implemented in a fast and easy-to-use package, it may find its public as illustrated by the positive evaluation of Reviewer 1 from a potential user point of view.
Decision: Reject