Factoring out Prior Knowledge from Low-Dimensional Embeddings
Authors: Edith Heiter
Jonas Fischer
Jilles Vreeken
Venue:  (2021)
Keywords: embedding
visualization
prior
tsne
umap

Abstract: Low-dimensional embedding techniques such as tSNE and UMAP allow visualizing high-dimensional data and therewith facilitate the discovery of interesting structure. Although they are widely used, they visualize data as is, rather than in light of the background knowledge we have about the data. What we already know, however, strongly determines what is novel and hence interesting. In this paper we propose two methods for factoring out prior knowledge in the form of distance matrices from low-dimensional embeddings. To factor out prior knowledge from tSNE embeddings, we propose JEDI that adapts the tSNE objective in a principled way using Jensen-Shannon divergence. To factor out prior knowledge from any downstream embedding approach, we propose CONFETTI , in which we directly operate on the input distance matrices. Extensive experiments on both synthetic and real world data show that both methods work well, providing embeddings that exhibit meaningful structure that would otherwise remain hidden.

Under review as a conference paper at ICLR 2021
FACTORING OUT PRIOR KNOWLEDGE FROM LOW- DIMENSIONAL EMBEDDINGS
Anonymous authors Paper under double-blind review
ABSTRACT
Low-dimensional embedding techniques such as ISNE and UMAP allow visual- izing nigh-dimensional data and therewith facilitate the discovery of interesting structure. Although they are widely used, they visualize data as is, rather than in light of the background knowledge we have about the data. What we already know, however, strongly determines what is novel and hence interesting. In this paper we propose two methods for factoring out prior knowledge in the form of distance matrices from low-dimensional embeddings. To factor out prior knowl- edge from iSNE embeddings, we propose JEDI that adapts the tSNE objective principled way using Jensen-Shannon divergence. To factor out prior knowledge from any downstream embedding approach, we propose CONFETTI, in which we directly operate the input distance matrices. Extensive experiments both synthetic and real world data show that both methods work well, providing em- beddings that exhibit meaningful structure that would otherwise remain hidden.
INTRODUCTION
Embedding high dimensional data into low dimensional spaces, such with tSNE (van der Maaten & Hinton, 2008) or UMAP (McInnes et al., 2018), allow us to visually inspect and discover mean- ingful structure from the data that would otherwise be difficult or impossible to see. These methods are popular as they are useful, but, at the same time limited in that they are one-shot only: they embed the data as is, and that is that. If the resulting embedding reveals novel knowledge, all is well, but, what if the structure that dominates it is something we already know, something we are no longer interested in, or, if we want to discover whether the data has meaningful structure other than what the first result revealed? In word embeddings, for example, we may already know that certain words are synonyms, while in single cell sequencing we may want to discover structure other than known cell types, factor out family relationships. The question at hand is therefore, how obtain -iwddmennsioal embeddings that reveal structure beyond what we already know, i.e. how to factor out prior knowledge from -iwddiinniioall embeddings?
For conditional embeddings, research so far mostly focused on emphasizing rather than factoring prior knowledge (De Ridder al., 2003; Hanhijärvi et al., 2009; Barshan et al., 2011), with conditional tSNE as notable exception, which, however, can only factor out label information (Kang al., 2019). Here, we propose two techniques for factoring out a more general form of prior knowledge from low-dimensional embeddings of arbitrary data types. In particular, we consider background knowledge in the form of pairwise distances between samples. This formulation allows us to cover a plethora of practical instances including labels, clustering structure, family trees, user- defined distances, but also, and especially important for unstructured data, kernel matrices.
To factor out prior knowledge from (SNE embeddings, we propose JEDI, in which we adapt the 1SNE objective in a principled way using Shasen-Shanoon divergence. It has an intuitively appealing information theoretic interpretation, and maintains all the strengths and weaknesses of tSNE. One of these is runtime, which is why UMAP is particularly popular in bioinformatics. To factor out prior knowledge from embedding approaches in general, including UMAP, we hence propose CONFETTI, which directly operates on the input data. An extensive set of experiments shows that both methods work well in practice and provide embeddings that reveal meaningful structure beyond provided background knowledge, such as organizing flower images according to shape rather than color, or organizing single cell gene expression data beyond cell type, revealing batch effects and tissue type.

---

Under review as a conference paper at ICLR 2021
2 RELATED WORK
Embedding high dimensional data into a low dimensional spaces a research topic of perennial interest that includes classic methods such as principal component analysis Pearson (1901), multi- dimensional scaling (Torgerson, 1952), self organizing maps (Kohonen, 1982), and isomap (Tenen- baum al., 2000), all of which focus on keeping large distances intact. This is inadequate for data that lies on a manifold that resembles a Euclidean space only locally, which is the case for high dimensional data (Silva & Tenenbaum, 2003) and for which we hence need methods such as locally linear embeddlng (LLE) (Roweis & Saul, 2000) and stochastic neighbor embedding (SNE) (Hinton & Roweis, 2003) that focus on keeping local distances intact. The current state of the art methods are t-distributed SNE (ISNE) by van der Maaten & Hinton (2008) and Uniform Manifold Approx- imation (UMAP) by McInnes et al. (2018). Both are by now staple methods for data processing, e.g. in biology (Becht et al., 2019; Kobak & Berens, 2019) and NLP (Coenen et al., 2019). As they often yield highly similar embeddings (Kobak & Linderman, 2019) it is a matter of taste which one to use. While tSNE has an intuitive interpretation, despite recent optimizations (van der Maaten, 2014: Linderman et al., 2019) compared to UMAP it suffers from very long runtimes.
Whereas the above consider only the data as is, there also exist proposals that additionally take user input and/or domain knowledge into account. For specific applications to gene expression, it was proposed to optimize projections of gene expression to model similarities in corresponding gene ontology annotations (Peltonen et al., 2010). More recently, attention has been brought to removing unwanted variation (RUV) from data using negative controls in particular in the light of gene expression, assuming that the expression can be modeled as linear function of factors of variation and a normally distributed variable "Tagnon-Bartsse & Speed, 2012). This approach has been successfully applied to different tasks and domains of gene expression (Risso et al., 2014; Buettner al., 2015; Gerstner et al., 2016; Hung. 2019). Here, we are interested to develop a domain independent method to obtain low-dimensional embeddings while factoring out prior knowledge. For that, we neither want to assume a functional relationship between prior and input data, nor do we want to assume particular distribution of the input, but keep the original data manifold intact. Furthermore, we do not want to rely on negative samples that have to be known and present in the data to be able to factor out the prior.
The general, domain independent methods supervised LLE (De Ridder et al., 2003), guided LLE (Alipanahi & Ghodsi, 201 1), and supervised PCA (Barshan et al., 201 1) all aim emphasize rather than factor out the structure given as prior knowledge. Like us, Kang et al. (2016; 2019); Puolamäki al. (2018) factor out background knowledge, but are much more limited in the type of prior knowl- edge. In particular, Puolamäki et al. (2018) requires users to specify clusters in the embedded space, Kang al. (2016) requires background knowledge for which a maximum entropy distribution can be obtained, while Kang et al. (2019) extend tSNE and propose conditional tSNE (ctSNE) which accepts prior knowledge in the form of class labels. In contract, we consider prior knowledge in the form of arbitrary distance metrics, which can capture relative relationships which appears naturally in real world data, such difference in age, geographic location, or level of gene expression. We pro- pose both information theoretic extension to (SNE, and an umbedding-algoridee independent approach to factor out prior knowledge.
THEORY
We present two approaches, with distinct properties, that both solve the problem of embedding high dimensional data while factoring out prior knowledge. We start with an informal definition of the problem, after which we introduce vanilla ISNE. We then present our first solution, JEDI, which extends the (SNE objective to incorporate prior information. We then present CONFETTI, which uses an elegant yet powerful idea that allows us to directly factor out prior knowledge from the distance matrix of the high dimensional data, which allows CONFETTI be used in combination with any embedding algorithm that operates on distance matrices.
3.1 THE PROBLEM - INFORMALLY
Given a set of n samples X from high dimensional space, e.g. IR our goal is find a low dimensional representation Y in IR. that captures the local structure in X while factoring out prior
2

---

Under review as a conference paper at ICLR 2021
knowledge Z about the samples. Here, consider both high dimensional data X and prior Z to be given as distance matrices, thus allowing for data from typical spaces such as Euclidean, but also images, up unstructured data such texts graphs. for which distance matrices can be specified using a kemel. When embedding X, our goal is to embed the samples such that the pairwise low dimensional distances D resemble high dimensional distances D locally, but distinet the prior distances D Informally, we can state this goal as finding an embedding Y subject to
D ~ D & D
We could formally define this as a multi-objective problem composed of a minimization over the difference between D and DY and maximization of the difference between D and D Be- sides how to measure these differences, there are two problems that render classic multi-objective optimization impractical. First, the two functions are highly imbalanced, with the minimization ob- jective obtaining its optimum at and the maximization at +oo, hence we need to constrain the optimization. Second, we want to put emphasis on correctly reconstructing local structure, as this yields superior visualizations (van der Maaten & Hinton, 2008; McInnes et al., 2018).
3.2
THE PROBLEM INFORMATION THEORETICALLY
The t-distributed Stochastic Neighbor Embedding (tSNE) (van der Maaten & Hinton, 2008) is a state-of-the-art approach for embedding data into low dimensional spaces that preserves the local structure of the high dimensional data. In particular, it models the local neighborhood of a point by casting the pairwise distances into similarity distributions that express for each point the likeli- hood of observing point, j as neighbor, given by Pyls- For the high dimensional distances D this likelihood is approximated by Gaussian kernel centered at point
e /2
Pjjs Exys exp(-
To account for varying densities of points in the space, the variance o; is dependent on where the kernel is centered. Given the user specified parameter perplexity, which can be thought of as an estimate of the neighborhood size, we can solve perplezity = 2f()) for o; for each point i, where (P.)) E, Pyls Pyj; is the entropy. By symmetrizing the conditional probabilities, the joint probability of a pair of points is given as Piy PactPos which yields the desired local similarity representation of high dimensional points.
2n
The low dimensional point similarities qıj are represented by a I-distribution instead of Gaussian, which solves the crowding problem' due its heavy tails. We thus get low dimensional similarities
(1+ (D)
4g = ExS((1 (D
The goal of tSNE model pairs of points exhibiting high similarity in the high dimensional space to have a high similarity in the low dimensional space. This is achieved by minimizing the Kullback-Leibler Divergence (KI), given by DKI.(P Q) Σity Pij log Pit, for the pairwise probabilities. This information theoretic measure yields the number of excess bits needed if we would encode P using code optimal for encoding Q and thus models how well Q approximates P. Minimizing the KL divergence with respect to Y, we get a non-convex objective that we can practically optimize using gradient descent. Using similar notion of neighborhood distributions, we can now define a new objective that instantiates our objective using tools from information theory.
qu
3.2.1 FACTORING OUT PRIOR INFORMATION WITH JEDI
To incorporate prior information into the tSNE objective, we first need to model the neighborhood distribution p' of the prior. Similar to the high dimensional data, we hence put emphasis on samples that are close according to the prior, defined as
Gaussian kernel by which
exp( cxp(-(D2))2 //20)
P/R Ener exp( (Dp(-(DD)2/2003
'The crowding problem is the phenomenon of assembling all points in the center of the map, due to the accumulation of many small attractive forces moderate distances are accurately modelled.
3

---

Under review as a conference paper at ICLR 2021
with o; perplexity parameter which describes the desired neighborhood size in the prior space.
We thus search for similarity distribution Q of points Y in the low dimensional space, that is similar to the high dimensional similarities P but different from the prior similarities P'. Similar to ISNE, the first term of our objective corresponds to minimizing the KL divergence, thus a natural extension would be to add second term that rewards maximizing the KL divergence between the distances of embedding Q and prior P'. This would be naive, however, as this second term would dominate the optimization because KL divergence is unbounded. Furthermore, would allow us to exploit the asymmetry of divergence in the one or the other direction.
To mitigate the unboundedness, the skewed KL divergence has been proposed, mixing the two distributions D (P || Q) = ( [P || (1 P) + SQ) with E [0. 1] controlling skewness and thus boundedness (see e.g. Yamano (2019)). To obtain symmetry, the
divergence defined as JS, (P || Q) 1 ( (P Q) Ki. (Q P)) was introduced. Based on these ideas, we propose a new divergence, which we call parameterized Jensen- -hasnnnn Divergence (pJSD), which allows control for both, the level of skewness well the level of symmetry, and prove that pJSD is bounded.
Definition (Parameterized Jensen Shannon Divergence). For two probability distributions P' and Q we define the parameterized Jensen-Shannu divergence as
(P Q) DK (P' || 1QQ P) + (1 aa)-))KKL(( (Q + +
where 0 S a S 1 determines the level of symmetry and 0 < B < 1 determines the level of skewness. Theorem (Upper bound on pJSD). For S a a and 0 3 < the parametrized JS divergence is bounded from above by
JSg S - log(1111 - 3).
We provide a proof in App. A.1.2. Putting the pieces together, we can now formulate our objective as the minimization of the KL divergence between the similarity distributions of X and Y. and the maximization of the pJSD between the similarity distributions of Y and Z. which is
arg min DEL (Pl (1 (S' (1Q). Y
While the parameters 8 give the user flexibility on how much of the prior distribution should be factored out, we will later discuss a good default parameter instantiation based on synthetic data. This objective, similar to vanilla tSNE and related approaches such as ctSNE, is not convex, and is optimized using gradient descent. We provide the derivation of the gradients in App. A.1.1. This provides us with method that solves our problem of factoring out prior knowledge on information theoretic grounds, which we call JEDI resemblance of the Jensen Shannon Divergence.
Computational Complexity The computational and memory complexity of JEDI is in O(k")), for n samples and iterations, which comes from the summation over all pairs of sample in the divergences in each iteration. Due to the interactions in the gradient of pJSD, standard algorithmic optimizations of tSNE (van der Maaten, 2014; Linderman et al., 2019) are not directly applicable.
Overall, JEDI is powerful, theoretically appealing approach to factor out prior knowledge. Based on tSNE, JEDI inherits many of its strengths and weaknesses. In particular runtime and strong emphasis on local structure make it hard to successfully apply tSNE, and therewith JEDI, to datasets that are either very large and/or contain structure at different scales, i.e. data as typically considered in bioinformatics. In the next section, we therefore revisit the problem, and propose an embedding algorithm independent approach that is applicable to such settings.
3.3 THE PROBLEM ALGORITHM INDEPENDENTLY
UMAP is one of the state-of-the-ar competitors to ISNE that alleviates its drawbacks for large data that not only contain local structure. Rather than presenting dedicated solution for UMAP, we here propose a general, embedding- formulationn independent approach. To do so, we have to revisit the original problem formulation, where the goal is to approximate high dimensional distances with embedding distances while simultaneously keeping them far from the prior. The key idea here is that, if we factor out the prior knowledge from the high dimensional distances directly,
4

---

Under review as a conference paper at ICLR 2021
independent of the actual embedding process, and hence any embedding algorithm can be used. Informally, we can state this as (D* e D2) = DV, where O describes some yet to be defined way of factoring out prior knowledge Z from the distances over high dimensional data X. Once we have this operator, we can use any distance metric based embedding algorithm on its result to obtain high quality embeddings Y from which Z has been factored out. Clearly, the operator should result in proper distance metric, discard any structure that is evident and keep any structure that is not evident given prior knowledge Z. w.l.o.g., for the remainder of this section we assume that the distances D are scaled to D'
Dmux D, with D Dmaz the maximum value in D. We define operator e as
D D E AND
+
i i i i=j.
(D* GA
which is to say. we subtract the information given by the prior distances from the high dimensional distances in a linear form, with À controlling how much prior to be removed. Although surprisingly simple, this elegant definition has very convenient properties that render it very powerful. First, there is only a single parameter A, which due to linearity gives the user direct and interpretable control over how much the prior information should be taken into account. Second, the distance matrix we obtain by applying the operator maintains metric properties - the proof can be found in App. A.1.3 that are required to properly optimize downstream embedding algorithms. Without the guarantee of symmetry and triangle inequality optimizing over pairwise distances is not possible.
Theorem 2 (Metric). Assuming that D and D2 based on valid metrics, for any X O. (Dx θx D2) fulfills the metric axioms of non-negativity, symmetry, identity, and triangle inequality.
Furthermore, the operator has the property of maintaining the original structure under unin- formative prior. More formally, we define N (i) = {j E (NN( according to D} to be the A-neighborhood of sample i according to distances Dij- For simplicity of notation, will assume that all distances are distinct, the results and definitions can directly be generalized to the case of equal distances. Assuming
uninformative prior, which was generated independently of the high dimensional data D 1 D*x on expectation the neighborhoods of each point stay the same.
Theorem 3 (Uninformative prior (proof in App. A.1.4)). Assume the prior is uninformative, that is D 1. D Furthermore, the distances are normalized to D E [0, 1]. For fixed X > 0 let (Fx)g D θx DZ, On expectation, the neighborhoods in X and in X with factored out prior are the same, that is Vi, k. N
FA (1) =E NE (i).
This theorem proves that the embeddings obtained from this distance matrix are expected to be robust against priors unrelated to the input, and thus knowledge lost and no spurious knowlegde is generated. Both the metricity as well as the robustness against uninformative priors are essential for generating good embeddings, which more complex formulations can not provide that easily. Normalizing the distances as discussed above, and applying the e to factor out prior knowledge, we obtain an embedding algorithm independent method to factor out prior knowledge. In reminiscence of how the plots look, we refer to this method CONFETTI and give its pseudocode Alg. 1.
Complexity CONFETTI runs in time O(n²), which includes the normalization of the distance ma- trix and computation of the operator, which only counts towards very small constant, Additionally, we will need to run an embedding algorithm, which respective runtime added to O(z²).
EXPERIMENTS
We evaluate both synthetic and real world data. We make the implementations of JEDI and CONFETTI available online? Since there does not exist any direct competitor that can factor out arbitrary distance matrices from an embedding, we compare to closest competitors. The first is ctSNE (Kang et al., 2019), which extends the tSNE objective and can factor out prior information given as cluster labels. The second supervised LLE (De Ridder et al., 2003), which although originally designed to emphasize structure in the embedding given as labels, we can modify such that it instead emphasizes any structure not in the prior. We give the details for this modification, which we refer to as SLLE-! in App. B.I. For fair comparison, we optimize all parameters via grid search on a synthetic data hold out set, and use these throughout all experiments (see App. B.2).
Available after publication.
5

---

Under review as a conference paper at ICLR 2021
(a) etSNE
(b) SLLE
(c) CONFETTI.
(d) JEDI.
Synthetic data. Shown are conditional embeddings synthetic data given resp. ground truth labels for ctSNE and SLLE- (a,b), and euclidean distance for CONFETTI and JEDI (c.d). Colors correspond to cluster assignment over dimensions 1-8, shape (circle, square, triangle, and cross) to cluster assignment dimensions 9-12. In b),c), and d), each of the four clusters consists of samples of one shape.
Figure 1:
4.1 RELIABLY FACTORING OUT DISTANCE PRIORS
We first consider synthetic data with known ground truth. In particular, we consider synthetic data of n 200 samples over 14 dimensions, where dimensions 1-8 and 9-12 both exhibit 4 clusters of different sizes, while dimensions 13 and 14 are Gaussian noise. We give more details, as well as a iSNE plot App. B.3. The cluster structure over the first dimensions dominates the tSNE embedding. To discover information beyond these clusters, we provide JEDI and CONFETTI the euclidean distances over these dimensions. To enable fair comparison etSNE and SLLE-1, which are limited to label priors, provide the ground truth cluster assignment background knowledge. All methods finished within minutes, and we plot the results Fig. 1. Although given the true labelling, etSNE fails to satisfactorily factor out the prior knowledge, whereas methods yield the distinct clusters from dimension 9-12. Notably, when we provide JEDI with the ground truth label assignment, it yields similarly sharp clusters SLLE (see App. 13).
To objectively quantify how well prior knowledge is factored out from an embedding, we pro- pose measure the similarity neighborhood size k. we define the neighbourhood overlap score (NOS) as NOS D', D) = 1 + (kNN of in D of D'H. Correspondingly, for a distance matrix D and label 11 & E | HANN of in D} n {L.)). While this score n
neighborhoods. For two distance matrices D, D' and
distribution L, we get NOS(D, L, k)
lends itself for evaluation, it is hard to directly optimize (see App. B.3). Plotting NOS(DX, DV, for all neighborhood sizes = 1...7 allows us to asses how well we preserve information of the original data, whereas plotting NOS( D2) allows knowledge from an embedding. As the id-line corresponds to a random neighbor encounter, we can measure the area between the NOS curve and id-line proxy for how well we preserve informa- tion, respectively how well we factor out prior knowledge.
asses how well we factor out prior
For the synthetic data, the area between the curves and the id-line for etSNE, SLLE-1 JEDI, and CONFETTI are .237, .344, .340, .344 when we compute the NOS between embedding and input data without prior. For the NOS between embedding and prior labels we have .037, .005, .003. .022, re- spectively (plots are given in App. Fig. 14). We see that JEDI factors out the prior almost idcally, as well - and that CONFETTI performs slightly worse especially small neighborhoods. When evaluating the NOS with regard to the euclidean distance prior, CONFETTI beats all other methods with an area between curves only 0.002. Overall, etSNE shows the worst NOS per- formance, which is also evident in the embedding (see Fig. 1). As for information captured in the embedding from the non-prior dimensions, JEDI, CONFETTI, and SLLE do equally well, putting etSNE at a distance. Overall, JEDI and CONFETTI perform par with the state of the art given label priors, but perform at least as well given continuous priors, for which NE and SLLE are not applicable.

---

Under review as a conference paper at ICLR 2021
(a) tSNE embedding.
(b) JEDI embedding.
(c) CONFETTI tSNE.
Figure 2: Real world data. Embedded are the sum of chi-square matrices for color, local shape and texture, boundary shape, and spatial petal distribution of the Oxford flower data. Shown are vanilla 1SNE (left), and JEDT (middle) and CONFETTI (right) with HSV color distances as prior knowledge.
4.2 RECOVERING FLOWER GEOMETRY
To evaluate on real data, we consider the Oxford flower dataset from Nilsback & Zisserman (2008), which consists of over 8000 images of flowers of 102 different classes. The data is given as set of four pairwise distance matrices which are the Chi-squared distances of the color (HSV), the local shape and texture, the shape of the boundary, and the spatial distribution of petals of the flower, all computed on segmented flower images. The available implementations for etSNE and SLLE-1 are not applicable to distance matrices as imput X, and hence we could not compare to these methods. We will use the sum of all four matrices as high dimensional input. To keep the results interpretable, we subsample 40 images from 25 different classes each, by which we have n 1000 samples. We are interested whether our algorithms is able to factor out known prior that dominates an embedding in a real world setting, which why we specify the HSV color matrix as background knowledge. While other priors could be specified, we would not know if that prior would be present the input data and thus would not be able to evaluate success. CONFETTI and JEDI terminate in seconds, respectively two minutes.
We give the vanilla tSNE embedding in Fig. 2a, which besides a clustering according to colour con- veys little other information. When factor the color information with JEDI and CONFETTI, we see that colors mix and new clusters form according to other features. For example, spiky petals arrange at the side (Fig. 2b bottom, Fig. 2c bottom right), whereas rounded petals assemble the respective other side the space. Similarly, flowers with few but large petals gather on one side (Fig. 2b right, Fig. 2c top) and flowers with many thin petals on the respective other side. Apart from these visual changes, we can evaluate based on the NOS plot, which shows that also for this metric prior our methods are able factor out the background knowledge (see App. Fig. 15).
4.3 BATCH EFFECTS IN SINGLE CELL SEQUENCING
One of the major applications of embeddings such as ISNE and UMAP is in single cell sequencing, where it is a standard routine to visualize the sequencing data, allowing e.g. assess the quality of sequencing, to easily remove outliers from the data, or highlight differences between cohorts.
To test the methods on such data, we use a recent single cell data set of cerebrospinal fluid (CSF) and whole blood of multiple sclerosis patients and control group (Schafflick et al., 2020). We generate a vanilla UMAP embedding using their original jupyter notebooks, and plot it as Fig. 3a. shows the typical clustering according to the cell types in blood and CSF, when coloring the samples according to gene expression of standard marker genes (for more information refer to the original paper). Here, we are interested whether thus provide again the gene expression as input, and additionally take the euclidean distance between the marker gene expression levels of each sample prior, corresponding to how the coloring was obtained. As etSNE cannot deal with continuous priors, we instead provide the cluster labels from an agglomerative clustering of the marker gene expression levels, which yielded the same number
methods can reveal information beyond cell type. We

---

Under review as a conference paper at ICLR 2021
(a) UMAP embedding, (b) CONFETTI UMAP (c) cISNE with labels of (d) SLLE with labels of coloring according to cell with marker gene expres- clustering marker gene ex- clustering marker gene ex- type.
sion prior.
pression as prior.
pression as prior.
Figure 3: Real World Data Visualizations of the embedding of single cell sequencing data of UMAP, CONFETTI UMAP, etSNE, and SLLE-! sLLE-! failed to produce an embedding for the full data, results are obtained for random subset of the data. Coloring in b), c), and d) according to batch ID, shape according to sample type (case vs control, and blood vs CSF).
of clusters as the original paper (see App. Fig. 16b). SLLE-1 requires a matrix inversion, and for this data gives an error hinting that the data matrix is too large. Hence, we provide sLLE- with only random subset of -6000 samples to obtain an embedding. While this is not practical, as important tasks such as outlier detection or assignment of cell types or states can only be done for that particular subset, it allows to compare the obtained embeddings. We provide SLLE with the same cluster labels as for etSNE, etSNE terminated within 23 minutes, SLLE-! in 32 minutes, CONFETTI in 10 minutes, and JEDI in 100h, the resulting embeddings are given in Figure 3, vanilla UMAP plot with coloring according to batch is given in App. Fig. 17c.
The results of CONFETTI show surprising separation of samples from different batches and ac- cordingly separation of the different tissue (blood and CSF), and of case and control along a man- ifold. Encouraged by these results, we also apply CONFETTI with tissue information prior, and then observe that the previously separated clusters for actually equal cell types (B1,B2, NKI,NK2, mDC1,mDC2) are now merged (see App. Fig. 17a), while all other information is kept. For etSNE, there are no clusters or manifolds directly visible and the result is a rather homogeneous ball, which show only clear separation of CD4 cells, which make up the largest proportion of cells (compare App. Fig. 16a). Similar to etSNE, JEDI shows no clear separation of cell types (see App. Fig. 17b), but there no bias towards CD4 cells for etSNE. SLLE does not show any discernable struc- ture beyond the provided cell types.
5 DISCUSSION AND CONCLUSION
In the experiments, we show that both JEDI and CONFETTI correctly factor out prior knowledge and result embeddings that reveal previously hidden structure. On synthetic data, when provided with euclidean distances capturing the structure which dominates the vanilla tSNE embedding, both our methods reveal the clustering of the input data that is independent of the background knowledge. Our closest competitor etSNE, is not able to factor out the background knowledge well, and its embeddings still show structure of the prior even when given the ground truth cluster labels. When we provide our methods the same information, they do recover the correct clustering, demonstrating their ability to reliably factor out both distances as well as label priors.
On real world data of flower images, where only distance matrices between attributes of the images are available, both JEDI and CONFETTI are able to factor out the property dominating the vanilla (SNE embedding, and organizing flowers according to number and shape of petals rather than by color. This shows that both are able applicable to real world settings, and unlike their competitors, consider both input data and background knowledge in the form of different distance metries. This provides us with the advantage to be applicable in domains where the input is only available as distance matrices, e.g. dernel-derived distances for unstructured data such graphs strings.
8

---

Under review as a conference paper at ICLR 2021
On the perhaps most widespread application of low dimensional embeddings, single cell gene pression data, we confirm that tSNE based approaches are a bad fit: both ctSNE and JEDI get lost in local detail, fail to factor out the prior knowledge, and are slow. When we combine algorithm-indeppnnde
approach, CONFETTI, with UMAP we do arrive at meaningful embeddings from which the background knowledge has been factored out. In particular, when we provide it with prior knowledge of marker gene expression which is used to determine cell type we arrive at an embedding that organizes samples according to batch id and tissue type. Conversely, if provided with tissue type as prior, we observe that previously separate clusters, that actually contain cells of the same type, are merged.
In practice, distance based prior is usually the most natural representation of the background knowledge that cannot be cast easily into class labels. With binning or clustering using distances, information is lost for differences within class, and the relative information is lost between classes. For example, effects based on geographic locations, age differences, or income that should be fac- tored out from an embedding should be considered as is rather than cast into labels, which is not possible with current approaches. Our approaches do take into account these relative relationships by considering distances between pairs of samples, such as the geodesic distance between locations of two individuals or their difference in age, which would be lost when encoded by labels.
Overall, both proposals solve the problem of factoring prior knowledge from low dimensional embeddings, and by allowing the prior knowledge to be specified as distances they are both applica- ble large number of data types and domains. While JEDI has theoretically appealing objective, it inherits all strength and weaknesses of (SNE by which especially its runtime is an open problem. For (SNE, several methods have been proposed that significantly speed up the optimization, yet none of them seem directly applicable to JEDI. We leave their adaptation for future work. Furthermore, although yielding overall good results, we would like more closely investigate the parameters of ISNE and JEDI in more principled way, as for new domains parameter settings for e.g. tSNE are mostly found by trial and error. Here, we tested our methods on different types distance metrics that cover different applications, and with single cell gene expressions one of the most important domains for low dimensional embeddings. In the future, we would also like to investigate how our methods perform when provided with more exotic types of background knowledge.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Babak Alipanahi and Ali Ghodsi. Guided Locally Linear Embedding, Pattern recognition letters, 32(7):1029-108) 1035, 2011.
Elnaz. Barshan, Ali Ghodsi, Zohreh Azimifar, and Mansoor Zolghadri Jahromi. Supervised Principal Component Analysis: Visualization, Classification and Regression on Subspaces and Submani- folds. Pattern Recognition, 44(7):1357-1377 2011.
Etienne Becht, Leland McInnes, John Healy, Charles-Antoine Dutertre, Immanuel WH Kwok, Lai Guan Ng. Florent Ginhoux, and Evan W Newell. Dimensionality reduction for visualizing single-cell data using umap. Nature biotechnology. 37(1):38 44, 2019.
Bucttner, K. N. Natarajan, P. Casale, V. Proserpio, A. Scialdone, F. J. Theis, S. A. Teichmann, J. C. Marioni, and O. Stegle. Computational analysis of cell-to-cell heterogeneity in single-cell RENA-seqenniig data reveals hidden subpopulations cells. Nat Biotechnol, 33(2):155-160 Feb 2015.
Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, and Martin Wat- tenberg. Visualizing and measuring the geometry of bert. arXiv preprint arXiv: arXiv:1906. 1906. 200600275, 2019.
Dick De Ridder, Olga Kouropteva, Oleg Okun, Matti Pietikäinen, and Robert PW Duin. Super- vised locally linear embedding. In Artificial Neural Networks and Neural Information Processing ICANW/CĆŌNP 2003, pp. 333-341. Springer, 2003.
Johann Gagnon-Bartsch and Terence P Speed. Using control genes to correct for unwanted vari- ation in microarray data. Biostatistics, 1333):539-552. 2012.
R. Gerstner, J. N. Koberstein, A. J. Watson, N. Zapero, D. Risso, T. P. Speed, M. G. Frank, and L. Peixoto. Removal of unwanted variation reveals novel patterns of gene expression linked to sleep homeostasis in murine cortex. BMC Genomics, 17(Suppl 8):727, 10 2016.
Sami Hanhijärvi, Markus Ojala, Niko Vuokko, Kai Puolamäki, Nikolaj Tatti, and Heikki Mannila.
Tell me something I don't know: randomization strategies for iterative data mining. In KDD, pp. 379-388. ACM, 2009.
Geoffrey E Hinton and Sam T Roweis. Stochastic neighbor embedding. In Advances neural information processing systems, pp. 857-864, 2003.
H. Hung. robust removing unwanted variation-testing procedure via >-divergence. Biometrics, 75 (2):650-662, 06 2019.
Bo Kang, Jefrey Lijffijt, Raúl Santos-Rodríguez, and Tijl De Bie. Subjectively Interesting Com- ponent Analysis: Data Projections that Contrast with Prior Expectations. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp- 1615-1624, 2016.
Bo Kang, Dario Gareía- Carefa-Gareía, Jefrey Lijffijt, Raúl Santos Hantos-Rodriguez, and Tijl De Bie. Condi- tional t-SNE: Complementary t-SNE embeddings through factoring out prior information. CoRR, abs/1905.10088 2019. URL h org/ aas//ab///// 905. 10086.
Dmitry Kobak and Philipp Berens. The art of using t-sne for single-cell transcriptomics. Nature communications, 10(1):1114, 2019.
Dmitry Kobak and George C Linderman. Umap does not preserve global structure any better than t-sne when using the same initialization. bioRxiv, 2019.
Teuvo Kohonen. Self-organized formation of topologically correct feature maps. Biological cyber- netics, 43(1):59-69 69, 1982.
George Linderman, Manas Rachh, Jeremy Hoskins, Stefan Steinerberger, and Yuval Kluger. Fast interpolation- based t-SNE for improved visualization single-cell RNA-seq data. Nature Meth- ods, 16:1, 03 2019.
10

---

Under review as a conference paper at ICLR 2021
Leland McInnes, John Healy, and James Melville. Umap: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv preprint arXiv:1802.03326, 2018.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphies Image Processing, pp. 722-729. IEEE, 2008
Karl Pearson. On lines and planes of closest fit to systems of points in space. The London, Edin- burgh, and Dublin Philosophical Magazine and Journal of Science, ((11):559-572, 1901.
J. Peltonen, H. Aidos, N. Gehlenborg, A. Brazma, and S. Kaski. An information retrieval perspective on visualization of gene expression data with ontological annotation. In 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2178-2181, 2010. ICASSP2200.5495665.
doi: 10. 109/
Kai Puolamäki, Emilia Oikarinen, Bo Kang, Jefrey Lijftijt, and Tijl De Bie. Interactive Visual Data Exploration with Subjective Feedback: An Information-Theoredl Approach. In 2018 IEEE 34th International Conference on Data Engineering (ICDE), pp. 1208-122 1211. IEEE, 2018.
D. Risso, J. Ngai, T. P. Speed, and S. Dudoit. Normalization of RNA-seq data using factor analysis of control genes or samples. Nat Biotechnol, 32(9):896 902, Sep 2014.
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embed- ding. science, 290(5500):2333-23366 2000.
D. Schafflick, C. A. Xu, M. Hartlchnert, M. Cole, A. Schult-Meckkenbeek, Lautwein, J. Wol- bert, M. Heming, S. G. Meuth, T. Kuhlmann, C. C. Gross, H. Wiendl, N. Yosef, and G. Meyer Zu Horste. Integrated single cell analysis of blood and cerebrospinal fluid leukocytes multiple sclerosis. Nat Commun, 11(1):247, 01 2020.
Vin D Silva and Joshua B Tenenbaum. Global Versus Local Methods in Nonlinear Dimensionality Reduction. In Advances in neural information processing systems, pp. 721-728, 2003.
Joshua B Tenenbaum, Vin De Silva, and John C Langford. A Global Geometric Framework for Nonlinear Dimensionality Reduction. science, 29065000) 2000.
Warren S Torgerson. Multidimensional scaling: I. Theory and method. Psychometrika, 17(4):401- 419, 1952.
Laurens van der Maaten. Accelerating t-SNE using Tree-Based Algorithms. Journal of Machine Learning Research, 15()):32113245520g) 2014.
Laurens van der Maaten and Geoffrey Hinton. Viualizing data using t-SNE Journal of Machine Learning Research, 9:2579-2605, 11 2008.
Takuya Yamano. Some bounds for skewed --hennen-Shannon divergence. Results in Applied Math- ematics, 3:100064, 2019.
A METHOD APPENDIX
A.1 THEORETICAL RESULTS
A.1.1 WORKING TITLE: THE HIGH GROUND
In the following, we derive the gradient of the conditional cost function
C(Y) DKL Q)
(P' Q1
We split the gradient into two parts
iC
5DRL. (P (1)))
6JS3 (P' (1
8y:
dyf
6y
11

---

Under review as a conference paper at ICLR 2021
as the first is given by the tSNE gradient with
4 Des - - (1+ l1y: -,5113) (y/ 8z).
dzh
First, we write out the full parametrized
Divergence:
JS Q1 (D || 11 + (1-8)P)) + (1 (KL 3P' -1--3)Q)
1/j
a
Σ
P'y log 3ys; + (1 B)yis
+ (1 -1) a))
Σ
Vig
log
4ki
Bp + (1 - B)4b
is
i.
Σ
Ply log 1/s -Pís log (Byss + (1 B3)pas)
+
a)
Σ
4ig log diş Qúj log (Bpls (1 -B)gs)))
The expected similarities Ply symmetrized probabilities
P/y + P/1s
(xp(-(D6)//2(01)))
P/g
with
P/J
2n
kitt exp( 2xp(-(DD(9//(003)
and the low-dimensional similarities qiy are defined as
(1 + 11ys My 11
46 Enyr(1 + 117e y1 12)--1
where we abbreviate the distances by diy 11 ツlا and the normalization term by Z (1
leading - leading to the succinet notation Øij (1 + (D
As 1/4 only affects the distances Dj,, and D by
5dgy
61 Mill
y1 Yi
8y
8y:
D
will derive the gradient with respect DI₁
8C
8C DD
8C 8
Σ
dy:
8D sj Vi
DDY ji
dy:
8C y1 yj
2
D
Note that the first term of the Jensen- Massen-Shannon cost function is constant with respect to D.. Thus, split the gradient
6JS0 P' )1Q)
8p/4 log (3qu + (1 - 3)pp))
-
8D)
Equa log qkl
(1 a)
8D 19 8qut log (3p/., + (1 3)ga)
1
(1 0
8D
1
12

---

Under review as a conference paper at ICLR 2021
and compute the three parts separately. The building block are the derivatives of Z and qat with respect Dyy
82
5(1 (DK))))
DDY
6
keft
5(1 + (Dz)))-1
-(-1)(1 (D1))3)-2213
=-2D + (1(())))))))
--2D 2 + (D1)))-1
(1+(D))))
6gul
8DY
1
8(1 (DZ)))
(1+ 87
1
8(1 + (D
&DY
1
8(1 2D
7+ (1+0 (D
8DY,
6((1+
(2
+ (1+ (1+()))-2222) (1 d
Z
Z
8((1 + (DK)))-1
+ +1+ (
8D
Z
[(++ ( -2 2
4ig (:5 1)
if ij
1(1+ 2
il
13

---

Under review as a conference paper at ICLR 2021
Next derive the gradients of the three parts:
6pl log (3qua + (1 - -1)ppk)
5 log (3qkt + (1 - 3)pp(a)
Pket
8di5
Ödij
kz1
3p'11
8qM1
3qus + (1 - B)yb's Ōdaj
1
B1
BPhi(1 + +dhi)- -Ha)-12dss00f
(Bqis + +(1 -3)pp's)
Bqua (1 - B3yp'r
Bply (1 + +dly)
Byke(1 + dlzi)
2dds
4
S9. + +(1- 33)pks
3qkt + (1 -3)pks
light
Bplygis(1 (1+ddt))
Bple 7. 3qua + (1 33)
2dg
Th
Σ
89. + +11 (1-3)p's
Righl
(- [[3qs + (1 3)ples
8pls
Spha Bqsa + (1 3)p²k
= =2d.91h(1 + +d,z)
+ Σ
Aigh
8gui log que
8(og qu))
Jqja
/D
Σ
9h1
8D
D sj
log Qki
1
8gul
6qul
Σ
9kl
DDY
+
DDY
log gki
dgut
(1 + log qur)
oDY
kii
(1 (2 91 loggg)) +
> (DK)) (1 + log qut)
= 20 -(1+(D)))) g0, + log q0) + 2
(1++ log qu))
-2D (1 (D))))- ((1 (-11+ (1g1og (G)) + Σ gw (1 + log qua)
14

---

Under review as a conference paper at ICLR 2021
squi log (3ppr + (1 - 3)qma)
8D
blog (8phr (1 3)qaa)
dqkl
qli
+
log (3pkr + (1 3))ms)
(1 B)gks
ōgut
5gkl
+
log (3y/r + (1 33)mm)
By/11 (1 (3) D
8qua D
(1 -3)9a
+ log (Sphr + (1 -1- (1-1))qm)) 3))
Bp/s + (1-B)gw
kt1
(1 3)4b
-1 (D2))
log (3plg -))ss))
Bply +1 -3)gis
(1 -8)ym
+ Σ
(1 ( 22
+ log (Bpkr + (1 3)
Bp4. -8))gu
((11 (-1- (D)))))-
(1 -3))as
= 2D
+ log (Sp's + (1 - -3)as))
Bply + (1 -3)yss
(10) (1 - 33)qu
+ +2D E(1 (DK))))-
(ppks + (1- 3)qks + log (3pka + (1 - 3)qk))
kzl
2D + (D%))
(-
(1 - -B)gss
log (3ply (1 B))cs)
Byly (1 (1-18) -3)4ks
(1 3)q
log (Bp'a - (B)aa))
qki
8pk + (1 3)qka
Finally we combine them into the derivative of JSB with respect to D
(1 +1+ (D!))) 8JS (P' || Q) -a
8ply
3ph
Σ
2Di,00 96
8
[Bqi +1--3)p,
-3p'ss
Bqua + (1 3)
kel
+ (1 a)
-(1 + log Qis)) +
Σ
gha + log qua)
1ipht
(1 -B))gs
-(1 a)
log (Bpls + +11 - (1- (B))sss))))))))))))}{)))))))(-)
Bpl, + (1 -B)qis
(1 (1-B)gu
- (1 a) Σ qkl
+ log (8phs + (1 - -8)ga))
Bp'a + (1 -B)gat
Aigh
a,3p's
a:3pka9m
3q. + +11
Bqat + (1 - 33)Pk
+ (1 (1-10) ai) an( -(1 + llog (1s)) + 3ph5 + (1 -3))gs
(1 (1-3)
+ log (Bplg + (1 ())sss)
(1 3)qma
+ +(1 - a) Σ
1 log qui
log (8ple + -11 B)gua))
3r4s + (1 33)mol
light
15

---

Under review as a conference paper at ICLR 2021
and substitute this result in the gradient with respect y/1
( Q1))
(
|| Q) (y/ - v1)
8y/i
6D kg
Dij
a.p'
a:pp'
= 4 Σ
y - 15)(1 + (D2))))-10 4s
89is (1 -B)p's
3qu + (1 - B)P
(1 B)0ss
+ (1 - -a) - -(1 log 4:²) + Byls (1 B)q.
+ log (Bpl, + +11-10 -B)gss)
log (3pkr + - -8)gaa) )
(1 -3aaa
(1 (x) 9ht
+ log qui
BPks (1 B)qxa
lighl
A.1.2
BOUNDING PJSD
Here, we will provide the proof for the upper bound of the pJSD following the idea of Yamano (2019).
Theorem (Upper bound on parametrized JS divergence) For 0 S S and 0 < 3 the parametrized JS divergence is bounded from above by
S Σ log(1 3).
Proof. Let first define a known bound for general f-divergences Yamano (2019).
Lemma 4. Let (x) be the conjugate function of f(x) defined as f*(x) xf(.). The f divergence satisfies the upper bound
D2(P11Q) < lim +
We will the definition of our parametrized Jensen Shannon Divergence JSJ = aKL(P || 3Q + (1 -1- (1-3)P) + (1 a)KL(QQ P (1 B)Q)
to bound the individual terms, which are essentially skewed KL divergences, to then obtain the upper bound for PISD. The skewed 3-KL divergence given by
p(z)
(P Σ p(x) log (1 B)p(x) + 3q(x))
Choosing (x) = x log (1- 3 we can see that the skewed B-KL divergence belongs to the family of -ddiergences. Similarly, choosing f2(x) -3 log (3x+ 1-11)11)))) we get the reverse 6-KL divergence KL || P).. We thus get
JSB aKL (P (1 (1- P)
S a lim
(fi(z)+ + fi()) + (1 a)) lim (J2(z) + fz)))
==
x
1
a lim
xlog (1
+ log 1 8 + xB
8)x + 1
3 +1 +1 B)
+ (1
a) lim x=20
- log (3x + 1- 1- (3) x log
a log
+ (1 a) -log(1 3)
1- 3
-log(1 - -}),
16

---

Under review as a conference paper at ICLR 2021
where the first inequality is achieved by bounding the individual skewed 3-KL divergences using Lemma 4, the following equality is obtained by plugging in the definitions the corresponding E-divergences fi and f2 with their respective conjugate, and the remainder is obtained by plugging in x = O and using basic math. This completes the proof.
A.1.3 CONFETTI MAINTAINS METRIC PROPERTIES
Theorem S. Assuming that Dx and D2 are based on valid metrics, for any >> O, (D* Ox D fulfills the metric axioms of non-negativity, symmetry, identity, and the triangle inequality.
Proof. We will prove that
- +
( i ==
fA(1,))
fulfills the metric axioms and use the fact that the sum of two metrics is a metric.
Non negativity; Vi, vi.j.fx(i..) O. Using the definition of () get À 2 L.DGG Since DJ (0, 1), we know that the inequality always holds.
2. Symmetry: Vi.j.ff( = ().
Follows directly from the symmetry property of D
3. Identity: ji.j-fx(t..)
= i=
We know that for i / j. E max + + - min + AJ. Using D2 E (0, 1), we get (x.j.) E { A]. The other direction follows directly from the definition.
4. Triangle Inequality: ji,j, (x,.i)) < (a(i,) fa(kjj).
Using the insight fx( E [ from before, we get fx(1.) I max (fa(,g))
A = 2 2 = min.,y (fa (I, v)) & (I + fa(
Thus, (Dx Ox D implements a metric.
A.1.4 CONFETTI IS ROBUST AGAINST UNINFORMATIVE PRIORS
Theorem 6. Assume the prior is uninformative, that is D D Furthermore, the distances are normalized to D/ [0, 1]. which is ensured by our method. For fixed X > Olet (F ex DZ be the high dimensional distances with factored out prior. On expectation, the neighbourhoods of X and X with factored out prior are the same, that is N-Fx (i) =E] NP* (i), for any i and k.
Proof. Let us first look at what on expectation the distance of some nearest point of i is under Fx. We now pick the j E NrA (i) with the largest distance. Note that the distances define a fixed ordering and thus for each j there is a k' such that has the largest distance in N (1). We prove by contradiction. Let j & NP* (i), hence there are k indices given by N (i) with VI L. D D Let Px and Pz be the generating distributions of Dx and Dz, respectively. We get
DN D
DN 1
^Epp [ 1 D 11 MEp, (D +
1
Ep. 1D Ab X] Ep, 1D 1 ADDZ +
Ep (FF)a Ep, [(F)bb
where line to 2 is obtained using that D 11. D and Erz ( E |0. 1). and line 2 to 3 is obtained by using linearity of expectation and D 1. D In other words, for all l on expectation the adapted distance to i, (Fx)as is smaller than the adapted distance between and j. (Fx)az- This means that contains indices that have smaller distances under FX than j. hence j & N (t). This is a contradiction.
17

---

Under review as a conference paper at ICLR 2021
A.1.5 CONFETTI PSEUDOCODE
Algorithm 1: CONFETTI
Data: Data distances Dx prior distances D penalty X
Result: Adjusted distances D Gx D
D
D*
max D D
D)
max D
foreach [I,n] do
foreach j E 1. do
(Dx θ D kj t D 2DD +1
(Dx @x @x D² + θx D²)g
return (D* 0x D2)
B EXPERIMENT APPENDIX
B. INVERSE SUPERVISED LLE (SLLE-1)
In SLLE (De Ridder al., 2003), class labels given as additional input are used to emphasize the structure within the classes. Thus, SLLE improves the separation between classes by homogenizing the local neighbourhood
each sample with respect to the class labels. This done by pushing away data points with different class labels, increasing the distance of such points by fraction of the maximum distance observed in the data. This can be summarized by the following update procedure for the sample distances, using the same notation in the main paper
D' D + max max(D*)D, 0 05a5
with Đ the binary matrix with D. = 1 if samples 2; and xy are from different classes
In our work, are interested in factoring out the prior, which is essentially the opposite of SLLE. By manipulating the objective from above, we can however achieve our goal of factoring our prior knowledge, by optimizing for
D' D max D), o
We call this method SLLE-1 This method can still only deal with labels as input, but for that it works surprisingly well in our experiments, yielding better results than the current state of the art etSEE
B.2 CHOOSING THE HYPERPARAMETERS
All methods involved in the experiment section come with set of hyperparameterss that the user has to fix. We decided to generate a small synthetic data set, generated differently than the one used for our experiments, to settle for a good set of parameters for each of the methods. In particular, we explore for CONFETTI coupled with iSNE the parameter À (0.2, 0.4, 0.5, 0.6, 0.8, 1.2 1,4, 1.5, 1.6, 2,8., 4} and perplexity 30, 50, 200, 400, for JEDI we search for a E {0, 0.2, 0.5, 0.8, 1). B E {0.8, 0.99, 1}. and perplexity and prior per- plexity 50, 200, 400 each. For etSNE we search for B E {le (1e-2, le- ..., le-"), and for SLLE the number of neighbours & E {10, 30, 50, 100, 200, 400, 500, 600} and parameter a e (0.0001, 0.001, 0.1, 0.2...... .... 1}. We note that the authors of ctSNE suggest to use small value 3 = 0.01, but use 8 = 0.0001 for their experiment, which led us to the search grid for their method.
We generated simple 10-dimensional dataset with 1 data points, with samples assigned to of four clusters in the first 4 dimensions, and one of two clusters in dimension 5 and 6. The clusters in the first 4 dimensions
centered at the four unit vectors along the four axes of the dimensions. The clusters in the other dimensions are centered at (8), (Q) The remaining four dimensions are gaussian noise (0, 1). Each data point is randomly assigned one cluster of the first four, and
18

---

Under review as a conference paper at ICLR 2021
Figure 4: Visualization of the synthetic data set for parameter tuning based on tSNE. We samples with the same colors clustered together (clustering planted in dimensions A) as well as samples with the same shape (clustering planted in dimensions B).
19

---

Under review as a conference paper at ICLR 2021
0.00
0.25
0.50
0.75
Euclidean distance
0.6 High dimonsional
Euclidean distances other samples computed only over dimensions dI to d4. We observe the clus- tering planted into those dimensions.
(a) Buclidean distances to all other samples computed over all dimensions.
jiller
0.00
0.55
3,59
0.76
1.00
Dendrog am distance
(d) Scaled dendrogram distances from the pink refer- ence sample all other samples. It is defined the height of their lowest common ancestor node dendrogram given in (c).
(c) Hierarchical clustering with average linkage based on dimensions di to d4. The four different clusters are easily identifiable.
Figure 5:
Visualization of different distances for sample from the pink cluster all other sam- ples in the synthetic data set for parameter tuning.
20

---

Under review as a conference paper at ICLR 2021
( a
(b) 0.99, a =
(0) = 1,
3 0.99,
( 1,0=1
Figure 6: Visualizations of the synthetic data set for parameter tuning by JEDI with Euclidean distance prior based on the dimensions in A (color clusters). Here we compare the influence of the smoothing parameters and B.
cluster in dimension 5 and 6, and takes the coordinates of the cluster centers plus some gaussian noise (0, 0.01). iSNE visualization is given in App. Fig. 4. For each method, we provide embeddings for subset of their search grid that gives a good idea how the embedding changes when varying the parameters in Figures 5, 6, 7, 8, 9, 10, and 11.
Looking the results, observe that for CONFETTI, as expected, the gradual increase in À is directly reflected in the embedding with how strongly differently colored points merge. Here, we settle for a perplexity of 50 for the prior and 2, as this allows take into account much information of the prior while yielding nicely separated and mixed colored clusters.
For JEDI, we have to investigate a good setting of different parameters in a principled manner. Fixing the perplexities, we observe that a 0 and / 0.99 yield much better mixtures of the similarly colored points, than any other combination, with JEDI not converging with Fig. 6). Yamano (2019) already pointed out that 3 = 0.99 worked well as a choice for the divergence mixing. Fixing 3 0.99 we proceed analyse the impact of and the perplexity parameters, noting that the latter is highly dependent on the data set size, and should be adjusted by the user as with original tSNE. Here, we settle for a = 0, perplexity of n and prior perplexity n,n where n is the number of samples.
(App-
For etSNE, 3 = le worked best, which is also close to what the authors use for their original experiments. In the colored points are not clustered together anymore. Furthermore, only with & 2 200 the two desired clusters clearly visible. We thus propose use 0.5 and fnn as default parameters, where k should be carefully adjusted depending on the data set at hand.
of SLLE-1, observe that for 0.3, independent of k. similarly
B.3 SYNTHETIC DATA
The data has 14 dimensions in total, where each sample belongs to one of 4 clusters (A1-A4) in dimension 1-8 and one of four clusters in dimension 9-12 (B1-B4). We first draw cluster centers
21

---

Under review as a conference paper at ICLR 2021
(b) prior perplexity 200, a
(c) prior perplexity
400, a
(a) prior perplexity- = 50, a =
(e) prior perplexity 200, a prior perplexity 400,
(d) prior perplexity = 50, a =
Figure 7: Visualizations of the synthetic data for parameter tuning by JEDI with Euclidean distance prior based on the dimensions in A (color clusters). We compare the effect of an increasing prior perplexity on both divergences determined by a.
22

---

Under review as a conference paper at ICLR 2021
(a) perplexity = 50, a
(b) perplexity = 200, a 0
(c) perplexity 400, a = 0
(d) perplexity 50, a = 0.5
(e) perplexity 200, a = 0.5
(1) perplexity 400, a 0.5
(g) perplexity 50, a
(h) perplexity 200, 1
perplexity 400, a
Figure& 8: Visualizations of the synthetic data set for parameter tuning by JEDI with Euclidean dis- tance prior based on the dimensions in A (color clusters). We see the effect of different perplexities in combination with different values for the mixture of divergences a.
23

---

Under review as a conference paper at ICLR 2021
(a) perplexity 50, = 0.4
(b) perplexity 200, A = 0.4
(c) perplexity 50, A 0.6
(d) perplexity 200, A 0.6
(e) perplexity 55, À
(f) perplexity 200,
(g) perplexity = 50, A =
(h) perplexity 200, À
Figure 9: Visualizations of the synthetic data for parameter tuning by CONFETTI with Eu- clidean distance prior based on the dimensions in A (color clusters), For two different perplexity values increase the maximum penalty À from 0.4 to 2.
24

---

Under review as a conference paper at ICLR 2021
0.1, 50
0.5, 50
(c) 1, 50
(a)= 0.1, 200
0.5, 200
= 200
(g) 0.1, = 500
(h) 0.5, k 500
(1 1, = 500
Figure 10: Visualizations by SLLE with labels prior based on the clusters in A (color). The amount how much the distances are adjusted determined by and the number of neighbors to consider is k.
(a) -3
(b) le-5
(0) 3 17
Figure 11: Visualizations by eSNN with label prior based on the clusters A (color). We try different values for 8 which influences how strong the prior is factored out.
25

---

Under review as a conference paper at ICLR 2021
from (0, 2) for each of the eight clusters. Then, the feature values for each sample are generated in three steps as follows:
1. Pick cluster a from A1-A4 with probability 0.1, 0.2, 0.3, or 0.4, respectively. Add Gaus- sian noise to the cluster center a with standard deviation 0.1, 0.2, 0.3, or 0.4, respectively. Noise drawn and added for each dimension independently.
2. Pick cluster a from B1-B4 with probability 0.1, 0.2. 0.3, or 0.4, respectively. Add Gaus- sian noise to the cluster center a with standard deviation 0.1, 0.2, 0.3, or 0.4, respectively. Noise drawn and added for each dimension independently.
3. The remaining dimensions of every sample Gaussian noise from from.N((. ./(00,).
A 1SNE visualization of the data is given in App. Fig. 12. For this dataset, we take the euclidean distances between all samples the first dimensions as prior for our methods, and the ground truth label assignment A1-A4 prior for the other methods. Here, we additionally provide the embed- dings CONFETTI and JEDI with label assignment prior, given App. Fig. 13, and the NOS plots for both the input data and the prior in App. Fig. 14. While the NOS plots and corresponding between curves give insight in how well an embedding factors out or maintains certain information, it is hard to optimize. In particular, it unclear how to combine and balance the two scores for how much prior is removed and how much information of the input is maintained. Even if these issues are resolved. the main problem will be that the kNNs that define this score are essentially describing binary relationships between entities, thus optimizing for optimal neighbourhoods does not give any clue how to arrange the samples in 2D continuous space.
26

---

Under review as a conference paper at ICLR 2021
Figure 12: iSNE of synthetic data. Visualized is (SNE plot for the synthetic data set using perplex- ity 50. Color corresponds to cluster assignment in dimension 1-8 and shape corresponds to cluster assignment in dimension 9- 12.
27

---

Under review as a conference paper at ICLR 2021
(a) CONFETTI À = 1.
(b) JEDI 0, .99.
Figure 13: JEDI and CONFETTI with label prior. Visualized are the plots generated by our methods when fed with the ground truth labels corresponding to the cluster assignment in the first 8 dimen- sions of the synthetic data set. Color corresponds to cluster assignment in dimension 1-8 and shape corresponds to cluster assignment in dimension 9-12.
100
neightortood overlsp in %
GNE Conren
ONE
neighborhood
Corrent
25-
0-
25
neighborhood size in % of the data
neighborhood size in % of the data
(a) NOS plot between prior labels and embedding (smaller better).
(b) NOS plot between non-prior dimensions and em- bedding (larger is better).
Figure 14: NOS score visualizations. Given are the NOS plots for prior labels and embedding (a) and non-prior dimensions 9-12 and embedding (b) on the synthetic data set. On the x-axis, the neighbourhood & is varied between 0% and 100% of data points, the y-axis is the NOS score.
28

---

Under review as a conference paper at ICLR 2021
B.4
FLOWER DATA
100
neighborhood overlap in %
75
50
Confetti
Jedi
1.SNE
25
50
75
100
neighborhood size in % of the data
Figure 15: NOS plot for flowers. Visualized is the NOS score for the Oxford flower data set for JEDI and CONFETTI against the prior, On the x-axis, the neighbourhood i: is varied between 0% and 100% of data points, the y-axis is the NOS score.
B.5 SINGLE CELL DATA
Here, we provide additional plots for the single cell experiments. In particular, we carried out an agglomerative clustering based marker gene expression, which arrives the same number of clusters as the original paper (App. Fig. 16b) Additionally, we show how the etSNE embedding looks when coloring the visualization according to cell type of the original paper (App. Fig. 16a), and new result when we use tissue information as prior knowledge for CONFETTI (App. Fig. 17a), as well the vanilla UMAP plot with coloring according batch ID.
29

---

Under review as a conference paper at ICLR 2021
(a) cISNE embedding on single cell data with color (b) Dendogram complete linkage clustering of accordint to cell labels.
samples according to marker gene expression.
Figure 16:
Embedding of cSNE and clustering according to marker genes. a) Visualized is the embedding obtained from etSNE for the single cell data with our cluster labels as prior. The coloring according to cell type. b) Dendogram of agglomerative clustering with complete linkage on marker gene expression. A natural cutting point is at 1.8 (blue horizontal line), which retrieves the same number of clusters as the orginal paper.
(a) Embedding CONFETTI with (b) Embedding of JEDI with marker tissue prior.
(c) Embedding of vanilla UMAP with batch ID coloring.
gene prior.
Figure 17: Additional SC embeddings. Visualized is the embedding of CONFETTI for the single cell data with the tissue type (blood vs CSF) as prior with coloring according to cell labels (a), the embedding obtained from JEDI for the single cell data with marker gene expression as prior, with coloring is according to the cell labels (b), and the original UMAP embedding with coloring according to batch ID (c).
30


Reviews:
[{"review_id": "z1qTygAX_C", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Description:\n\nThis paper aims to \"factor out\" existing prior knowledge from embeddings, by adapting a tSNE objective or from other methods by operating on input distances. The prior knowledge is assumed to be pairwise distances.\n\nIn the tSNE case, a neighborhood distribution is derived from the prior pairwise distances, and a \"parameterized Jensen-Shannon divergence\" is proposed to measure deviation of the output neighborhood from that. The objective is then a difference of the tSNE objective and that divergence, optimized by gradient descent.\n\nIn the general case, it is proposed to edit the pairwise distances of the data by, substracting out the pairwise distance given in the prior information (up to some constants), for all pairs of different points. A metric property of the definition is proven, and a proof that independent prior distances would not on expectation change the neighborhoods.\n\nExperiments compare the result to conditional tSNE and to a modification of supervised LLE, on synthetic gaussian-cluster data and on two real data (flowers and single cell sequencing data). \n- On synthetic data, results are better than conditional tSNE but difference to the modified supervised LLE seems small. \n- On flower data, there are no comparisons to others except unmodified tSNE, but the proposed methods may be able to show structure beyond the prior as claimed.\n- In the single cell sequencing data, the comparisons are to unmofidied UMAP and to ctSNE. The UMAP with modified distances seems to show some batch-d and tissue-type differences.\n\nEvaluation:\n\nThe idea of factoring out prior knowledge seems very meaningful, although not completely new. \n\nThe approaches here seem reasonable but are somewhat simplistic:\n- The tSNE approach seems reasonable but not very striking: essentially it is a weighted sum of two cost functions, and most of the detail is just to ensure the new divergence does not overwhelm the original one.\n- The distance editing seems rather ad-hoc; while the result may have a metric property, it is hard to say in what sense this is the \"right\" way to combine the prior information to the distances.\n\nThe experiments are rather limited, which is somewhat disappointing; in the flower case no comparison to ctSNE or SLLE-1 is done, or to SLLE-1 in the single cell sequencing data (too large matrix inversion issues are claimed - could those have been resolved e.g. by computing the result for a subset instead?). The results do seem to show ability to show information beyond the prior knowledge, but for a modern dimensionality reduction paper I would expect a more thorough evaluation.\n\nComments:\n\n- Neighbor embeddings that explore beyond known annotation have been proposed long before conditional tSNE: e.g. the method of [1] includes extraction of gene expression features unrelated to an ontological annotation, to embed structure not explained by that annotation.\n\n[1] Peltonen J, Aidos H, Gehlenborg N, Brazma A, Kaski S. An information retrieval perspective on visualization of gene expression data with ontological annotation. ICASSP 2010.\n\n- The equation in definition 1 seems to be wrong. In the second divergence term, \"P' + (1-beta)Q\" is not a proper distribution that sums to 1.\n\n- In the single cell sequencing data, you used marker gene expression for \"Z\" data, but what was the X data? Clarify.\n\n- Figure 1: marker shapes are far too small to be seen at a normal viewing factor or in a printed paper.\n\n- In experiments authors write that JEDI is \"not designed for labels as priors\" but in the introduction you write \"we consider background knowledge in the form of pairwise distances between samples. This formulation allows\nus to cover a plethora of practical instances including labels\". In what sense is JEDI not designed for labels as priors?\n\n- The flower experiment seems somewhat biased because the input distance is a linear sum of distances including color-distance: therefore the distance-editing by subtracting the color-distance is of course well matched to this construction.\n\n- In the single-cell sequencing case, to use ctSNE authors \"provide it the cluster labels from an agglomerative clustering\": why was this not done for the flower data too (and for SLLE-1 too)?\n\n- In the single-cell sequencing case, Figure 3(a) is hard to compare to (b) and (c) since the coloring means different things.\n\n- In Figure 3(c), why are there smooth changes of the batch-id colors? Are batches with close-by bacth numbers somehow expected to be more similar to each other?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ok6zsjZFAk", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents JEDI and CONFETTI, two approaches to the task of factoring prior knowledge when creating low dimensional embeddings for data exploration. The authors allow the prior knowledge to be specified as distances based on an information theoretic extension to tSNE and UMAP.\n\nThe task is interesting and paper is well-presented. The theory section is a bit weak, although it provides an intuition of why those specific extra terms were added to the optimization problems. \n\nI found the experiments interesting -- the three case studies are well thought out. However, I find the comparison with the competitors unfair in most cases, as the information that it is provided to those is quite different. In this case, I think the paper would benefit from not comparing to those methods but to use simple baselines of projections that de-bias the embeddings to illustrate the advantages of using either JEDI or CONFETTI.\n\nFinally, I would like to have seen a discussion on how realistic it is that the prior knowledge would come in the form expected by the authors in real-world problems. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "M1eZJwHVGg0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThe author(s) provide two methods for factoring out specific covariates from tSNE, UMAP or other distance matrices. The first one is JEDI, an extension of tSNE that minimizes a parameterized divergence (that takes into account the information to be factored out) instead of the simple KL divergence between high dimensional data and low-dimensional embedding. Because tSNE has inherent limitations, the author(s) also propose CONFETTI, a simple approach to create a distance matrix based on the distance matrices of the covariates and the input data. This produces a proper distance metric and can be used upstream of any embedding procedure. On synthetic data, the proposed methods perform to the level of ctSNE as well as a second baseline sLLE-1. However, both methods are not directly meant to solve the original problem. Only JEDI and CONFETTI may factor out continuous variations (ctSNE is designed for discrete clusters). On real world data, the method seems to effectively reorganize embeddings (either of images, or of single cells) by factoring out variations of interest. \n\nStrong points:\n1. The paper is well-written and simple to understand\n2. The idea is conceptually interesting \n\nWeak points:\n1. The manuscript does not acknowledge foundational work. The idea of factoring nuisance variation out of representations is rather common, either in the setting of matrix factorization, Gaussian processes or variational autoencoders [1, 2]. This is especially true in the setting of high-dimensional genomics data, in which all the work around \"removing unwanted variations\"  (RUV) became a common theme (> 10 papers).  \n2. I am skeptical of the applicability of the method, especially regarding runtime. Much of the paper describes JEDI. However, JEDI takes 100 hours to run on the scRNA-seq data compared to tSNE (that should finish in less than 10 minutes, or at least be faster than ctSNE that runs in 20 minutes). \n3. The construction of the distance matrix for CONFETTI is interesting, but there is no effort in understanding in which sense such construction is efficient, optimal or even why it seems to be a reasonable choice. I do not think that the presented theory is helpful for understanding when the method would work well or not. \n4. In terms of experiments, a stronger point could be made by choosing a challenging setting (removing cell-cycles in single-cell RNA-sequencing data in order to better recover cell types as in [2]), and comparing against RUV models, to justify why corrections on the distance matrices is more efficient than linear corrections on the input space \n\n[1] Gagnon-Bartsch, J., Jacob, L. & Speed, T.P. Removing unwanted variation from high dimensional data with negative controls. Tech. Rep. 820, Department of Statistics, University of California, Berkeley (2013).\n[2] Buettner, F. et al. Computational analysis of cell-to-cell heterogeneity in single-cell RNA-sequencing data reveals hidden subpopulations of cells. Nat. Biotechnol. 33, 155–160 (2015).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AyE47WgzzW", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n=======\nThe distance metric learned by low-dimensional embeddings typically captures the knowledge that we already know. This paper proposes a principled way of factoring out prior knowledge (in the form of distance matrices) from tSNE and UMAP embeddings. Two algorithms are proposed for factoring out prior knowledge. JEDI (for tSNE embeds) uses a parameterized JS divergence-- the objective is to learn a low-dimensional distance metric that preserves high-dimensional distances but is orthogonal to the prior distance matrix. CONFETTI is the second algorithm which also optimizes a similar objective to JEDI but doesn't employ JS divergence and is algorithm independent, so one can use it for tSNE or UMAP. Results are shown on synthetic and real-world flower and cell-sequencing data and they highlight the superior ability of JEDI and CONFETTI algorithms in factoring-out prior knowledge compared to the baselines. \n\n\n\nComments:\n==========\nThe paper is well written and puts itself nicely in context of previous work. Given the ubiquity of low-dimensional embeddings these days, the paper addresses an important problem of factoring out prior information from the embeddings.\n\n\n1). The paper doesn't describe the details of optimizing the parameterized JS Divergence (pJSD) metric that they propose. Is it even convex?\n\n2). How is the beta parameter of pJSD chosen?\n\n3). I didn't fully understand why the paper makes a big deal about UMAP, when JEDI is based on tSNE and CONFETTI can work with any embedding formulation? How not just mention a \"general embedding.\"\n\n4). The formulation of the CONFETTI method seems a little arbitrary. Why are the prior distances factored out linearly? Again, no optimization details are provided regarding the CONFETTI method.\n\n5). It seems that, as an application, the proposed methods can also be used for factoring out demographic information from word embeddings. For such applications how would one define the prior distance matrix? The datasets used in the paper and the broader setup seems a little contrived. It would be nice to provide guidelines on how one can readily define such prior distance matrices for other applications. \n\n\nTypo:\n\nConclusion: \"This shows that both are able applicable to real world...\"  ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: A method is proposed for removing prior knowledge, presented as a
distance matrix, from low-dimensional embeddings, to focus them on
what is new.

The task of visualizing novely in data is interesting and good
solutions would potentially be highly useful.

The proposed method essentially substracts a distance matrix from
another. While this is sensible, it is not completely clear in what
sense this is _the_ right solution for what the embeddings will be
used for.

In final discussions among the reviews, the main remaining concerns
were considered severe: comparisons to other methods being limited,
and possible problems in one of the experiments.

Decision: Reject