dataesr/scientific-paragraphs-categorization huggingface dataset:
Third, it contains no information about the quality of preservation of global similarities, e. g. positions of clusters relative to each other in the hierarchy. Instead, we use a scale-independent quality criteria  34  (see Methods and Supplementary Note 2), which was demonstrated to be a good metric to compare embeddings in an unbiased way. The criteria consists of estimating two scalar values Q local and Q global reflecting local and global properties of the dataset. We follow the assumption of Lee et al. 34  that a single-cell dataset comprises a smooth manifold and a good dimensionality reduction method would preserve local and global distances on this manifold. An important result from our experiments is that Poincaré maps was the only method that demonstrated the ability to visualize the correct branching structure of developmental processes for all datasets in terms of this quality metric (Fig.
3   2 ). Separate visual comparison of various embeddings (Supplementary Fig. 4 5 6 7 8 9 10 ) demonstrates the superior readability advantages of Poincaré maps. For example, on the dataset Paul et al. 2  only Poincaré maps and t-SNE identify the lymphoid cluster, while this important population remained invisible during exploratory data analysis when using UMAP or ForceAtlas2. Although t-SNE visualizes separate clusters well for Paul et al. dataset, it disregards the hierarchical structure between clusters (see also example in Supplementary Fig. 8 ). Knowledge of the position of a newly identified cluster in the developmental hierarchy could be further exploited for assigning labels (e. g. "lymphoid population") or, when the population was not known, for designing experiments to test morphological properties.
Finally, Poincaré maps places the 16Neu cluster downstream of 15Mo in the hierarchy -in contrast to the canonical hierarchy, where neutrophils and monocytes are located at the same level. This result is in line with the analysis of Wolf et al. 21 , indicating that the inconsistency is due to a faulty labeling of the clusters. A back-to-back comparison of a quality metric with visual inspection gives a good intuition about the meaning of the metric quality 4 Figure  3  | Comparison of embedding quality metric (best case) for various datasets. Poincaré maps perform consistently well on all synthetic and real-world datasets in our evaluation. Planaria and C. elegans datasets -which exhibit the highest complexity in terms of number of branching trajectories-are datasets where Poincaré maps perform significantly better than other methods.
scores and embedding properties. In particular, on datasets with simple trajectories (e. g. , ToggleSwitch, MyeloidProgenitors), methods such as PCA or SAUCIE show strong performance since they preserve local similarities well. However, the performance of these methods drops significantly for the datasets with many different cell types and branches, such as Planaria and C. elegans. Poincaré maps in their turn, don't suffer from this limitation (Fig. 3 ) and significantly outperforms other methods in terms of both local and global metrics. This allows us to summarize the whole C. Elegans cell atlas in a single Poincaré maps embedding (Fig. 2 ). This was not possible with UMAP with any choice of parameters, as reported by the authors in their original study. This makes Poincaré maps a strong candidate for visualization of single-cell atlases.
Poincaré maps generate new hypothesis about early blood development in mice As a deeper case study, we analyze the dataset of early blood development in mice, previously studied by Moignard et al. 1  , using Poincaré maps. This dataset contains measurements of cells captured in vivo with qRT-PCR at different development stages: primitive streak (PS), neural plate (NP), head fold (HF), four somite GFP (Runx1) negative (4SG-) and four somite GFP positive (4SG+) (Fig. 4 (a) ). The stages correspond to different physical times of the experiment between embryonic day 7 and day 8. 25. We compare our results obtained with Poincaré maps to Moignard's diffusion maps study  1   pseudo-time  16. Poincaré maps provide a qualitatively different visualization of the developmental process, where we are able to visualize the whole spectrum of the heterogeneity arising from the onset of the process.
However, as we demonstrate in Supplementary Fig. 13  -14, assigning this cluster as the root of the hierarchy would lead to a contradiction with the physical direction of time. By virtue of the Poincaré visualization, we reassigned the root of the developmental process to the furthest PS cell not belonging to the "mesodermal" cluster. We picked up a root cell from PS as to ease clustering by angle for lineage detection. More 6 specifically, we chose the most "exterior" cell from the PS cluster, by visual inspection. Given our reassigned root, we separate the dataset into five potential lineages (see Methods), to find the asynchrony in the developmental process in terms of marker expressions (Fig. 4 (b) ). Analysis of the composition of cells belonging to each lineage (Fig. 4 (c )) indicates that erythroid cells belong only to lineage 0 and this lineage contains no endothelial cells.
For this purpose, Poincaré maps aids the discovery of complex hierarchies from single-cell data by embedding large-scale cell measurements in a two-dimensional Poincaré disk. The resulting embeddings are easy to interpret during exploratory analysis and provide a faithful representation of similarities in hierarchies due to the underlying geometry. This property makes Poincaré maps stand out among other embeddings as it allows to simultaneously handle visualization, clustering, lineage detection, and pseudo-time inference. Poincaré maps do not need to be constrained to two dimensions, and would have the same implementation for 3 or dimensions. However, for the datasets used in this study two dimensions were sufficient; using more dimensions would reduce readability and harden interpretation.
g. close neighbours should be placed close to each other, while maintaining large distances between distant points. Lee et al. 34  proposed to use two scalar quality criteria Q local and Q global focusing separately on low and hight dimensional qualities of the embedding. The quantities of Q local and Q global range from 0 (bad) to 1 (good) and reflect how well are local and global properties of the dataset are preserved in the embedding (see details in Supplementary Note 2). To estimate distances in the high-dimensional space ij , we use geodesic distances estimated as length of a shortest-path in a k-nearest neighbours graph. We fixed k = 20 for all the datasets as there is no objective way to decide on a correct k and visually results looked good for all the embeddings for this choice of k.
Choice of Hyperparameters Computational complexity and time Memory complexity of Poincaré maps is O(n 2 ), where n is the number of samples. Time complexity consits of three parts: estimation of kNNG -O(n 2 ) (this part could be replaced with FAISS  38  for scalability), estimation of RFA -O(n 2 ) and minimization of KL-divergence -O(neb), where e -maximum number of epochs, b -batch size. As we need to minimize KL till convergence, we can in advance estimate the number of epochs needed. For all the datasets used here the number of epochs was less than 2000 and we also used early stopping upon convergence. Typical running time on 1 GPU for all the small-medium datasets is less than a minute, and for large datasets around 15 min (Planaria) or 2-3 hours (C. elegans). Supplementary Notes for: Poincaré Maps for Analyzing Complex Hierarchies in Single-Cell Data February 25, 2020   Supplementary Note 1: Poincaré maps for learning hierarchical representations Hyperbolic space is a Riemannian manifold whose structure is well-suited to represent hierarchical and tree-like relationships.
Scale-independent quality criteria To quantitatively compare the performance of different embedding approaches, we use a scaleindependent quality criteria proposed by Lee et al. [12] The main idea is that a good dimensionality reduction approach will have a good preservation of local and global distances on the manifold, e. g. close neighbors should be placed close to each other, while maintaining large distances between distant points. Below we provide a short summary of how to compute this metric. All the details can be found in the original paper of Lee et al. [12]  Let X = {x i } N i=1 be a high-dimensional dataset of N samples x i 2 R p (e. g. , individual cells) with p features (e. g. , gene expression measurements) and Y = {y i } N i=1 be a low-dimensional representation of this dataset in m = 2 dimensions.
In this work, to estimate distances ij in the high-dimensional space, we use geodesic distances estimated as the length of a shortest-path in a k-nearest neighbors graph. We fixed k = 20 for all the datasets, as there is no objective way to decide on a correct k, and visually results looked good for all the embeddings for this choice of k. For the distances ij in the low-dimensional space we use euclidean space for all the embeddings except Poincaré maps, for which we use hyperbolic distances. As all the embeddings involve an element of stochasticity in their output, we run every embedding three times with a different seed. We run all the embeddings with a different set of parameters in the range proposed by the authors of each method. Supplementary Figure  2  demonstrates the comparison of Q local and Q global for all the datasets described below.
Robustness of Poincaré maps to random seed and choice of hyper-parameters choice We used the quality criteria described above and visual inspection to address the robustness of Poincaré maps to hyper-parameters choice and random seed. Supplementary Figure  3 (a)  demonstrates that good values of vary for different datasets. However, the parameter has a less strong effect and rather controls how much the embedding will be scattered on the disk. We advise to set to 1. 0 or 2. 0 depending on the dataset size: larger datasets typically have better visualization with = 2. 0 (Supplementary Figure  3 (b-c )). Supplementary Figure  3 (a, d ) demonstrates that Poincaré maps are very robust to random seed and that it doesn't significantly affect neither quality, nor visual interpretation. Supplementary Figure  2.
Synthetic datasets To demonstrate the performance of Poincaré maps we used several synthetic datasets available as Jupyter notebooks with Scanpy [13]: a simple toggle switch, myeloid progenitors and myeloid progenitors with Gaussian blobs. These datasets were previously used to demonstrate the performance of diffusion maps and graph abstractions, and constitute great examples of manifolds with a hierarchical structure of increasing levels of complexity. All models consist of Boolean equations, which were translated into ordinary differential equations and simulated with Scanpy as stochastic differential equations with Gaussian noise [14]. A simple toggle switch model [15, 16] is a process with two branches, which are defined by the expression of two markers. Supplementary Figure  4  demonstrates that all competing methods produce rather correct results for this simple problem.
However, Poincaré maps gives a more clear separation of the intermediate states of terminal fates (inter1 vs inter2). In this example, only tSNE, diffusion maps, and Poincaré maps produce embeddings with meaningful pairwise distances. A synthetic dataset for myeloid differentiation [17] represents cell differentiation progresses of a common myeloid progenitor state towards one of four different branches: erythrocyte, neutrophil, monocyte and megakaryocyte. Supplementary Figure  5  shows the provided embeddings for all methods. Poincaré maps produce an embedding which is visually similar to the other methods, but has neither discontinuities, nor overlaps in the trajectories, since it preserves all the pairwise distances. Given the known root, the rotation of the Poincaré map (by means of translation) allows to easily read out the hierarchy.
Diffusion maps produce embeddings consistent with one main branch, but more Euclidean dimensions would be necessary to separate the rest. Monocle 2 produce a tree layout consistent with the hierarchical structure of the data, but is not able to reconstruct the temporal connection (trajectory) of the cell differentiation process. The third dataset shows the stability of Poincaré maps with respect to the existence of clusters not related to the main cell development process. To this end we use the synthetic dataset of myeloid differentiation with two Gaussian blobs, added as proposed by Wolf et al. [9] (Supplementary Figure  6 ). None of the benchmark methods except ForceAtlas2 is able to capture the hierarchy.
Mouse myelopoesis dataset (single-cell RNA seq) To demonstrate the performance of Poincaré maps on single-cell RNA seq data, we used the mouse myelopoesis dataset (wild type only) from Olsson et al. [18]. The data was downloaded and preprocessed according to the pipeline from Qiu et al. [11]. The processed dataset contained 532 features for 382 cells. Nine cell types were annotated corresponding to the original study: HSCP-1 (haematopoietic stem cell progenitor), HSCP-2, megakaryocytic, erythrocytic, Multi-Lin* (multi-lineage primed), MDP (monocyte-dendritic cell precursor), monocytic, granulocytic and myelocyte (myelocytes and metamyelocytes). In order to obtain the best results for Monocle 2, we used the analysis pipeline provided by the authors (https://github. com/cole-trapnell-lab/ monocle2-rge-paper).
Mouse myeloid progenitors dataset (MARS-Seq) As an example of a dataset with multiple intermediate populations, we use a dataset provided by Paul et al. [20]. Myeloid progenitor cells were separated by sorting the c-Kit+ Sca1 lineage from mouse bone marrow and sequenced with MARS-seq. We followed the data preprocessing procedure recipe_zheng17 (Scanpy-recipe [21]), which selects the 1000 most highly-variable genes for 2730 cells. In the original study, the authors identify 19 clusters. We use these labels and canonical hematopoetic cell lineage tree (Supplementary Figure  8 (a) ) to compare the performance of all methods. We run all methods except Monocle 2 on the 20 top principal components of the preprocessed data. For Monocle 2, we used the Jupyter notebook provided by the authors (the lymphoid cluster was separated as described in the original study).
Supplementary Figure  8 (b)  shows the embeddings provided by Poincaré maps. For this dataset, the root is supposed to be at CMP cluster, which is not observed. We chose the root as the medoid (with respect to Poincaré distances) of the MEP and GMP clusters combined. Supplementary Figure  8 (c)  shows the hierarchy that could be read out from the Poincaré map. We would like to point out that Poincaré maps clearly separate lymphoid cells and dendritic cells as outliers, which agrees with the canonical tree as they are part of lymphoid lineage. None of the other methods (Supplementary Figure  8 (d) ) were able to capture this fact. Poincaré maps also suggest that some of the clusters (13-15) could be relabeled to better reflect the canonical hierarchy. After the removal of the lymphoid cluster, Monocle 2 captures the main lineage branching between the MEP and GMP lineages, but it does not separate dendritic cells, and destroys the eosonphils cluster.
Planaria dataset (Drop-seq) To demonstrate scalability of Poincaré maps to large datasets, we analyzed the entire Planaria dataset of Plass et al. [22]. The dataset comprises 11 individual experiments capturing a total of 21,612 cells with droplet-based single-cell transcriptomics (Drop-seq). To obtain the Poincaré maps we used the pre-processed data provided by the authors: https://nbviewer. jupyter. org/ github/rajewsky-lab/planarian_lineages/blob/master/paga/planaria. ipynb. The preprocessed dataset comes in the form of 50 principal components, which were used by the authors to apply tSNE, PAGA and ForceAtlas2. (Supplementary Figure  9 ) illustrates that Poincaré maps agree with tSNE and ForceAtlas2 embeddings, significantly outperform PCA and UMAP, and agrees with the PAGA hierarchy annotation (Figure  4  in Plass et al.
C. elegans dataset (10X Genomics) The C. elegans dataset from Packer et al. [23] is the largest dataset used in our experiments. Original dataset contained 84,625 single cells measured with 10x Genomics platform. We used the preprocessed version (batch corrected, 100 PCA components) of the data provided by the authors at https://github. com/qinzhu/VisCello. In the original study the authors used UMAP to visualize the data. We loaded the UMAP coordinates provided by the authors to perform fair comparison with Poincaré maps. 37 manually annotated labels of cell types were provided together with the dataset. We randomly down-sampled the dataset to 40,000 cells. The whole dataset represents > 60x oversampling of the 1,341 branches in the C. elegans embryonic lineage, therefore down-sampling should not destroy statistical properties of the dataset.
Clustering Poincaré maps provide embeddings useful beyond visualization. Since Poincaré maps preserve pairwise similarities, their embeddings are suitable for downstream tasks, such as clustering. We compared several clustering approached using Poincaré maps and benchmark embeddings. We also provide Louvain clustering and clustering in the original gene expression space. Since the datasets comprise several continuous trajectories and there is no true separation for progenitor populations of different branches, we used the Adjusted Rand Index (ARI) and Fowlkes-Mallows scores (FMS) to measure cluster quality. Adjusted Rand Index. The Adjusted Rand Index (ARI) is a function that measures the similarity between two cluster assignments. ARI is bonded bewteen [ 1, 1], where negative values correspond to independent labelings, similar clusterings have a positive ARI, and 1.
0 is the perfect match score. Lets denote C a ground truth class assignment and K the clustering. Adjusted Rand Index is defined through raw Rand Index (RI): RI = a + b C n samples 2 , ( 9 ) where a is the number of pairs of elements that are in the same set in C and in the same set in K, b is the number of pairs of elements that are in different sets in C and in different sets in K, and C n samples 2 is the total number of possible pairs in the dataset (without ordering). ARI is after adjusting for random labelings: ARI = RI E[RI] max(RI) E[RI] , ( 10 ) where a is the number of pairs of elements belonging to the same cluster in predicted and true labels, b is the number of pairs of elements belonging to different clusters in predicted and true labels, and C n samples 2 is the number of all possible combinations of pairs of elements in the dataset.
Fowlkes-Mallows scores. The Fowlkes-Mallows score FMI is defined as the geometric mean of the pairwise precision and recall: F MI = T P p (T P + F P )(T P + F N) , (11) where TP is the number of pairs of points that belong to the same cluster in both the true labels and the predicted labels (true positives), FP is the number of pairs of points that belong to the same clusters in the true labels but not in the predicted labels (false positives), and FN is the number of pairs of points that belong in the same clusters in the predicted labels but not in the true labels (false negatives). The FMI ranges from 0 to 1. A high value indicates a good similarity between two clusterings. More details on these metrics can be found at: https://scikit-learn. org/stable/modules/ clustering. html#clustering-evaluation Supplementary Table  1  shows the clustering results on synthetic datasets.
Pseudotime We demonstrated Poincaré pseudotime performance by comparison with real time and diffusion pseudotime on synthetic datasets. Supplementary Table  2  demonstrates that Poincaré pseudotime as well as diffusion pseudotime achieve high correlation scores with actual time on all synthetic datasets. This is unsurprising, since these two measures are related in their nature. The performance of both pseudotime approaches is probably bounded by the construction of kNNG. Technical details. In the myeloid progenitors dataset, to obtain a pair of points between which to draw the interpolation, we first perform a Louvain clustering of the shrinked dataset. Afterwards, we select two clusters from which we sample a corresponding pair: one cluster corresponding to mature neutrophils, and another cluster corresponding to early neutrophil progenitors.
In the Olsson et al. dataset, we use the previously annotated clusters, and randomly sample several pairs of points belonging to "HSPC-1" cluster, and to either the "Multi-lin", "Eryth" and "Meg" clusters. This corresponds to a potential position of "HSPC-2" cluster in the hierarchy. In the Plass et al. Planaria dataset, we removed parenchymal progenitors. We interpolate between the original cluster of parenchymal progenitors and either the "psap+ parenchymal cells", "pgrn+ parenchymal cells", or "ldlrr-1+ parenchymal cells". As for the network architecture, we used 5 fully connected layers with ReLU non-linearities. For UMAP and ForceAtlas2, we added batch normalization, as it showed better performance. demonstrate how Poincaré maps could be used to reveal the developmental structure in this process.
Robustness of Poincaré maps to random seed and hyperparameters choice for a fixed k = 20 (a) Scale-independent quality criteria for various hyperparameters and three random seeds. (b -c) Change of visual qualities of the embedding for a fixed and varying for ToggleSwitch (b) and Olsson (c) datasets. Red frame represents best quality score. (d) Comparison of robustness to random seed: and are fixed.
Comparison of various embeddings for the scRNAseq dataset of mouse myelopoesis (Olsson et al. ). (a) Canonical hematopoetic cell lineage tree. Colored circles correspond to the population colors from the dataset. (b) Poincaré map rotated with respect to the root. (c) Benchmark methods. Supplementary Figure 8. Comparison of various embeddings for the mouse myeloid progenitors MARS-seq dataset (Paul et al. ). (a) Canonical hematopoetic cell lineage tree. Colored nodes correspond to the population colors from the dataset. White nodes correspond to intermediate annotated states. (b) Rotated Poincaré map with respect to the root (medoids of MEP and GMP cluster). (c) Hierarchical relationships suggested by the Poincaré map. (d) Benchmark methods. To reproduce the Monocle 2 tree, the lymphoid cluster was removed.
Data availability Several public datasets were used in this study: three synthetic datasets generated with Scanpy, Olsson et al. (synapse ID syn4975060), Paul et al. (accession code GSE72857), Moignard et al. (accession code GSE61470), Plass et al. (accession code GSE103633, preprocessed data available at https://shiny. mdc-berlin. de/psca/), Packer et al. (preprocessed data available at https://github. com/qinzhu/VisCello).
Supplementary Figure 3. 3 Robustness of Poincaré maps to random seed and hyperparameters choice for a fixed k = 20 (a) Scale-independent quality criteria for various hyperparameters and three random seeds. (b -c) Change of visual qualities of the embedding for a fixed and varying for ToggleSwitch (b) and Olsson (c) datasets. Red frame represents best quality score. (d) Comparison of robustness to random seed: and are fixed.
2 Supplementary Figure 7. d 27 Comparison of various embeddings for the scRNAseq dataset of mouse myelopoesis (Olsson et al. ). (a) Canonical hematopoetic cell lineage tree. Colored circles correspond to the population colors from the dataset. (b) Poincaré map rotated with respect to the root. (c) Benchmark methods. Supplementary Figure 8. Comparison of various embeddings for the mouse myeloid progenitors MARS-seq dataset (Paul et al. ). (a) Canonical hematopoetic cell lineage tree. Colored nodes correspond to the population colors from the dataset. White nodes correspond to intermediate annotated states. (b) Rotated Poincaré map with respect to the root (medoids of MEP and GMP cluster). (c) Hierarchical relationships suggested by the Poincaré map. (d) Benchmark methods. To reproduce the Monocle 2 tree, the lymphoid cluster was removed.