dataesr/scientific-paragraphs-categorization huggingface dataset:
This general procedure of flat clustering, which is also employed for the two remaining cognate detection methods provided by LingPy, is illustrated in Fig 1A  and 1B. Cognate Detection using the Sound Class Algorithm. A third method available in the LingPy package, the SCA method, uses the same threshold-based clustering algorithm as the Edit Distance but employs distance scores derived from the Sound-Class Based Alignment Table  3. Recent approaches to cognate detection. A plus "+" indicates that the algorithm meets the requirement, a minus "-" indicates that its failure. ML (multilingual) refers to the ability of an algorithm to identify cognate words across more than two languages at the same time. RQ (requirements) refers to additional requirements apart from the raw word list data, such as needing reference phylogenies or extensive training data.
Cognate Detection Cognate Detection using the LexStat method. The last publicly available method we tested, the LexStat method, is again based on flat UPGMA clustering, but in contrast to both the Edit-Distance method and the SCA method, it uses language-specific scoring schemes which are derived from a Monte-Carlo permutation of the data  [19]. This permutation, by which the wordlists of all language pairs are shuffled in such a way that words denoting different meanings are aligned and scored, is used to derive a distribution of sound-correspondence frequencies under the assumption that both languages are not related. The permuted distribution is then compared with the attested distribution, and converted into a language-specific scoring scheme for all language pairs. Using this scoring scheme, the words in the data are aligned again, and distance scores are derived which are then used as the basis for the flat cluster algorithm.
Differences between algorithms. In order to illustrate the differences between these four algorithms, we analysed the test set by  Kessler [49]. This dataset is particularly interesting for the task of cognate detection, since the sample of languages contains not only four Indo-European languages with different degrees of genetic affiliation, but also unrelated languages from different language families. When running the algorithm with default thresholds as proposed in List  [19] , LexStat performs best, showing the smallest amount of false positives and false negatives, followed by SCA, Edit-Distance, and Turchin. When looking at specific results of this analysis, like the cognate judgments for the concept 'there', given in Table  4 , for example, we can immediately see the shortcomings of the language-independent methods.
The Turchin method (T), for example, links Albanian [aty] and Navajo [ʔaːdi] as cognate, where these are a clear chance resemblance in the consonant class structure. Note that initial vowel is treated identical with initial glottal stop in the Turchin method, following the original sound class proposal by  [45]. The Edit Distance (E) method also identifies a chance resemblance by proposing that French [la] and Hawaian [laila] are cognate. The Edit-Distance method is especially prone to identifying chance similarity as cognacy, and this risk increases as languages get more and more different  [15]. The threshold of the SCA method (S) is too low to identify any cognate set for the concept 'there'. Only the LexStat method (L) correctly identifies English [ðεr] and German [daː] as cognates, but not due to the phonetic similarity of the words, but due to the fact that matches of English [ð] and German  [d]  recur frequently in the dataset.
Scholars often use pairwise scores  [32]  for evaluation. In these scores, all words in a concept slot are assembled into pairs. The pair score is then calculated by comparing how many pairs in the gold standard are identically clustered by the algorithm, and vice versa. This is simple and straightforward, since, for pairs, there are only two possible decisions, namely whether they are cognate or not. We can then simply count how many pairs in the gold standard are also judged to be cognate by the algorithm, or how many pairs proposed to be cognate by the algorithm are also cognate according to the gold standard. The advantage of this score is that we can directly convert it into an intuitive notion of false positives and false negatives versus true positives and true negatives. Breaking down the comparison of two different clusters into pairs is, however, problematic, since it has a strong bias in favoring datasets containing large amounts of non-cognate words  [19].
Threshold and Parameter Selection Apart from the Turchin method, all analyses require a threshold which ranges between 0 and 1, denoting the amount of similarity needed to judge two items as cognate. In order to find the most suitable threshold for each of the three methods, we used the expert cognate decisions in our training set and ran the analyses on these data with varying thresholds starting from 0. 05 up to 0. 95. the threshold. As can be seen from this figure, all methods show a definite peak where they yield the best results for all datasets. In order to select the best threshold for each of the four methods, we selected the threshold which showed the best average B-Cubed F-Score (i. e. the best accuracy at recovering the known cognate sets). For the Edit Distance Method, the threshold was thus set to 0.
Results We analyzed the datasets with each of the five methods described above, using the individual thresholds for each method, setting the number of permutations to 10,000, and using the default parameters in LingPy. For each analysis, we further calculated the B-Cubed scores to evaluate the performance of each method on each dataset. Table  7  shows the averaged results of our experiments. While the LexStat method shows the highest precision, the Infomap method shows the highest recall and also the best general performance. The results are generally consistent with those reported by List  [19]  for the performance of Turchin, Edit Distance, SCA, and LexStat: The Turchin method is very conservative with a low amount of false positives as reflected by the high precision, but a very large amount of undetected cognate relations as reflected by the low recall.
The Edit Distance method shows a much higher cognate detection rate, but at the cost of a high rate of false positives. The SCA method outperforms the Edit Distance, thus showing that refined distance scores can make a certain difference in automatic cognate detection. However, as the performance of LexStat and Infomap shows: Language-specific approaches for cognate detection clearly outperform language-independent approaches. The reason for this can be found in the specific similarity measure that is employed by the methods: the better performing methods are not based on surface similarities, but on similarities derived from previously inferred probability scores for sound correspondences. These methods are therefore much closer to the traditional comparative method than methods which employ simple surface similarities between sounds.
Dataset Specific Results There are interesting differences between method performance across language datasets, with marked variation in cognate identification accuracy between different languages. Fig  3  shows the performance of the methods on the individual test sets, indicating which method performed best and which method performed worst. These results confirm the high accuracy of the LexStat method and the even better accuracy of the Infomap approach. All methods apart from the Turchin method perform the worst on the Chinese data. Since compounding is very frequent in Chinese, it is difficult to clearly decide which words to assign to the same cognate set. Often, words show some overlap of cognate material without being entirely cognate. This is illustrated in Fig  4 , where cognates and partial cognates for Germanic and Sinitic languages are compared.
This shortcoming of the Turchin approach is illustrated in Fig  5 , where the amount of true positives and negatives is contrasted with the amount of false positives and negatives in each dataset and for each of the five methods. This figure indicates that the Turchin method shows exceptionally high amounts of false negatives in Bahnaric and Romance. The clear advantage of the Turchin method is its speed, as it can be computed in linear time. Its clear disadvantage is its simplicity which may under certain circumstances lead to a high amount of false negatives. The Edit-Distance method also performs very poorly. While, on average, it performs better than the Turchin approach, it performs considerably worse on the Chinese and Huon test sets. The reason for this poor performance can be found in a high amount of false positives as shown in Fig 5.
Discussion In this study we have applied four published methods and one new method for automated cognate detection to a set of six different test sets from five different language families. By training our data on an already published dataset of similar size, we identified the best thresholds to obtain a high accuracy for detecting truly related words for four out of the five methods (Edit-Distance: 0. 75, SCA: 0. 45, LexStat: 0. 6, Infomap: 0. 55). Using these thresholds, we tested the methods on our new gold standard, and found that most methods identified cognates with a considerable amount of accuracy ranging from 0. 82 (Tuchin) to 0. 89 (Infomap). Our new method, which builds on the LexStat method but employs the Infomap algorithm for community detection to partition words into cognate sets, outperforms all other methods in almost all regards, slightly followed by the LexStat approach.
Given that the LexStat method and our Infomap approach are based on language-specific language comparison, searching for similar patterns in individual language pairs, our results confirm the superiority of cognate detection approaches which are closer to the theoretical foundation of the classical comparative method in historical linguistics. The Consonant Class Matching method by Turchin et al. confirmed worst in our experiment, followed by the Edit-Distance approach, which was criticized in earlier work  [15]. While the major drawback of the Turchin approach is a rather large amount of false negatives, the Edit-Distance approach shows the highest amount of false positives in our test. The method of choice may well depend on the task to which cognate detection is to be applied. If the task is to simply identify some potential cognates for future inspection and annotation, then a fast algorithm like the one by Turchin et al.
should provide enough help to get started. This practice, which is already applied by some scholars  [64] , is further justified by the rather small amount of false positives. While the use of the Turchin method may be justified in computer-assisted workflows, the use of the Edit-Distance approach should be discouraged, since it lacks the speed advantages and is very prone to false positives. When searching for deeper signals in larger datasets, however, we recommend using the more advanced methods, like SCA, LexStat or our new Infomap approach. LexStat and Infomap have the great advantage of taking regular sound correspondences into account. As a result, these methods tend to refuse chance resemblances and borrowings. Their drawback is the number of words needed to carry out the analysis.
As we know from earlier tests  [65] , language-specific methods require at least 200 words for moderately closely related languages. When applied to datasets with higher diversity among the languages, the number of words should be even higher. Thus, when searching for cognates in very short word lists, we recommend using the SCA method to achieve the greatest accuracy. However, as demonstrated by the poorer performance of all methods on the Chinese language data where compounding has played a major role in word formation, language family specific considerations about the methods and processes need to be taken into consideration. Our results show that the performance of computer-assisted automatic cognate detection methods has advanced substantially, both with respect to the applicability of the methods and the accuracy of the results.
A key problem to solve is the performance of these methods on larger datasets that trace language relationships to a greater depth. Most of our test cases in this paper are shallow families or subgroups of larger families. Deeper relationships between languages spoken in more complicated language situations are where the real challenge lies. Currently automatic cognate detection algorithms are highly accurate at detecting a substantial proportion of the cognates in a lexical dataset. Tools like LingPy are already at a stage where they can act as a computer-assisted framework for language comparison. These tools therefore provide a powerful way of supplementing the historical linguistics toolkit by enabling linguists to rapidly identify the cognate sets which can then be checked, corrected, and augmented as necessary by experts.
Acknowledgments We thank the anonymous reviewers for helpful advice. We thank  Outi Vesakoski  and  Jury Lehtinen  ( Uralic data  from the  Uralex  project),  Paul Sidwell (Bahnaric), George Starostin (Tujia) , and  M. Saenko  ( Romance ) for sharing their data with us by either exchanging them directly or making them accessible online.
However, several authors have reported on shortcomings of PCA for scRNA-seq data. In particular, for real datasets, the first or second principal components often depend more on the proportion of detected genes per cell (i. e. , genes with at least one read) than on an actual biological signal  [22, 23]. In addition to PCA, dimensionality reduction techniques used in the analysis of scRNA-seq data include independent components analysis (ICA)  [15] , Laplacian eigenmaps  [18, 24] , and t-distributed stochastic neighbor embedding (t-SNE)  [2, 4, 25]. Note that none of these techniques can account for dropouts, nor for the count nature of the data. Typically, researchers transform the data using the logarithm of the (possibly normalized) read counts, adding an offset to avoid taking the log of zero.
Recently, Pierson & Yau  [26]  proposed a zero-inflated factor analysis (ZIFA) model to account for the presence of dropouts in the dimensionality reduction step. Although the method accounts for the zero inflation typically observed in scRNA-seq data, the proposed model does not take into account the count nature of the data. In addition, the model makes a strong assumption regarding the dependence of the probability of detection on the mean expression level, modeling it as an exponential decay. The fit on real datasets is not always good and, overall, the model lacks flexibility, with its inability to include covariates and/or normalization factors. Here, we propose a general and flexible method that uses a zero-inflated negative binomial (ZINB) model to extract low-dimensional signal from the data, accounting for zero inflation (dropouts), over-dispersion, and the count nature of the data.
ZINB-WaVE leads to biologically meaningful clusters We applied the ZINB-WaVE procedure to several publicly-available real datasets, from microfluidics, platebased, and droplet-based platforms (see Methods). As previously shown Hicks et al. [22]  and Finak et al. [23] , the first few principal components of scRNAseq data, even after normalization, can be influenced by technical rather than biological features, such as the proportion of genes with at least one read (detection rate) or the total number of reads (sequencing depth) per sample. Figure  2  illustrates this using the V1 dataset: although the first two principal components somewhat segregated the data by layer of origin (Fig. 2a ), the clustering was far from perfect. This is at least partly due to unwanted technical effects, such as sequencing depth and amount of starting material.
To quantify such technical effects, we computed a set of QC measures, such as detection rate and total number of reads (see Methods). The first two principal components are especially correlated with detection rate (Fig. 2b ). ZIFA suffered from the same problem: the clustering of the samples in two dimensions was not qualitatively different from PCA (Fig. 2c ) and the second component was highly correlated with detection rates (Fig. 2d ). Conversely, ZINB-WaVE led to tighter clusters, grouping the cells by layer of origin (Fig. 2e ). Furthermore, the two components inferred by ZINB-WaVE showed lower correlation with the QC features (Fig. 2f ), highlighting that the clusters shown in Figure  2e  are not driven by technical effects. We repeated these analyses on the S1/CA1 dataset (Supplementary Fig.
S1 ), mESC dataset (Supplementary Fig. S2 ), and Glioblastoma dataset (Supplementary Fig. S3 ). For all datasets, ZINB-WaVE led to tighter clusters in two dimensions. However, it did not always lead to a decrease in correlation with the QC measures. See also Hicks et al. [22]  for additional datasets in which principal components are strongly correlated with detection rate. As a measure of the goodness of the clustering results, we used the average silhouette width (see Methods), computed using the labels available from the original study: these were either known a priori (e. g. , the patient ID in the Glioblastoma dataset) or inferred from the data (and validated) by the authors of the original publication (e. g. , the cell types in the S1/CA1 dataset). For all four datasets, ZINB-WaVE led to generally tighter clusters, as shown by an increased per-cluster average silhouette width in the majority of the groups (Supplementary Fig.
See Perraudeau et al. [31]  for a complete workflow that involves ZINB-WaVE for dimensionality reduction and Slingshot for lineage reconstruction. We next focused on a set of 68,579 peripheral blood mononuclear cells (PBMCs) assayed with the 10x Genomics Chromium system  [32]. This example allowed us to demonstrate how ZINB-WaVE can be applied to a state-of-the-art dataset that comprises tens of thousands of cells, while also illustrating that existing clustering algorithms can be used on the low-rank matrix inferred by ZINB-WaVE. In particular, we applied a popular clustering method, based on the identification of shared nearest neighbors  [33] , similar to that of Macosko et al. [2]  and implemented in the R package Seurat  [34]. We modified the clustering procedure to use ZINB-WaVE (with K = 10) instead of PCA as the dimensionality reduction step.
g. , clustering, differential expression analysis, or pseudotime ordering). In this work, we considered three popular between-sample normalization methods: total-count (TC), trimmed mean of M values (TMM), and full-quantile (FQ) normalization (see Methods); we also compared the results to unnormalized data (RAW). TC, along with the related methods transcripts per million (TPM) and fragments per kilobase million (FPKM), is by far the most widely used normalization in the scRNA-seq literature; hence, TC-normalized data were used for the results shown in Figure  2  and in Supplementary Figures  S1, S2,  and S3. Normalization highly affected the projection of the data in lower dimensions (Supplementary Fig. S10  -S15 ) and, consequently, the clustering results varied greatly between normalization methods in all four datasets (Fig.
4 ). Strikingly, the choice of normalization method was more critical than the choice between PCA and ZIFA. For instance, for the mESC dataset (Fig. 4d ), FQ normalization followed by either PCA or ZIFA led to very high average silhouette width. Critically, the ranking of normalization methods was not consistent across datasets (Fig. 4 ), highlighting that the identification of the best normalization method for a given dataset is a difficult problem for scRNA-seq  [11, 36]. One important feature of the ZINB-WaVE model is that the sample-level intercept γ (corresponding to a column of ones in the gene-level covariate matrix V , see Methods) acts as a global-scaling normalization factor, making a preliminary global-scaling step unnecessary. As a consequence, ZINB-WaVE can be directly applied to unnormalized read counts, hence preserving the distributional properties of count data.
ZINB-WaVE applied to unnormalized counts led to results that were comparable, in terms of average silhouette width, to PCA and ZIFA applied using the best performing normalization (Fig. 4 ). In particular, ZINB-WaVE outperformed PCA and ZIFA on the S1/CA1 (Fig. 4b ) and Glioblastoma (Fig. 4c ) datasets, while it was slightly worse than PCA (after FQ normalization) on the mESC dataset (Fig. 4d ) and than ZIFA (after FQ normalization) on the V1 dataset (Fig. 4a ). Interestingly, the overall average silhouette width was lower for the S1/CA1 and Glioblastoma datasets than it was for the V1 and mESC datasets, suggesting that ZINB-WaVE leads to better clustering in more complex situations (e. g. , lower sequencing depth). Although the overall average silhouette width is a useful metric to rank the different methods in terms of how well they represent known global biological structure, looking only at the average across many different cell types may be misleading.
For the V1 dataset, for instance, ZINB-WaVE led to a slightly lower overall average silhouette width than ZIFA (Fig. 4a ). However, ZINB-WaVE yielded higher cluster-level average silhouette widths for the L6a, L5b, and L5 samples, while ZIFA produced higher average silhouette widths for the L4 and L5a samples (Supplementary Fig. S5a ). In fact, certain cell types may be easier to cluster than others, leading to silhouette widths that differ greatly between different clusters, as is the case for the S1/CA1 dataset: oligodendrocytes are much easier to cluster than the other cell types and all methods were PCA and ZIFA were applied after normalization: unnormalized (RAW), total-count (TC), trimmed mean of M values (TMM), and full-quantile (FQ). For the mESC dataset (d), we fitted the ZINB-WaVE model with batch as a sample-level covariate (ZINB-Batch) in addition to the default model with only a sample-level intercept (see Fig.
Accounting for batch effects Although the sample-level intercept γ (corresponding to a column of ones in the gene-level covariate matrix V ) can act as a global-scaling normalization factor, this may not be sufficient to accurately account for complex, non-linear technical effects that may influence the data (e. g. , batch effects). Hence, we explored the possibility of including additional sample-level covariates in X to account for such effects (see Methods). We illustrate this using the mESC dataset, which is a subset of the data described in Kolodziejczyk et al. [37]  and which comprises two batches of mESCs grown in three different media (see Methods). ZINB-WaVE applied with no additional covariates led to three clusters, corresponding to the three media (Fig. 5a  and Supplementary Fig.
S16 ). The advantage of ZINB-WaVE is the ability of including batch effects in the same model used for dimensionality reduction, without the need for prior data normalization. The mESC dataset is an example of good experimental design, where each batch includes cells from each biological condition (this is known as a factorial design). Hence, it is relatively easy to correct for batch effects and, unsurprisingly, both ComBat and ZINB-WaVE successfully do so. The Glioblastoma dataset is an example of a more complex situation, in which there is confounding between batch and biology, each patient being processed separately  [22]. Luckily, the Glioblastoma design is not completely confounded, as patient MGH26 was processed in two batches (Fig. 5e ). We used this feature of the data to test whether ZINB-WaVE was able to adjust for batch effects even in the presence of confounding.