dataesr/scientific-paragraphs-categorization huggingface dataset:
Phone recognition Similarly to LENA  (Xu et al. , 2008) , we explore the use of automatic phone recognition as a feature extractor for ALUC estimation. For this purpose, we use Allosaurus 2 , a languageindependent phone recognizer  (Li et al. , 2020). Allosaurus is a multilayer Long Short-Term Memory (LSTM) neural network model trained on 12 different languages for phone and phoneme recognition. Its output consists of a "universal" phone 2 Allosaurus is available for Python as a pip package called ipa-recognizer. This paper uses version 0. 0. 3 of the system. layer with an inventory of phone units that are shared among all training languages, followed by language-dependent mappings from the detected (allo)phones to language-dependent phonemes. The present system uses an Allosaurus configuration that first produces the most likely string of universal phones and silent breaks for a given input utterance, and then, for dimensionality reduction, maps them onto English phoneme categories that best match with the phone activations.
Each intermediate layer and the deepest convolutional layer feed into a shared integrator module ("PostNet") with a recurrent layer that is responsible for transforming the inputs from the convolutional layers into a final syllable count estimate (see  Seshadri & Räsänen, 2019, for details). The output of SylNet is the estimated number of syllables in each input utterance, and this number is directly used as a feature fi gd ,5 in ALUC estimation. ALICE uses the SylNet baseline model from the original paper that is trained on approximately 10 hours of handannotated Estonian and Korean speech, but further adapting the model to the present daylong child-centered data using the standard adaptation procedure of the algorithm (see Experimental Setup for details). In our initial experiments on child-centered audio, SylNet was also compared to syllabifiers from  Wang and Narayanan (2007) ,  Räsänen, Doyle, and Frank (2018a), and Räsänen, Seshadri, and Casillas (2018b).
Data The data for our experiments consisted of seven childcentered daylong audio recording corpora. These include the Bergelson corpus ("BER") from US English-speaking families from New York state  (Bergelson, 2016) , the LuCiD Language 0-5 corpus ("L05") consisting of UK Englishspeaking families from Northwest England  (Rowland et al. , 2018) , the Casillas Tseltal corpus ("TSE") of Tseltal-speaking families from a rural Mayan community in Southern Mexico  (Casillas, Brown, & Levinson, 2017a) , the McDivitt and Winnipeg corpora (so-called McDivitt+; here "MCD") of Canadian English-speaking families  (McDivitt & Soderstrom, 2016) , the Warlaumont corpus ("WAR") of US English-speaking families from central California  (Warlaumont et al. , 2016) , the Rosemberg corpus ("ROS") of Argentinian Spanish-speaking families from the Buenos Aires metropolitan area  (Rosemberg et al.
, 2015) , and the Casillas Yélî corpus ("YEL") of Yélî Dnye-speaking families from a remote island community in Papua New Guinea  (Casillas, Brown, & Levinson, 2017a). The data consist of 4-16-hour at-home recordings from microphones worn by children under age three from diverse linguistic and socioeconomic contexts. BER, MCD, L05, and WAR recordings were collected with the LENA recorder, while the Casillas corpora (TSE and YEL) were recorded with an Olympus WS-382 or WS-852 recorder, and ROS was recorded with a mix of Olympus, Panasonic, Sony, and LENA recorders. In our experiments, we use the subset of each corpus that has been sampled and manually annotated as a part of the ACLEW project  (Bergelson et al. , 2017). From each corpus, daylong recordings of 10 target children between the ages of 0 and 36 months with a representative spread of assigned sex and maternal education were chosen for manual annotation.
Gold-standard phoneme and syllable counts were obtained by automatic phonemization and syllabification of the handannotated orthographic transcripts. First, the transcripts were cleaned of all non-lexical entries such as incomprehensible speech, non-linguistic communicatives and other non-speech sounds (e. g. , &=yawns), and paralinguistic markers (e. g. , <hello> [=! sings] to denote singing while speaking). The resulting word strings were then converted into sequences of phonemes either using the Phonemizer tool 3 (for English) or using hand-crafted letter sequence-to-phoneme conversion rules that, while not reflecting a one-to-one-match between letters and phones, preserved the phonotactic constraints for syllable onsets in each non-English language tested (Spanish, Tseltal, Yélî Dnye).
Finally, phoneme strings were syllabified either using the Festival toolkit 4 tokenizer (for English corpora) or using the maximum onset principle (see, e. g. ,  Kahn, 1976)  with language-specific lists of valid syllable onsets, again hand-crafted to work with the phone-converted text (for Spanish, Tseltal, and Yélî Dnye). Gold-standard word counts were derived from the cleaned-up transcripts using white spaces as word boundaries. Some recordings in BER, and all recordings in TSE, YEL, MCD, and WAR are available from the HomeBank repository  (VanDam et al. , 2016).
In short, correlation describes whether the ALUC estimation algorithm is sensitive to relative differences in ALUCs across the given pool of test subjects, but does not show whether the absolute unit counts are accurate. In contrast, ERR median reveals whether the absolute counts are accurate, but is not as descriptive of the relative increases in estimated counts with an increasing amount of true counts across the full range of possible values. Since ALICE and both metrics are deterministic for any given data set, we report the numbers without measures of uncertainty (all reported correlations being p <. 001).
For instance, if 50% of a LENA-detected adult segment with 10 LENA-estimated words overlapped with the clip, 5 words were added to the LENA AWC estimate for that clip. Also note that a substantial majority of the LENA segments were fully within the annotated clip limits, as typical adult segments are much shorter than the annotated clips in question, and therefore this approximation affects only 1. 3% of all adult segments. SylNet features were calculated after adapting the baseline SylNet model to the training data of each fold using the goldstandard syllable counts as the reference and following the adaptation procedure described in  Seshadri and Räsänen (2019). Allosaurus was used as provided in the original Python library.
9 language-specific recordings). Comparison to LENA on English corpora (Fig. 5 ) also shows how ALICE outperforms LENA in all cases and on both metrics. As for the features used (Fig. 4  and Table  1 ), there are again no major differences in the performance of different feature sets, except that SylNet performs worse than the others when used in isolation. In terms of correlation, even the basic signal-level features (duration, total energy, total zero-crossings) are competitive with the more sophisticated phone recognition and syllable counting models. In terms of relative errors with algorithmic diarization, the best individual feature extractor is the Allosaurus phone recognizer. However, as a drawback,  Allosaurus is also the most computationally expensive of all three feature extractors.
In terms of different languages, there is some variation in performance, with English corpora and YEL being generally more accurate to estimate (in terms of correlation). Argentinian Spanish is the most difficult in terms of relative counts, whereas MCD and YEL are the most challenging in terms of ERR median. The high correlation but poor ERR median on YEL also illustrates how the two metrics are complementary to each other. Previous LENA studies have reported correlations between gold-standard and LENA AWCs, such as r = 0. 64 for French (10-minute segments), r = 0. 67 for Swedish (5-minute segments), r = 0. 99 for Finnish (1-hour segments), r = 0. 88 for Dutch (5-minute segments), r = 0. 80 for Mexican Spanish (1-hour segments), and r = 0. 73 for Mandarin Chinese (15-minute segments). The corresponding ERR median are 36.
The primary reason for using the current set of corpora was that they had all been annotated using exactly the same annotation protocol  (Casillas et al. , 2017b) , ensuring maximal mutual compatibility and comparability in our experiments. We also took some steps to mitigate the impact of English-heavy distribution in our experiments by measuring the overall ALICE performance with an equal weight on the four unique languages. In addition, we experimented with frequencybalanced training of ALICE to see whether that would impact the cross-language generalization performance of the system, but it did not lead to any performance gains over the baseline approach (not reported separately). Finally, direct performance comparison to LENA was not possible in three of the four tested languages. This was because LENA analysis software outputs were only available for the English corpora, as Tseltal, Argentinian Spanish, and Yélî Dnye data were collected with non-LENA audio recorders; LENA software only accepts audio files that are recorded with the LENA recorder.
Acknowledgements This research was supported by an  Academy of Finland  grant nos. 314602  and  312105  to OR and by an  NWO   Veni Innovational Research Scheme grant  to MC ( 275-89-033 ). ML and AC were funded by  Agence Nationale de la Recherche  ( ANR-17-CE28-0007   LangAge ,  ANR-16-DATA-0004   ACLEW ,  ANR-14-CE30-0003   MechELex ,  ANR-17-EURE-0017 ) and the  James S. McDonnell Foundation   Understanding Human Cognition Scholar Award. This work builds on research using the  Extreme Science and Engineering Discovery Environment (XSEDE) , which is supported by  National Science Foundation  grant number  OCI-1053575. Specifically, it used the Bridges system, which is supported by  NSF  award number  ACI-1445606 , at the  Pittsburgh Supercomputing Center (PSC). The authors would like to thank the  ACLEW project consortium  for kindly providing the daylong data and their annotations,  Xinjian Li  and  Florian Metze  for their help with the Allosaurus algorithm, and  Anant Mital  for his support with the technical development.
An open-source Python implementation of ALICE for Linux and MacOS is available for download at https://github. com/orasanen/ALICE under MIT License. Some of the daylong recordings in BER, and all recordings in TSE, YEL,  MCD , and WAR, are available from HomeBank repository (VanDam et al. , 2016). Inquiries about access to the ROS and L05 corpora should be directed to their respective authors. None of the experiments were preregistered.
Biochemical demands constrain the range of amino acids acceptable at specific sites resulting in across-site compositional heterogeneity of the amino acid replacement process. Phylogenetic models that disregard this heterogeneity are prone to systematic errors, which can lead to severe long branch attraction artifacts. State-of-the-art models accounting for across-site compositional heterogeneity include the CAT model, which is computationally expensive, and empirical distribution mixture models estimated via maximum likelihood (C10 to C60 models). Here, we present a new, scalable method EDCluster for finding empirical distribution mixture models involving a simple cluster analysis. The cluster analysis utilizes specific coordinate transformations which allow the detection of specialized amino acid distributions either from curated databases, or from the alignment at hand.
In contrast to the above, the CAT model  (Lartillot et al. , 2004)  uses one set of exchangeabilities for all mixture model components and a Dirichlet process prior over the stationary distributions and their weights in a Bayesian Markov chain Monte Carlo framework. The CAT model is widely used, and greatly improves model fit. However, computational requirements are high, such that convergence times are long, and convergence may be beyond reach, especially for larger data sets  (Whelan et al. , 2016). Inspired by the CAT model, and consistent with the derivation of widely used empirical transition rate matrices from curated databases, empirical stationary distribution mixture (EDM) models, which use a fixed set of stationary distributions, have been developed. The rationale is that site-specific amino acid constraints may be caused by universal biochemical constraints (e.
g. ,  Jimenez et al. , 2018). In particular, composition heterogeneity and site-specific amino acid constraints have already been used to estimate protein structure  (Goldman et al. , 1996)  and the association of protein structure with evolution  (Goldman et al. , 1998). Previously,  Quang et al. (2008)  used an expectation maximization algorithm to find EDM models with 10, 20, up to 60 components, which we collectively call CXX models, from alignments of the HSSP database  (Schneider et al. , 1997). Each mixture model component is defined by the used stationary distribution and weight. Accordingly, we use the term component to refer to a stationary distribution with corresponding weight. For computational reasons, the Poisson model  (Felsenstein, 1981) , which exhibits uniform exchangeabilities, was used when searching for the components.
These strategies include (1) restricting the analysis to sites with high rate, (2) penalizing low frequencies of amino acids, (3) down-weighing contributions from species-rich clades, and (4) phylogeny-based estimation. Here, we describe EDCluster, a new method for obtaining stationary distributions that can be used to construct EDM models. EDCluster can be used on any set of alignments ranging from large databases of homologous genes to more specific data sets. We employ the CAT model implemented in Phylobayes  (Lartillot et al. , 2013)  to estimate site-specific posterior distributions of the stationary distributions of amino acids. In this way, specialized treatment of the expected variation in the divergence between the sequences is not required. The site distributions are analyzed as is, or transformed using linear transformations developed specifically for compositional data.
The transformations aid the clustering method in finding stationary distributions of amino acids with different specialized features. The use of a clustering algorithm seemed natural because clustering is a simple machine learning approach for feature discovery. Although EDCluster does not directly use biochemical information, the inferred components are found to correspond to specific biochemical traits of amino acids, such as hydrophobicity, size, or mass. Using EDCluster, we provide sets of 4, 8, 16,. , 4096 components estimated from subsets of the HOGENOM database  (Dufayard et al. , 2005) , the HSSP database, and the union of both. We present extensive analyses of EDM models based on these sets of components which we collectively call universal distribution mixture (UDM) models. For the same number of components, we demonstrate that the UDM models outperform the CXX models not only in terms of likelihood but also in parametric bootstrap analyses, were they exhibit improved amino acid compositions and branch lengths.
Moreover, EDCluster allows construction of EDM models with a large number of components. In particular, the UDM models with more components show even further increases in accuracy. However, the number of components is still limited by the associated linear increase of computational requirements during inference. In conclusion, the UDM models minimize systematic errors caused by constraints in amino acid usage in a fraction of the run time of CAT. We provide ready-to-use implementations for several established phylogenetic software packages such as IQ-TREE  (Nguyen et al. , 2015) , Phylobayes, or RevBayes  (Höhna et al. , 2016). Further, we provide user-friendly scripts implementing EDCluster to construct EDM models specific to the data set at hand. Finally, we employ a simulation study to reproduce a well known LBA artifact of classical substitution models, and show that application of EDM models successfully recovers the correct topology.
New Approaches EDM models assume that evolution occurs along a phylogeny according to a mixture of N amino acid substitution models. The transition rate matrices of the different components share a single set of exchangeabilities. In this contribution, Poisson exchangeabilities were used exclusively although the method could in principle be generalized directly to any other set of exchangeabilities. In contrast, the stationary distributions (or equilibrium frequencies over the 20 amino acids) differ between each component of the mixture model. Here, the stationary distributions are inferred from alignments obtained from curated databases. Each alignment was analyzed with Phylobayes under the CAT model with Poisson exchangeabilities. For each alignment and each site, the expectation of the posterior distribution of the stationary distribution of amino acids (site distribution), was calculated.
Each site distribution is a point in 20-dimensional space with elements summing up to 1. 0. For each database, the site distributions of all sites were used as is or transformed before further analysis. Application of linear transformations is a standard procedure in analyses of compositional data. The two employed transformations were: (1) the centered log ratio transformation (CLR;  Aitchison, 1982) , and (2) the log centered log ratio transformation (LCLR;  Godichon-Baggioni et al. , 2018). In our case, the transformations ensure that site distributions exhibiting specific different features are moved further apart from each other, so that they fall into different groups in the subsequent analysis. K-means clustering was used to group the prepared site distributions into N ∈ {4, 8, 16,.
, 4096} clusters. The stationary distributions and weights of the different components to be used in the UDM models were set to the determined cluster centers and their relative weights, respectively. We combine the components obtained from a subset of the HOGENOM database with Poisson exchangeabilities and refer to the mixture model resulting from a specific set of components as UDM-XXX-Trans, where XXX is the number of components, and Trans is the used transformation (None, CLR, or LCLR). The usage of Poisson exchangeabilities is implicitly assumed and not mentioned specifically. For example, the UDM model with four components obtained from the LCLR transformed site distributions is referred to as UDM-004-LCLR model. Although the presented analyses exclusively focus on components estimated from the HOGENOM database, components estimated from the HSSP database, and the union of both databases are provided for further reference.
, 2018)  of the HOGENOM database  (Dufayard et al. , 2005)  are shown. On the far right, the distribution of the effective number of amino acids of the site distributions obtained with the CAT model  (Lartillot et al. , 2004)  using Poisson exchangeabilities  (Felsenstein, 1981)  is displayed. The width of the violin plots was normalized such that all areas are equal. Horizontal bars display the means of the distributions. the number of components. Additional components exhibit stationary distributions with K eff values usually well below 10. UDM models with more than 128 components tend to include more than one general stationary distribution with K eff > 10. The results for the stationary distributions obtained from untransformed, and CLR transformed site distributions are almost identical (Figure  S1  and Figure  S3 ).
Further, the distribution of K eff values of the site distributions inferred from the HOGENOM database using the CAT model with Poisson exchangeabilities and the corresponding mean value are shown. As expected, the more components are used, the closer the distribution of K eff values of the stationary distributions of the UDM components to the distribution estimated directly from the HOGENOM database (see also Figures  S2, S4,  and S5 ). The mean K eff values exhibit the same tendency. Exact K eff values for UDM models up to 16 components including all three transformations are available in Section S3. Strikingly, in most cases components with higher weight also have higher K eff values. Further, the K eff values of the site distributions associated with the components with the most general stationary distributions are usually much lower than the K eff value of the stationary distribution of the respective component.
, 1987; Kamaishi et al. , 1996)  marked them as a candidate for an ancient eukaryotic lineage predating the acquisition of mitochondria. However, more sophisticated phylogenetic analyses have recovered microsporidia being relatives of fungi, rather than being basal eukaryotes  (Hirt et al. , 1999; Keeling et al. , 2000; Van de Peer et al. , 2000; Keeling et al. , 2002)  and subsequently remnants of mitochondria were found experimentally  (Williams et al. , 2002). Here, as an illustration, we consider the data set of  Brinkmann et al. (2005, referred to  as microsporidia data set), which spans 40 sequences, and comprises 133 genes corresponding to 24294 amino acid sites. We show analyses of the concatenated alignment, as well as of the separate genes. For the concatenated alignment of the microsporidia data set, site homogeneous substitution models such as the LG model favor the former topology in terms of likelihood.
In contrast, if the number of components is large enough, the UDM and CXX models reject the topology exhibiting the LBA artifacts in all three data sets. For the data set involving microsporidia, compared to the CXX models the UDM models require a higher number of components to reject the topology with LBA artifacts. For the data sets involving nematodes and platyhelminths, the situation is reversed in that the differences of the maximum log-likelihoods of the UDM models are more positive than the ones of the CXX models. Also, the difference in maximum log-likelihood does not increase substantially for the CXX models when applied to the data set involving nematodes. The performance of the UDM models on shorter alignments compared to the LG model, and the CAT model with Poisson exchangeabilities was tested on the separate genes of the microsporidia data set.
In general, the dif-  Similarity between the across-site compositional heterogeneity of the microsporidia data set  (Brinkmann et al. , 2005) , and simulated alignments for the maximum likelihood parameter estimates of the WAG  (Whelan et al. , 2001) , LG  (Le et al. , 2008a) , CXX  (Quang et al. , 2008) , and universal distribution mixture (UDM) models. Results of UDM models obtained from untransformed (None), center log ratio (CLR;  Aitchison, 1982)  transformed, and log center log ratio (LCLR; Godichon-Baggioni et al. , 2018) transformed site distributions are shown. Similarity is measured by the Wasserstein distance between the distributions of effective number of amino acids per site between empirical data and the sequences simulated using parametric bootstrap. For all three data sets, the WAG and LG models with a single amino acid transition rate matrix produce alignments with inflated diversity measured in effective number of amino acids (Figure  4  for the microsporidia data set; Figures S25, and S26 for the nematode and platyhelminth data sets, respectively).
Numeric values of the average effective number of amino acids per site in the alignment, as well as the Wasserstein distances are given in Table  S1. Component-rich UDM and CXX models typically exhibit lower Wasserstein distances than UDM or CXX models with fewer components. The UDM models consistently outperform the CXX models. The C50 model exhibits large deviations in all three data sets. UDM models with 256 components sometimes exhibit higher Wasserstein distances than their equivalents with 128 components. For example, compare the UDM-256-LCLR model with the UDM-128-LCLR model in the microsporidia data set (Figure  4 ). The reason for the increase of    (Felsenstein, 1981) , and stationary distributions of amino acids obtained from analyses of the HOGENOM database  (Dufayard et al. , 2005).