dataesr/scientific-paragraphs-categorization huggingface dataset:
ASTRAL has high accuracy even without binning, and binning with either threshold has only a small impact on its accuracy. When MP-EST is used, binning with B = 50% leads to relatively small improvements in accuracy, whereas B = 75% results in much larger improvements. Thus, the choice of the threshold can have an impact, but for the two model conditions we studied here both choices of the threshold are beneficial. Effects of binning on gene tree and species tree error for 15-taxon datasets We explored the impact of statistical binning on gene tree estimation error using two sequence lengths and two values for B, the bootstrap support threshold parameter (S1 Table ). For the shorter sequence lengths (100bp), binning increases gene tree estimation error (from 77% to 80% when B = 50%, and from 77% to 86% when B = 75%).
For the longer sequence lengths (1000bp), binning with B = 50% has no impact on gene tree estimation error, but using B = 75% increases error from 36% to 40%. Thus, statistical binning increases gene tree estimation error for these very high ILS 15-taxon datasets, but the amount of the increase depended on the parameter B (with larger increases for B = 75% and small increases for B = 50%) and sequence length (where the impact on gene tree estimation error is much reduced for the 1000bp alignments). Fig. 8  shows the impact of weighted and unweighted statistical binning on species tree accuracy for the 15-taxon dataset. We apply statistical binning with two support thresholds (50% and 75%), and we use both MP-EST and ASTRAL as the summary method. In all cases, weighted and unweighted binning have similar accuracy, with no statistically significant differences (Tables  2  and 3 ).
The relative accuracy of unbinned and binned analyses depends on the support threshold, so that with B = 50%, there are no statistically significant differences, but with B = 75%, binning significantly improves accuracy (p = 0. 04 for MP-EST and p = 0. 008 for ASTRAL; Tables  2  and 3 ). The extent of the improvements seems larger for more genes and smaller alignments, but the impact of these factors are not statistically significant for MP-EST (p = 0. 24 and p = 0. 17 respectively) and only impact of sequence length was significant for ASTRAL (p = 0. 02 ; Tables  4  and 5 ). The biggest gains are obtained when the 75% threshold is used with 1000 genes of 100bp, where binning reduces the error of MP-EST from 21% to only 7%. Thus, the choice of the threshold can matter, and on this dataset, the effects of binning can range from neutral to highly beneficial, depending on the threshold used, number of genes, and gene sequence length.
Effects of binning on species tree error for 10-taxon datasets Fig. 9  shows the impact of binning on species tree accuracy on the 10-taxon datasets with two choices of the threshold B for the statistical binning pipeline (B = 50% and 75%), two choices of the summary method (MP-EST and ASTRAL), and two levels of ILS (high and very high). No statistically significant differences are observed on these data between weighted and unweighted binning, or between weighted binning and unbinned analyses (see Tables  2  and 3 ); nevertheless, some patterns can be observed in terms of the average error (Fig. 9 ). Both weighted and unweighted statistical binning are close to neutral (regardless of the choice of method or level of ILS) when applied with a 50% threshold. When the 75% threshold is used, the impact of binning depends on the level of ILS: binning improves accuracy with low ILS levels and reduces accuracy with high ILS levels, especially when MP-EST is used, but these differences are not statistically significant (Tables  2  and 3 ).
Analysis of biological datasets We compared weighted and unweighted binning of MP-EST and ASTRAL on MLBS gene trees on the avian and mammalian biological datasets studied in  [23]. Results for MP-EST on these datasets showed the following trends. First, for the avian dataset, there are no topological differences between MP-EST trees estimated using weighted or unweighted statistical binning, and extremely small differences in branch support (less than 3%; see Fig. 10 ). Thus, although  [26]  only explored unweighted statistical binning with MP-EST, the main conclusions they drew about the evolutionary history of modern birds are also found in the weighted statistical binning analysis using MP-EST. The unbinned MP-EST analysis violates several subgroups established in the avian phylogenomics project and other studies (indicated in red in Fig.
10 ), but the binned MP-EST analyses do not violate any of these subgroups. Of these violated subgroups, the failure of the unbinned MP-EST analysis to recover Australaves is the most significant, since it has been recovered in many prior analyses  [37] [38] [39] [40]. On the mammalian dataset, weighted and unweighted MP-EST again produce the same exact tree, with small differences in support (less than 3%; see S11 Fig. ). The unbinned MP-EST tree, however, has one topological difference (the position of treeshrews; compare S10 and S11 Figs. ). with binned analyses, as discussed in  [23]. Results for ASTRAL on the biological datasets show generally similar trends. Unbinned ASTRAL on the avian dataset (S8 Fig. ) recovers Australaves and hence is more in line with the prior literature than unbinned MP-EST; however, just like unbinned MP-EST, the unbinned ASTRAL does not recover some key clades recovered by concatenation and other analyses reported in  [26].
Using weighted and unweighted statistical binning with ASTRAL on the avian dataset produces almost identical results, and are also almost identical to the binned MP-EST tree (the only change is the position of hoatzin which has low support in all trees; see supporting information S9 Fig. ). On the mammalian dataset, trees produced by binned ASTRAL analyses with weighted or unweighted binning pipelines are topologically identical to each other, and to the tree produced by the unbinned analysis, and have rather small differences in bootstrap support (see S12 Fig. ). Binned and unbinned ASTRAL analyses and binned MP-EST analyses all put treeshrews as sister to Glires, while unbinned MP-EST puts them as sister to primates. The placement of treeshrews is of substantial debate, and so the differential placement is of considerable interest in mammalian systematics.
Summary of observations Across all our analyses, results for both ASTRAL and MP-EST are very similar with respect to how they responded to statistical binning. Weighted statistical binning produces nearly identical results to unweighted statistical binning on the biologically-based simulated datasets, and topologically identical results (with very similar bootstrap support values) on the biological datasets we explored in this study, and so this study generally supports the conclusions about statistical binning in  [23]. In addition, because weighted and unweighted statistical binning produce topologically identical trees on the avian dataset, this study supports the findings about the avian phylogeny reported in  [26]. The fact that weighted and unweighted binning typically produced similar results is not surprising, since the unweighted binning technique strives to create "balanced" bins as much as possible, and largely achieves this on the datasets we explored.
These improvements are largest when gene sequence alignments have low phylogenetic signal, the gene trees exhibit at most moderately large ILS levels, or there are many genes. The impact of statistical binning on the 15-taxon datasets is somewhat different than for the biologicallybased simulations. Gene tree estimation accuracy is reduced for both sequence lengths (though the impact is small for the longer sequence lengths and only substantial for the short sequence lengths with B = 75%). Nevertheless, the impact on species tree estimation on these data tends to be neutral, but there are also conditions where binning was beneficial. On the lower ILS 10-taxon datasets, statistical binning reduces gene tree estimation error, and both weighted and unweighted binning reduce species tree estimation error for B = 75%.
However, species tree estimation error is unchanged when B = 50%. The results on the higher ILS 10-taxon datasets stand out from the other analyses: statistical binning slightly increases gene tree estimation error when B=50% but substantially increases gene tree estimation error when B=75%. Furthermore, while species tree estimation error is not increased for B = 50%, when B = 75%, the error increases. The difference in impact for statistical binning in this case is interesting, and points out the significance of how B is set. To understand this, note that when B is very small, then bin sizes will tend to be very small, since any pair of incompatible branches with support above B will be considered to be evidence of statistically significant discord; thus, small settings for B produce results that are similar to unbinned analyses.
Conversely, very large settings for B are more likely to bin genes together, since only the strongest supported conflicting branches will prevent genes from being binned. Therefore, if all the gene trees have low support then statistical binning could tend to produce results that are similar to concatenation. Thus, the choice of the threshold matters. To better understand the difference in impact of statistical binning on these simulated datasets, it is helpful to consider the ILS levels and gene tree bootstrap support values for these data. As shown in Table  1 , the average distance between the true gene trees and the species trees ranges for these datasets from as low as 18% (for the Mammalian 2X collection) to above 80% (for the 10-taxon higher ILS collection and the 15-taxon collection).
Fig. 3  shows how the effect of statistical binning used with MP-EST is impacted by ILS level on the avian datasets: statistical binning provided an improvement at all ILS levels, with the largest improvement for the lowest ILS level (2X branch lengths) and the smallest improvement on the highest ILS level (0. 5X branch lengths). S3-S5 Figs. evaluate this issue on the mammalian datasets, and shows large improvements provided by statistical binning under the lowest (2X branch lengths) ILS level, smaller improvements under the middle (1X branch length) ILS level, and then no improvement under the highest (0. 5X branch lengths) ILS level. Thus, statistical binning provided an improvement except for a small number of model conditions: some of the 15-taxon conditions (which have discordance of 82%), the higher ILS 10-taxon conditions (which have discordance of 84%), and the highest ILS mammalian condition (which have discordance of 54%).
S2 Table shows that the average bootstrap support for the higher ILS 10-taxon datasets is quite low -only 37%. Thus, statistical binning seems to be beneficial when both ILS level and gene tree bootstrap support are not too high, will be neutral when bootstrap support values are high (so little or no binning occurs), but can be detrimental when ILS levels are extremely high but gene tree bootstrap support is low enough that binning occurs. Thus, one consequence of this study is the suggestion that when ILS levels are very high and the average gene tree bootstrap support is low, then either statistical binning should not be used, or it should be used in a very conservative fashionwith the parameter B set very low.
On the biologically-based simulated datasets, weighted and unweighted statistical binning generally improved estimated gene tree distributions and led to improvements for MP-EST and ASTRAL estimations of species tree topologies. The use of statistical binning with MP-EST also improved estimated species tree branch lengths, increased bootstrap support for true positive edges, and reduced the number of highly supported false positives, compared to unbinned MP-EST analyses. These improvements increased when gene sequence alignments had low phylogenetic signal, the species tree had low ILS, or there were many genes. The estimation of species tree branch lengths is biologically significant since these lengths are used to infer the amount of ILS in the data. Unbinned MP-EST analyses tended to substantially underestimate branch lengths (and thus over-estimate ILS), but both weighted and unweighted binning reduce this problem and produce branch lengths that are much closer to their true lengths.
Since MP-EST tends to over-estimate ILS in the presence of gene tree estimation error, this means that predictions of ILS levels for biological datasets may have been over-estimated. Another consequence of this observation is that the biologically-based model species trees used here and in  [23, 43]  may have inflated levels of ILS, since they used MP-EST to construct the model species tree. If so, then performance under the lower ILS levels (species tree branch lengths of 2X or larger) might be closer to the biological dataset conditions than the default 1X condition and higher ILS conditions. The improvement in branch support is biologically relevant, especially since unbinned MP-EST analyses sometimes produced highly supported false positive branches in the presence of poorly estimated gene trees and low levels of ILS, but binning reduced the incidence of these false positive branches with high support.
The results on small numbers of species, and in particular on the higher ILS 10-taxon datasets, show somewhat different trends. While results on the 15-taxon datasets showed binning generally being helpful or neutral, statistical binning ranged from neutral to detrimental on the higher ILS 10-taxon datasets (however, the differences were not statistically significant). Both the higher ILS 10-taxon and 15-taxon datasets had extremely high levels of ILS (the two highest we examined -average topological distance between true gene trees and the true species trees of 84% and 82%, respectively). Given that statistical binning ranged from neutral to highly beneficial for all the other model conditions, these data suggest that statistical binning may not be suitable to datasets with extremely high ILS levels.
Clearly, further research is therefore needed to understand the conditions under which binning will be beneficial and where binning may reduce accuracy. This study also did not examine model conditions in which gene tree estimation error is due to model misspecification, nor other biological causes for gene tree discord, such as gene duplication and loss or horizontal gene transfer. Furthermore, while we examined sequence datasets with varying numbers of sites for each locus (including some with 100bp), even shorter sequences may be needed to avoid loci that include any recombination  [24]. This study mainly examined the impact of statistical binning on MP-EST, and examined its impact on ASTRAL only for a subset of the data and only with respect to species tree topology estimation (instead of the full set of criteria).
Thus, an important direction for future study is to consider other coalescent-based methods for estimating the species tree from multiple loci. As a simple example, Mirarab et al. [43]  showed that the accuracy of MP-EST species trees depended on whether MLBS or best maximum likelihood (BestML) gene trees were used, and that MP-EST trees based on BestML gene trees generally produced more accurate species tree topologies for datasets with large numbers of genes (such as some of the model conditions studied in this paper). The explanation offered for this is that BestML gene trees are generally closer to the true gene tree than MLBS gene trees, and that this helps coalescent-based species tree estimation. Hence, the evaluation of the impact of binning on MP-EST with BestML gene trees is also needed.
Hence, it is also possible that methods that construct species trees directly from the sequence data, rather than by combining gene trees, will have the best accuracy (see, for example,  [50] [51] [52] ), since they can avoid the analytical and empirical challenges caused by gene tree estimation error. However, as observed in this and other studies  [24, 42] , concatenation often produces more accurate trees than even the best coalescent-based methods when the level of ILS is low enough. Therefore, an important question is whether a given biological dataset has a sufficiently high level of ILS that a coalescent-based analysis is needed. Conversely, coalescent-based methods that are not only more accurate than concatenation under conditions with high ILS, but also comparably accurate even under low levels of ILS, would be very helpful tools.
, t p } be a set of p rooted gene trees sampled from the distribution defined by T sp under the multi-species coalescent model. Let {θ 1 , θ 2 ,. , θ p } be a set of numeric GTR model parameters (gene tree branch lengths and 4 × 4 substitution matrices) so that T i = (t i , θ i ) is a GTR model tree for each i = 1, 2,. , p. Let T = {T 1 , T 2 ,. , T p }. For each i, 1 ≤ i ≤ p, let sequence dataset S i evolve down the GTR model tree T i. Let < 1 and bootstrap support threshold B < 1 be given. Then, there is a sequence length L (that depends on T and ) such that if at least L sites evolve down each gene tree, then with probability at least 1 -, the following will be true: • For each i = 1, 2,. , p, the gene tree estimated using GTR maximum likelihood on S i will have the same unrooted topology as t i (the true gene tree for S i ), and will have bootstrap support greater than B for all its branches, • For every bin produced by statistical binning based on GTR maximum likelihood analyses of the gene sequence alignments, the estimated gene trees for genes in the bin will have the same topology, and • All genes with the same true gene tree topology will be in the same bin.
Proof: Since GTR maximum likelihood is statistically consistent for sequences generated by GTR model trees, then for any > 0, there is a sequence length L i such that given sequence dataset S i with at least L i sites generated on T i , the GTR maximum likelihood tree topology for S i is t i (i. e. , the true gene tree) and has bootstrap support greater than B, with probability at least 1 -. Letting L = max i {L i }, it follows that all estimated gene trees will be the true gene trees and have bootstrap support greater than B with probability at least 1 -p. Therefore, when = p and the sequences are all of length at least L, the result then follows by Lemma 1. Fully partitioned GTR maximum likelihood: In a fully partitioned GTR maximum likelihood analysis, the input is a set of p multiple sequence alignments, {S 1 , S 2 ,.
Evaluation We explored the performance of MP-EST and ASTRAL with weighted and unweighted statistical binning, and also without binning. We also examine concatenation of the entire set of gene sequence alignments using an unpartitioned maximum likelihood analysis using RAxML. We explore performance on a collection of simulated and biological datasets originally studied in  [23]. We applied MP-EST and ASTRAL to a set of RAxML gene trees computed on bootstrap replicates of each gene sequence alignment. With bootstrap ML gene trees for each gene, summary methods were applied with the site-only multi-locus bootstrapping (MLBS) procedure  [53] , implemented as follows. For each gene or supergene, 200 replicates of bootstrapping are performed using RAxML. Next, 200 replicates (R 1 , R 2 ,. , R 200 ) of input datasets to the summary methods are created such that R i contains the i th bootstrap tree across all genes/supergenes.
Therefore, we have The average bootstrap support (BS) in the biological data was 71%, and so  [23]  generated sequence lengths that produced estimated gene trees with bootstrap support bracketing that value -500bp alignments produced estimated gene trees with 63% average BS and 1000bp alignments produced estimated gene trees with 79% BS. We also generated model conditions with very short sequence lengths (250bp), which have 43% average BS. Here, we varied the number of genes from 50 to 800 to explore both smaller and larger numbers of genes than the biological dataset (which had roughly 400 genes). In total, we generated 17 different model conditions specified by the ILS level, the number of genes, and the sequence length. For each of these model conditions,  [23]  created 20 replicates.
Avian simulated datasets Mirarab et al. [23]  used the species tree estimated by MP-EST on a subset of the avian dataset with 48 species and 14,446 loci studied by  [26] , and simulated gene trees by varying different parameters (similar to the mammalian simulated datasets). Three types of genomic markers were studied in  [26] : exons, UCEs, and introns. The average bootstrap support (BS) of the gene trees based on exons, UCEs, and introns, was 24%, 39% and 48%, respectively; the longest introns had the highest average BS (59%). Mirarab et al. varied sequence lengths (250bp, 500bp, 1000bp, and 1500bp) to produce four model conditions with patterns of average bootstrap support that resemble these four marker types. Mirarab et al. varied the number of genes from 200 to 2000, but here, we augmented the dataset to also look at fewer genes (50 and 100).
15-taxon simulated datasets We simulated a collection of 15-taxon datasets. The model species tree is a caterpillar-like ultrametric tree (i. e. , the substitution process obeys a strict molecular clock) with 15 taxa; hence, it has two leaves x and y that are siblings in the tree. The lengths of all internal branches and the two branches incident with leaves x and y are all set to 0. 005 substitutions per site; note that the assumption of ultrametricity defines the remaining branch lengths. The population size parameter (θ = 4N µ) is set to 0. 05 for all branches, and this results in 12 short internal branches (0. 1 in coalescence units) in succession. Ultrametric gene trees were simulated down this tree using McCoal  [56]  and commands given in S13 Fig. Sequence data were simulated down each gene tree using bppseqgen  [57]  according to GTR+Γ parameters given in S13 Fig.
10-taxon simulated datasets We used simPhy  [33]  to simulate species trees using the Yule process with two different maximum tree length settings: 200K generations, resulting in short trees and high levels of ILS, and 1. 8M generations, resulting in relatively longer trees and lower levels of ILS. We generated 20 species trees per model condition, and used simPhy to simulate 200 gene trees for each species trees using the multi-species coalescent process (simPhy parameters and commands are given in S14 Fig. ). The gene trees (with branch lengths in substitution units) deviate from the strict molecular clock, and the rates of evolution vary across genes. We used Indelible to simulate GTR+Γ sequence evolution down these gene trees with 100 sites, with parameters given in S14 Fig.
Biological datasets We studied two biological datasets also studied in  [23] : the avian dataset  [26]  containing 14,446 loci across 48 species, and a reduced version of the mammalian dataset studied by Song et al. [31]  with 447 loci across 37 species, from which  [23]  deleted 23 erroneous genes and re-estimated gene trees using RAxML (see  [23, 43]  for discussion of these loci).
Methods and commands Gene tree estimation: RAxML version 7. 3. 5  [58]  was used to estimate gene trees under the GTRGAMMA model, using the following command: Supergene tree estimation: For the biological datasets and the 10-and 15-taxon simulated datasets, we used a fully partitioned maximum likelihood analysis. All other analyses were based on unpartitioned maximum likelihood analysis, using the command given above for gene tree estimation. For the fully partitioned analysis, we used the following command: raxmlHPC-SSE3 -m GTRGAMMA -s [input alignment] -m GTRGAMMA -n [output name] -N 20 -M -q [partition file] -p [random seed number] Concatenation: We concatenate the alignments of all genes into one supermatrix, and then estimate a tree from the supermatrix using unpartitioned maximum likelihood.