dataesr/scientific-paragraphs-categorization huggingface dataset:
We used BlobTools 66 in combination with SILVA  67  and UniProt 68 databases to infer the taxonomy for each scaffold. We removed all scaffolds that were classified as bacteria or fungi and also those that had a skewed GC content and read coverage. The completeness of each final assembly was assessed by BUSCO  69  with the Plants set, and by using BWA  70  and HISAT2  71  to map Illumina reads to the assemblies (Supplementary Table  2 ). Repeat annotation. RepeatModeler 72 was used to generate species-specific repeat libraries for masking and annotation. Consensus repeat sequences with homology to known plant genes were removed from the repeat libraries. Homology was defined as having a significant (E-value < 1 × 10 -5 ) blastx  73  hit to a subset of the PlantTribes 74 v1. 1 database that does not contain transposable element-related terms.
Filtered RepeatModeler libraries were annotated with the name of the highest-scoring significant Repbase 75 v22. 04 full database sequence (tblastx  73  , E-value < 1 × 10 -5 ) and the highest-scoring significant Dfam 76 v2. 0 profile hidden Markov model (HMM) (hmmsearch  77  , E-value < 1 × 10 -5 ). LTR-RTs were discovered using structural criteria by the GenomeTools 78 program LTRHarvest  79  with the following modifications to the default settings: a LTR similarity threshold of 0. 01, an allowed LTR length range of 100-6,000 bp, an allowed distance between LTRs of a single element range of 1,000-25,000 bp and the number of bases outside LTR boundaries to search for target-site duplications set to 10. The GenomeTools program LTRDigest 80 was used with a set of 138 transposable element-related Pfam profile HMMs to annotate protein-coding domains in the internal regions of LTR-RTs.
We next used hmmsearch  77  to search the genome assemblies again to identify all regions with similarity to the centromere profile HMMs. Genomic regions with significant HMM matches were identified and these regions were annotated in a GFF3 format. Gene prediction. Protein-coding genes were predicted using MAKER-P 90 (v2. 31. 8), and three MAKER-P iterations were performed: (1) repeat masking and creation of initial gene models from transcript and homologous protein evidence; (2) refinement of initial models with SNAP  91  ab initio gene predictor trained on initial models; and (3) final models generated using SNAP and the ab initio gene predictor AUGUSTUS  92  trained on gene models from the second iteration. Masking was performed by RepeatMasker 93 (v4. 0. 5) using the previously described species-specific repeat libraries and the full Repbase v22.
04 database. After masking, gene models were inferred from transcripts and homologous protein sequences by first aligning to the genomes using BLAST+  73  (v2. 2. 31) blastn or blastp, and then refined using the functions est2genome and protein2genome from the splice-site aware alignment program Exonerate 94 (v2. 2. 0). We included the previously described A. filiculoides or S. cucullata transcriptomes and the set of protein sequences consisting of the full Swiss-prot database (downloaded 18 June 2016), Amborella trichopoda v1. 0 proteins, A. thaliana TAIR10 proteins, Chlamydomonas reinhardtii v5. 5 proteins, Oryza sativa v7. 0 proteins and Physcomitrella patens v3. 3 proteins (from Phytozome  95  ). Gene models with an annotation edit distance (AED) score of < 0. 2 were used to train SNAP, which was used during the second iteration of MAKER-P.
The parameters for each software package were the default options for PASTA. For each gene family phylogeny, we ran PASTA until we reached three iterations without an improvement in the likelihood score using a centroid breaking strategy. We used the best-scoring PASTA tree for each multi-species nuclear gene family to infer and locate WGDs using MAPS. For the null simulations, we first estimated the mean background gene duplication rate (λ) and the gene loss rate (μ) with WGDgc 101. Gene count data were obtained from OrthoFinder clusters associated with each species tree. λ = 0. 0031 and μ = 0. 0039 were estimated using only gene clusters that spanned the root of their respective species trees, which has been shown to reduce biases in the maximum likelihood estimates of λ and μ  101. We chose a maximum gene family PPR annotation.
We conducted a targeted annotation for PPR genes because they are generally only weakly expressed and thus often lack transcriptome support. Open reading frames from the nuclear genome assemblies were translated into amino acid sequences using the "getorf " tool from the EMBOSS (v. 6. 5. 7) package  113  with a minimum size restriction of 300 nucleotides. These open reading frames were searched for PPR motifs using the hmmsearch tool from the HMMER3 package  77. The PPR motif models and parameters used follow those of Cheng et al. 27. Motifs were assembled into full PPR tracts and the best model for each PPR was determined  27. To study the prevalence and location of RNA editing, non-poly(A)-enriched RNA-seq data were filtered to remove adapters, low-quality reads and reads with ≥ 5% Ns. Clean reads were aligned against the assembled plastid and mitochondrial genome assemblies using TopHat 2.
0 (ref. 114 ). One of the inverted repeat regions in the plastid genomes was removed before mapping. Only uniquely mapped reads were retained as input for SAMtools  115  to call mismatches between RNA and the corresponding DNA. Differences between corresponding RNA and DNA sequences were identified as the putative RNA-editing sites. The RNA-editing level was defined as the number of altered reads divided by the total mapped reads for each site. Phylogeny of the insecticidal protein Tma12. We used BLASTp  73  to search for Tma12 (Genbank accession: JQ438776) homologues in Phytozome  95  , 1KP transcriptomes  32  and the NCBI Genbank non-redundant protein database. Although Tma12 homologues are present in fern transcriptomes and in the S. cucullata genome, no significant hit was found in any other plant genomes or transcriptomes.
In addition, the majority of the Tma12 protein is composed of a chitin-binding domain that belongs to the PF03067 Pfam family. This family does not contain any plant genes but is predominantly represented in the genomes of Actinobacteria, insects and fungi. To trace the origin of fern Tma12 genes, we downloaded representative sequences containing PF03067 and PF08329 (as the outgroup) from UniProt and Genbank and reconstructed the phylogeny using IQ-TREE  116. We then used this preliminary phylogeny (Supplementary Data) to construct a more focused data set to narrow down the phylogenetic affinity of Tma12. PartitionFinder 117 was used to infer the optimal codon partition scheme and substitution models, and RAxML 100 was used for maximum likelihood phylogeny inference and to calculate bootstrap branch support.
We then filtered out genes with more than two species missing or having an average sequence length shorter than 75% of the one in A. filiculoides. This resulted in a final data set of 2,108 genes. Sequence alignments and gene tree inferences were done in PASTA  98  , with the default setting, except that RAxML 100 was used to estimate the best tree on the final alignment. To infer the species tree from these gene trees, we used the multi-species coalescent method implemented in ASTRAL-III (v5. 6. 1)  119. The tree topology from the plastome and nuclear data sets were identical, and all nodes received bootstrap support of 100 and a local posterior probability of 1. 0. Cyanobiont phylogeny. To compare the host and symbiont phylogenies, we assembled the cyanobiont genomes from five additional Azolla species (Supplementary Table  12 ) using the resequencing data generated from total DNAs, including sequences derived from both the host and the cyanobiont.
To extract the cyanobiont genomes from each of the Azolla species, we first filtered out chloroplast sequences by using BWA  70  (default parameters) to map the total clean DNA reads against each chloroplast genome reference. In this step, ~3-4% of the reads were filtered out, which is necessary to remove plastid ribosomal RNAs that are highly similar to ones in the cyanobionts. For each of the five Azolla species, we then mapped the filtered reads to the published cyanobiont reference (N. azollae 0708 isolated from A. filiculoides  11  ; Genbank accession: NC_014248) using BLAST  73  (alignment criteria: E-value ≤ 1 × 10 -5 , sequence identity of ≥ 90% and an aligned coverage of ≥ 90%). Only the aligned reads were assembled by Mitobim 64 (iterations = 5) using N. azollae 0708 (ref. 11 ) as a reference.
Data analysis We employed a number of software for data analyses in this study, which was described in detail in the materials and methods, including the parameters used, versions (if applicable), and citations. For manuscripts utilizing custom algorithms or software that are central to the research but not yet described in published literature, software must be made available to editors/reviewers upon request. We strongly encourage code deposition in a community repository (e. g. GitHub). See the Nature Research guidelines for submitting code & software for further information. The axis labels state the marker and fluorochrome used (e. g. CD4-FITC).
Acknowledgements We are grateful to 123 backers from  Experiment. com who crowdfunded the initial work. We thank  Z. Fei  for providing comments and suggestions,  M. Harrison  for discussion on Azolla transporter genes,  J. Shaw  and  D. Weston  for providing access to the  Sphagnum genome  and  T. Nishiyama  for the Jungermannia genome, and  P. -F. Lu  for providing the image for Fig. 1c. This project was partly supported by the  Shenzhen Municipal Government of China  (no. JCYJ20150529150409546 ), the  National Science Foundation Doctoral Dissertation Improvement  Grant  DEB-1407158  (to K. M. P. and F. -W. L. ) and the  German Research Foundation   Research Fellowship   VR132/1-1  (to J. d. V. ). Computational support was provided by the  Duke Compute Cluster  and the  Centre for Information and Media Technology at University of Düsseldorf.
Environmental sequencing has greatly expanded our knowledge of micro-eukaryotic diversity and ecology by revealing previously unknown lineages and their distribution. However, the value of these data is critically dependent on the quality of the reference databases used to assign an identity to environmental sequences. Existing databases contain errors and struggle to keep pace with rapidly changing eukaryotic taxonomy, the influx of novel diversity, and computational challenges related to assembling the high-quality alignments and trees needed for accurate characterization of lineage diversity. EukRef (eukref. org) is an ongoing community-driven initiative that addresses these challenges by bringing together taxonomists with expertise spanning the eukaryotic tree of life and microbial ecologists, who use environmental sequence data to develop reliable reference databases across the diversity of microbial eukaryotes.
EukRef organizes and facilitates rigorous mining and annotation of sequence data by providing protocols, guidelines, and tools. The EukRef pipeline and tools allow users interested in a particular group of microbial eukaryotes to retrieve all sequences belonging to that group from International Nucleotide Sequence Database Collaboration (INSDC) (GenBank, the European Nucleotide Archive [ENA], or the DNA Data-Bank of Japan [DDBJ]), to place those sequences in a phylogenetic tree, and to curate taxonomic and environmental information for the group. We provide guidelines to facilitate the process and to standardize taxonomic annotations. The final outputs of this process are (1) a reference tree and alignment, (2) a reference sequence database, including taxonomic and environmental information, and (3) a list of putative chimeras and other artifactual.
Building a better database Environmental sequencing may be transformative in all the ways mentioned above, but the resulting datasets are only as good as the reference database used to annotate the data. Reference databases of ribosomal DNA bring together sequences from known isolates as well as Sanger-sequenced environmental datasets. The two main databases for eukaryotic ribosomal DNA sequences are SILVA  [12] , a general database that also includes Bacteria and Archaea ribosomal DNA, and the more specialized Protist Ribosomal Reference Database (PR 2 )  [13] ; many researchers also use the International Nucleotide Sequence Database Collaboration (INSDC), which encompasses the DNA DataBank of Japan (DDBJ), GenBank, and the European Nucleotide Archive (ENA) database resources  [14]. These existing databases differ in numbers of sequences, taxonomic annotations, number of taxonomic ranks, and even inclusion of major lineages of eukaryotes  [15]  (Fig  1 ).
Thus, the database used for annotation strongly influences the taxa and the taxonomic resolution reported in a given study. While these databases have improved significantly in recent years, substantial challenges remain, largely because of changing views on eukaryotic relationships  [16]  and the influx of vast amounts of data from environmental sequencing that continues to reveal new lineages. This influx is reflected in the majority of sequences in INSDC that are annotated as unknown or environmental sequences from a given location (Fig  1A ). These should ideally be incorporated into reference databases and novel clades assigned stable names so that they inform refinements in taxonomy of relevant eukaryotic clades. Stable names also ensure that new studies do not "rediscover" the same lineages, but rather refine what we already know about their diversity and distribution.
Ideally, databases should flexibly handle several taxonomic ranks in a way that enables researchers to use standardized levels when necessary. To illustrate the impact of database choice on interpretation of a dataset, we annotated one dataset-BioMarKs, a survey of coastal marine microbial communities  [9] -with the three major databases. At the highest taxonomic rank (the rank below "Eukarya"), this coastal marine survey appears to be dominated by unknown organisms identified as environmental sequences when annotated with GenBank, but to varying degrees by lineages in Stramenopiles (e. g. , diatoms), Alveolates (e. g. , dinoflagellates and ciliates), and Rhizaria (SAR) when annotated with SILVA and PR 2 (Fig  1A ). Going into greater detail, we compared the taxonomic annotation of the dominant taxa in BioMarKs using the three databases (Fig 1C  , S1 Table).
First of all, we had to normalize the annotation, because the ranks in the different databases do not represent the same taxonomic level. As expected, the different databases annotated a different number of ranks for each taxon, but we also found that the number of ranks between taxa was not the same for GenBank and Silva (PR 2 has a defined number of ranks). Furthermore, for the taxonomic levels for which the annotation completeness was better, i. e. , Genus and Species, the degree of agreement between databases was low; annotations agreed in only 15% at the Genus level and 5% at the Species level when comparing the three databases. The agreement between two of the annotations was 40% at the Genus level and 50% at the Species level. The situation was similar for the rest of the taxonomic ranks that showed a completeness higher than 50% (Class, Phylum, "Supergroup," and the unranked level below Supergroup).
The rest of the taxonomic levels were poor in terms of retrieved annotations and agreement between databases. EukRef is a community effort funded by the Moore Foundation, the United States National Science Foundation, and the International Society of Protistologists. It aims to improve the taxonomic information associated to 18S rRNA sequences as well as assemble environmental metadata that provide context, and to create better reference databases for metabarcoding/ amplicon studies. EukRef is part of the UniEuk project (unieuk. org), which aims to provide a comprehensive taxonomic framework for eukaryotes  [21]. EukRef (Box 1) complements existing efforts curating multicellular eukaryotic taxa by filling in major gaps in our knowledge for the protists that comprise the breadth of the eukaryotic tree of life, and its products will be incorporated back into the established SILVA and PR 2 databases.
To be useful, database curation must be initially done by those with knowledge of the taxonomy, phylogeny, and ecology of a given group of interest so that informed decisions can be made during the curation process. Curators can incorporate relevant information from diverse sources such as traditional classification schemes, phylogenomic studies, historical literature, morphological observations, and distribution data from high-throughput sequencing studies  [22]. Such expert knowledge enables researchers to generate robust classification schemes for lineages known only from sequences, such as the diverse marine stramenopiles (MASTs  [23] ), and can provide a mechanism for workers to link described organisms to proposed environmental MAST lineages (e. g. , Solenicola setigera and Incisomonas marina, which belong to the MAST-3  [24, 25]  clade, or Pseudophyllomitus vesiculosus, which belongs to the MAST-6 clade  [26] ).
Many parts of the eukaryotic tree of life are currently known either from sequences or from morphological records, but not both  [27, 28] , so enabling morphological data to inform molecular classification is crucial  [21]. While ribosomal DNA is invaluable for taxonomic classification, this information alone is unable to reliably disentangle deep eukaryotic relationships and is most powerful when combined with insight from multigene analyses (including phylogenomics) and/or morphological data. To address this, EukRef supports a flexible curation approach that can incorporate expert knowledge as well as insight from multigene molecular analyses and morphological studies. This approach differs from that used by SILVA and Greengenes  [29] , which either rebuild the ribosomal RNA tree for the whole dataset from scratch (Greengenes) or insert sequences into an existing and fixed alignment and tree (SILVA).
The use of backbone constraints based on published and robustly established relationships from phylogenomics and morphology facilitate incorporation of this knowledge and will be helpful in cases in which this is warranted by existing data. For example, Fungi is a very well-established group of eukaryotes but generally appears as polyphyletic in ribosomal DNA trees without backbone constraints  [30]. EukRef guidelines recommend a tiered approach assessing multiple analyses to compare phylogenetic structure, with and without constraints, to understand their impact. In addition, we should be aware that novel lineages may have new insertions in ribosomal DNA or other differences that require rebuilding of the alignments. The improved reference database and phylogenetic tree will in turn enable better annotation of subsequent high-throughput sequencing studies.
Box 1 Definitions as used by EukRef: • Low level: less inclusive taxon (e. g. , genus), • High level: more inclusive taxon (e. g. , phylum), • Database: refers to tab-delimited file constructed by EukRef curators containing information about the identity of a sequence, its classification, and environmental metadata, • Clade: used here to refer to a clade in a phylogenetic tree, • Taxon: a group of organisms that has been assigned a name in previous literature (e. g. , a genus or a species), • Group: a lineage or clade in a phylogenetic tree being curated, • Chimera: DNA sequence that stems from two or more distinct sequences generated as a product of the DNA amplification process. framework for eukaryotes. Here, we present the standardized guidelines and open source operational tools developed by this initiative, also available through eukreg.
org. The final outputs of EukRef for each group are (1) a phylogenetic reference tree and alignment, (2) a curated reference database with accession numbers, curated classification string, and curated metadata, and (3) a list of sequences known to be problematic (such as chimeras). To enable efficiency and consistency, the EukRef pipeline was developed to curate and annotate diverse eukaryotic lineages by researchers who are experts in that group, to comprehensively capture its existing sequence diversity (Fig  2 ). Curation starts with a broadly sampled alignment and corresponding 18S rRNA phylogeny. EukRef targets the18S rRNA gene because existing databases rely on this marker and because it best captures the breadth of eukaryotic diversity, but other markers could theoretically be used in a similar manner.
This initial set of sequences is prepared by the curator and becomes the input to the EukRef workflow. The first step is an iterative retrieval of sequences from GenBank (INSDC) by BLAST  [31]  using a similarity threshold defined by the user, depending on the targeted lineage. During this step, sequences shorter than 500 bp and chimeras automatically detected by VSEARCH  [32]  are excluded. The expanded set of sequences retrieved from GenBank, together with the input sequences and relevant outgroups, are aligned using MAFFT  [33]  (a widely used and accurate alignment program  [34] , although curators can use another multiple sequence alignment program of their choice), then automatically trimmed using trimAl  [35]  and used for phylogenetic inference with RAxML  [36] , which can readily handle typical datasets of hundreds to several thousands of sequences.
The resulting tree is the starting point for curation and classification of sequences. Curators then manually examine the tree to identify discrepancies such as long branches, which may be potential artifacts or chimeras that escaped the initial filtering, that should be removed (Fig  2 ). Following removal of these problematic data, a new alignment and tree are constructed with the remaining sequences. EukRef scripts then use the GenBank accession numbers for these sequences to retrieve the classification string and relevant metadata from GenBank and to organize this information in a tab-delimited file. This information, together with the tree, is the starting point for classification of the group and for each sequence. These outputs are combined with previous taxonomic knowledge, and improved metadata are manually incorporated throughout this process.
As a community, we have established guidelines for annotating sequences based on the phylogenetic tree and classification as part of the curation process, including guidelines for naming environmental clades. The curation process brings previous literature and expert knowledge to bear in annotating clades on the 18S tree generated in the EukRef pipeline, but current informal names are largely ad hoc, especially those referring to environmental sequence clades; some degree of consistency would be both simpler and more informative. The proposed annotation guidelines are designed to be practical, stable, and compatible with downstream analyses that will use the curated databases. We recommend a conservative approach that minimizes the introduction of new names by relying on published literature and only assigning names to well-supported clades.
4. Only annotate to the level for which there is support. Fill in blank ranks by propagating down from the higher levels (more inclusive) to lower levels. 5. Do not name clades that are not supported or clades for which the applicability of a name is ambiguous (see website for examples and detailed guidelines). 6. Novel environmental clades may be named following the "Naming environmental clades" guidelines (below). Our approach also provides tools for attaching biological and environmental information to each sequence in the curated database, including basic habitat information and whether a sequence came from a culture or morphologically identified isolate, or an environmental survey. Host associations are reported in the case of host-associated lineages (Box 3). To make this tool, we adopted standardized metadata annotation: Minimum Information about any (x) Sequence (MIxS)  [37]  and Environment Ontology (EnvO)  [38].