dataesr/scientific-paragraphs-categorization huggingface dataset:
Data curation and quality control. Starting from assembled contigs, we combined all genomic fasta files by datasets and obtained 6 and 1. 9 billion contigs from terrestrial and plant-associated categories, respectively. The length distribution of assembled contigs is highly heterogeneous between metagenomes due to variation in sequencing technologies, experimental protocols, pipeline version used and biological features. Probably because most data initially consisted of short sequencing reads, half of the contigs were shorter than 296 bp (Table  1 ). These short contigs increase the volume to be processed and are unlikely to contain complete genes, hence providing no more information on gene diversity  27. Thus, we filtered data on assembly length, and kept contigs at least 1 kb long or containing at least three genes predicted by Prodigal in JGI files.
Only 763 million contigs (10%) passed this filter and were retained for further analysis (Fig. 1 ). These remaining contigs were distributed in 6,610 metagenomes (Fig. 1 ): hence data from 262 starting metagenomes were entirely removed due to a too high level of fragmentation. This quality filtering drastically reduced the dataset volume and ensured we only worked with contigs on which complete eukaryotic genes have a chance to be predicted.
Detection of contigs from eukaryotic organisms. The JGI provides taxonomic information for genes predicted using Prodigal, which is not suitable for eukaryotic genes. Moreover this taxonomic information has been transfered solely from that of the best BLAST hit, which can be misleading. Thus, this information cannot be used to identify potential eukaryotic contigs and no further taxonomic information is provided for contigs. Therefore, we scanned all the contigs and identified those from eukaryotic origins using Kraken2 28 , a taxonomic classification tool based on exact kmer matches, designed to process in a fast and sensitive way large data sets such as those from metagenomics analyses. Among the taxonomic classifiers dedicated to metagenomic data, we selected Kraken2 because it provides taxonomically labeled contigs and it is designed to work on reads but can also process contigs.
As a consequence, this software maintains a good sensitivity on short sequences, representing an ideal choice in our case. Indeed, as indicated in Table  1 , our data mainly contains short contigs with an average size of 1. 9Kb, which would be sub-optimal for usage with a contig-centered software such as Eukrep  20  ,that performs better on contigs at least 3Kb long, as mentioned by the authors. As a reference database, we combined all RefSeq libraries of complete genomes [Archaea, Bacteria, Plasmid, Viral, Human, Fungi, Plant, Protozoa]  29  , complemented by the NCBI's nt library and ran Kraken2 with default parameters. This allowed assigning taxonomic information to 82% of contigs, among which 113 million were classified with a eukaryotic taxonomic identifier 'TaxID' (Fig. 1 ).
Eukaryotic gene prediction. For all contigs identified as eukaryotic by Kraken2, we used Augustus (v3. 3), a software dedicated to de novo eukaryotic gene prediction  30. The gene structure is complex in eukaryotes and changes across species  27. Thus, Augustus provides ab initio models for 73 different species (Fig. 2 ) and one must be selected to perform gene prediction. Due to the conservation of genomic features across closely related organisms, we assigned, to each eukaryotic contig, a model based on its Kraken2 taxonomic annotation. Note that this model selection step does not aim at a definitive taxonomic annotation; here we used a sensitive approach to predict as accurately as possible putative eukaryotic genes that will then be filtered by a more selective Fig. 1  Our eukaryotic protein prediction pipeline from soil metagenomic contigs to a final dataset of taxonomically annotated proteins with contigs, proteins and metagenomes number at each step.
homology-based taxonomic annotation approach at the protein level. Selection of the phylogenetically closest model for gene prediction on each contig was done using a custom python script  31  which functions as follows: • First, we browsed the 73 model species tree from the leaves to the root assigning a non-ambiguous parental taxonomic term to each model species as long as no bifurcation with a branch containing another model species was found (Fig. 2 ). For example, in plants, Arabidopsis thaliana is the sole representative of the Brassicales; so the Brassicales parental term was associated with the A. thaliana model. Consequently, we used the A. thaliana Augustus model for all eukaryotic contigs assigned with a taxonomic ID belonging to the Brassicales branch. Similarly, Homo sapiens is the only representative of mammals, so any contig identified by Kraken2 as a mammalian organism will be assigned the H.
sapiens model. At this point, an Augustus gene prediction model could be assigned to 7. 1% (ca. 8 million) of contigs. The rest of the contigs (ca. 105 million) could not be assigned an unambiguous closest model species because they belonged to a bifurcating branch in the tree leading to several equally close model species. • Therefore, in a second step, for all these remaining eukaryotic contigs, we selected among the children branches the most frequently assigned model in the whole dataset the contig belongs to (i. e. Plant-associated, Terrestrial 1 or Terrestrial 2) at the previous step (first pass). To continue the previous example, the next more ancestral branch in the phylogeny of Brassicales is the clade 'Malvids' that displays a polytomy of eight children branches of which only two contain an Augustus model species (Malvales and Brassicales).
Hence, no model could be unambiguously assigned to contigs with a Malvids taxonomic ID other than Malvales or Brassicales. Therefore, all contigs from other Malvids orders are processed with the most frequently assigned species model for each of the three datasets (Fig. 3 ). For example, they are processed with the cocoa gene model (Malvales, Theobroma cacao) in the dataset Terrestrial 1, or the Arabidopsis thaliana gene model in the Plant-associated and Terrestrial 2 datasets (Fig. 3 ). The distribution of contigs across the models is available in Supplementary Data, Fig. 1  32. Overall, our pipeline allowed assigning an Augustus model to ca. 101 million possibly eukaryotic contigs (Fig. 1 ). The most assigned ones were Metazoa and Viridiplantae models, with respectively 49% and 44% of contigs in plant-associated metagenomes and 76% and 16% in terrestrial data.
In both datasets, we assigned fungal models to 6. 5% of contigs (Supplementary Data, Table  1 33  ); and the majority of other contigs were assigned to SAR, Discoba or Rhodophyta models. Although these last taxonomic groups were assigned at a relatively low proportion, this still corresponds to tens or hundreds of thousands of contigs. Unsurprisingly, the less assigned are gene models of aquatic animals such as some benthic animals, sharks, or also lamprey models. At this point, we could not assess whether the numerous assignments to metazoan and plant models came from mis-annotated contigs or contamination, therefore further analyses were performed after gene and protein prediction. Then, once a model species has been assigned to contigs we ran the eukaryotic gene predictor Augustus 30 , with default parameters, which allowed predicting 93 million protein-coding genes (Fig.
1 ). The number of proteins predicted per contig ranges from 1 to 410 with 2 protein predicted per contig on average for all datasets together. Consistent with the model assignment across kingdoms, the highest numbers of proteins were predicted for contigs assigned to Metazoan and Viridiplantae Augustus models. Moreover, we predicted 8. 7 million proteins with Augustus fungal models and 1. 8 million with different protist models (Fig. 4a ).
Confirmation of eukaryotic origins and improvement of the taxonomic information. To filter-out false-positive eukaryotic classification and assign a more reliable taxonomic annotation to the proteins predicted by our pipeline than simple inheritance from the Kraken2-based contig annotation, we used the last common ancestor algorithm of Diamond  34. The homology search was run at the protein level with an E-value threshold of 10 -6 and using the January 2020 release of the NCBI nr database  29  as protein reference. In the LCA mode, Diamond will assign an NCBI taxonomic identifier (i. e. TaxID) based on the last common ancestor of all the hits with a score not diverging by more than 10% from the best hit score. Using an LCA approach constitutes a substantial gain in taxonomic annotation reliability compared to approaches based on the best BLAST hit alone, this single best hit being potentially mis-annotated itself, or sharing only low identity with the query sequence.
This LCA approach is usually employed for taxonomic assignment of sequences distantly related to those of known organisms present in public sequence libraries, such as ancient or actual metagenomic data  28, [35] [36] [37] [38]. Consequently, the improved quality and completeness of protein sequences combined with a more accurate taxonomic assignment method is expected to yield a more reliable taxonomic annotation. From the 93 million proteins predicted by Augustus, 8,001,326 (present on 5,724,823 contigs from 4,083 metagenomes, Fig. 1 ) were assigned a eukaryotic taxonomic annotation by the Diamond LCA approach (Table  2 ) and are made available as a curated dataset of eukaryotic soil proteins  39  with taxonomic informations  40.
Data Of these 8 million proteins, 45% were assigned a Opisthokonta taxonomy (Fungi + Metazoa), of which 96% were fungal and only 4% Metazoa (Fig. 5 ). These proportions are consistent with eukaryotic taxonomic distribution previously described in the literature, reporting fungi as the most abundant eukaryotic microorganisms in studied soil  2, 41. Actually, in soil metagenomes, fungal organisms are often second to bacteria in number and account for a comparable proportion of the biomass. Here, we retrieved 1,657 different fungal TaxIDs covering granularity levels ranging from species to the whole kingdom. Taxonomic annotations at deeper taxonomic nodes indicate the protein is equally related to proteins from multiple different and phylogenetically distinct fungal species. Among Metazoa, the dominant categories were Arthropoda, then Nematoda and Rotifera, respectively representing 48%, 9% and 8% of Metazoa (Supplementary Data  42  ), again consistent with these species being the most abundant animals in soil environment.
Besides Opisthokonta, Viridiplantae was actually the most represented kingdom, with 49% of all eukaryotic taxonomic assignment (Fig. 5 ). This suggests plant material is frequently present in soil samples and this is particularly expected for the plant-associated samples. Besides Opisthokonta and plants, other eukaryotes mainly belonged to the category SAR (1% of all) and most of the rest (5%) were unclassified eukaryotes (category other eukaryota, Fig. 5 ). These last taxa show small percentage of the whole dataset of soil eukaryotic proteins but still represent several thousand of proteins due to the size of the dataset. The rest of the 93 million soil proteins were either assigned a non-eukaryotic TaxID with 47. 6% and 0. 1% being assigned a prokaryotic and viral taxonomy, respectively (Table  2 ), or had undetermined taxonomic annotation (43.
Identification of potential orphan eukaryotic proteins. More than 40 million proteins, representing 43. 7% of the total Augustus predictions, could not be assigned a prokaryotic, eukaryotic or viral TaxID. Among them 27,269,572 (67%) were assigned untraceable taxonomic identifiers such as 'unclassified' (e. g. '12908' TaxID) or 'other' (e. g. '32644' TaxID), and the rest of the proteins (13,561,566 or 33%) simply returned no hit at all against  the NCBI's nr library in our Diamond homology search. Because these proteins might represent false positives from Augustus, they were not blindly added to the dataset of 8 million eukaryotic soil proteins. However, these proteins might as well represent orphan eukaryotic proteins lacking homology in public databases, constituting an important resource for new discoveries.
To discriminate potential eukaryotic from non-eukaryotic orphan proteins, we assessed whether they were distributed on otherwise mostly-eukaryotic contigs. Thus, from the initial dataset of 113 million Kraken2-assessed eukaryotic contigs, we only retained orphan proteins present on contigs that contained at least 50% of Diamond-confirmed eukaryotic protein-coding genes. This yielded a total of 3,657,380 contigs distibuted on 4,059 metagenomes Data, Table  3 33  ). A total of 354,243 orphan proteins were distributed on these contigs and represent potential novel eukaryotic proteins. We made this additional dataset of potential novel orphan eukaryotic proteins also available  43. Reducing redundancy of soil eukaryotic proteins. Some redundancy was expected because we used metagenomic data from thousands of individual studies, and some sequencing data came from the same sampling location.
Therefore, we clustered Fasta files using the Linclust software of the MMseq2 metagenomic toolkit  44. For both eukaryotic and orphan datasets, we clustered proteins with at least 99% sequence identity and covering at least 90% of the target. With these parameters, the 8 million eukaryotic proteins were clustered in 4,624,994 representative sequences  45  , and the 354,243 orphan proteins were clustered in 288,612 proteins  46. For both clusterings, we provide the correspondence files to link original protein predictions to their respective representative clusters  47, 48.
Technical Validation Comparison of protein prediction and taxonomic annotation quality to original JGI annotation. To determine whether using Augustus in our pipeline allowed improving eukaryotic protein predictions, we compared them to the predicted proteins obtained by the JGI using Prodigal for the same set of contigs. For this comparison, we used the same 3,657,380 contigs (covering 4,059 different metagenomes) containing at least 50% of predicted proteins with a eukaryotic taxonomy assigned by Diamond-LCA (defined above). Our pipeline allowed predicting 5. 6 million proteins in these contigs. In comparison, on the same dataset, Prodigal initially predicted a total of 16 million proteins, covering 3,294,764 of these contigs and 3,979 metagenomes (Supplementary Data, Table  3 33  ). First, although the number of protein predicted is higher with Prodigal, this software was unable to predict proteins in more than 360,000 contigs (3,657,380-3,294,764).
The proportion of BUSCO genes found in complete length in metagenomes was significantly higher for the Augustus predictions than for the initial Prodigal predictions (Fig. 7 ; paired Wilcoxon-test, n = 1,093 metagenomes, Complete T = 1. 132 10 5 , p ≤ 10 -4 ; Fragmented T = 2. 039 10 4 , p ≤ 10 -4 ). Similarly, the proportion of fragmented and missing BUSCO genes were significantly lower in Augustus predictions as compared to Prodigal predictions; this trend is identical for all datasets (Supplementary Data, Fig. 2 32  ). BUSCO completeness scores from our Augustus gene predictions are as good or better than Prodigal for more than 98% of metagenomes. Furthermore, we have predicted more universal single-copy genes than Prodigal for 574 metagenomes, or more than half of the 1,093 metagenomes containing at least one BUSCO gene in one of both predictions.
Validation of taxonomic assignment and gene prediction strategy. Taxonomic assignment methods. Comparing the Kraken2-assigned Augustus models for gene prediction on contigs to the taxonomic assignment at the protein-level based on Diamond LCA, we observed substantial differences in the relative proportion of taxonomic groups (Figs. 4, 5). For instance, while metazoan Augustus models were assigned to 49 and 76% of contigs in plant-associated and terrestrial datasets, respectively, only 2% of the eukaryotic proteins were assigned a metazaon taxonomy via Diamond-LCA. Conversely, while fungal Augustus models were assigned to only 6. 5% of contigs, fungi represented 43% of taxonomic assignments obtained by Diamond-LCA. These Diamond-LCA taxonomic assignments are more consistent with the expected dominant taxa in the soil and illustrate the interest of our two-steps strategy with the first sensitive step aiming at identifying as many putative eukaryotic contigs as possible and the second specific step aiming at assigning an as reliable as possible taxonomic annotation to the genes and proteins.
This fact may cast doubt on Augustus model selection procedure. To evaluate our soil eukaryotic gene prediction strategy, we compared the quality of proteins obtained by a mixture of Augustus models selected by our pipeline with prediction using either only one Augustus model, chosen as (i) Fusarium (a fungal model retrospectively corresponding to the most taxon in the final Diamond-based taxonomic assignment), or (ii) Zebrafish (a metazoan model corresponding to a taxon with low chance to be actually present in soil contigs), or (iii) another gene prediction software, MetaEuk  54  with NR database as a reference and finally (iv) MetaEuk with SwissProt database as a reference. These four strategies were used to predict proteins on Kraken2-assigned eukaryotic contigs within the same dataset: metagenome '3300031471' from the terrestrial dataset.
This metagenome was randomly chosen among those containing complete eukaryotic BUSCO genes and thus representing an easy reference to check whether our mutli-model Augustus approach was relevant compared to single-model or reference database approaches. We compared eukaryotic BUSCO scores as well as the number of predicted proteins and the number of amino-acids per protein. Concerning Augustus, we observed the best recovery of universally conserved genes using our procedure (mixture of phylogenetically assigned models) (Table  4 ; line 1,2,3). Hence, although fungi and in particular Fusarium were the most numerous taxa in soil metagenomes, a mixture of models chosen by our procedure allowed a better recovery of BUSCO proteins. Thus, despite a necessarily substantial portion of imperfect model assignments, due to discrepancy between a priori Kraken2 taxonomic assignment and a posteriori Diamond taxonomic confirmation, a mixture of models seem to yield better results than a single phylogenetically close model.
Usage Notes Current microbiology investigations are focused on addressing the factors shaping the structure of microbial communities. To drive the development of tomorrow's biotechnology it is essential to understand biological pathways both at the organism level and at the inter-microbial relationships scale, for prokaryotic and eukaryotic organisms together. This dataset provides a more complete and comprehensive view of the pool of genes and proteins, genetic diversity and distribution of eukaryotic microbes in soil and plant-associated microbiomes. At the molecular level, the use of this data is relevant to address biological questions in both fundamental research on plant-microbe interactions and applied, agronomical research, such as the study of potential metabolic functions of telluric eukaryotes, or of the interaction pathways between microbial members of the community.
At a broader scale, the more accurate taxonomic annotation provides an unparalleled opportunity to assess how microbial eukaryotes are distributed across the soil and plant-associated microbial-environments. As illustrated in the data validation section, this improvement of microbial eukaryote representation has allowed us to increase by a factor of six times the detection of the ubiquitous AMF species, which are of high agronomic and economic interest. Any research involving study of soil eukaryotes from evolutionary research on gene flow and transfers within the biome to more translational research aiming at deciphering important soil functions and biochemical pathways will benefit from this improved dataset of soil proteins with more accurate taxonomic annotation. In addition, our data can be cross-referenced with the metadata provided by the JGI (downloadable from the IMG/M portal) which includes geo-tracking and a wealth of environmental, sampling and processing information on each metagenome.
They can be linked to proteins and annotations by searching for the 'metagenomeID' , as each protein name in our dataset has a nomenclature based on the following pattern: 'contigName_metagenomeID. geneID' , to offer this possibility. On one hand, the ecological metadata provides an unprecedented potential to study the effect of the environment on community structures and to have a better, more comprehensive view on how external factors influence the eukaryotic soil microbial communities. On the other hand, metadata on sampling and processing could be useful to assess which parameters affect the diversity and sequencing of eukaryotes in metagenomes and help to shape future protocols. Moreover, in this study, we provided a fully documented pipeline and protocol available as python scripts from the detection of putative eukaryotic contigs to the ab initio model selection for Augustus gene prediction and further Diamond-based taxonomic annotation, that can be re-used to improve the annotation of eukaryotes on any microbiome data, including in other biomes than the soil.
acknowledgements We would like to warmly thank for they help and support all members of the bioinformatics platform of the  Institute Sophia Agrobiotech, Sophia Antipolis, France. We also thank for his advice on soil microbial analyses  Samuel Mondy  from  INRAE, Dijon ,  France. Finally , we would like to thank all the persons who worked to generate the data publicly available on the IMG/M platform. In particular, we thank the following labs for having granted us the right to use their data despite the use-restrictions:  Dr. Rich V. and the  NSF Biology EMERGE Integration Institute , ( NSF-BII 2022070 );  Dr. DiRuggiero J. and  The Johns Hopkins Department of Biology ;  Dr. Pietrasiak  and  NMSU Plant and Environmental Sciences department of College of Agricultural, Consumer, and Environmental Sciences (ACES).
tsv 48 TSV 27MB. Composition of orphan protein clusters eukaryotic_proteins_taxonomy_krona. html 42 html 1,7MB. /Supplementary Data Krona representation of 8 M of validated eukaryotic protein taxonomy from last common ancestor algorithm of Diamond Supplementary_data_1. txt 24 text file 158KB. /Supplementary Data List of metagenome identifier of processed data from JGI Fig. 1: Informations on eukaryotic proteins Supplementary_data_Figures. pdf 32 PDF 323KB. /Supplementary Data prediction processing Fig. 2: BUSCO scores by dataset Supplementary_data_tables. pdf 33 PDF 51KB. /Supplementary Data.
Abstract-Android security has received a lot of attention over the last decade, especially malware investigation. Researchers attempt to highlight applications' security-relevant characteristics to better understand malware and effectively distinguish malware from benign applications. The accuracy and the completeness of their proposals are evaluated experimentally on malware and goodware datasets. Thus, the quality of these datasets is of critical importance: if the datasets are outdated or not representative of the studied population, the conclusions may be flawed. We specify different types of experimental scenarios. Some of them require unlabeled but representative datasets of the entire population. Others require datasets labeled with valuable characteristics that may be difficult to compute, such as malware datasets.