dataesr/scientific-paragraphs-categorization huggingface dataset:
Background: Viruses are a significant player in many biosphere and human ecosystems, but most signals remain "hidden" in metagenomic/metatranscriptomic sequence datasets due to the lack of universal gene markers, database representatives, and insufficiently advanced identification tools. Results: Here, we introduce VirSorter2, a DNA and RNA virus identification tool that leverages genome-informed database advances across a collection of customized automatic classifiers to improve the accuracy and range of virus sequence detection. When benchmarked against genomes from both isolated and uncultivated viruses, VirSorter2 uniquely performed consistently with high accuracy (F1-score > 0. 8) across viral diversity, while all other tools under-detected viruses outside of the group most represented in reference databases (i.
Given the sheer magnitude and significant importance of the virosphere across diverse ecosystems, establishing a broad genomic catalog of Earth's viral diversity is critical. These efforts will rely on automated detection of viral genomes across a broad range of sequencing datasets  [9]. Currently, two general computational approaches exist to identifying viral sequences. One set of tools rely on a combination of gene content and genomic structural features to distinguish viral from microbial sequences, including Prophinder  [27] , PhiSpy  [28] , VirSorter  [29] , the Earth's Virome pipeline  [17] , PHAS TER  [30] , MARVEL  [31] , and VIBRANT  [32]. These genomic features are either statistically compared to a null model (Prophinder, VirSorter, PHASTER), or more recently have been used as input for automatic machinelearning classifiers (MARVEL and VIBRANT).
The other approach uses the frequencies of DNA "words" (i. e. , kmers) found in known viral and cellular genomes as signatures to train machine-learning classifiers to recognize new viral and microbial sequences (e. g. , VirFinder and DeepVirFinder  [33, 34] ). Both approaches efficiently detect common viruses that are well represented in databases, such as dsDNA bacteriophages from the Caudovirales order  [31, 32] , but they struggle with less welldocumented viruses like ssDNA viruses  [35] , RNA viruses  [36, 37] , and viruses that infect archaea  [38, 39]. One reason for this is that current approaches consider viruses as a single cohesive group for detection purposes, which is potentially problematic given the varied ecological and evolutionary rules that govern the diversity and evolution of different viral genomes  [40].
While some features may span the virosphere (e. g. , enrichment of uncharacterized genes, relative to microbes), others are group-specific (e. g. , hallmark genes, specific genomic structure, or the presence of metabolic genes). Though machine learning approaches based on nucleotide composition would not suffer as much from viral database representation, they tend to confuse any unusual sequence as viral (e. g. , plasmids or eukaryotic genome fragments  [39] ). Together, these observations call for the field to move beyond a single model to represent the virosphere for virus identification. Here, we develop and introduce a new viral sequence identification tool, VirSorter2, which leverages recent sequencing efforts for under-represented viral groups to develop customized automatic classifiers that improve detection of viruses in the order Caudovirales (the focus of the original VirSorter tool  [29] ), while also identifying other major virus groups across a broad range of hosts, genome lengths, and genome complexity.
Results and discussion The VirSorter2 framework Viral sequence identification in VirSorter2 occurs via three steps: (i) input sequences are automatically annotated and relevant features are extracted, (ii) each sequence is scored independently using a set of classifiers customized for individual viral groups, and (iii) these scores are aggregated into a single prediction provided to the user (Fig. 1 ). Annotation of the input sequences follows current standards in the field  [9]  including coding sequence (CDS) identification with Prodigal (version 2. 6. 3) [41] , and annotation of predicted CDS using HMMER3 (version 3. 3)  [42]  against Pfam (release 32. 0)  [43]  and a custom comprehensive viral HMM database including Xfams (described in the "Methods" section) and viral protein families (VPF) from the JGI Earth's virome project  [17].
Within this custom viral HMM database, profiles corresponding to viral hallmark genes were manually identified based on functional annotation and distribution across viral and microbial genomes for different viral groups. These include structural genes such as major capsid protein (MCP) or terminase large subunit for viruses in the order Caudovirales (as done in the original VirSorter); RNA-dependent RNA polymerase (RdRP) for the viral kingdom Orthornavirae (RNA viruses); MCP for viruses in the family Microviridae; replication-associated protein for viruses in the phylum Cressdnaviricota  [44] ; ATPase for viruses in the family Inoviridae; MCP, pATPase, primase, transcription elongation factor (TFIIS), and viral late gene transcription factor (VLTF3) for viruses in the phylum Nucleocytoviricota (i.
To train the different classifiers, reference genomes for each viral group were split into training and testing sets with a 9 to 1 ratio, respectively, and 5 random fragments were generated from each genome starting at a random position and with size ranging from 1 kb to complete genomes. Separate classifiers were trained for the following viral groups (i) caudovirids (Caudovirales) and comparable dsDNA phages (e. g. , corticovirids), (ii) NCLDVs, (iii) RNA viruses, (iv) ssDNA viruses, and (v) lavivirids (Lavidaviridae). Separately, a negative (non-viral) dataset was generated from genomes of three groups: (i) bacteria and archaea, (ii) eukaryotes (fungi and protozoa), and (iii) plasmids. The training and testing sets for non-viral genomes were generated the same way as the viral set described above.
For each input sequence to be classified, all random forest classifiers are first applied to the entire sequence. If the score obtained with one or more of the classifiers is above the cutoff set by the user (the default was 0. 5, as used in the classifier training step), the score is considered significant and the sequence was considered as entirely or near-entirely viral. To identify potential host regions on the edge Fig. 1  Overview of the VirSorter2 framework. Schematic of the viral prediction pipeline used in VirSorter2. "hmmDB" represents databases of HMM profiles including viral HMMs from Xfam (described in the "Methods" section) and viral protein families (VPF) from JGI Earth's Virome  [17] , and cellular HMMs (archaeal, bacterial, eukaryotic) as well as "mixed" HMMs (not specific to either virus or cellular organisms) from Pfam  [43].
Expanded and manually curated databases enable a robust detection of viruses in the order Caudovirales We first evaluated the new VirSorter2 approach on genomes in the order Caudovirales. As described above, test sequences included both phage isolate sequences from NCBI Viral RefSeq  [49]  and uncultivated phage genomes obtained from metagenomes or identified as proviruses in bacterial and archaeal whole genome shotgun sequencing (  [15, 16, 48] , see "Methods" section). Test sequences also did not overlap with the references used to train the classifier. As negative control, the same number of sequences from the non-viral testing set were included. The same test sequences were also processed with established viral sequence detection tools including VirSorter  [29] , VirFinder  [33] , DeepVirFinder  [34] , MARVEL  [31] , and VIBRANT  [32].
8 with sequences 5 kb and longer (Fig. 3a , Figs. S1 & S2). As previously observed  [29, 39] , all approaches also displayed a decreased accuracy with shorter sequences, especially when reaching lengths of ~1-2 kb (Fig. 3a ). While increased accuracy with sequence length was also observed when evaluating the same tools with uncultivated viral sequences of the order Caudovirales, the overall performance of several tools was lower (Fig. 3b, c, Figs. S1  & S2 ). Specifically, the performance of VirSorter, VirFinder, DeepVirFinder, and MARVEL was decreased by 10-30%, while VIBRANT and VirSorter2 displayed F-score comparable to those when evaluated with the RefSeq sequences (± 5%), likely due to their larger reference database and their ability to use hallmark genes to identify viruses only distantly related to RefSeq genomes.
Similar results were obtained when analyzing viral sequences, either from RefSeq or uncultivated, for which < 25% of the genes were annotated as viral based on the custom HMM database (Fig. S3 ), confirming that the differences in tool performances were mostly due to the ability to identify novel viruses. To verify that the VirSorter2 approach was similarly efficient for integrated proviruses, we processed bacterial genomes from a gold-standard set of 51 microbial genomes  [51]  for which 82 integrated proviruses have been manually curated. As for viral contigs, VirSorter2 recall and precision was comparable or higher to all other tools (Table  2 ). Overall, the approach used by VirSorter2 to identify Caudovirales is thus on par or more efficient than other recently published tools for both isolate and environmental sequences, across a broad range of contig length, and for sequences both entirely and partially viral.
Plasmids and eukaryotic genomes represent unique challenges for viral detection tools While initial tests were performed using random fragments of bacterial and archaeal genomes as non-virus sequences, previous studies showed that eukaryotic genome and plasmid fragments were especially prone to be mis-identified as a putative viral sequence  [39]. We thus evaluated how VirSorter2 and other tools handled both eukaryotic and plasmid sequences, by processing datasets entirely composed of eukaryotic genome fragments or plasmid fragments and measuring at which frequency these would be wrongly considered as viral (Fig. 5 ). For eukaryotic genomes, both MARVEL and VirSorter showed high specificity with 2. 4% and 0. 1% false positives on average respectively, while VirSorter2 and VI-BRANT displayed on average 12.
VirSorter2 is easily available and can be scaled to handle large-scale datasets Beyond accurate and sensitive virus identification, scalability to process large (> 100,000 sequences) datasets and broad availability for users regardless of their knowledge of command-line or scripting are desirable traits in virus sequence detection tools. For the former, VirSor-ter2 can process an input dataset of 100,000 sequences totaling 4. 2 Gb in 28. 6 h using 32 threads. Its memory usage stays nearly constant with increasing data size, and scales nearly linearly with threads used (Figs. S8 & S9). Its total CPU time is higher than other tools (typically 2-10× higher, Fig. S8 ), but VirSorter2's internal processes are highly parallelizable, and written with snakemake, a cutting-edge pipeline management tool designed for computer clusters  [58].
VirSorter2 is thus able to utilize multiple cores in the same node and across multiple nodes efficiently, so that compute time would not be a limiting factor even for (ultra)large-scale datasets if state-of-the-art computing cluster resources are available (Fig. S9 ). Further, increased CPU time is mostly due to the annotation step with larger viral HMM database (> 90% of CPU time) and not to the use of several distinct classifiers, suggesting that additional viral groups and classifiers could be added to the VirSor-ter2 framework with minimal impact on performance. Finally, users can explore the annotation and per-model score computed by VirSorter2 for each individual sequence, and can generate new output files using different options and cutoffs based on an existing VirSorter2 directory without having to re-compute the annotation step.
To illustrate this and other differences between VirSorter and VirSorter2, we provide a detailed analysis of viral sequence detection from an ocean virome using both VirSorter and VirSorter2 (Supplementary Text, Fig. S10 , Additional files 4 and 5). In order to make VirSorter2 useful for the largest community possible, we provide it through multiple implementations and accompanied with extended documentation. Specifically, VirSorter2 is freely available at bitbucket (https://bitbucket. org/MAVERICLab/ VirSorter2), as an App on the CyVerse (https://de. cyverse. org/de), and available for local installation through a bioconda package (https://anaconda. org/ bioconda/virsorter). In addition, we provide an extensive step-by-step tutorial on how to run VirSorter2 and how to design new classifiers for additional viral groups and integrate this within a local VirSorter2 instance on protocols.
The addition of other features would also be a possible avenue for extending the range and/or accuracy of VirSorter2. Contrary to the addition of new viral groups, integrating new feature(s) would require substantial modification of Vir-Sorter2 code, and would thus be only associated with the release of new versions of the tool. Finally, while run times and computational resources required by VirSorter2 are compatible with modern omics datasets, the size of these datasets will likely keep increasing over the next few years, and may eventually limit the usefulness of VirSorter2. Given that most of the computation time is currently dedicated to the annotation step, this issue will require integration of scalable and improved sequence comparison tools (e. g. , MMSeqs2  [60] , HHBlits  [61] ) and/or new versions of the tools currently used.
Conclusion The automatic extraction of viral sequences from largescale sequencing data is now a cornerstone of the current viral ecogenomics toolkit. Identifying viral genomes from omics data enables unprecedented studies of viral taxonomic diversity  [62] , viral population distribution and associated eco-evolutionary constraints  [15, 16, 47] , and viral potential for microbial metabolic reprogramming  [20]. While current tools typically detect known and "standard" viruses, i. e. , viruses in the order Caudovirales, pretty well, the approaches available to date remain challenged by novel and/or unusual viral groups. VirSorter2 now provides a framework to go beyond the detection of viruses in the order Caudovirales and enables robust detection of all types of viruses in sequencing data. By defining subgroups of viruses with consistent genome features and unique markers, the Vir-Sorter2 framework is designed to grow as databases do such that new viral diversity can be readily detected in large-scale datasets.
Viral HMM database The viral HMM database includes viral HMMs from viral protein families (VPF) of JGI earth's virome project  [17]  and Xfams (https://de. cyverse. org/dl/d/648936 0D-1126-413B-A667-D18E39D5F2F1/viral_db_default. hmm). Xfams were generated from a large collection of viral sequences from the Global Ocean Viromes 2. 0 (GOV 2. 0)  [16]  and the Stordalen Mire Viromes (SMV)  [63]. Briefly, viral contigs were identified by VirSorter  [29] , DeepVirFinder  [34] , and MARVEL  [31]. The intersection between the three tools (Category 1 in VirSorter with "--virome" mode, score ≥ 90% in Marvel, and a score of ≥ 0. 9 with a p value of < 0. 05 in DeepVirFinder), were kept. Open reading frames (ORF) were predicted using prodigal with "--meta" mode  [41] , filtered by removing ones with ≥ 95% similarity to RefSeq's bacterial and archaeal proteins, and then clustered by ClusterONE  [64]  to get rid of singletons (-d 0.
3 -s 0. 2 --max-overlap 0. 8). Multiple sequence alignments were generated for each cluster by MUSCLE (--maxiters 4)  [65] , and then made into HMMs by hmmbuild in HMMER3 package  [66]. More details of Xfams can be found at iVirus in CyVerse's Discovery Environment (https://de. cyverse. org/de/): /iplant/ home/shared/iVirus/Xfams/version_0. 5/Xfams-XC.
For bacteria, archaea, and eukaryotes, five fragments were generated from a representative genome from each genus in the same way as described above; for plasmids, because no systematic global classification is available and the total number of plasmids in the input dataset  [67]  was only 6642 sequences, five fragments were generated from each sequence. The generated fragments from each of the three non-viral groups (bacteria and archaea, eukaryotes, and plasmids) were subsampled to the minimum within these three negative groups (bacterial and archaea, eukaryotes, and plasmids) and combined as a final "negative set. " For each viral group, the viral "training set" and the non-viral "training set" were evenly subsampled to the smaller size between the two, and then used for training the classifier.
Then the following 27 features were extracted from each sequence fragment: 1. "Hallmark gene count" is the count of hallmark genes in a viral sequence 2. "% of viral genes" is calculated as the percent of genes annotated as viral (best hit to viral HMMs) of all genes 3. "% of archaeal genes" is calculated as the percent of genes annotated as archaeal (best hit to archaeal HMMs) of all genes 4. "% of bacterial genes" is calculated as the percent of genes annotated as bacterial (best hit to bacterial HMMs) of all genes 5. "% of eukaryotic genes" is calculated as the percent of genes annotated as eukaryotic (best hit to eukaryotic HMMs) of all genes 6. "% of mixed genes" is calculated as the percent of genes with best hit to HMMs shared between viruses and cells 7. "% of unaligned genes" is calculated as the percent of genes with no hits to HMMs in the VirSorter2 HMM database (bit score cutoff 30) 8.
"SD of GC content" is the standard deviation of GC content of all genes in a sequence 17. "% of RBS motifs" is percent of ribosomal binding sites (RBS) with a specific motif. There are 11 types of motifs included: SD_Canonical, SD_Bacteroidetes, TATATA_3-6, OnlyA, OnlyT, DoubleA, DoubleT, Other_GA, NoA, Other, and None (no motif found)  [19]. Features 8-27 are all extracted from the gff output from prodigal  [41]. With the above features, "RandomForestClassifier" in scikit-learn package  [50]  was used to train the random forest classifier. "MinMax" scaler was used to scale all the feature data, and "GridSearchCV" was used to find optimal parameter sets among "n_estimators" of 20, 50, 100, 150, 200, and "criterion" of gini or entropy.
Accuracy To compare the accuracy of VirSorter2 to other viral identification tools, datasets with equal number  (50)  of viral and non-viral random DNA fragments were generated from the testing set created above and with different lengths (1. 5, 3, 5, 10, and 20 kb) to discern the impact of sequence length on accuracy. VirSorter2 (version 2. 0. beta) was run with "--include-groups dsDNAphage, NCLDV, RNA, ssDNA, lavidaviridae, --min-score 0. 5," and sequences in "final-viral-combined. fa" were considered as viral. VirSorter (version 1. 1. 0) was run with "--db 2 --virome -diamond," and sequences in categories 1, 2, 4 and 5 were considered as viral; VIBRANT (version 1. 2. 1) was run with "--virome -f nucl", and sequences in "VIBRANT_*/VI-BRANT_phages_*/*. phages_combined. fna" were considered as viral; VirFinder (version 1.
1) was run with VF. pred function within R as described in (https://github. com/jessieren/VirFinder); DeepVirFinder was installed directly from GitHub (https://github. com/jessieren/DeepVirFinder) with last commit ID of ddb4a9433132febe5cda39548cb9332669e11427 and was run with default parameters; for both VirFinder and DeepVirFinder, sequences with p value < 0. 05 were chosen as viral. Since MARVEL (version 0. 2) required each viral genome as a separate file, input sequences were first split so that each sequence was an individual sequence file, and then default parameters were used, and the sequences in "results/phage_bins" were considered as viral. F1 score (harmonic mean of recall and precision) was used as a metric for tool accuracy comparisons. For measuring false positives on eukaryotic and plasmid sequences, the fraction of eukaryotic or plasmid sequences classified as viral were calculated.
Computational efficiency To measure how tools scale with input data size, datasets with 10, 100, and 1000 sequences of 10 kb in length were generated in the same way as in accuracy benchmarking described above. To measure multithreading efficiency, datasets with 1000 sequences of 10 kb were generated and used as an input for VirSorter2, VirSorter, MARVEL, and VI-BRANT were run with 1, 2, 4, 8, 16, 32 threads. Vir-Finder was not included since it did not have a multi-threading option. DeepVirFinder had unexpected behavior with its multi-threading option ("-c"), i. e. , the "-c" option could not control thread number, so was also not included. CPU time and run time was measured with "/usr/bin/time -v" command available in the Linux operating system and peak memory was measure with "memusg" script (https://gist.
Funding Funding was provided by  NSF  (# OCE1829831 , # ABI1758974 ), the  U. S. Department of Energy  (# DE-SC0020173 ), and the  Gordon and Betty Moore Foundation  (# 3790 ) to MBS. The work conducted by the  U. S. Department of Energy Joint Genome Institute  is supported by the  Office of Science of the U. S. Department of Energy  under contract no. DE-AC02-05CH11231.
Availability of data and materials Source code of VirSorter2 is available at bitbucket (https://bitbucket. org/ MAVERICLab/VirSorter2). VirSorter2 can be installed via bioconda  [71]  (https:// anaconda. org/bioconda/virsorter). A web version can be accessed at CyVerse/iVirus (https://de. cyverse. org/de). The databases used in VirSorter2, high quality non-RefSeq genomes used for training classifier and benchmark (except for ssDNA viral genomes [Varsani, unpublished data]), benchmark result tables (recall, precision, and F1 score), and ocean virome contig data used for manual curation and its VirSorter2 and VirSorter results are available at zenodo (https://zenodo. org/record/4297575; see instruction in README file). The VirSorter2 database can be automatically downloaded through Vir-Sorter2 "setup" subcommand.
However, despite its effectiveness, OpFuzz has several limitations. First, OpFuzz is limited by its finite mutation space: the seed formulas have a fixed set of operators and for each of them there are often only 2-3 Fig. 1. Almost four-year latent soundness bug in CVC4's string logic. https://github. com/cvc5/cvc5/issues/5940 type-conforming choices for mutation. Furthermore, as fuzzing campaigns and especially OpFuzz have led to hundreds of bug fixes in the SMT solvers Z3 and CVC4, the solvers have matured. Because of this saturation effect  [Amalfitano et al. 2015] , SMT solver fuzzers are finding fewer and fewer critical bugs. Yet, important bugs have been missed. Consider the formula in Fig. 1  which manifests a long-latent soundness bug in CVC4. The "declare-fun" statements specify two string variables and one integer variable respectively, the "assert" specifies the constraints, and the "check-sat" queries the solver.
However, FuzzChick applied in the SMT setting limits the mutation of operators. Generative Type-Aware Mutation overcomes both limitations by combining mutation and grammar-based fuzzing. Main Contributions. We make the following contributions: • We introduce Generative Type-Aware Mutation, a novel, effective approach for stress-testing SMT solvers; Generative Type-Aware Mutation is a hybrid of mutation-based and grammarbased fuzzing and has an infinite space, overcoming one of OpFuzz's key limitations. • Based on Generative Type-Aware Mutation, we have realized TypeFuzz, a highly practical testing tool customizable by a configuration file of SMT-LIB's theory specification language. • From end of January 2021 to mid September 2021, we reported 237 bugs in the state-of-the-art SMT solvers Z3 and CVC4 with TypeFuzz.