dataesr/scientific-paragraphs-categorization huggingface dataset:
As a counterpoint, using the same approach as above, inspecting now the functional annotation of 200 randomly selected phages from the training dataset revealed that on average 70% (stdev 33%) of the phage ORFs were annotated and only 2% (stdev 6%) of these affiliated with reference archaea or archaeal viruses (Supplementary Table  1 ). While several of these phages were derived from well-curated public databases (NCBI  [62]  and the Virsorter curated dataset  [64] ), and represent exceptionally well-annotated viruses, this nevertheless suggests that phages will have a relatively low proportion of ORFs affiliating with archaea or archaeal viruses, relative to genuine archaeal viruses. Next, manual curation of the benchmarking archaeal viruses used to evaluate the performance of the random forest model once again leverages the network analytics  [68]  and functional annotations  [69].
Network analysis with the benchmarking dataset revealed that 649 of the 1402 total sequences (183 archaeal viruses and 465 phages) clustered into 234 VCs (56 archaeal viruses, 178 phages) with 354 VC outliers (68 archaeal viruses and 287 phages) and 399 singletons (23 archaeal viruses and 376 phage) (Fig. 2 , Supplementary Table  2 ). Out of the 1003 clustered or cluster outlier viruses, 201 archaeal viruses and 582 phages fell into modules with corresponding archaeal viruses or phages from the OcAVdb, NCBI  [62] , or pVOGs databases  [63]. Hereafter, we refer to those archaeal viruses that fell into the same module with reference archaeal viruses as the "verified archaeal viruses", while those phages that cluster with reference phages as the "verified phage". Benchmarking archaeal viruses that cluster amongst themselves but with no reference virus, were considered as "putative archaeal viruses", and those that did not resolve in the network at all are referred to as "archaeal virus singletons", again with equivalent nomenclature for the phages.
Surprisingly, 20 additional IMG/VR phages  [67] , clustered with archaeal viruses from OcAVdb, NCBI  [62] , or the VirSorter database  [64] , possibly indicating a mis-annotation of these viruses in IMG/VR and bringing the total number of verified archaeal viruses up to 221 (Supplementary Table  2 ). Functional and taxonomic annotations for the archaeal viruses in the benchmarking dataset revealed that an average of 47% (stdev 33%) of the ORFs per sequence receive an annotation from DRAMv  [69]. Out of the ORFs receiving an affiliation, 76% (stdev 30%) matched either archaea or archaeal viruses from the KEGG  [70]  or NCBI viral databases  [62]. Only 10 of these sequences encoded no detectable archaeal signal, 7 of which have <10% ORFs receiving any affiliation. Among the 20 IMG/VR  [67]  phages predicted as archaeal viruses, 78% (stdev 28%) of the annotated genes matched either archaea or archaeal viruses (Supplementary Table  1 ).
Development of the random forest archaeal virus classifier-MArVD2. With the now sufficient reference, training, and benchmarking data in hand, we next sought to develop the tool, MArVD2, for more scalable, user-friendly, and sensitive archaeal virus identification by incorporating machine learning. To this end, MArVD2 first populates a feature table consisting of a set of 27 genomic features, which we have predetermined to be informative for archaeal virus identification (Supplementary Table  3 ), leveraging several databases and tools as follows. First, ORFs are predicted with Prodigal  [72] , yielding information regarding gene length, gene density, and strand bias. Second, functional and taxonomic annotations are provided by using (i) MMseq2  [73]  to search protein-coding regions against viruses in the NCBI nr database  [62]  (ii) hmmsearch  [74]  to search against the pVOGs  [63]  database, and (iii) iterative jackhmmer  [74]  searches against OcAVdb (Fig.
1 ). A fivefold cross-validation is then used to recursively identify and retain only the most important features based on the Gini importance index  [75]  (Supplementary Fig. 1A, B ). Finally, MArVD2 then implements the resulting feature table in the development of a random forest machine-learning model for archaeal virus identification, splitting the training data into training and out-of-bag test datasets at a 70:30 ratio, respectively  [75]. Evaluation of the random forest model development reveals that MArVD2 exhibits a high degree of performance with the training dataset. Using permutations of the training and out-of-bag test dataset, the F1 score (harmonic mean of the precision and recall, with a score of 1 indicating perfect precision and recall) for the model's development plateaued at 0.
98 with the inclusion of only 8 of the most important features even though all 27 features were identified as contributing to optimal model performance (Supplementary Fig. 1A ). This is also reflected by a considerably higher Gini importance score for these 8 features (Supplementary Fig. 1B ), indicating that only a subset of the 27 features was required for accurate archaeal virus identification. While building the random forest model (not to be confused with later implementation with the benchmarking dataset), only 19 out of the 857 training sequences (10 archaeal viruses and 9 phages) had inconsistent classifications, according to hierarchical clustering analysis with the random forest proximity matrix (Supplementary Fig. 2 ). The proximity matrix in this instance is a measure of similarity among the terminal nodes per all decision trees in the random forest model among the given sequences.
All 10 of these were viruses of either Halobacteria, Methanobacteria, or Thermococci. Closer inspection of particularly the Thermococci sequences revealed this and one other sequence to be pTN2-like plasmids which extensively share replication and regulation genes with other Thermococcales viruses  [76, 77] , further highlighting the value of iterating between model classifications and manual inspection. The main difference between the rest of these proximity outliers and the other training data was a reduced number of hits to the OcAVdb (Mean 3. 74 ± 1. 72 vs Mean 18. 02 ± 20. 6, ANOVA p = 0. 002), the second most important feature in the models' performance, suggesting that the OcAVdb reference database is not representative of these outliers. Further, out of these 19 poorly characterized proximity outliers, 16 were either singletons or outliers in the vConTACT2 network analysis  [68] , again indicating that these sequences represent poorly covered sequence space, often with incomplete representatives, in the reference databases (Supplementary Tables  2  and 3 ).
Evaluation of MArVD2s performance Benchmarking MArVD2. Random forest classification is drawn from the collective designations of all decision trees per input query, whereby the prediction probability is the proportion of trees agreeing on a particular classification  [61]. These prediction probabilities can be interpreted as confidence intervals and provide a high degree of resolution to discern the range of predictions in which MArVD2 will be reliable (Fig. 1 )  [61]. These confidence metrics are derived from the training set however, and it is good practice to verify these using an independent benchmarking dataset, here including both archaeal viruses and phage from a wide range of environments (see above). Of the 221 verified archaeal viruses in the benchmarking dataset, MArVD2 correctly classified 212, including 13 of the IMG/VR predicted phage that cluster with reference archaeal viruses, while only 9 verified archaeal viruses were missed (Supplementary Table  4 ).
3B )  [32]. Together, with the fact that the original MArVD is no longer functional due to its reliance on unsupported software, and its relative inflexibility to grow as archaea virus discovery progresses, this makes MArVD2 far superior to its predecessor. To better assess the performance of MArVD2 and determine which probability thresholds yield the most optimal results, we evaluated the receiver operating characteristic curve, relative to the prediction probabilities of MArVD2. The verified archaeal viruses from the benchmarking dataset had an average MArVD2 prediction probability of 0. 87 (Fig. 4 ), with a very high area under the receiver operating curve (AUROC) value (0. 99) (Fig. 5A ). Above this conservative probability threshold, 71% of the verified archaeal viruses (n = 157) were identified with only one false positive among the verified phage.
Decreasing the probability threshold to 0. 80 facilitated the correct identification of 85% of the verified archaeal viruses (n = 188) with only 2 false positives among the verified phage. The FPR does not exceed 2% until the MArVD2 probability threshold drops below 0. 55, at which point MArVD2 correctly classifies 95% of the true archaeal viruses (n = 210) with 13 false positives among the verified phage and another 20 among the putative phage. When used with unbalanced datasets (i. e. , more phage than archaeal viruses), classifiers with a low FPR on benchmark datasets can still yield as many or even more false-positive predictions than true positives, thus rendering the model ineffective. In addition to the detection of archaeal viruses, we also evaluated whether MArVD2 could correctly classify viruses that were not archaeal viruses using a precision-recall curve.
Here, the area under the precision-recall curve (AUPRC) value again is high (0. 99) where the precision of the model does not drop below 98% until sensitivity exceeds 80% (Fig. 5B ). Hence the performance of MArVD2 should not be significantly impacted by potentially unbalanced datasets which would include many more phages than archaeal viruses. Together these analyses indicate that with a permissive prediction probability (we suggest 0. 80), MArVD2 will identify most of the archaeal viruses (~85%) from marine, hypersaline, and hot spring environments with very few falsely classified phages. How much genomic information is needed?. Many viral datasets are plagued by short sequences or considerable amounts of microbial contamination which can have major impacts on viral identification and classification  [78].
To determine how well MArVD2 would perform on realistic datasets, we split our benchmarking dataset into three test groups to examine the effect of variable dataset size, sequence length, and microbial contamination. The first test dataset included randomly selected sequences from the benchmarking dataset with sequence counts of between 5 and 75% (at 25% intervals) of the original count. The second test dataset includes genome fragments with variable sequence sizes between 1 kb and 10 kb (at 2. 5 kb intervals) from the benchmarking dataset. The third test dataset includes varying proportions, between 10 and 75% (at 25% intervals), of randomly selected microbial genomic fragments from IMG/M  [79]  (equal parts bacteria and archaea) of sizes between 10 kb and 200 kb. Dataset size in terms of the number of sequences included had a negligible impact on the performance of MArVD2.
Across all dataset size fractions (5%, 25%, 50%, 75% number of original sequences) there was minimal variation in TPR, ACC, SPEC, MCC, and FDR relative to the original dataset (average 0. 96, 0. 97, 0. 97, 0. 92, 0. 8 respectively) (Supplementary Fig. 4 ). Other viral identification machine learning tools such as DeepVirFinder  [57] , MARVEL  [58] , VIBRANT  [59] , and Virsorter2  [60]  have reduced performance as virus genome fragment length diminishes. Not surprisingly, we found that MArVD2s performance is impaired on progressively smaller fragments with TPR, ACC, MCC, AUROC, and AUPRC values only exceeding 90% on datasets with contigs>10kbp (Fig. 6A  and C ). The exception to this was SPEC which remained high and nearly unchanged across the variable sequence size fractioned datasets (1 kb, 2.
5 kb, 5 kb, 7. 5 kb, 10 kb, >10 kb). Further, the FDR stayed relatively low across all fragment sizes, never exceeding 15% (Fig. 6A, C ). Likewise, increasing amounts of "contaminating" microbial fragments in the viral dataset introduced a higher likelihood of misidentifying a non-viral sequence as an archaeal virus. Even with a 10% inclusion of microbial sequences, MArVD2's MCC was reduced to less than 90%, while the FDR increased reaching a maximum of 53% with 75% of the input data being microbial. Interestingly, MCC and FDR were the only values influenced by the inclusion of microbial sequences, indicating that the true archaeal viruses were still identified, but that the false positive rate was driven up due to archaeal virus classifications of non-viral sequences (Fig. 6B, D ). Notably, when applying the recommended 0.
80 prediction probability threshold from above, and using the 75% microbial dataset, the FDR is reduced to 16%, and of the false positives identified above this threshold, all were from Archaea derived from metagenomic datasets. Pragmatically, this means that for the most optimal performance of MArVD2, we suggest using datasets comprised of contigs no smaller than 10 kb and which have previously been identified as viral by the various available viral identification tools currently available, as well as, an archaeal virus probability threshold of 0. 80 (Fig. 6B, D )  [57] [58] [59] [60]. Beyond these minimal recommendations, we note that the underlying training and test datasets used to develop MArVD2 are predominantly derived from marine, hypersaline, and hot-spring environments. MArVD2 will potentially be ineffective at predicting archaeal viruses from other untested environments or other taxonomic lineages not represented in the current training datasets as it is yet undetermined if these viruses are substantially different from those in the current training datasets.
Additionally, it is unclear whether MArVD2 will function with datasets composed of ssDNA viruses, as it has not yet been tested in this capacity. Though currently, this represents a "next frontier" development need, MArVD2 is designed such that it should handle them once appropriate reference genomes become available. In addition, while untested, there is potential that DNA Eukaryotic viruses may be incorrectly identified as archaeal viruses by MArVD2. While the vast majority of dsDNA viruses are thought to infect prokaryotes, we caution users to be aware of these factors. Contaminating microbial sequences, issues stemming from discerning provirus boundaries, and the potential to miss novel, divergent viral types present considerable challenges to any viral identification effort. These challenges may be exacerbated when further searching for archaeal viruses with relatively unknown sequence space, and which in some cases may share considerable portions of their genome with host elements.
OcAVdb development The database of marine archaeal viruses (OcAVdb) was created by collecting all the putative archaeal viruses published from marine metagenomic, single-cell genomes, and viral isolation studies up to 2019  [18, 32, [34] [35] [36] [37] [38] [39] [40] 42]. This included a total of 226 archaeal viruses which were further manually curated using vConTACT2  [68]  to provide a taxonomic context for each of the putative archaeal viruses, and DRAMv  [69]  to provided functional annotations. Only those viruses larger than 10 kb which fell into a network module (a collection of related genus scale taxonomic clusters) comprised of only other archaeal viruses, and included archaea or archaeal virus like ORFs were retained in the final database.
The training and benchmarking dataset development The training dataset used to develop the MArVD2 random forest model for archaeal virus identification was created by using a combination of public reference databases and databases created by the original MArVD (described below)  [32] , each vetted by vConTACT2  [68]  to include only sequences which fall into the same network module as a reference archaeal virus, and functional and taxonomic annotations affiliating with archaea or archaeal viruses from DRAMv  [69]. In total, the training dataset includes 857 virus sequences larger than 10 kb with roughly equal parts archaeal virus and phage. This includes 194 phages from the RefSeq version 85 database  [62] , 112 phages, and 70 archaeal viruses from the VirSorter database  [64]  and 131 phages, and 350 archaeal viruses from a published marine environmental virome from the ETSP  [65, 66].
Each of these phages and archaeal viruses were selected for inclusion in the training dataset because they cover as much of the taxonomic sequence space as possible according to a network analysis by vConTACT2 and were derived from a variety of environments including hot springs, hypersaline ponds, and the oceans. Training data were implemented in the model creation by Scikit-learn at a ratio of 70 and 30% training and testing datasets  [75]. The benchmarking test dataset was created by mining the IMG/VR-db v2. 0  [67]  for all archaeal viruses from enrichment cultures, the marine environment, hypersaline or alkaline habitats, and thermal hot springs among others. Phages were selected randomly from the same environments with the addition of phages from soils, freshwater, and freshwater sediments.
To account for a lack of archaeal viruses from the open ocean in the IMG/VR dataset, an additional 25 putative archaeal viruses from 2 open ocean mesopelagic samples in the Tara oceans GOV2. 0 dataset  [17]  were identified by the original MArVD  [32]  as described below and included in this test dataset. All viruses in the benchmarking dataset were >10 kbp.
Re-design of the original MArVD The original MArVD  [32]  was recreated as a python 2. 7+ script to use the output information from the widely accessible viral identification software VirSorter  [64]. This redesigned version of the original MArVD, first uses MetaGeneAnnotator  [85]  predicted proteins from the VirSorter identified viruses and uses BLASTp  [62]  to search against the Refseq (version 77) database  [62]. Functional and taxonomic annotations are then prescribed in concordance with the highest scoring target sequence with a bitscore >50 and evalue >0. 001. These annotations are then integrated into the VirSorter "affi_contigs. csv" gene annotation file retaining the VirSorter derived Pfam  [86]  designations >40 bitscore and <0. 00001 evalue. Using this updated per gene annotation file, MArVD functions exactly as its first inception  [32].
Only MArVD category 1 and 2 putative archaeal viruses, corresponding to viruses having over 66 or 50% of their annotated genes affiliating with archaeal viruses respectively, and with bitscore >75 and higher than those for the phage affiliations, were retained as MArVD predicted archaeal viruses. This updated version of MArVD enabled the creation of the new environmental archaeal virus datasets from the ETSP and GOV2. 0 datasets needed to train and test MArVD2 as well as allows for a means to compare the performance of MArVD with MArVD2. Fig. 6  MArVD2 performance with different categories of viral data types. A MArVD2 performance relative to contig size using a probability threshold of 0. 50. Values were calculated on the IMG/VR and GOV2. 0 benchmarking datasets where contig sizes were fragmented into different size categories and randomly selected the same number of input contigs as the original dataset.
B MArVD2 performance relative to varying proportions of cellular contamination, using a probability threshold of 0. 50, with the IMG/VR and GOV2. 0 benchmarking datasets supplemented with cellular gene fragments of equal proportions archaea and bacteria of size between 10-200 kb. C, D These represent the same analysis with an adjusted probability threshold of 0. 80, reflecting our recommended threshold. The red line indicates 0. 90 where performance is considered acceptable. Performance metrics are described in detail in Supplementary Fig. 3.
Feature table, databases, and MArVD2 development Informative features distinguishing archaeal viruses from phages were first identified by generating a feature table containing numerous genome attributes (e. g. , average gene length, gene density, strand bias, etc. ) (Supplementary Table  3 ) and combining this with aggregated results from searches against various databases. ORFs were predicted using prodigal  [72]  with the "-p meta" option. Each of the final set of features were derived either from genomic attributes of the input sequences, an MMseq2  [73]  comparison with the NCBI nr  [62]  database, hmmsearch  [74]  comparisons against the pVOGs  [63]  database, or comparison with OcAVdb using jackhmmer  [74] , each with default parameters. Values and attributes for each feature per input sequence were created and tabulated into a comprehensive feature table which becomes the basis for the random forest model generation  [61].
To avoid potential bias introduced in the random forest model by co-correlating features, a co-correlation analysis was performed. Features with greater than 0. 95 correlation coefficients were removed. Finally, each virus was designated as archaeal virus or phage and fed into python's scikit-learn  [75]  implementation of the random forest model. A manual examination of all archaeal viruses used herein (with the exception of known archaeal viruses in public repositories) is listed in Supplementary Table  1. MArVD2 first creates the feature table as described above, including only MMseq2 hits with evalues <1e-5, hmmsearch hits with full protein length evalues of <1e-10, and jackhmmer hits with evalues of <1e-5. The MArVD2 random forest model is then built by the python scikit-learn package  [75].
To obtain the optimal number of features to create the highest F1 score, recursive feature elimination was used. Features with the lowest Gini importance scores were iteratively removed, with a minimum of five features being retained. fivefold cross-validation of the model's final accuracy is then calculated using a permuted set of training and out-of-bag test datasets. Multiple additional machine learning algorithms were also tested, but almost always with random forest performing the best. The final random forest model and the preliminary feature table for the training dataset are saved for later implementation with other novel datasets. Re-running the model with new input data will generate a new feature table with archaeal virus or nonarchaeal virus predictions and the probabilities associated with those predictions.
MArVD2 benchmarking Using the benchmarking test dataset derived from IMG/VR  [67]  and the GOV2. 0 data  [17] , we next evaluated the performance of MArVD2 in distinguishing archaeal viruses from phage. The test IMG/VR dataset was first confirmed to be of viral origin by VirSorter  [64]. Distinctions between archaeal viruses and phage were next verified by MArVD and confirmed by vConTACT2  [68]  network analysis and manual curation of the functional annotations provided by DRAMv  [69]. The verifed phage and archaeal virus dataset from IMG/VR were then size fractioned to include contigs of 1 kb, 2. 5 kb, 5 kb, 7. 5 kb, 10 kb, and >10 kb lengths. For the >10kbp size fraction, a second test dataset with various amounts of microbial sequences was included with equal proportions of bacteria and archaea.
Genomic fragments from microbial sequences were randomly selected from the IMG/M  [79]  database and only included if their size was between 10 kb and 200 kb. Microbial sequences were added at 10, 25, 50, 75, and 95% of the total data. Microbial sequences were ensured not to be viral by use of VirSorter. Dataset size in terms of the number of contigs was also tested with the benchmarking dataset being broken into sets 10, 25, 50, 75, and 95% of the total number of contigs from the original validation dataset. Sensitivity analysis was then conducted on the unaltered benchmarking dataset and each of the datasets of various size fractions and with various proportions of included cellular sequences. For each dataset, the true positive rate (TPR), specificity (SPEC), accuracy (ACC), Matthews correlation coefficient (MCC) and false detection rate (FDR) was calculated using the R package "EvaluationMeasures"  [89].