dataesr/scientific-paragraphs-categorization huggingface dataset:
Data Description To document various aspects of AmpliconTagger, we processed 7 published and publicly available datasets of rRNA amplicon sequencing data of various sequencing configuration, targeting various marker genes and regions (Table  1 ). We aimed to include datasets from a variety of ecosystems: indoor, human gut and oral cavity, soil and water. We also included a novel dataset consisting of a commercial mock community. These are the reads that are sent for OTU/ASV generation after having been paired-end assembled (for paired-end sequencing) and controlled for quality as described in Methods.
Experimental design Each of the datasets described in Table  1  was processed into the AmpliconTagger pipeline, which contains from 91 to 94 jobs depending on the OTU/ASV generation algorithm used and on sequencing configuration (paired vs single-end sequencing data). All datasets were processed following a common core of qualityfiltering procedures, but submitted to 2 different methods of OTU generation (Fig. S1 ) (VSEARCH  [29]  and DNAclust  [30] ) and 2 ASV methods (Deblur  [31]  and DADA2). In addition, short amplicon mock community datasets were also entirely processed in QIIME2 using a VSEARCH, Deblur, and DADA2 workflow to compare our pipeline against a third-party reference method. We also present and discuss community profiling results of long PacBio amplicons using a mock community and a published oral microbiome sequencing dataset (Table  1 ).
More in-depth analyses of common microbial ecology metrics were assessed for each project for each OTU/ASV generation method and are available in Additional File 1 (Fig. S2  to S7). Although we do present some high-level analysis of ASV vs OTU end results, the present study primarily aims at demonstrating the modularity and methodology implemented in AmpliconTagger and less at performing an exhaustive comparison of the OTU/ASVgenerating packages that we used. The complete data processing description of AmpliconTagger is described in the Methods section, and the complete set of commands of each job for each data analysis run is available in Additional Files 2 (AmpliconTagger command traces) and 3 (QIIME2 command traces for the 16S V4 region mock community). An exhaustive user guide is available in Additional File 4.
Validation with mock communities and comparison with third-party reference pipeline To validate AmpliconTagger, we processed 2 defined mock communities, 1 with even concentrations of 20 bacterial strains (Table  S1 ) and 1 with staggered concentrations of 9 genomes (Table  S2 ). We obtained community profiling results of each mock sample using AmpliconTagger (VSEARCH, DNACLUST, Deblur, and DADA2) and compared them with the end results of the same sequencing libraries, but entirely processed with QIIME2 using VSEARCH, Deblur, and DADA2 workflows (DNACLUST is not implemented in QIIME2). Taxonomic profiles are highly similar across all tested methods (Fig. 1a  and b ), and minor differences are probably caused by the 2 different Silva R128 training sets used by AmpliconTagger (100% identity sequences) vs QI-IME2 (database, clustered at 99% identity).
Performance We compiled compute resources consumed for each of the datasets considered for this study (Fig. 2 ). The common core of AmpliconTagger consists of steps 1-6 as detailed below in the Methods section. The indoor microbiome, Lake Michigan, and antibiotic-associated diarrhea (AAD) studies consumed similar amounts of core hours (22. 7, 20. 2, and 26. 6 core hours, respectively), while the mock community, oral microbiome, and rhizosphere microbiome transplant studies took significantly less resources, with respective values of 0. 53, 2. 2, and 5. 2 core hours. OTU/ASV generation jobs consumed the most resources for the oral microbiome (PacBio long reads) project followed by the indoor microbiome data, which again is the dataset containing both the highest number of (short) reads and base pairs.
Microbial ecology metrics obtained for each dataset Globally, ecological patterns were similar for the 4 tested methods for all projects (Table  3   In general the amounts of generated ASVs/OTUs were similar, but ASVs represented fewer total reads. To highlight differences amoung the 4 tested OTU/ASV generation methods, we compared their taxonomic summaries and αand βdiversities. These results are included in Figures  S2-S7 , and for clarity and practicality purposes, we narrowed sample selection for each project to a subset of experimental variables. We also computed the Mantel r statistic to assess for correlation between weighted UniFrac and Bray-Curtis distance matrices among all 4 tested methods (Table  3 ). Overall, and consistent with results reported in Figures  S2-S7 , these results show that weighted UniFrac and Bray-Curtis distances/dissimilarities obtained with all 4 tested methods are quite similar, with r statistic values >0.
9 for the indoor microbiome, AAD, and Lake Michigan data types. The ITS data (rhizosphere transplant) showed low r statistics in the range of 0. 4-0. 6 between the weighted UniFrac matrices. However, Mantel tests between Bray-Curtis dissimilarity matrices of all 4 OTU/ASV ITS data types gave r statistics >0. 9 for all comparisons. For the mock community, weighted UniFrac distances of all 4 methods were similar with r statistics values >0. 9. However, Bray-Curtis dissimilarity matrices were more divergent, with r statistics values of 0. 580 (Deblur vs VSEARCH), 0. 580 (VSEARCH vs DADA2), and 0. 724 (DNACLUST vs VSEARCH). Overall, this indicates that the type of distance metric used had a considerable effect on the microbial population structure assessment.
Backbone for rRNA sequence processing methodology The primary objective of this report was to showcase Amplicon-Tagger, a highly modular HPC-oriented pipeline geared for performing bioinformatic analyses of rRNA amplicon data. To facilitate implementation and testing, we shared a Docker repository of a CentOS-7 image with a fully working implementation of AmpliconTagger, which includes data of 20 sequencing libraries from the AAD study, all the databases, training sets, and fully detailed commands of each job for each step of our pipeline. Our workflow also relies on a set of perl scripts and libraries labeled nrc tools for which the complete code is also available in a code repository. We wish to emphasize that we do not present our methodology as a gold standard but rather as a blueprint of an end-to-end open source working modular pipeline that is able to incorporate virtually any in-house scripts or third-party packages (see Additional File 4 for more details).
Resources and time consumption With bioinformatics entering many spheres of research and service fields, there is a need for flexible, scalable, and robust methods to systematically analyse high-throughput nucleic acid sequencing data of all types. Here we present in detail our gene amplicon production workflow with performance metrics and actual results from published rRNA marker gene datasets. We executed our workflow using 2 OTU generation methods, VSEARCH and DNACLUST; and 2 ASV generation methods, Deblur and DADA2. We included VSEARCH because it represents the open source version of the popular USEARCH closed source software. As such, VSEARCH is increasingly being used by the research community as a suitable replacement for USEARCH. Our group started to use DNACLUST as a suitable open source alternative to replace USEARCH some years ago, and since then, we have processed many rRNA marker gene sequencing datasets using this package, which is the reason for its inclusion in this study.
The goal of the present study was not necessarily to benchmark these 4 OTU/ASV generation methods but rather to show the modular nature of the pipeline and offer useful metrics of what microbiome profiling results and resource consumption  can be expected from using these software packages in the context of our whole workflow methodology. While short amplicon sequence data processing is not that demanding from a computational resource point of view, a complete working pipeline requires many steps to get from the raw data to key end results. Here we show that quality control (QC) and fastq preprocessing steps of sequencing data prior to OTU/ASV generation consumed the most core hours for datasets that had an appreciable number of reads (indoor microbiome, Lake Michigan, and AAD). The rhizosphere transplant project had a comparable number of reads, but possibly because of short ITS read length (once paired-end assembled), jobs relying on Qscore metrics such as quality filtering and quality score profile compilation quickly completed (Fig.
2b ). Even though the 2 training sets used to classify OTUs/ASVs are built from the Silva 128 release, the QIIME2 training set was trained on a 99% identity clustered database and the AmpliconTagger training set was trained on a 100% identity (i. e. , unclustered) fasta database, which probably explains the differences observed in taxonomy, which, all things considered, are quite minimal.
Robustness of ecological patterns between OTU-and ASV-based results One of the key advantages of the pipeline introduced here is that it is modular and can be customized to fit the research needs of the user. For instance, we have shown here that both ASV and OTU generation methods can be used interchangeably, making the AmpliconTagger pipeline agnostic to the current heated debate about ASV vs OTU. ASV generation is a method that has recently gained traction  [33]  and is increasingly being adopted as an alternative to OTU-based analyses. Here we processed 7 types of datasets and compared end results of OTU clustering procedures done with DNACLUST and VSEARCH and 2 ASV-based methods (Deblur and DADA2). In most cases in which short reads are being analysed, results given by Deblur and DADA2 are essentially identical to what is obtained by VSEARCH and DNA-CLUST.
One notable exception is for the ITS data type, where the weighted UniFrac Mantel correlations among the 4 methods are significantly lower than what is observed for short 16S and 18S amplicon data types. This discrepancy might be due to the fact that ITS amplicon sequence lengths are quite variable as shown in Fig. S8 , with significant amounts of reads dispersed from 200 to 350 bp. During the clustering process, many of the shorter reads are "absorbed" by the larger reads. For ASVs, all these reads of various lengths eventually form a distinct ASV and in consequence many ASVs do not reach the cut-off of ≥25 reads per ASV. This holds true for all projects, where the number of ASV reads is often lower than the number of OTU reads (Table  2 ), but is probably exacerbated in the ITS data where many of the amplicon sequences are of different lengths (Fig.
S8 ). That said, one would still have expected the weighted UniFrac mantel correlation to be higher between DNACLUST and VSEARCH OTUs-as observed in other data types-which was not the case. On the other hand, Bray-Curtis dissimilarity matrices were highly concordant between Deblur, DADA2, VSEARCH, and DNACLUST for the ITS data type. This implies that global alignments, inherent to generating the Unifrac distance matrix, may be impractical because of the high variability in sequence and length of ITS amplicons, as previously suggested  [34].
Some differences were also observed in less abundant species: Deblur was enriched in Streptococcus vestibularis while DADA2 contained Streptococcus mitis and Streptococcus sanguinis. Our results highlight the challenge of properly assigning species taxa to long 16S reads using general databases, in our case the Silva DB. This suggests that in some instances, PacBio CCS reads contain at least enough errors to cause misclassification at the species level when processed as ASVs and that clustering reads (OTU methods) in the objective of correcting these errors also results in misclassification. Deblur ASVs in the mock community gave poor Mantel r correlation with other methods because although it held a total of 6,535 reads, 4 samples did not meet the cut-off of 1,000 reads required in the rarefaction procedure to normalize ASV tables, hence resulting in the rejection of 4 samples and lowering Man-tel r-correlations involving Deblur.
Compared to short reads, long PacBio amplicon data showed lower Mantel correlations among the 4 tested methods (Table  3 ). PacBio CCS reads have an accuracy of 99. 999% according to the manufacturer's specifications. However, with long amplicons of around 1,400 bp, even such a high percentage of accuracy will eventually translate into actual errors, inevitably reflecting a high proportion of low-abundance sequences in OTU/ASV tables. This problematics is partially compensated for with clustering methods because reads having ≥97% identity will be merged in the same OTU. With the ASV paradigm, reads showing differences in a single base will form distinct ASVs, resulting in a more scattered abundance table, which inevitably affects indexes based on abundance such as the Bray-Curtis dissimilarity index.
However, re-usability across studies assumes that DNA of these different studies have been extracted using the same method  [42, 43]  and DNA amplified using the same primer sequences. If sequencing libraries to be analysed meet these criteria, in practice, it is probably more opportune to pool all libraries together and re-initiate ASV generation to make sure that samples have been processed the same way (e. g. , exact same parameters) in upstream steps of ASV/OTU generation. Given the constant improvement of compute hardware and efficiency of OTU clustering methods, re-generating OTUs or ASVs as new datasets for a given project become available can be considered a viable option: our results (Fig. 1 -right panels) show that VSEARCH completed in ∼4. 5 hours compared to 5. 2 hours for Deblur for the indoor microbiome project data, but that VSEARCH significantly outperformed Deblur for the other datasets.
Regardless of the ASV or OTU generation method used, the use of short amplicon sequencing should mainly aim at offering a broad snapshot of the microbial communities at stake in a biological system. In all cases, the AmpliconTagger pipeline can be customized and accommodate any OTU or ASV generation method, being agnostic to the current debate. In conclusion, microbial ecology is more than ever relying on high-throughput sequencing technologies. Bioinformatic pipelines used for analysing these data loads are increasing in complexity and there is a need for increased flexibility in the systematic analysis of short rRNA amplicon data. End-to-end pipelines do exist, but these solutions are not necessarily conducive to an easy integration of third-party packages or in-house software. Moreover, these pipelines, as they are provided, are mostly geared toward interactive or single-batch job processing and can generate inessential intermediate files, which can be constraining in a production context.
This is usually a fairly small proportion of all reads. This step is not performed if reads are single ended (MiSeq single ended, PacBio, IonTorrent, or 454). 3. If reads are of single-end configuration (i. e. , 454, IonTorrent, or PacBio data types), they are trimmed to a fixed length that is variable depending on the quality of sequencing run and amplicon length. If reads are paired end (Illumina), trimming can be optional and should be performed in such a way that enough bases are left on the 3 end of each read pair to allow assembly using forward (read 1) and reverse (read 2) common overlapping parts during merging of read pairs in the next step. Nevertheless, in the case of paired-end reads, it is arguably preferable to trim the 3 portion of reads that shows low quality before reads performing the overlapping paired assembly.
The remaining reads will be referred to as filtered reads from now on. Filtering parameters are customizable and should be adapted to the quality profile of each dataset. 10. A summary of read counts throughout the different steps of the pipeline is generated. This is useful to get a global outlook on the sequencing run: how many reads were sequenced, how many reads were filtered out after QC, how many OTUs/ASVs were generated, etc. 11. From these classified OTUs/ASVs, a multiple sequence alignment is then obtained by aligning OTU/ASV sequences on a Greengenes core reference alignment  [52]  using the PyNAST aligner  [17]. If data type is ITS or 18S, OTU/ASV sequences are aligned against a Unite or Silva eukaryote core alignment, respectively. For short amplicon data, alignments are filtered to keep only the hypervariable region of the alignment.
Read clustering and ASV methodology Our OTU generation procedure was implemented on the basis of a procedure previously described (Lundberg et al. 2012  [55] ) and uses either DNACLUST or VSEARCH for the read clustering step. Briefly, quality controlled reads/sequences are dereplicated at 100% identity. The dereplication step is necessary to lower data load for the clustering software because only 1 representative of many thousands of identical sequences is kept for clustering. Counts of each unique sequence representative are kept in sequence headers after the dereplication process. For instance for the AAD study, the fastq file holding quality controlled paired-end assembled reads holds 9,209,510 sequences. Once dereplicated, these sequences are actually regrouped into 831,570 sequences, which represent a 11.
RDP classifier training sets The RDP classifier is a Bayesian classifier whose purpose is to classify sequences against a training set. Existing training sets are based on 99% identity clustered versions of either Greengenes or Silva databases. The RDP database (not to be confused with the RDP classifier software) was also built in a similar manner. To improve resolution of classification, we built our own custom training sets using the whole Silva SSU (release 128) database. We had to semi-automatically and manually alter the classification of certain taxa in order to make the lineages unique and non-conflicting. At the time of writing, we are using a training set based on the Silva 128 release. We also built our own training sets for 18S and ITS sequences. The taxonomic classification system for eukaryotic organisms is far more complex than that for the simpler bacterial kingdom.
As such, additionally to the common kingdom, phylum, class, order, family, and genus fields found in prokaryotic taxonomy, eukaryotic taxonomy includes ranks such as subphylum, subdivision, subclass, superorder, suborder, and subfamily, which makes the task of generating consistent values for each rank challenging. Importantly, in order to obtain more resolution from taxonomic classifications, our training sets were generated using the entirety of the Silva and Unite databases and not the clustered or "OTU" versions (i. e. , clustered at various identity thresholds ranging from 95 to 99%) of these databases. Perl code used to generate training sets and training sets themselves are available in the following repository: https://github. com/jtrembl ay/RDP-training-sets. Normalizing OTU/ASV tables with a multi-rarefaction procedure Normalization of ASVs or OTUs is a controversial topic  [50, 56].
A good pipeline framework should generate jobs, manage their dependencies, and have a smart restart mechanism in case of job failure. In the context of a complex pipeline with hundreds to thousands of jobs, a smart restart mechanism is indispensable to gain productivity and save time determining which job failed. The GenPipes core modules fill these requirements and is the reason for their adoption  [21]. For instance if the execution of AmpliconTagger gets interrupted because of a job failure, it should be straightforward to identify exactly which job failed to properly execute. With a smart restart mechanism implementation, the pipeline framework should find, upon re-execution, which job actually failed to successfully complete and effectively rewrite it for resubmission. Bioinformatics pipeline frameworks are also critical in that they allow sequencing data to be systematically analysed in reproducible ways and that each step or job generated is parameterizable.
Sequencing library preparation for mock community DNA Mock communities purified DNA was purchased from BEI resources (Manassas, VA, USA) as HM-782D (even spike-in of total mock community). 16S rRNA gene amplicon libraries were prepared as described  [58]. under SRR559331  [4] [5] [6] [7] , SRR55933  [19] -  [20] , and SRR559333  [2] -  [3]. All raw data and intermediate files used and generated for this study are available in GigaDB  [59]. All commands used to process all of the 6 datasets are also available in GigaDB  [59].
Availability of Source Code and Requirements Project Name: AmpliconTagger Project Home Page: http://jtremblay. github. io/amplicontagger. h tml Operating System: CentOS 7 Programming Languages: Python, Perl, R Other requirements: pynast/1. 2. 2; perl/5. 26. 0; rdp classifier/2. 5; fasttree/2. 1. 10; FLASH/1. 2. 11; qiime/1. 9. 1; duk/1. 051; DNACLUST/3; fastx/0. 0. 13. 2; python/2. 7. 5; python/3. 6. 5; java/jdk1. 8. 0 144; blast/2. 6. 0+; Deblur/1. 0. 4; VSEARCH/2. 7. 1; R/3. 6. 0, DADA2/1. 12. 1 License: GNU GPL The AmpliconTagger pipeline wrapper code and Python, Perl, and R scripts that are being called by AmpliconTagger are available here: https://bitbucket. org/jtremblay514/nrc pipeline public/src/1. 1/ https://bitbucket. org/jtremblay514/nrc tools public/src/1. 1/ External software packages module install scripts are available here: https://bitbucket.
org/jtremblay514/nrc resources public/src/1. 1/ A Docker image built on the CentOS 7 operational system that contains all necessary modules for full pipeline functionality is available for testing/evaluation purposes and running small datasets (https://cloud. docker. com/u/julio514/repository/docke r/julio514/centos). Scripts used to generate RDP training sets are available here: https://github. com/jtremblay/RDP-training-sets, and the training set files are available on the Docker image. The PipelineViewer web page is located here: http://jtremblay. github. io/PipelineViewer/amplicontagger. html and its source code is available here: https://github. com/jtremblay/PipelineViewer.
Availability of Supporting Data and Materials Sequencing for the indoor microbiome project is available through the ENA portal under accession number ERP005806. 16S rRNA amplicon sequence data for the AAD study are available in the NCBI's SRA portal under accession number SRP120170. PacBio full-length 16S rRNA amplicons for the oral microbiome project are available under SRR56217. ITS amplicons from the plant root microbiome transplant study are available under PRJNA301462. 18S rRNA gene amplicon data from the Chicago Lake Michigan study are available under PR-JNA294919/SRP063479. The even mock community reads are available under PRJNA510326. The staggered mock community is available under SRR2082918-20 and the PacBio mock libraries.
Spectral analysis for traffic characterization Several research studies have been conducted for traffic characterization using spectral analysis. One of the most consistent is  [16]. The authors have demonstrated how Long Range Dependence (LRD) can be an efficient parameter to quantify the level of variability of Internet traffic. They have developed a specific method (including a Matlab toolbox) to process data traffic. This process uses wavelet analysis (see  [17]  for details) which is an efficient tool to obtain the variability level of any data series for different time scales and different moments of analysis. In this paper, we will use an enhanced version of the method introduced in  [17]  developed by H. Wendt more recently called Wavelet Leader Multi-fractal (WLM) analysis toolbox(see  [18, 19, 20, 21]  for details).