dataesr/scientific-paragraphs-categorization huggingface dataset:
Beyond metagenomics, there is also a clear need to obtain functional readouts in the form of other omics data. The sequencing of reverse transcribed RNA (cDNA) yields metatranscriptomic (MT) data, which provides information about gene expression and therefore allows a more faithful assessment of community function  [4] [5] [6]. Although both MG and MT data allow unprecedented insights into microbial consortia, the integration of such multi-omic data is necessary to more conclusively link genetic potential to actual phenotype in situ  [4, 6]. Given the characteristics of microbial communities and the resulting omic data types, specialized workflows are required. For example, the common practice of subsampling collected samples prior to dedicated biomolecular extractions of DNA, RNA, etc. has been shown to inflate variation, thereby hampering the subsequent integration of the individual omic datasets  [7, 8].
For this purpose, specialized wet-lab methods which allow the extraction of concomitant DNA, RNA, proteins, and metabolites from single, unique samples were developed to ensure that the generated data could be directly compared across the individual omic levels  [7, 8]. Although standardized and reproducible wet-lab methods have been developed for integrated omics of microbial communities, corresponding bioinformatic analysis workflows have yet to be formalized. Bioinformatic analysis methods for MG and MT NGS data can be broadly classified into reference-dependent or reference-independent (de novo) methods  [5]. Referencedependent methods are based on the alignment/mapping of sequencing reads onto isolate genomes, gene catalogs, or existing MG data. A major drawback of such methods is the large number of sequencing reads from uncultured species and/or divergent strains which are discarded during data analysis, thereby resulting in the loss of potentially useful information.
g. , Trinity  [12, 33] ). In all cases, the available assemblers are able to handle the uneven sequencing depths of MG and MT data. Although dedicated assembly methods have been developed for MG and MT data, formalized pipelines allowing the integrated use of both data types are not available yet. Automated bioinformatic pipelines have so far been mainly developed for MG data. These include MOCAT  [34]  and MetAMOS  [10] , which incorporate the entire process of MG data analysis, ranging from preprocessing of sequencing reads, de novo assembly, and post-assembly analysis (read alignment, taxonomic classification, gene annotation, etc. ). MOCAT has been used in large-scale studies such as those within the MetaHIT Consortium  [35, 36] , while MetA-MOS is a flexible pipeline which allows customizable workflows  [10].
Although these studies clearly demonstrate the power of multi-omic analyses by providing deep insights into community structure and function, standardized and reproducible computational workflows for integrating and analyzing the multi-omic data have so far been unavailable. Importantly, such approaches are, however, required to compare results between different studies and systems of study. Due to the absence of established tools/workflows to handle multi-omic datasets, most of the aforementioned studies utilized non-standardized, ad hoc analyses, mostly consisting of custom workflows, thereby creating a challenge in reproducing the analyses  [10, [45] [46] [47]. Given that the lack of reproducible bioinformatic workflows is not limited to those used for the multi-omic analysis of microbial consortia  [10, [45] [46] [47] , several approaches have recently been developed with the explicit aim of enhancing software reproducibility.
These include a wide range of tools for constructing bioinformatic workflows  [48] [49] [50]  as well as containerizing bioinformatic tools/pipelines using Docker  [29, [46] [47] [48]. Here, we present IMP, the Integrated Meta-omic Pipeline, the first open source de novo assembly-based pipeline which performs standardized, automated, flexible, and reproducible large-scale integrated analysis of combined multi-omic (MG and MT) datasets. IMP incorporates robust read preprocessing, iterative coassembly of metagenomic and metatranscriptomic data, analyses of microbial community structure and function, automated binning, as well as genomic signature-based visualizations. We demonstrate the functionalities of IMP by presenting the results obtained on an exemplary data set. IMP was evaluated using datasets from ten different microbial communities derived from three distinct environments as well as a simulated mock microbial community dataset.
The input to IMP consists of MG and MT (the latter preferably depleted of ribosomal RNA prior to sequencing) paired-end reads in FASTQ format (section "Input data"). MG and MT reads are preprocessed independently of each other. This involves an initial quality control step (Fig. 1  and section "Trimming and quality filtering")  [52]  followed by an optional screening for host/contaminant sequences, whereby the default screening is performed against the human genome while other host genome/contaminant sequences may also be used (Fig. 1  and section "Screening host or contaminant sequences"). In silico rRNA sequence depletion is exclusively applied to MT data (Fig. 1  and section "Ribosomal RNA filtering"). The customized assembly procedure of IMP starts with an initial assembly of preprocessed MT reads to generate an initial set of MT contigs (Additional file 1: Figure  S1 ).
MT reads unmappable to the initial set of MT contigs undergo a second round of assembly. The process of assembling unused reads, i. e. , MG or MT reads unmappable to the previously assembled contigs, is henceforth referred to as "iterative assembly". The assembly of MT reads is performed, first as transcribed regions are covered much more deeply and evenly in MT data. The resulting MTbased contigs represent high-quality scaffolds for the subsequent co-assembly with MG data, overall leading to enhanced assemblies  [43]. Therefore, the combined set of MT contigs from the initial and iterative MT assemblies are used to enhance the subsequent assembly with the MG data. MT data are assembled using the MEGAHIT de novo assembler using the appropriate option to prevent the merging of bubbles within the de Bruijn assembly graph  [23, 36].
1  and section "Annotation and assembly quality assessment"). MaxBin 2. 0  [20] , an automated binning procedure (Fig. 1  and section "Automated binning") which performs automated binning on assemblies produced from single datasets, was chosen as the de facto binning procedure in IMP. Experimental designs involving single coupled MG and MT datasets are currently the norm. However, IMP's flexibility does not forego the implementation of multi-sample binning algorithms such as CONCOCT  [16] , MetaBAT  [18] , and canopy clustering  [15]  as experimental designs evolve in the future. Non-linear dimensionality reduction of the contigs' genomic signatures (Fig. 1  and section "Non-linear dimensionality reduction of genomic signatures") is performed using the Barnes-Hut Stochastic Neighborhood Embedding (BH-SNE) algorithm allowing visualization of the data as two-dimensional scatter plots (henceforth referred to as VizBin maps  [13, 56] ).
Further analysis steps include, but are not limited to, calculations of the contig-and gene-level depths of coverage (section "Depth of coverage") as well as the calling of genomic variants (variant calling is performed using two distinct variant callers; section "Variant calling"). The information from these analyses are condensed and integrated into the generated VizBin maps to produce augmented visualizations (sections "Visualization and reporting"). These visualizations and various summaries of the output are compiled into a HTML report (examples of the HTML reports available via Zenodo  [57] ). Exemplary output of IMP (using the default IDBA-UD assembler) based on a human fecal microbiome dataset is summarized in Fig. 2. The IMP output includes taxonomic (Fig. 2a ) and functional (Fig.
Assessment and benchmarking IMP was applied to ten published coupled MG and MT datasets, derived from three types of microbial systems, including five human fecal microbiome samples (HF1, HF2, HF3, HF4, HF5)  [28] , four wastewater sludge microbial communities (WW1, WW2, WW3, WW4)  [43, 44] , and one microbial community from a production-scale biogas (BG) plant  [29]. In addition, a simulated mock (SM) community dataset based on 73 bacterial genomes  [12] , comprising both MG and MT data was generated to serve as a means for ground truth-based assessment of IMP (details in section "Coupled metagenomic and metatranscriptomic datasets"). The SM dataset was devised given the absence of a standardized benchmarking dataset for coupled MG and MT data (this does solely exist for MG data as part of the CAMI initiative (http://www.
Resource requirement and runtimes IMP is an extensive pipeline that utilizes both MG and MT data within a reference-independent (assemblybased) analysis framework which renders it resourceand time-intensive. Therefore, we aimed to assess the required computational resource and runtimes of IMP. All IMP-based runs on all datasets were performed on eight compute cores with 32 GB RAM per core and 1024 GB of total memory (section "Computational platforms"). IMP runtimes ranged from approximately 23 h (HF1) to 234 h (BG) and the IMP-megahit runtimes ranged from approximately 21 h (HF1) up to 281 h (BG). IMP was also executed on the Amazon cloud computing (AWS) infrastructure, using the HF1 dataset on a machine with 16 cores (section "Computational platforms") whereby the run lasted approximately 13 h (refer to Additional file 1: Note S1 for more details).
The analysis of IMP resulted in an increase in additional data of around 1. 2-3. 6 times the original input (Additional file 2: Table  S1 ). Therefore, users should account for the disc space for both the final output and intermediate (temporary) files generated during an IMP run. Detailed runtimes and data generated for all the processed data sets are reported in Additional file 2: Table  S1. We further evaluated the effect of increasing resources using a small scale test dataset (section "Test dataset for runtime assessment"). The tests demonstrated that reduced runtimes are possible by allocating more threads to IMP-megahit (Additional file 2: Table  S2 ). However, no apparent speed-up is achieved beyond allocation of eight threads, suggesting that this would be the optimal number of threads for this particular test dataset.
Contrastingly, no speed-up was observed with additional memory allocation (Additional file 2: Table  S3 ). Apart from the resources, runtime may also be affected by the input size, the underlying complexity of the dataset and/ or behavior of individual tools within IMP. (See figure on previous page. ) Fig. 2  Example output from the IMP analysis of a human microbiome dataset (HF1). a Taxonomic overview based on the alignment of contigs to the most closely related genomes present in the NCBI genome database (see also HTML report S1  [57] ). a, b Abundances of predicted genes (based on average depths of coverage) of various KEGG Ontology categories represented both at the MG (b) and MT (c) levels (see also Krona charts within HTML report S1). d-f Augmented VizBin maps of contigs ≥1 kb, representing contig-level MG variant densities (d), contig-level ratios of MT to MG average depth of coverage (e), and bins generated by the automated binning procedure (f).
The evaluation of the iterative assembly strategy was applied to MG and MT datasets. For both omic data types, it involved an "initial assembly" which is defined as the de novo assembly of all preprocessed reads. Additional iterations of assembly were then conducted using the reads that remained unmappable to the generated set of contigs (see section "Iterative single-omic assemblies" for details and parameters). The evaluation of the iterative assembly procedure was carried out based on the gain of additional contigs, cumulative contig length (bp), numbers of genes, and numbers of reads mappable to contigs. Table  1  shows the evaluation results of four representative data sets and Additional file 2: Table  S4  shows the detailed results of the application of the approach to 11 datasets. In all the datasets evaluated, all iterations (1 to 3) after the initial assembly lead to an increase in total length of the assembly and numbers of mappable reads (Table  1 ; Additional file 2: Table  S4 ).
Despite being developed specifically for the analysis of coupled MG and MT datasets, the iterative assembly can also be used for single omic datasets. To assess IMP's performance on MG datasets, it was applied to the simulated MG datasets from the CAMI challenge (http://www. cami-challenge. org) and the results are shown in Additional file 1: Figure  S2. IMP-based MG assembly using the MEGAHIT assembler on the CAMI dataset outperforms well-established MG pipelines such Results for all datasets available in Additional file 2: Table  S2  as MOCAT in all measures. In addition, IMP-based iterative assemblies also exhibit comparable performance to the gold standard assembly with regards to contigs ≥1 kb and number of predicted genes (http://www. camichallenge. org). Detailed results of the CAMI assemblies are available in Additional file 2: Table  S5.
Data usage: multi-omic iterative co-assembly In order to assess the advantages of integrated multi-omic co-assemblies of MG and MT data, IMP-based iterative co-assemblies (IMP and IMP-megahit) were compared against MG-only-based assemblies which include single-omic iterative MG assemblies generated using IMP (referred to as IMP_MG) and standard MG assemblies by MOCAT (hereafter referred to as MOCAT_MG) and MetAMOS (hereafter referred to as MetAMOS_MG). Furthermore, the available reads from the human fecal microbiome dataset (preprocessed with IMP) were mapped to the MetaHIT Integrated Gene Catalog (IGC) reference database  [35]  to compare the data usage of the different assembly procedures against a reference-dependent approach. IMP-based iterative co-assemblies consistently recruited larger fractions of properly paired MG (Fig.
3a ) and/or MT (Fig. 3b   S5  assemblies. The resulting assemblies also produced larger numbers of contigs ≥1 kb (Fig. 3c ), predicted nonredundant unique genes (Fig. 3d ), and, even more important, complete genes as predicted with start and stop codon by Prodigal  [61]  (Additional file 2: Table  S5 ). Using the reference genomes from the SM data as ground truth, IMP-based iterative co-assemblies resulted in up to 25. 7% additional recovery of the reference genomes compared to the single-omic MG assemblies (Additional file 2: Table  S5 ). IMP-based iterative co-assemblies of the human fecal microbiome datasets (HF1-5) allowed recruitment of comparable fractions of properly paired MG reads and an overall larger fraction of properly paired MT reads compared to those mapping to the IGC reference database (Table  2 ).
The total fraction (union) of MG or MT reads mapping to either IMP-based iterative co-assemblies and/ or the IGC reference database was higher than 90%, thus demonstrating that the IMP-based iterative co-assemblies allow at least 10% of additional data to be mapped when using these assemblies in addition to the IGC reference database. In summary, the complementary use of de novo co-assembly of MG and MT datasets in combination with iterative assemblies enhances overall MG and MT data usage and thereby significantly increases the yield of useable information, especially when combined with comprehensive reference catalogs such as the IGC reference database.
Assembly quality: multi-omic iterative co-assembly In order to compare the quality of the IMP-based iterative co-assembly procedure to simple co-assemblies, we compared the IMP-based iterative co-assemblies against co-assemblies generated using MetAMOS  [10]  (henceforth referred to as MetAMOS_MGMT) and MOCAT  [34]  (henceforth referred to as MOCAT_MGMT). Although MetAMOS and MOCAT were developed for MG data analysis, we extended their use for obtaining MG and MT co-assemblies by including both MG and MT read libraries as input (section "Execution of pipelines"). The assemblies were assessed based on contiguity (N50 length), data usage (MG and MT reads mapped), and output volume (number of contigs above 1 kb and number of genes; Additional file 2: Table  S5 ). Only the SM dataset allowed for ground truth-based assessment by means of aligning the generated de novo assembly contigs to the original 73 bacterial genomes used to simulate the data set (section "Simulated coupled metagenomic and metatranscriptomic dataset")  [12, 54].
This allowed the comparison of two additional quality metrics, i. e. , the recovered genome fraction and the composite performance metric (CPM) proposed by Deng et al. [62]. Assessments based on real datasets demonstrate comparable performance between IMP and IMPmegahit while both outperform MetAMOS_MGMT and MOCAT_MGMT in all measures (Fig. 4a-c ). The ground truth assessment using the SM dataset shows that IMP-based iterative co-assemblies are effective in recovering the largest fraction of the original reference genomes while achieving a higher CPM score compared to co-assemblies from the other pipelines. Misassembled (chimeric) contigs are a legitimate concern within extensive de novo assembly procedures such as the IMP-based iterative co-assembly. It has been previously demonstrated that highly contiguous assemblies (represented by high N50 lengths) tend to contain higher absolute numbers of misassembled contigs compared to highly fragmented assemblies, thereby misrepresenting the actual quality of assemblies  [38, 62, 63].
Therefore, the CPM score was devised as it represents a normalized measure reflecting both contiguity and accuracy for a given assembly  [62]. Based on the CPM score, both IMP and IMP-megahit yield assemblies that balance high contiguity with accuracy and thereby outperform the other methods (Fig. 4c, d ). In summary, cumulative measures of numbers of contigs ≥1 kb, N50 lengths, numbers of unique genes, recovered genome fractions (%), and CPM scores (the latter two were only calculated for the SM dataset), as well as the mean fractions (%) of mappable MG and MT reads, show that the IMP-based iterative co-assemblies (IMP and IMP-megahit) clearly outperform all other available methods (Fig. 4e ; Additional file 2: Table  S5 ).
Use-cases of integrated metagenomic and metatranscriptomic analyses in IMP The integration of MG and MT data provides unique opportunities for uncovering community-or populationspecific traits, which cannot be resolved from MG or MT data alone. Here we provide two examples of Average fractions (%) of properly paired reads from the human microbiome datasets (HF1-5) mapping to various references, including IMP-based iterative co-assemblies (IMP and IMP-megahit) and single-omic co-assemblies (IMP_MG, MetAMOS_MG, and MOCAT_MG) as well as the IGC reference database. IMP + IGC and IMP-megahit + IGC reports the total number of properly paired reads mapping to IMP-based iterative co-assemblies and/or the IGC reference database. Refer to Additional file 2: Table  S3  for detailed information insights gained through the direct inspection of results provided by IMP.
Tailored preprocessing and filtering of MG and MT data The preprocessing of the datasets HF1-5 included filtering of human-derived sequences, while the same step was not necessary for the non-human-derived datasets, WW1-4 and BG. MT data analyzed within this article included RNA extracts which were not subjected to wetlab rRNA depletion, i. e. , BG  [29] , and samples which were treated with wet-lab rRNA removal kits (namely HF1-5  [28]  and WW1-4  [43] ). Overall, the removal of rRNA pairs from the MT data showed a large variation, ranging from as low as 0. 51% (HF5) to 60. 91% (BG), demonstrating that wet-lab methods vary in terms of effectiveness and highlighting the need for such MTspecific filtering procedures (Additional file 1: Note S2; Additional file 2: Table  S6 ).
Identification of RNA viruses To identify differences in the information content of MG and MT complements, the contigs generated using IMP were inspected with respect to coverage by MG and MT reads (Additional file 2: Table  S7 ). In two exemplary datasets HF1 and WW1, a small fraction of the contigs resulted exclusively from MT data (Additional file 2: Table  S7 ). Longer contigs (≥1 kb) composed exclusively of MT reads and annotated with known viral/bacteriophage genes were retained for further inspection (Table  3 ; complete list contigs in Additional file 2: Table  S8  a d e b c Fig. 4  Assessment of the IMP-based iterative co-assemblies in comparison to MOCAT-and MetAMOS-based co-assemblies. Radar charts summarizing the characteristics of the co-assemblies generated using IMP, MetAMOS, and MOCAT pipelines on: a human fecal microbiome, b wastewater sludge community, c biogas reactor, d simulated mock community.
IMP co-assemblies were performed with two de novo assembler options, IDBA_UD and MEGAHIT, whereas MetAMOS and MOCAT were executed using default settings. Assessment metrics within the radar charts include number of contigs ≥1 kb, N50 length (contiguity, cutoff 500 bp), number of predicted genes (unique), and fraction of properly mapped MG and MT read pairs. N50 statistics are reported using a 500-bp cutoff. Additional ground truth assessments for simulated mock dataset included recovered genome fractions (%) and the composite performance metric (CPM) score with a cutoff of 500 bp  [62]. e Summary radar chart reflecting the cumulative measures and mean fraction of properly mapped MG and MT read pairs from all analyzed 11 datasets while incorporating ground truth-based measures from the simulated mock dataset.
Higher values within the radar charts (furthest from center) represent better performance. Detailed information on the assembly assessments is available in Additional file 2: Table  S5  and S9). A subsequent sequence similarity search against the NCBI NR nucleotide database  [64]  of these candidate contigs revealed that the longer contigs represent almost complete genomes of RNA viruses (Additional file 2: Table  S10  and S11 ). This demonstrates that the incorporation of MT data and their contrasting to the MG data allow the identification and recovery of nearly complete RNA viral genomes, thereby allowing their detailed future study in a range of microbial ecosystems.
The MG-based average depths of coverage of the contigs from the entire community exhibited a bellshape like distribution, with a clear peak (Fig. 5b ). In contrast, MT depths of coverage exhibited more spread, with a relatively low mean (compared to MG distribution) and no clear peak (Fig. 5b ). The C. intestinalis subset displays similar distributions to that of the entire community, whereas the E. coli subset clearly exhibits unusually high MT-based and low MG-based depths of coverage (Fig. 5b ). Further inspection of the individual omic datasets revealed that the E. coli subset was not covered by the MG contigs, while approximately 80% of the E. coli genome was recoverable from a single-omic MT assembly (Fig. 5c ). In contrast, the C. intestinalis subset demonstrated genomic recovery in all co-assemblies (IMP, IMP-megahit, MOCAT_MGMT, MetAMOS_MGMT) and the singleomic MG assemblies (IMP_MG, MOCAT_MG, MetA-MOS_MG; Fig.