dataesr/scientific-paragraphs-categorization huggingface dataset:
As recently described, oligotyping  (Eren et al. , 2013a)  also employs a form of signature analysis by using Shannon entropy  (Shannon, 1948)  to distinguish biologically meaningful signals from noise without requiring the calculation of pairwise sequence similarity. By relying on information-rich variable sites and discarding low-entropy nucleotide positions in a group of sequencing reads, oligotyping facilitates the identification of closely related but distinct organisms that may differ by as little as one nucleotide over the sequenced region of the 16S rRNA gene. Oligotyping different high-throughput sequencing datasets has resolved unexplained diversity within taxa and OTUs  (Eren et al. , 2011; Mclellan et al. , 2013; Apprill et al. , 2014; Eren et al. , 2014; Maignien et al. , 2014; Reveillaud et al.
, 2014). However, oligotyping has characteristics that sometimes limit its applicability to environmental datasets. First, oligotyping performs optimally when applied to closely related taxa as the great number of high-entropy locations among distantly related organisms renders the supervision step arduous  (Eren et al. , 2013a). Second, oligotyping requires a preliminary OTU clustering or taxonomic classification to identify closely related taxa suitable for analysis. Oligotyping is not a stand-alone approach and its improvements cannot be applied directly to the entirety of a sequencing dataset. Nevertheless, oligotyping shows the potential of entropy to partition mixed information into homogenous units by using a fraction of available sequencing data, and without pairwise sequence alignment and comparison (Figure  1 ).
We have developed and employed a new algorithm, 'Minimum Entropy Decomposition' (MED), to partition high-throughput sequencing datasets into ecologically meaningful and phylogenetically homogeneous units by extending the underlying principles of oligotyping to entire marker gene datasets. MED uses information uncertainty among sequence reads to iteratively decompose a dataset until the maximum entropy criterion is satisfied for each final unit (i. e. , until there is no entropy left to explain; Figure  2 ). In contrast to oligotyping, MED requires no user supervision, no preliminary classification or clustering result, and can be applied to the entire dataset instead of only a group of closely related taxa. We used two datasets to demonstrate the utility of MED: a new V4-V5 rRNA gene dataset from previously described host microbiomes of deep-sea sponges  (Reveillaud et al.
, 2014)  and a publicly available human oral microbiome dataset (The Human Microbiome Project Consortium, 2012b). We compared MED analysis results to taxonomic analysis and de novo OTU clustering using QIIME with UCLUST, which among several methods, has often been used in studies of the Human Microbiome Project (The Human Microbiome Project Consortium, 2012a) and the Earth Microbiome Project  (Gilbert et al. , 2010).
Minimum Entropy Decomposition The algorithm iteratively partitions a dataset of amplicon sequences into homogenous OTUs ('MED nodes') that serve as input to alpha-and betadiversity analyses. MED inherits the core principle of oligotyping  (Eren et al. , 2013a)  and uses Shannon entropy to identify information-rich positions within an internal node. Entropy increases proportionally to the amount of variability in a nucleotide position and MED uses high entropy positions to decompose a node into child nodes. A nucleotide position that directs a decomposition step will have zero entropy in child nodes. Hence, the increasing number of identified nodes decreases the cumulative entropy in the dataset (Figure  1 ). For each cycle of decomposition, an entropy profile that does not contain any discernible entropy peak defines the stop condition of the algorithm for a given node.
If a node's entropy profile is not uniform, indicating that it has not converged, MED will decompose the node further. Figure  2  gives an overview of the algorithm. The MED algorithm operates on high-throughput sequencing datasets without requiring an initial DNA sequence alignment step. Like oligotyping, MED requires length differences among sequenced reads to represent biologically meaningful variation rather than indels derived from systematic sequencing errors such as homopolymer indel events commonly encountered with 454/Roche technology. If read length differences represent biologically meaningful variation, appending gap characters to the short reads would be sufficient to finalize the data; otherwise, an alignment step would be Figure  1  Decomposing a mock dataset of 10 reads using Shannon entropy.
The algorithm can detect biologically meaningful differences between closely or distantly related sequences in large datasets without requiring CPUintensive alignment. Although subsampling is not mandatory for MED, it must be considered if the number of reads across samples in a dataset differ by multiple orders of magnitude to avoid biases in entropy results. The MED pipeline assumes that the input dataset is quality-filtered. Most widely used quality-filtering approaches trim low-quality ends of sequencing reads  (Minoche et al. , 2011; Bokulich et al. , 2013) ; to eliminate artificial read length variation for MED analysis, reads that require trimming should be discarded prior to analysis. MED (see Figure  2 ) initially adds the input dataset (Step 1) to the decomposition pool as a node for analysis.
We recommend setting M to N/10 000 or larger, where N equals the number of reads in the dataset. A node that satisfies M is subjected to Shannon entropy analysis to identify information-rich positions (Step 4), and the node is identified as a terminal node if it has a flat entropy profile. Owing to sequencing errors, entropy is rarely zero for a given nucleotide position in large nodes, hence MED uses a threshold (m), below which entropy is treated as zero. The m parameter is determined dynamically during the runtime for each node, and by default it decreases from 0. 0965 (the expected entropy generated by a sequencer with 1% error rate) proportional to the ratio of the size of a node and the frequency of the most abundant unique read in the dataset. The following equation demonstrates the default m heuristics in the MED algorithm developed empirically based on our experience with Illumina HiSeq and MiSeq datasets, m 0 ¼ m À ffiffiffiffiffiffiffi ffi mn 2N r where m' is the normalized m for a node with n reads found in a dataset where the frequency of the most abundant unique read is N.
Sample collection and handling Sponge data. We sequenced the V4-V5 region of bacterial rRNA genes from 19 deep-sea sponge and 5 control water samples. Samples were collected at locations along 45000 km of the European margins and spanned wide bathymetric gradients (130-958 m)  (Reveillaud et al. , 2014). Human Microbiome Project data. The Human Microbiome Project Consortium (2012a) describes the sample collection and pyrosequencing of the oral microbiome dataset. Of the 242 individuals who participated in the study, we included only individuals who were sampled at each oral site (n ¼ 148 for V3-V5 data)  (Eren et al. , 2014).
Quality filtering We merged and quality-filtered the partially overlapping V4-V5 paired-end reads of the sponge microbiome dataset using our Illumina Utilities library (https://github. com/meren/illumina-utils; see Supplementary Methods). To minimize the impact of read count variation, we randomly subsampled the sponge dataset to a maximum of 20 000 reads per sample. After quality filtering and subsampling, the sponge dataset represented 373 474 reads from 24 samples. Supplementary Table  S2  reports the number of raw and quality-filtered reads per sample. VAMPS  (Huse et al. , 2014)  posts the filtered sponge datasets under project ID MER_MED_Bv4v5 and the NCBI Sequence Read Archive hosts the unprocessed sequences under accession SRP042371. The Human Microbiome Project oral microbiome dataset from 1332 samples (9 oral sites sampled from 148 individuals) initially contained 47 M quality-filtered reads.
The Human Microbiome Project Consortium's quality filtering method removed reads with one or more ambiguous base calls and those with a homopolymer of 8 nt or longer. Reads trimmed from the 3 0 end used a sliding window of average quality score (The Human Microbiome Project Consortium, 2012a). We re-trimmed the data to a consistent length of B235 nt and discarded short reads  (Eren et al. , 2014) , resulting in 5 926 860 quality-filtered reads.
Taxonomic classification, OTU clustering and MED analyses We used GAST  (Huse et al. , 2008)  to assign taxonomy to our reads individually. For OTU clustering, we used QIIME v1. 5  (Caporaso et al. , 2010)  with UCLUST  (Edgar, 2010)  in de novo mode via the pick_otus. py script. We ran UCLUST with default parameters and generated clusters with 97% similarity threshold. We used the open-source MED pipeline version 1. 2 for MED analysis. To filter noise, we set the minimum substantive abundance criterion (M) to 100. We removed GAST taxa and OTUs that were represented by fewer than 100 reads for comparability to the MED results. To assess the performance of MED with respect to UCLUST, we subsampled the oral microbiome dataset incrementally and analyzed each file separately with both methods.
Statistical analyses and visualization We evaluated sample group variances in the sponge microbiome and human oral microbiome datasets explained by each method using the betadisper function in the vegan package  (Anderson, 2006)  for R (R Core Team, 2013) and reduced distances produced from Horn dissimilarity coefficients to principal coordinates, which embeds sample coordinates in Euclidean space. Non-parametric multivariate analysis of variance on the dataset coordinates (using the ADONIS function in the vegan package identified the ratio of between-group variance to within-group variance and the proportion of total variation associated with different sites in the human oral cavity. To investigate the recovery of previously identified taxa, we searched the representative sequences of taxa, OTUs and MED nodes in the oral microbiome dataset against the Human Oral Microbiome Database (HOMD)  (Chen et al.
Improved partitioning of oral microbiome sites by MED and oligotyping We re-analyzed an oral microbiome dataset of B6 million V3-V5 rRNA sequences (The Human Microbiome Project Consortium, 2012a) using taxa, de novo OTUs and MED nodes. The previous report of 481 oligotypes  (Eren et al. , 2014)  compares to GAST and clustering detection of 122 taxa and 329 OTUs, respectively. Using the same dataset, MED identified 858 terminal nodes. UCLUST analyzed the oral microbiome dataset two times faster than MED, which took B25 min in our simulation. Runtimes for both algorithms increased linearly with increasing number of reads (Supplementary Figure  S1 ). Figure  4  demonstrates the tree-like topology of the decomposition process with intermediate and terminal nodes identified in the oral microbiome dataset.
Multivariate analysis of variance compared the captured proportion of variation in this dataset associated with the oral sites using all four methods. Multivariate analysis of variance analysis of the same dataset produced two comparison metrics: F-ratio and R 2. Increasing similarity of samples collected from the same oral site compared to samples collected from other sites increases the F-ratio. The R 2 value indicates the proportion of total variation captured by the model. Oligotyping and MED partitioned the dataset with increased resolution relative to taxon-based and OTU-based analyses (Figure  5 ).
MED recovers more organisms from the HOMD The curated HOMD holds 688 near-full-length rRNA gene reference sequences for microbial taxa isolated from healthy and diseased human oral cavities  (Dewhirst et al. , 2010). We evaluated the ability of the four analytical methods to identify known oral microbes. We compared each sequence in the oral microbiome dataset with the HOMD and recovered 516 matches with 100% identity and coverage. Of these, 248 occurred more than 100 times in our dataset. This defines the maximum number of HOMD taxa that we can identify in the oral microbiome dataset that contain at least 100 members in a taxonomic group, cluster or node. The remaining perfect hits to the HOMD that occurred fewer than 100 times represented 0. 056% of the oral microbiome dataset. BLAST queries using representative sequences of each taxon, OTU and MED node identified 67, 112 and 235 matches, respectively, to sequences in the HOMD database using the criteria of 100% identify and coverage.
When we limited our comparison to only the 329 most abundant MED nodes, numerically equal to the 329 OTUs found, the MED nodes still matched more HOMD taxa (138) than did the OTUs (112). Representative sequences of taxa, OTUs and MED nodes, respectively, identified 27%, 45% and 95% of the strains described in HOMD using sensitivity settings of 100 or more occurrences. By partitioning a dataset of B6 M reads into 858 terminal nodes, MED recovered 95% of what the unique reads would have recovered at the same sensitivity.
MED nodes are ecologically relevant The larger number of exact sequence matches to the HOMD reference database for MED nodes (235 matches) relative to OTUs (112 matches) from the oral microbiome dataset reflects MED's ability to partition the data into more phylogenetically homogeneous units. To investigate the impact of an increased number of units on ecological inference, we explored the source OTUs and taxa for reads that make up the most abundant 100 MED nodes using an alluvial diagram (Figure  6 ). A total of 4 936 104 reads (83% of the entire dataset) accounted for the 100 most abundant MED nodes. For clarity, we did not draw connections that corresponded to fewer than 1000 reads. This analysis accounted for 29 taxonomic groups and 57 OTUs. In most cases, individual taxa split into multiple OTUs, which then split into multiple MED nodes.
Discussion MED partitions large marker gene datasets into terminal nodes that can share higher levels of sequence similarity than OTU clusters. MED identified organisms in two example datasets that differ by only a few nucleotides, yet distribute differently across environments, and recapitulated published oligotyping results. Instead of requiring pairwise sequence alignment or taxon assignments, MED uses a fraction of the available nucleotide variation in an iterative process that relies upon the most information-rich sites to decompose large datasets. In our sponge microbiome analysis, MED detected substantial differences in microbial communities between two closely related deep-sea sponge species, H. cf. dedritifera and H. dedritifera that other approaches missed. Cluster analysis identified the highly abundant OTU #0, which included a large number of reads that GAST mapped only to the class Gammaproteobacteria.
MED analysis of the dataset split OTU #0 into two distinct nodes with representative sequences that showed only 3 nt differences (99. 2% sequence identity). Although H. cf. dedritifera and H. dedritifera can occur sympatrically (i. e. , on the Irish continental margin), MED revealed highly specific sponge-Proteobacteria associations at a very fine taxonomic scale. Gammaproteobacteria reads that cluster together in OTU #0 represent abundant and ecologically distinct organisms that might serve different functions for the sponge host or out-compete other variants in response to a given host environment. Speciesspecificity sometimes reflects complex genetic interdependence between a host and its associated bacteria  (Bulgheresi et al. , 2006; Gourdine et al. , 2007; Mandel et al. , 2009; Franzenburg et al.
, 2013). Unlike oligotyping, MED identifies subtle nucleotide variation among high-throughput sequencing reads without user supervision. We recently used oligotyping to explore the diversity of very closely related commensal and pathogenic organisms within distinct phyla that represent more than 99% of the sequencing data in a human oral microbiome dataset  (Eren et al. , 2014). Our supervised analysis of each major phylum revealed that some organisms differing by as few as a single nucleotide showed dramatically different distributions among oral sites and among individuals  (Eren et al. , 2014). Here we re-analyzed the same dataset to Figure  5  Comparison of oral microbiota sample distribution with network analysis and multivariate analysis of variance results. The top panel shows network analysis results for each method.
Proper sequence partitioning will reduce the number of observed units in a given dataset without sacrificing ecological inference. In the absence of PCR or sequencing errors, unique sequences in a dataset would serve as ideal proxies for genomic signatures in microbial communities. However, even the most stringent quality-filtering methods applied to large marker gene datasets will retain many more unique sequences than the true number of different organisms. Hence the dilemma: partitioning strategies must account for subtle differences between highly similar sequences to accurately represent every organism, while not inflating the number of observed groups artificially. A thorough comparison of partitioning methods that strive to achieve this goal requires determining the minimum number of units in a dataset that accurately represent every organism, which de novo investigations cannot determine.
However, a curated database can serve as a reference to demonstrate how many well-described organisms each method recovers. In our analysis of the oral microbiome, we used the phylogenetically curated HOMD of 688 near fulllength 16S rRNA gene sequences from mostly cultivated organisms as a reference. Among the B230 000 unique sequences in our oral microbiome dataset, only 248 unique sequences with a frequency of 100 or higher matched an entry in HOMD. Hence, the goal for any partitioning method is to generate units with representative sequences that match 248 sequences in HOMD without inflating the total number of units. All three unsupervised methods reduced the number of observed units by orders of magnitude compared to the number of unique sequences; however, MED representative sequences recovered 95% of expected HOMD entries, whereas the representative sequences of taxa and OTUs recovered only 27% and 45%, respectively.
One solution is to increase the similarity threshold, as previously proposed  (Stackebrandt and Ebers, 2006) ; however, this approach does not scale. Although longer reads provide increased phylogenetic resolution, they will likely possess larger numbers of sequencing errors, precluding the use of higher similarity thresholds. Here, even a very conservative similarity threshold of 99%, while greatly increasing the number of observed OTUs, would have failed to identify distinct microbiomes of closely related sponge species or distinguish ecologically distinct members of the oral microbiome that share more than 99% sequence identity (Figure  6 ). The sensitivity of computational methods that partition sequencing datasets should be agnostic to read length and pairwise sequence similarity. Our MED results demonstrate the advantage of relying on information theory rather than pairwise sequence similarity to sensitively define ecologically relevant units in a dataset.
It differs in important ways from commonly used clustering approaches. MED iteratively partitions sequencing datasets of marker genes into phylogenetically homogeneous units using entropy, but imposes minimal computational heuristics by disregarding sequence similarity or phylogenetic relationships between reads. As MED does not perform pairwise sequence alignment and similarity assessment during decomposition, it utilizes a fraction of the available nucleotide data in a dataset. Instead of grouping reads based on sequence similarity, it splits groups of reads into more refined units incrementally based on the nucleotide positions that present dissimilarities identified by the entropy analysis. These differences allow MED to detect and explain subtle nucleotide variation more effectively and identify distinct organisms that are as few as one nucleotide apart regardless of the length of the sequenced region.
No currently available method addresses every potential artifact associated with large marker gene datasets. Sequencing errors, PCR errors, alignment artifacts and failure to resolve closely related taxa lead to inaccurate assessments of microbial community diversity and ecology. Analytical methods that mitigate the influence of sequencing errors through de novo clustering at a predefined level of sequence similarity generally sacrifice sensitivity as they fail to resolve very closely related taxa that track ecological context. In contrast, MED provides very a high-resolution depiction of microbial communities while recapitulating the oligotyping results without user supervision. To eliminate 'noise' or false nodes caused by sequencing error, it employs the criterion 'minimum substantive abundance' (M) to dictate the removal of any intermediate or terminal MED nodes with a representative sequence that occurs in the dataset less than M times.
One notable feature of MED from a computational perspective is that individual nodes are analyzed independently from each other throughout the decomposition process (Figure  4 ). Thus the algorithm eliminates the need for shared memory space for subsequent steps of analysis, which makes significant performance gains possible through distributed and parallel implementations. MED is a de novo approach; therefore MED nodes, in theory, are not comparable across studies. However, as a fully resolved MED node will have minimal phylogenetic mixture, to what extent the representative sequences of MED nodes can be compared across studies warrants further investigation. In this study, we demonstrated the capacity of MED, a sensitive approach that explains the diversity of closely related organisms in high-throughput marker gene datasets regardless of read length and percent similarity.
Acknowledgements We thank  Loı ¨s Maignien ,  Tom O. Delmont ,  Sharon L. Grim ,  Gary G. Borisy  and  Ines Martı  ´nez for helpful discussions, our anonymous reviewers for their suggestions, and  Richard Fox  for systems support. AME was supported by a  G. Unger Vetlesen Foundation  grant to the  Marine Biological Laboratory  and the  Alfred P Sloan Foundation.
2. 1. Patients. Gene expression microarray data from three independent cohorts of patients diagnosed with CLL were used. Publicly available gene expression data from 2 cohorts with newly diagnosed CLL patients were used to construct GE-based risk score  [15]. The first cohort, used as the training cohort, comprised 107 patients, and the second one as the validation cohort comprised 44 patients  [15]. Peripheral blood or bone marrow samples were analyzed by Affymetrix oligonucleotide microarrays  [15]. A third cohort of 130 newly diagnosed patients, with available Affymetrix gene expression data, was used as validation cohort for time to treatment analyses  [16]. Clinical characteristics of patients and number and schedules of treatments were previously published  [15, 16]. Interphase FISH data of the training cohort were previously published  [17].