dataesr/scientific-paragraphs-categorization huggingface dataset:
Whole-genome databases are rapidly growing, however. Since the publication of the first genome sequence of a free-living bacterium, Haemophilus influenzae, in 1995  (49) , the number of sequenced microbial genomes has increased exponentially  (137) , with 129,346 sequencing projects at various stages of completion registered in the Genomes Online Database (GOLD) as of March 2017  (125). Currently, microbial genomes are available through multiple-often overlapping and cross-referenced-databases that have been customized for different purposes, ranging from those that aim to encompass all microbes, such as GOLD and the National Center for Biotechnology Information's RefSeq database  (131)  to those that focus on specific groups of bacteria, such as EnteroBase (http:// enterobase. warwick. ac.
uk) and the HOMD  (27). Pipelines that utilize whole-genome data for taxonomic assignments include the MEGAN Alignment Tool (MALT)  (71)  and MIDAS (129). 6. 1. 2. The query strategy-Metagenomic tools differ not only in the way that their reference databases are constructed but also in the way those databases are queried, which depends on whether read alignment is performed. As such, query strategies can be divided into alignment-based and alignment-free approaches.
Alignment-based approaches: One important strategy for the assignment of query sequences to large reference databases is based on classical sequence alignments. The algorithms applied are usually modifications of the Needleman-Wunsch (130) or Smith-Waterman (177) algorithms designed to perform semiglobal or local alignments, respectively. However, the calculation of precise alignments is time consuming, even for more efficient variants of these algorithms. For this reason, most software tools use so-called seed-and-extend approaches. Here, candidate matches are first determined in a time-effective but imprecise manner, and precise alignments are then calculated only for these candidates. Well-known examples of this algorithm type include BLAST (5) and its faster variant MEGABLAST  (124) , as well as UBLAST and USEARCH  (43).
Alignment-free approaches: Not all methods for metagenomic analysis rely on calculating precise alignments. Rather than taxonomically assigning sequencing reads or other sequences, alignment-free methods determine the relative abundances of taxa based on other features. Kraken  (201) , for example, uses an approach that is based on exactly matching k-mers. Both the sequences present in the reference database and the query sequences are split into overlapping subsequences of length k. This allows fast matching of queries to the database by hashing algorithms, a concept that is often applied in the seeding step of seed-and-extend alignment-based methods. Other methods are based on similarities and dissimilarities of nucleotide composition  (74) , comparison of Burrows-Wheeler transformed sequence information  (6) , or machine learning approaches  (156).
Data type- The input data type for metagenomic analysis is DNA sequencing reads or assembled DNA sequences, such as contigs. The data type for the reference database can be either DNA or amino acid sequences. In the case of a protein database, the DNA input sequences must be translated from all six possible reading frames, which is analogous to BLASTX (  5 ). An advantage of using a protein database is its higher level of sequence conservation, which allows the exploration of deeper evolutionary relationships. Furthermore, tools such as DIAMOND  (19) , lambda  (69) , and RapSearch2 (206) provide extremely fast algorithms for performing DNA-to-protein alignments. However, protein alignments can be challenging to use for ancient metagenomic data because they are incompatible with DNA alignment-based authentication procedures.
Algorithms, Pipelines, and Environments A versatile selection of software is available for metagenomic analysis. In the context of aDNA, these software packages are often not directly comparable; some of them represent specific algorithms that perform specific steps of an analysis, whereas others combine algorithms into analysis pipelines that perform multiple steps of the metagenomic analysis. Algorithms can be divided into different categories according to the analytical step they are performing. If they apply an alignment-based concept, then the first step is alignment, or mapping. In principle, any DNA or protein alignment tool can be used, including sequence similarity search tools such as BLAST (5), as well as read mappers. The next step is to classify the alignments using a taxonomy in order to assign the query sequences to specific taxa, a process known as binning.
Examples of binning algorithms include UCLUST  (43) , RDP Classifier  (194) , naive lowest common ancestor (LCA), and weighted LCA  (80). MALT (  71 ) is a tool that combines alignment and binning. It is based on a seed-and-extend approach and uses spaced seeds for higher sensitivity. Seeds are matched to a DNA sequence database that can contain single or multiple loci or whole genomes. MALT calculates precise local or semiglobal alignments and uses (weighted) LCA for binning. It integrates well with the MEGAN environment, which does not contain alignment functionality on its own. MetaPhlAn  (172)  is an example of a metagenomic analysis pipeline. It uses a selection of multiple loci as a target database. The selection of loci varies for different taxa, thus ensuring specificity on different taxonomic levels and for different parts of the taxonomy.
For its alignment algorithms, MetaPhlAn integrates MEGABLAST or Bowtie 2; however, other alignment algorithms could be integrated as well. The taxonomic assignment algorithm is specific to MetaPhlAn owing to its unique database structure. Another example of a metagenomic pipeline is MIDAS  (129) , which combines multiple databases, aligners, and binning algorithms into an integrated pipeline. In some cases, single algorithms and more complex analytical procedures are made accessible in interactive environments that also provide an integrated visualization of the results. MEGAN (  80 ) is an example of such an environment. In other cases, multiple pipelines are made accessible through an environment, as is the case with QIIME (96) and metaBIT  (110). MetaBIT, for example, implements MetaPhlAn-based microbial profiling and provides built-in modules for (a) visualizing profiles, including Krona plots (  134 ); (b) calculating diversity indices (e.
g. , alpha and beta diversity); and (c) performing a range of statistical analyses. These analyses include principal coordinate analysis, hierarchical clustering, and linear discriminant analysis, the latter of which is used to identify microbes that drive differentiation among user-defined groups, such as archaeological sites, samples, and/or DNA libraries  (171). MetaBIT also provides preprocessed microbial profiles from the Human Microbiome Project  (78, 79)  and cross-biome soil communities  (48) , allowing users to place their (ancient) microbial communities within the known diversity of human and soil microbiomes. Because each software strategy operates differently and makes different assumptions based on the algorithms and databases incorporated into the pipeline, the results generated by different software are not identical.
THE CHALLENGE OF RELATED TAXA Although numerous tools and growing databases are available for taxonomic identification, care must be taken to avoid taxonomic misassignment, especially for the short, damaged sequences typical of aDNA. Accurate taxonomic assignment can be particularly challenging if closely related environmental contaminants are also present. Additionally, clades of closely related species within endogenous microbiota can be difficult to resolve if database coverage is poor or sequencing effort is insufficient. Validation efforts focusing on the evenness of genome mapping coverage, sequence identity distributions, and sequence monomorphism (haploidy) are powerful approaches for identifying and eliminating false positive assignments in microbial archaeology studies.
Other pathogens with close, nonpathogenic environmental relatives include B. pertussis, S. enterica, Bacillus anthracis, Clostridium botulinum, Shigella dysenteriae, and Vibrio cholerae (see Supplemental Appendix 3). Figure  3  shows the extent of closely related cultured and uncultured relatives for each of these eight pathogens based on a 16S rRNA gene sequence identity of >97% in the RDP database of more than 3 million sequences. When working with very large data sets, as is typical for HTS, it is important to be aware of -and have a plan for handling-false positives. In pathogen studies, shotgun metagenomic sequence data generated from bone lesions or dental pulp (typically a few million sequence reads) are often initially mapped against a panel of pathogen genomes using standard mapping tools in order to screen for putative taxa of interest.
Although an important first step, this approach yields many false positives, and in fact, nearly any large metagenomic data set is expected to yield false positives under such an approach. For example, mapping metagenomic data from three sources-soil, ocean, and healthy human saliva-against a panel of 14 pathogen reference genome sequences resulted in numerous false positive hits to all 14 pathogens, ranging from a low of 2 Treponema pallidum hits per million soil reads to a high of 173 B. anthracis hits per million saliva reads (Supplemental Figure  1 ). Overall, healthy saliva yielded the highest number of false positives, with a mean of 67 false positive hits per million metagenomic sequences. None of these reads were true positives; rather, they resulted from mismapping at highly conserved regions and in regions of low complexity, mapping of mobile elements and other horizontal gene transfer sites, and mapping at orthologous sites in related taxa because of database bias.
aureus and Propionibacterium acnes, and food microbes, such as Propionibacterium freudenreichii and Lactobacillus rhamnosus, may also be present. Correctly assigning sequences within these genera is therefore important for understanding the dietary and potential health implications of past microbial communities. Database bias, however, can be a major challenge because of the limited number of species with sequenced genomes, and it can result in both the underestimation of important taxa in ancient microbiota and sequence misassignment to the closest sequenced genome (Figure  4 ).
DNA Fragmentation aDNA is highly fragmented through a process that is driven by depurination followed by hydrolysis of the DNA backbone  (105). This process, initially described in vitro, is easily identifiable using HTS by looking at DNA breakpoints, which in aDNA are enriched in purines (both A and G)  (17). Fragmentation patterns can be visually inspected and are produced by default in software such as mapDamage  (83). The length distribution of aDNA can be approximated by a lognormal distribution (4) and shows an exponential decline in the tail, which results from random fragmentation of DNA  (36). Upon logarithmic transformation of fragment length frequencies, the decline can be characterized by a linear function with slope λ. The fragmentation constant λ therefore represents the fraction of bonds broken in the DNA backbone  (4, 36)  and is a summary statistic of the magnitude of DNA fragmentation.
Estimating Source Contribution In addition to profiling DNA damage and decay patterns, source modeling is a powerful tool for authenticating well-preserved microbiome samples. Originally developed to detect contamination in metagenomic studies, the software tool Source Tracker (91) models the composition of a metagenome sink (i. e. , sample) as the product of different contributing metagenome sources using Bayesian inference. For example, the microbial composition of a given sample, such as archaeological dental calculus, can be modeled as a mixture of DNA originating from dental plaque, skin bacteria, soil, and other sources. Using reference metagenomic data sets selected for each of these sources, the algorithm then estimates the proportion of the dental calculus sample originating from each source (Figure  7 ).
Although all four pipelines generally identify the same phyla, predictable biases are also readily apparent. For example, QIIME estimates the largest proportion to be Firmicutes, a phylum known to have a high rrn operon copy number  (190). Euryarchaeota is absent in the MIDAS analysis because there are no reference genome sequences for this phylum in the database. MIDAS and metaBIT, which rely on genome-scale databases, also fail to detect the largely uncultivated phyla Saccharibacteria (TM7), Chloroflexi, and SR1. Explanations for other differences in phylum frequency abundance, such as the high proportion of Actinobacteria estimated by MALT and the absence of Fusobacterium detected by MIDAS, are not as clear. Supplemental Appendix 1 provides the specific parameters used for each analysis.
Abbreviations: MALT, MEGAN Alignment Tool; MIDAS, Metagenomic Intra-Species Diversity Analysis System; QIIME, Quantitative Insights into Microbial Ecology. Lack of signal for highly abundant endogenous bacteria resulting from database bias. A medieval dental calculus sample (G12)  (196)  was screened using MALT  (71)  and visualized in MEGAN6 (80) using two databases, one without the genome sequence of the oral bacterium Pseudopropionibacterium propionicum (database 1, red) and another with that genome sequence included (database 2, blue). The tree visualizes the results of both analyses, and nodes are scaled based on the summed number of hits to a log scale. The inclusion of P. propionicum results in hits being shifted away from related dietary (Propionibacterium freudenreichii ), skin (Propionibacterium acnes), and other species, as well previously nonaligned hits, toward the oral bacterial genome, revealing the presence of a previously unseen, highly abundant species.
Acknowledgments This work was supported by the  Max Planck Society  (C. W. , A. H. , and J. K. ), its  Donation Award  (C. W. ), and its  Presidential Innovation Fund  (H. A. B. ); the  US National Science Foundation  (grants  BCS-1516633  and  BCS-1643318  to C. W. ); the  US National Institutes of Health  (grant  2R01GM089886  to C. W. ); the  European Research Council  (starting grant APGREID to J. K. and consolidator grant PEGASUS to L. O. ); the  Danish Research Foundation  (grant  DNRF94  to L. O. ); and the  Villum Fonden  (grant miGENEPI to L. O. ). The authors thank the participants of the 2016 Standards, Precautions, and  Advances in Ancient Metagenomics (SPAAM) conference  for their thoughtful comments, ideas, and suggestions, as well as Floyd Dewhirst at the  Forsyth Institute  for providing helpful information about the Human Oral Microbiome Database.
Motivation: Large pharmacogenomic screenings integrate heterogeneous cancer genomic datasets as well as anti-cancer drug responses on thousand human cancer cell lines. Mining this data to identify new therapies for cancer sub-populations would benefit from common data structures, modular computational biology tools and user-friendly interfaces. Results: We have developed GDSCTools: a software aimed at the identification of clinically relevant genomic markers of drug response. The Genomics of Drug Sensitivity in Cancer (GDSC) database (www. cancerRxgene. org) integrates heterogeneous cancer genomic datasets as well as anticancer drug responses on a thousand cancer cell lines. Including statistical tools (analysis of variance) and predictive methods (Elastic Net), as well as common data structures, GDSCTools allows users to reproduce published results from GDSC and to implement new analytical methods.
Introduction Cancers occur due to genetic alterations in cells accumulated through the lifespan of an individual. Cancers are genetically heterogeneous and as a consequence patients with similar diagnoses may vary in their response to the same therapy. The path towards precision cancer medicine requires the identification of specific biomarkers, such as genetic alterations, allowing effective patient selection strategies for therapy. Large-scale pharmacological screens such as the Genomics of Drug Sensitivity in Cancer (GDSC)  (Garnett et al. , 2012)  and Cancer Cell Line Encyclopaedia projects  (Barretina et al. , 2012)  have been used to identify potential new treatments and to explore biomarkers of drug sensitivity in cancer cells. In particular, the GDSC project releases database resources periodically (www.
cancerRxgene. org) (Yang et al. , 2013). A recent installment of this resource (version 17) includes cancer-driven alterations identified in 11 289 tumors from 29 tissues across 1001 molecularly annotated human cancer cell lines, and cell line sensitivity data for 265 anti-cancer compounds. A systematic identification of clinically-relevant markers of drug response uncovered numerous alterations that sensitize to anti-cancer drugs  (Iorio et al. , 2016). Here, we present GDSCTools, a Python library that allows users to perform pharmacogenomic analyses as those presented in  (Iorio et al. , 2016). Our software complements an existing tool  (Smirnov et al. , 2016)  by giving access to the full GDSC dataset and providing a powerful platform for statistical analyses and data mining through visualization tools.
Data formats and data wrangling tools The GDSC database provides large-scale genomics and drug sensitivity datasets. The drug sensitivity dataset contains dose-response curves (e. g. cell viability for 5-9 drug concentrations) which can be used to derive drug sensitivity indicators  (Garnett et al. , 2012; Vis et al. , 2016) , such as the half-maximal inhibitory concentration (IC 50 ) or the area under the curve (AUC) (Fig. 1A ). In GDSCTools, logged IC 50 indicators are encoded as a N c Â N d matrix, where N c is the number of cell lines labeled with their COSMIC identifier (http://cancer. sanger. ac. uk/cosmic) and N d is the number of drugs. For a given drug, we denote with Y d the vector of logged IC 50 s across the N c cell lines. The genomic feature dataset X is also encoded as a N c Â N f matrix, where N f is the number of genomic features.
Implementation and future directions GDSCTools is available on http://github. com/CancerRxGene/ gdsctools. It is fully documented on http://gdsctools. readthedocs. io. Pre-compiled versions of the library are available on https://bioconda. github. io/. GDSCTools can be used via standalone applications to analyse a user defined set of drugs (and genomic features) and assemble the results in an HTML report. We also provide solutions based on the Snakemake framework (Ko ¨ster and Rahmann, 2012) to parallelize the analysis on distributed cluster farm architectures such as LSF or SLURM (Supplementary Material). Besides analysis of pharmacogenomic datasets, GDSCTools can provide the framework for discovering new biomarkers through integration/mining of novel and heterogeneous datasets, including pharmacological, RNA interference or increasingly available genetic screens (e.
Therefore, the patches in the noiseless image follow a similar distribution as the database constructed from standard images. Hence, we can use p(α) learned from the database to denoise an image patch. The idea of probability distribution estimation is described as follows. From the standard images, we randomly extract a set of N clean image patches to form a database of patches, where each patch is denoted as x = [x1,. , xm] T ∈ R m. Let D = [d1,. , dK ] ∈ R m×K denote a given dictionary. A good quality patch x in the database can be represented in dictionary space Ωα ∈ R K by its orthogonal projection: α = (D T D) -1 D T x = PDα (4) where PD = (D T D) -1 D T is the projection matrix. Hence, with a database of patches in the image domain, we can obtain a distribution of decomposition coefficients in Ωα.
Fig. 2  shows the distribution of α in 3D and 2D visualization, respectively, whose forms are different from sparse distribution in Fig. 1. There are two main contributions in this paper. First, we propose using the probability distribution function p(α), which is estimated from image patches in the database, as an image prior to replace the sparsity models in the denoising problem. Second, we propose a denoising strategy to take into account the obtained p(α) in solving the optimization problem in  (3). The remainder of this paper is organized as follows: Section II tackles the estimation of the probability distribution p(α) in which p(α) is a piece-wise constant function. Then, a scheme for solving the optimization problem indicated in (3) is proposed to find α. Experimental results presented in section III prove the out-performance of proposed method over the sparsity ones.
II. DENOISING BY PROBABILITY DISTRIBUTION ESTIMATION A. Estimation of probability distribution p(α) from the database As estimation is a difficult problem in high dimensional spaces, we consider the α-space with dimensions up to three (Ωα ∈ R 3 ). This is equivalent to choosing a dictionary with only three atoms (K = 3) D = [d1, d2, d3] ∈ R m×3. After that, we randomly extract a set of N image patches x ∈ R m from the standard images to create a database of patches. In the next stage, all of patches in the database are projected into α-space Ωα using the projection matrix PD. As a result, we obtain a distribution of N points in Ωα. To demonstrate, we used the Kodak PhotoCD Dataset (Fig. 4 ) as standard images and randomly extracted a set of 500000 image patches. After the decomposition in (4), we obtained 500000 corresponding points distributed in Ωα as shown in Fig.
p(α) =    1 V s α point ∈ Ω s α point ∈ Ωα = 1 V s α Ns N , if α ∈ Ω s α 0, otherwise (5) where Ns is the number of points inside cell Ω s α which contains α, V s α is the volume of Ω s α (as D ∈ R m×3 ), N is the total number of points in Ωα that is equal to the number of image patches in the database. An important notification in the estimation of p(α) is how to divide each dimension of Ωα into the set of intervals to restrict the number of empty subvolumes, in which the solution of α in (3) is undetermined. Our strategy is to split each axis of Ωα such that the same number of points falls into each interval. By that way, the length of each interval can adapt to the density of points. In the experiments, each axis of Ωα is separated into 2 n intervals by 2 n -1 median points using a recursive division (n times), so that each interval contains N/2 n points.
B. Solving the optimization problem In this section, we will present how to employ the estimated p(α) in image denoising task. Given the dictionary D in section II-A and the prior knowledge of probability distribution p(α) estimated from the database of patches, our aim is to recover a clean image patch x from its noisy version y. The denoised patch can be generated as a linear combination of atoms in the dictionary, x = Dα *. Our objective is equivalent to find the value of α * which satisfies the optimization problem in (3): As indicated in previous section, we have divided the α-space Ωα into disjoint rectangular cells Ω s α , and the value of p(α) is constant in each cell Ω s α. Consequently, the cost function J(α) in (  6 ) becomes a cell-wise quadratic function. A simple way for finding the solution of α is to minimize J(α) on each rectangular cell Ω s α and then select the best value α * from which we achieve the smallest cost.