dataesr/scientific-paragraphs-categorization huggingface dataset:
Nonetheless, we can obtain a reasonable approximation to the dPCA solution using PCA in each marginalization. Namely, U f can be taken to constitute the encoders F f. In turn, the decoders D f are obtained by a pseudo-inverse D ¼ U þ , where U is a matrix with 4k columns obtained by joining together all U f. We found that this procedure provides a close approximation of the actual decoder and encoder matrices, provided one chooses a reasonable value of k: choosing k too small results in poor demixing, and choosing k too large results in overfitting. In our datasets, k ¼ 10 provides a good trade-off. This approximate solution highlights the conditions under which dPCA will work well, i. e. , result in well-demixed components that capture most of the variance of the data: the main principal axes of different marginalizations X f need to be non-collinear.
Comparison of dPCA with targeted dimensionality reduction Next, we compare dPCA with 'targeted dimensionality reduction' (TDR), the method proposed by  Mante et al. (2013). Briefly, the algorithm underlying TDR works as follows: 1. Perform PCA of the trial-average neural data e X and define a 'denoising' matrix K as a linear projector on the space spanned by the leading principal axes. Here we used 20 principal axes: K ¼ U 20 U > 20. 2. For each neuron i, regress its firing rate at each time point t on stimulus, decision, and interaction between them: Figure  14. Some demixed components as given by three different demixing methods (rows) in various datasets and marginalizations (columns). Empty subplots mean that the corresponding method did not find any components. All projections were z-scored to make them of the same scale.
If the individual vectors in B are far from orthogonal, then the resulting axes will strongly depend on the order of stacking them into the matrix B. We found that we obtain best results if we select this order ad hoc for each dataset; the orders we used were stimulus ! decision ! interaction for the somatosensory working memory task, stimulus ! interaction ! decision for the visuospatial working memory task, and interaction ! decision ! stimulus for both olfactory tasks. The vector b o ðtÞ was always used last. We applied TDR to all our datasets and observed that dPCA consistently outperforms it in terms of capturing variance and demixing task parameters. First, unlike dPCA, TDR yields only one component per task parameter. Second, even this component tends to retain more mixed selectivity than the corresponding dPCA component.
A statistical test known as MANOVA can be seen as another possible factorial generalization of LDA. Given the same data and covariance decomposition, MANOVA tests if the effect of f is statistically significant by analyzing eigenvalues of C À1 noise C f. The eigenvectors of this matrix can in principle serve as decoders, but these projections are optimized to separate the contribution of f from noise, not from the contributions of noise and other parameters. Hence, MANOVA is not the appropriate method for demixing purposes. While the toy example of Figure  2  illustrates that dPCA and LDA will in principle have very different solutions, we note that in all datasets considered here factorial LDA and dPCA yielded very similar components. This may reflect several pecularities of the data: for instance, the population activity for different values of the same parameter was spaced rather evenly, and all decisions were binary.
In other words, the prior is chosen such that those components are favored that have variance in only one marginalization. Under the constraint that decoding directions W are orthogonal, the model can be fit using the expectation-maximization algorithm. However, the probabilistic formulation of  Brendel et al. (2011)  still suffers from the orthogonality constraint. As explained in the Discussion, the orthogonality constraint is too rigid and can prevent successful demixing if parameter subspaces are sufficiently non-orthogonal. Indeed, we applied dPCA-2011 to all our datasets and observed that dPCA-2015 showed better demixing (Figure  14 ). Moreover, dPCA-2011 failed to find any decision components in the visuospatial working memory task. In addition, the formulation of dPCA in this manuscript is radically simplified compared to  Brendel et al.
Data and code The dPCA code is available at http://github. com/machenslab/dPCA for Matlab and Python. All four datasets used in this manuscript have been made available at http://crcns. org (Romo et al. , 2016; Constantinidis et al. , 2016; Feierstein et al. , 2016; Uchida et al. , 2016). Our preprocessing and the main analysis scripts (Matlab) are available at http://github. com/machenslab/elife2016dpca.
Deletions of B2M (20%) and CD58 (10%) were restricted to LR DLBCLs. Deletions and duplications of HLA related genes, were observed frequently in our cohort and are most likely related to HLA system polymorphism. Losses in IgHV and IgKV clusters are presumably a sign of BCR rearrangement and cannot be considered as recurrent lesion in tumor samples. We made the choice to consider all CNVs, including those that are described in databases of genomic variants. Limiting our analysis to acquired lesions may avoid constitutional abnormalities that are prone to foster lymphoma development. Third, dysregulation of transcription through altering events in genes encoding regulators of chromatin compaction and DNA replication were associated with ER DLBCLs. Duplications of the linker histone H1 gene and/or deletions of its regulatory partner PTMS  [48]  were found in 21 to 31% of cases.
Copy number analysis CNVs of the 39 samples were determined using the Affymetrix SNP 6. 0 platform (Affymetrix, Santa Clara, CA, USA), which interrogates 1 800 000 copy number probes distributed evenly along the human genome (mean interval: 1. 8 kilobases)  [55]. The 1 800 000 probes were annotated with home-made tools querying the Ensembl 77 th version with Human Genome HG37 MYSQL database available at ftp. ensembl. org/pub/release-77/mysql/. Total copy numbers were computed using the Bioconductor crlmm package  [56]. A sliding window (length = 11 probes) representing the signal median value was used to smooth artifactual data. This strategy allowed CNV discovery at high resolution and identification of genes frequently altered in relapsed DLBCLs. Duplications were defined by a copy number ≥2.
(viii) if FDR > 0. 05, decrease the threshold D and repeat steps 6 and 7. Then, the aim of this method is not to provide an individual p-value for each gene of the genome but to identify of a list of genes which CNVs distribution is differential between ER and LR, with systematic correction of multiple testing and less than 5% of false positives. SNP microarray data acquired on Affymetrix SNP 6. 0 platform can be accessed online on Gene Expression Omnibus (GSE73791).
CNV impact on gene expression through integrative genomics Gene expression profiles were available for 30 out of the 39 patients. These data were acquired on an Agilent Whole Human Genome microarray 4x44K platform (Agilent technologies, Santa Clara, CA, USA). Gene expression levels for each sample were integrated with CNVs. The impact of deletions and duplications of the selected genes on their own expression level was evaluated by comparing the average gene expression levels of deleted, duplicated and normal samples using Student's t-test.
The explosion of bioinformatics technologies in the form of next generation sequencing (NGS) has facilitated a massive influx of genomics data in the form of short reads. Short read mapping is therefore a fundamental component of next generation sequencing pipelines which routinely match these short reads against reference genomes for contig assembly. However, such techniques have seldom been applied to microbial marker gene sequencing studies, which have mostly relied on novel heuristic approaches. We propose NINJA Is Not Just Another OTU-Picking Solution (NINJA-OPS, or NINJA for short), a fast and highly accurate novel method enabling reference-based marker gene matching (picking Operational Taxonomic Units, or OTUs). NINJA takes advantage of the Burrows-Wheeler (BW) alignment using an artificial reference chromosome composed of concatenated reference sequences, the "concatesome," as the BW input.
On the other hand, in habitats with mostly well-characterized microbes, we have the option of matching the generated amplicon sequences to reference databases containing example marker genes from known taxa  [5]. A hybrid approach may also be used, where sequences are first compared to a reference database, with subsequent de novo clustering of those that failed to match. As the number of published culture-independent amplicon-based surveys of microbial communities continues to grow, our ability to rely on reference sequences also increases. However, although the crucial analysis step of mapping generated amplicons to reference marker genes has received much attention from the microbial bioinformatics field, with a variety of solutions proposed  [6] [7] [8] [9] [10] , there is much room for improvement in terms of speed, accuracy, memory footprint, and openness of code.
NINJA-OPS, our portable, opensource OTU picking pipeline, realizes these goals. Originally conceived as a means to make data more compressible, the Burrows-Wheeler transform (BWT)  [11]  is a lossless, reversible transformation that effectively positions series of like characters close to each other in a way that can easily be undone to recover the original data. It involves creating a circular suffix array, sorting the final column lexicographically, and storing that column as the transformed data for later compression. This algorithm also has the interesting property of enabling rapid substring search, with O(1) order of growth in finding exact string matches. As long as there is an efficient indexing scheme that stores the indices of the transformed bases into the original string, the BWT can be used for fast database substring search amounting to binary searching (or looking up via rank matrix) the transformed reference string representation and mapping back to the original, and has hence been employed in a number of commonly used DNA alignment tools  [11] [12] [13] [14].
Although these tools are approximate methods due to the high additional computational cost of performing optimal local or global alignment search when mismatches occur, they are generally fast and widely used in the genome-enabled research community (http://bowtie-bio. sourceforge. net/bowtie2/other_tools. shtml). Here we demonstrate that BWT-enabled DNA alignment can be effectively used for accurate and fast assignment of marker-gene sequences to a reference database. We present the NINJA-OPS pipeline utilizing several novel contributions to achieve an order of magnitude speedup and higher accuracy when compared to commonly used approaches (or up to two orders of magnitude when combined with denoising). To test the accuracy and efficacy of our approach, we perform closed-reference OTU-picking on a wide range of biological data sets from varied environments.
Accuracy was evaluated using an optimal aligner which produces a BLAST-style %ID for each query sequence against the reference sequence chosen by the OTU picking method. Speed was assessed as the elapsed time from parsing the correctly-formatted input FASTA file, which is accelerated by NINJA's fast C parser and simple format requirements, until the alignment (against a pre-generated database) has terminated. However, it may be useful to note that NINJA also significantly speeds up the subsequent steps of tallying reads, incorporating taxonomic annotations, and producing an OTU table in sparse BIOM 1. 0 format, as well as other steps prior to the alignment such as reverse complementing and trimming reads. Hence, the NINJA pipeline accelerates many stages of the OTU-picking pipeline in addition to the alignment step.
Pipeline overview The pipeline follows three stages: filtering, aligning, and parsing. After forming the concatenated reference string, called the "concatesome," from the individual references, NINJA applies a powerful filtering step which uses a 3-way radix quicksort on string pointers to rapidly de-duplicate millions of reads, construct a sample dictionary, and output a reducedsize filtered FASTA file and sample dictionary (Fig  1 ). The program implements this lossless filtering approach as well as a lossy variant, making use of singleton filtering as well as statistical profiling over the entire set of reads to exclude reads with a user-defined number of duplicates or rare segments (k-mers) appearing below a user-defined threshold of prevalence. The lossy filtering, which is not enabled by default, is intended to identify reads with probable read error independently, and speeds up the resulting alignment by excluding such reads from the BWT aligner.
This adds an additional speedup because BWT string matching spends a disproportionate amount of search effort to align erroneous or low-identity reads. Because choice of k (from 8 to 14) and prevalence threshold are highly domain-and dataset-specific, it is difficult to issue a general recommendation for this setting. Although we have found k = 8 to k = 14 at a threshold of 0. 05%-0. 01% to be a safe minimum for 16S data, user experimentation is recommended. The NINJA filter step also performs reverse-complementing and sequence length trimming at the same time as the other filtering steps. Because of this simultaneous multi-step filtering, no intermediate files are created prior to the alignment stage, and all filtering steps are performed rapidly in optimized C code on data structures already in memory.
This takes a fraction of the time used by other filtering pipelines which perform sequential operations often written in general-purpose scripting languages and generate numerous large intermediate files after each step. Using the base NINJA filter parameters, the entire filtering process itself takes approximately 10-20% of the time it takes to align the resulting filtered file when using all optional filtering steps. Next, the filtered reads are aligned against a reference database containing the (the concatesome) via any BWT-derived short read aligner such as BWA  [11] , Bowtie  [13] /Bowtie2  [12] , hpg-aligner  [15] , SOAP2  [16] -or, more broadly, any read aligner whatsoever capable of outputting to headerless SAM format  [17]  and suppressing unmatched input reads. Utilizing SAM is much faster than BAM (binary compressed SAM) after deduplication, as the alignment step is not I/O bound and the overhead of BAM's additional compression/decompression step can be significant.
We have chosen to standardize NINJA around Bowtie2 for our tests and publish the command line options for Bowtie2 as we have found it to be suitable for the purposes of BLAST-identity-based OTU picking. Following alignment, the resulting SAM file is fed to the NINJA parsing step, which takes in the sample dictionary metadata as well as an optional taxonomy map to rapidly re-assign each de-duplicated read to the biological sample(s) in which it originally occurred, add taxonomy annotation to each picked OTU, bin all reads by their matched OTU into a sample-by-OTU matrix (OTU table), and output the result in sparse BIOM 1. 0 format or a tab-delimited human-readable legacy QIIME format. This can also be incorporated into an open-reference OTU-picking pipeline.
Burrows-Wheeler transform The BWT has received a lot of attention in the alignment of short reads to a reference genome, and now enjoys routine use in clinical and other settings as a well-vetted technique for mapping short DNA reads to a longer reference, where it is known as Burrows-Wheeler alignment. The BWT is based on the principle that a long string of text can be reversibly transformed to reduce the complexity of substring queries to effectively two binary searches into the transformed representation of the original string, which is then converted back to indices into the original reference string with a short walk-back (the BW Last-First, or LF walk) or lookup. The  efficacy of this approach in matching short reads to a reference database of numerous short reference marker genes has remained largely unexplored  [1].
The concatesome is output as a new single-line FASTA file, and the original indices are output as an index table of OTU names and IDs. The concatesome serves as the reference for the BWT-enabled aligner to construct its database. For bowtie2, which we have used as the pipeline's BWT aligner for the subsequent analyses, this is accomplished with the bowtie-build command. K-mer-based denoising. NINJA's filter step implements a variant of 3-way radix quicksort  [20]  that achieves high binning and de-duplication performance on amplicon datasets using minimal RAM. This is due to its general high performance on strings combined with a median pivot scheme and increasing performance on data with larger numbers of duplicates, which is especially true for amplicon datasets. Trimming is implemented in C and is performed before deduplication during the parsing process by halting the read pointer and skipping to the next sequence, speeding up the parse and using less memory.
We use k = 8 by default (adjustable at compile-time with a practical maximum of 14), giving a counter array of length 2 16 , occupying up to 2 16 × 64 bits or 4 MB of RAM. The counter is incremented per k-mer per read to represent the number of that k-mer's duplicates in the dataset. The resulting count array after all sequences are parsed forms the empirical k-mer frequency distribution across all input reads. For amplicon data, the sorted frequency distribution resembles an exponential curve, with a small number of k-mers well represented and others with fewer occurrences (and many with none, depending on k-mer size). With sufficient number of input reads, the likelihood of observing a k-mer of a given rareness can be estimated from the area under the empirical frequency distribution. By checking the number of occurrences of a given k-mer against the empirical duplicate count at that probability threshold, we can determine if a given k-mer is an outlier and hence discard the read.
In several data sets we have found that this eliminates approximately 10% of the total sequences due to high likelihood that they contain errors. The resulting OTU assignments are generally unchanged when excluding singletons (S1 Fig ), and yet the runtime achieves an additional 3-10x speedup over the non-denoised approach. We recommend these settings (parameter "-d 2") for most users unless they have very low read counts. Alignment. We have conducted most of our testing using bowtie2 as the BWT-enabled aligner for NINJA. To perform OTU-picking in a manner consistent with current tools, a custom set of command-line parameters was used. We sought to maintain concordance with various USEARCH operating assumptions which have become the standard for marker gene sequencing (as used in the QIIME pipeline).
The primary criterion used for matching should be percent identity, which weights all matches in an alignment 1, and mismatches of any type 0. The sum of all such scores across the alignment, spanning the length of the query sequence, is divided by the total length of that alignment. Matches to ambiguous bases are also not penalized, nor are sequential gaps weighted any differently than single gaps (no affine penalty). Further, once a reasonable match has been identified in the database matching these criteria, the search terminates. The following options for bowtie2 target this behavior: --np 0 --mp }1; 1} --rdg }0; 1} --rfg }0; 1} --score À min }L; 0; À0:03} À k 1 --norc The percent ID is specified by setting the third argument to the minimum scoring function (-score-min) as %ID/100-1. In this case, to match with 97% ID, the parameter is set to 97/ 100-1 = -0.
03. It is recommended that users not modify other parameters of bowtie2 pertaining to scoring criteria or output format, to maintain compatibility with the %ID match criterion and downstream NINJA parsing. It is further recommended to use the included python wrapper so that all steps from reading the formatted FASTA to OTU table creation are performed automatically for the user with a single command. Additionally, for compatibility with NINJA and to save space, reads that fail to align are suppressed and no headers are printed to the output SAM file and the concatesome built with bowtie2-build is used as the reference database. The input sequences are the result of the filtering step. Full examples of the bowtie2 command are given in the online documentation. By default, we include presets for fast or very sensitive matching.
Very sensitive matching typically improves the quality of matches noticeably at the expense of a 3-6x longer running time. Parsing. The final step in NINJA-OPS combines parsing the alignments, assigning OTU identity and taxonomy, and tallying OTU counts in each sample as an output OTU table (in BIOM 1. 0 or legacy QIIME format). This is performed by NINJA-OPS using the information provided in sample dictionary generated by the filtering step, as well as the alignment data produced by the aligner and (optionally) a user-supplied table of OTU taxonomic assignments (which is also provided pre-compiled with NINJA-OPS). This parsing step is I/O bound and runs in under a second on our test computer and dataset. Comparison. To compare NINJA-OPS against USEARCH 8 (an earlier version of USEARCH is used in the QIIME pipeline  [1] ), we built the USEARCH database using the same multiple reference FASTA used to create the concatesome, the Greengenes 97% OTU representative sequence database.
Preparation step The runtime performance of the database generation is significantly longer than is practical to perform on the fly. This step only needs to be performed once for each reference database. Although ninja_prep performs the concatenation of references rapidly (it is I/O bound on the Macbook's SSD), the BWT program may spend a long time generating the BW index. For bow-tie2 on our test machine, this takes over half an hour (with a maximum of one thread) on the Greengenes 97% OTU representative sequence database. For this reason, it is best to store and use pre-compiled databases for all subsequent alignments, and NINJA-OPS is distributed with a number of pre-compiled databases for commonly performed 16S bacterial marker-gene OTU matching.
Alignment step Bowtie2 with the settings mentioned in methods aligns the entire test dataset of 1. 6 million 175bp reads in under 40 seconds on a single thread of the test laptop. Performance for the default and maximum-fidelity ("max") NINJA presets were measured (Fig  2 ). The "max" preset not only demonstrates higher accuracy than either the default preset or USEARCH, but also retains significantly more reads. The "fast" preset displays similar accuracy characteristics to "default," but misses about 2% of the alignments detected by the latter, usually of the lowest identity. Total pipeline runtime on the same dataset decreases to less than 10 seconds when using the recommended singleton-based denoising option (parameter "D 2"), in combination with default preset. In addition, the speedup versus USEARCH 8 was measured using the default preset without denoising ("D 0") across different datasets (S2 Fig).
Accuracy To assess the accuracy of the alignments found by NINJA, and to compare them to existing tools, we calculated the optimal alignment, using a semi-global version of the Smith-Waterman algorithm, of each query sequence with the reference sequence assigned by a given tool. We found that NINJA (default preset) generally finds higher-accuracy matches than USEARCH 8 (Mann-Whitney U test p < 2. 2e-16) (Figs  3  and 4 ). In a published dataset containing healthy subjects and patients with Crohn's disease the two methods produced the same list of differentiated genera across disease conditions with occasional disagreements about the direction of.