dataesr/scientific-paragraphs-categorization huggingface dataset:
Approach We developed two R packages, bigstatsr and bigsnpr, that integrate the most efficient algorithms for the pre-processing and analysis of large-scale genomic data while using memory-mapping. Package bigstatsr implements many statistical tools for several types of FBMs (unsigned char, unsigned short, integer and double). This includes implementation of multivariate sparse linear models, Principal Component Analysis, matrix operations, and numerical summaries. The statistical tools developed in bigstatsr can be used for other types of data as long as they can be represented as matrices. Package bigsnpr depends on bigstatsr, using a special type of FBM object to store the genotypes, called "FBM. code256". Package bigsnpr implements algorithms which are specific to the analysis of SNP arrays, such as calls to external software for processing steps, I/O (Input/Output) operations from binary PLINK files, and data analysis operations on SNP data (thinning, testing, plotting).
Memory-mapped files The two R packages don't use standard read operations on a file nor load the genotype matrix entirely in memory. They use an hybrid solution: memory-mapping. Memory-mapping is used to access data, possibly stored on disk, as if it were in memory. This solution is made available within R through the BH package, providing access to Boost C++ Header Files 1. We are aware of the software library SNPFile that uses memory-mapped files to store and efficiently access genotype data, coded in C++  (Nielsen and Mailund 2008)  and of the R package BEDMatrix 2  which provides memory-mapping directly for binary PLINK files. With the two packages we developed, we made this solution available in R and in C++ via package Rcpp  (Eddelbuettel and François 2011). The major advantage of manipulating genotype data within R, almost as it were a standard matrix in memory, is the possibility of using most of the other tools that have been developed in R (R Core Team 2017).
For example, we provide sparse multivariate linear models and an efficient algorithm for Principal Component Analysis (PCA) based on adaptations from R packages biglasso and RSpectra  (Qiu and Mei 2016; Zeng and Breheny 2017). Memory-mapping provides transparent and faster access than standard read/write operations. When an element is needed, a small chunk of the genotype matrix, containing this element, is accessed in memory. When the system needs more memory, some chunks of the matrix are freed from the memory in order to make space for others. All this is managed by the Operating System so that it is seamless and efficient. It means that if the same chunks of data are used repeatedly, it will be very fast the second time they are accessed, the third time and so on. Of course, if the memory size of the computer is larger than the size of the dataset, the file could fit entirely in memory and every second access would be fast.
Data management, preprocessing and imputation Package bigsnpr currently takes as input a variety of formats (e. g. vcf, bed/bim/fam, ped/map). However, it uses PLINK for conversion to bed/bim/fam format and for Quality Control (QC) of the data, so that we first provide R functions that use system calls to PLINK for the conversion and QC steps (Figure  1 ). Then, fast read/write operations from/to bed/bim/fam PLINK files are implemented. PLINK files are then read into a "bigSNP" object, which contains the genotype Filebacked Big Matrix (FBM), a data frame with information on samples and another data frame with information on SNPs. We also provide another function which could be used to read from tabular-like text files in order to create a genotype in the format "FBM". We developed a special FBM object, called "FBM.
Population structure and SNP thinning based on Linkage Disequilibrium For computing Principal Components (PCs) of a large-scale genotype matrix, we provide several functions related to SNP thinning and two functions, for computing a partial Singular Value Decomposition (SVD), one based on eigenvalue decomposition, big_SVD, and the other on randomized projections, big_randomSVD (Figure  3 ). While the function based on eigenvalue decomposition is at least quadratic in the smallest dimension, the function based on randomized projections runs in linear time in all dimensions  (Lehoucq and Sorensen 1996). Package bigstatsr use the same PCA algorithm as FlashPCA2 called Implicitly Restarted Arnoldi Method (IRAM), which is implemented in R package RSpectra. The main difference between the two implementations is that FlashPCA2 computes vector-matrix multiplications with the genotype matrix based on the binary PLINK file whereas bigstatsr computes these multiplications based on the FBM format, which enables parallel computations and easier subsetting.
The second approach doesn't use univariate summary statistics but instead train a multivariate model on all the SNPs and covariables at once, optimally accounting for correlation between predictors  (Abraham et al. 2012). The currently available models are linear and logistic regressions and Support Vector Machine (SVM). These models include lasso and elastic-net regularizations, which reduce the number of predictors (SNPs) included in the predictive models  (Friedman et al. 2010; Tibshirani 1996; Zou and Hastie 2005). Package bigstatsr provides a fast implementation of these models by using efficient rules to discard most of the predictors  (Tibshirani et al. 2012). The implementation of these algorithms is based on modified versions of functions available in the R packages sparseSVM and biglasso  (Zeng and Breheny 2017).
Overview We present the results for three different analyses. First, we illustrate the application of R packages bigstatsr and bigsnpr. Secondly, we compare the performance of the R packages to the performance obtained with PLINK and FastPCA (EIGENSOFT). Thirdly, we present results of the two new methods implemented in these packages, one method for the automatic detection and removal of long-range LD regions in PCA and another for the imputation of missing genotypes. We use three types of data: a case-control cohort for the celiac disease, the European population cohort POPRES and simulated datasets using real genotypes from the Celiac cohort. We compare performances on two computers, a desktop computer with 64GB of RAM and 12 cores (6 physical cores), and a laptop with only 8GB of RAM and 4 cores (2 physical cores).
As illustrated in the supplementary data, the whole pipeline is userfriendly and requires only 20 lines of R code. To illustrate the scalability of the two R packages, we performed a GWAS analysis on 500K individuals and 1M SNPs. The GWAS analysis completed in approximately 11 hours using the aforementioned desktop computer. The GWAS analysis was composed of four main steps. First we read from PLINK files in our format "bigSNP" in 1 hour. Then, we removed SNPs in long-range LD regions and used SNP clumping, leaving 93,083 SNPs in 5. 4h. Then, the 10 first PCs were computed on the 500K individuals and these remaining SNPs in 1. 8h. Finally, we performed a linear association test on the complete 500K dataset for each of the 1M SNPs, using the 10 first PCs as covariables in 2. 9h.
Method Comparison We first compared the GWAS and PRS computations obtained with the R packages to the ones obtained with PLINK 1. 9 and EIGENSOFT 6. 1. 4. For most functions, multithreading is not available yet in PLINK, nevertheless, PLINK-specific algorithms that use bitwise parallelism (e. g. pruning) are still faster than the parallel algorithms reimplemented in package bigsnpr (Table  1 ). Overall, the computations with our two R packages for an association study and a polygenic risk score are of the same order of magnitude as when using PLINK and EIGEN-SOFT (Tables  1  and 2 ). However, the whole analysis pipeline makes use of R calls only; there is no need to write temporary files and functions have parameters which enable subsetting of the genotype matrix without having to copy it.
On our desktop computer, we compared the computation times of FastPCA, FlashPCA2 to the similar function big_randomSVD implemented in bigstatsr. For each comparison, we used the 93,083 SNPs which were remaining after pruning and we computed 10 PCs. We used the datasets of growing size simulated from the Celiac dataset. Overall, our function big_randomSVD showed to be almost twice as fast as FastPCA and FlashPCA2 and 8 times as fast when using parallelism (an option not currently possible with either FastPCA or Flash-PCA2) with 6 cores (Figure  6 ). We also compared results in terms of precision by comparing squared correlation between approximated PCs and "true" PCs provided by an exact singular value decomposition obtained with SmartPCA. FastPCA, FlashPCA2 and bigstatsr infer the true first 6 PCs but the squared correlation between true PCs and approximated ones decreases for larger PCs when using FastPCA (Fast mode of EIGENSOFT) whereas it remains larger than 0.
Automatic detection of long-range LD regions For the detection of long-range LD regions during the computation of PCA, we tested the function snp_autoSVD on both the Celiac and POPRES datasets. For the POPRES dataset, the algorithm converged in two iterations. The first iterations found 3 long-range LD regions in chromosomes 2, 6 and 8 (Table  S1 ). We compared the PCs of genotypes obtained after applying snp_autoSVD with the PCs obtained after removing pre-determined long-range LD regions 5  and found a mean correlation of 89. 6% between PCs, mainly due to a rotation of PC7 and PC8 (Table  S2 ). For the Celiac dataset, we found 5 long-range LD regions (Table  S3 ) and a mean correlation of 98. 6% between PCs obtained with snp_autoSVD and the ones obtained by clumping with removing of predetermined long-range LD regions (Table  S4 ).
For the Celiac dataset, we further compared results of PCA obtained when using snp_autoSVD and when computing PCA without removing any long range LD region (only clumping at R 2 > 0. 2). When not removing any long range LD region, we show that PC4 and PC5 don't capture population structure and correspond to a long-range LD region in chromosome 8 (Figures  S1  and S2 ). When automatically removing some long-range LD regions with snp_autoSVD, we show that PC4 and PC5 reflect population structure (Figure  S1 ). Moreover, loadings are more equally distributed among SNPs after removal of long-range LD regions (Figure  S2 ). This is confirmed by Gini coefficients (measure of dispersion) of each squared loadings that are significantly smaller when computing SVD with snp_autoSVD than when no long-range LD region is removed (Figure  S3 ).
Imputation of missing values for genotyped SNPs For the imputation method based on XGBoost, we compared the imputation accuracy and computation times with Beagle on the POPRES dataset. The histogram of the minor allele frequencies (MAFs) of this dataset is provided in figure  S4  and there is no missing value. We used a Beta-binomial distribution to simulate the number of missing values by SNP and then randomly introduced missing values according to these numbers, resulting in approximately 3% of missing values overall (Figure  S5 ). Imputation was compared between function snp_fastImpute of package bigsnpr and Beagle 4. 1 (version of January 21, 2017). Overall, snp_fastImpute made 4. 7% of imputation errors whereas Beagle made only 3. 1% of errors but it took Beagle 14. 6 hours to complete while our method only took 42 minutes (20 times less).
Discussion We have developed two R packages, bigstatsr and bigsnpr, which enable multiple analyses of large-scale genotype datasets in a single comprehensive framework. Linkage Disequilibrium pruning, Principal Component Analysis, association tests and computations of polygenic risk scores are made available in this software. Implemented algorithms are both fast and memoryefficient, allowing the use of laptops or desktop computers to make genome-wide analyses. Technically, bigstatsr and bigsnpr could handle any size of datasets. However, if the OS has to often swap between the file and the memory for accessing the data, this would slow down data analysis. For example, the Principal Component Analysis (PCA) algorithm in bigstatsr is iterative so that the matrix has to be sequentially accessed over a hundred times.
If the number of samples times the number of SNPs remaining after pruning is larger than the available memory, this slowdown would happen. For instance, a 32GB computer would be slow when computing PCs on more than 100K samples and 300K SNPs remaining after LD thinning. The two R packages use a matrix-like format, which makes it easy to develop new functions in order to experiment and develop new ideas. Integration in R makes it possible to take advantage of the vast and diverse R libraries. For example, we developed a fast and accurate imputation algorithm for genotyped SNPs using the widely-used machine learning algorithm XGBoost available in the R package xgboost. Other functions, not presented here, are also available and all the functions available within the package bigstatsr are not specific to SNP arrays, so that they could be used for other omic data or in other fields of research.
Dubois et al. 2010; Nelson et al. 2008). The Celiac dataset is composed of 15,283 individuals of European ancestry genotyped on 295,453 SNPs. The POPRES dataset is composed of 1385 individuals of European ancestry genotyped on 447,245 SNPs. For computation times comparison, we replicated individuals in the Celiac dataset 5 and 10 times in order to increase sample size while keeping the same population structure and pattern of Linkage Disequilibrium as the original dataset. To assess scalibility of our algorithms for a biobank-scale genotype dataset, we formed another dataset of 500,000 individuals and 1 million SNPs, also through replication of the Celiac dataset.
(Purcell et al. 2007)in each pair of individuals with a proportion of alleles shared IBD greater than 0. 08(Purcell et al. 2007). For the POPRES dataset, this resulted in 1382 individuals and 344,614 SNPs with no missing value. For the Celiac dataset, this resulted in 15,155 individuals and 281,122 SNPs with an overall genotyping rate of 99. 96%. The 0. 04%.
( Dubois et al. 2010; Nelson et al. 2008). The Celiac dataset is composed of 15,283 individuals of European ancestry genotyped on 295,453 SNPs. The POPRES dataset is composed of 1385 individuals of European ancestry genotyped on 447,245 SNPs. For computation times comparison, we replicated individuals in the Celiac dataset 5 and 10 times in order to increase sample size while keeping the same population structure and pattern of Linkage Disequilibrium as the original dataset. To assess scalibility of our algorithms for a biobank-scale genotype dataset, we formed another dataset of 500,000 individuals and 1 million SNPs, also through replication of the Celiac dataset.
(Purcell et al. 2007)in each pair of individuals with a proportion of alleles shared IBD greater than 0. 08(Purcell et al. 2007). For the POPRES dataset, this resulted in 1382 individuals and 344,614 SNPs with no missing value. For the Celiac dataset, this resulted in 15,155 individuals and 281,122 SNPs with an overall genotyping rate of 99. 96%. The 0. 04% , removing individuals and SNPs which had more than 5% of missing values, non-autosomal SNPs, SNPs with a minor allele frequency lower than 0. 05 or a p-value for the Hardy-Weinberg exact test lower than 10 -10 , and finally, removing.
Introduction The development of sequencing technologies has enabled to target the genome of complex non-model organisms and communities of organisms. Some of these non-model organisms can be challenging to isolate from their environment or cannot be cloned in vitro. They might alternatively be compulsorily associated with cognate commensal or parasitic organisms, or even embedded within a host. Consequently, genome assembly datasets sometimes include DNA from unexpected sources like mixture of untargeted species, but may also contain organelles or even laboratory contaminants. The presence of additional untargeted species was indeed reported in several recent genome assemblies, for instance in the draft assembly of domestic cow  (Merchant et al. , 2014) , in several isolates of the phytopathogenic fungi Magnaporthe oryzae  (Chiapello et al.
, 2015)  or recently in the tardigrade genome  (Delmont and Eren, 2016). Such mixed assemblies may produce several biases and problems in downstream bioinformatics analyses and raise the need for tools able to deal with mixed-organism DNA assemblies. Several tools were recently designed or used to detect and filter untargeted organisms from sequence datasets. A first type of approach, used by khmer software  (Crusoe et al. , 2015) , is to compute k-mer frequencies on short reads to pre-process and filter read datasets prior to de novo sequence assembly. Other packages like Blobtools  (Kumar et al. , 2013)  and Anvi'o  (Eren et al. , 2015)  combine sequence properties (GC content, oligonucleotide profile) and additional information such as depth of coverage, similarity to public databases and reference gene sets to identify untargeted species using both raw reads and assembled contigs of a genomic dataset.
Finally, a last type of approach is to use software dedicated to metagenomic species read binning, such as CONCOCT  (Alneberg et al. , 2014)  that use sequence composition and coverage across multiple samples to automatically cluster contigs into genomes or Kraken  (Wood and Salzberg, 2014 ) that relies on exact alignment of k-mers to a k-mer reference database to assign taxonomic labels to metagenomic DNA. Here we present PhylOligo, a toolset designed to explore, segment and subtract untargeted material from assembled sequences using an ab initio alignment-free approach relying only on the intrinsic oligonucleotide signature of an assembled genomic dataset. Compared to existing software, PhylOligo provides several features to explore assemblies, including: (i) a customizable oligonucleotide pattern, including continuous and spaced pattern k-mers  (B rinda et al.
Our strategy present several advantages. (i) Unlike approaches that process short read datasets prior to the de novo sequence assembly and use sequence homology information, PhylOligo allows the identification of potentially uncharacterised and distantly related sequences in already assembled genomic datasets, the handling of any type of genome assembly, shunning the dependency on the availability of raw sequencing reads data, additional data and patchiness of knowledge in databases; (ii) The double-threshold species-specific filtration prevents the removal of HGTs and the subsequent fragmentation of the assembly; (iii) Learning the compositional profile on longer and assembled sequences such as contigs compared to unassembled reads, allows for a refined oligonucleotide profile, unbiased from heterogeneous sequencing depth along the sequence.
Assembly exploration PhylOligo allows for a visual exploration of the compositional similarity distribution and structure of the contigs in an assembly based on either continuous (k-mers) or spaced-pattern oligonucleotide frequencies. The oligonucleotide profile of each contig is computed and a pairwise distance matrix based on metrics including Euclidean or Jensen-Shannon is produced (Fig. 1A ) to generate an interactive Neighbour-Joining tree. Branch width is drawn proportional to the cumulated length of the contigs in a clade, allowing the user to track where the main part of the assembly clusters (assumed to correspond to the targeted organisms) and what significant clades branch out as hint for separate organisms (see Fig. 1B ). Thanks to the Ape package  (Paradis et al. , 2004) , sequences from a clade are interactively selected on the tree and exported to fasta files to learn a prototype of their oligonucleotide profile.
Prototype learning ContaLocate then allows the learning of oligonucleotide profiles from the main and presumed additional organisms identified by user-selected subsets from the previous step. These subsets must cumulate at least 50 Kb in order to generate an accurate prototype sufficiently representative of an organism. Learning subsets can be generated or complemented from public sequences, specialised databases  (Menigaud et al. , 2012)  or other tools  (Alneberg et al. , 2014; Eren et al. , 2015; Kumar et al. , 2013).
Synthetic datasets We evaluated the performances of PhylOligo by generating artificial contaminations on 32 contig datasets generated by GRINDER  (Angly et al. , 2012)  from real Refseq genome data (see section 6. 1 of Supplementary Material for detailed protocol). The species were chosen to cover the main domain of life (archea, bacteria, fungi, protozoa and vertebrate) and different degrees of genome complexity, content, length and composition. We benchmarked the automatic version of PhylOligo that uses the unsupervised HDBSCAN clustering and evaluated performances by using three indicators: (i) the cluster specificity i. e. the maximum fraction of contaminant in a cluster, (ii) the cluster sensitivity, i. e. the fraction of the whole contaminant aggregated in the cluster and (iii) an hybrid score, which indicated the best computed value of the product of cluster specificity and sensitivity.
Real datasets PhylOligo has been successfully applied to identify untargeted large bacterial regions in four out of nine fungal genomic datasets of Magnaporthe  (Chiapello et al. , 2015). GOHTAM  (Menigaud et al. , 2012)  taxonomical assignment of these additional regions confirmed their homogeneity and origin from Burkholderiales. Targeted Blast comparisons indicated that some of these supplementary regions were almost identical to Burkholderia fungorum sequences (100% identity for 16S, recA and gyrB genes) suggesting an origin or relatedness to one or several bacterial isolate(s) of this species. PhylOligo was applied to filter genome assemblies, validated with BUSCO  (Simão et al. , 2015)  and DOGMA  (Dohmen et al. , 2016)  (see Supplementary Material) and allowed to continue further bioinformatics analyses without rebuilding the costly initial genome assembly and annotation processes.
Acknowledgments This work was funded by the  German Ministry for Education and Science (BMBF) , the  German Research Foundation (DFG)  via the  DFG collaborative research centre SFB/TRR 173 SPIN+X , the  Graduate School Material Science in Mainz  ( DFG/GSC 266 ),  EU   7th Framework Programme  ( WALL FP7-PEOPLE-2013-ITN 608031 ; MAGWIRE,  FP7-ICT-2009-5 ), the  European Research Council  ( MultiRev 665672 ) and the  Research Center of Innovative and Emerging Materials at Johannes Gutenberg University (CINEMA). A Pfeiffer  and  M Kläui  thank the participants of SpinMechanics 2 Workshop for valuable discussions and are grateful for financial support from the  German Academic Exchange Service (DAAD)  via the  SpinNet Program  ( 56268455 ). The device were fabricated in the  Plateforme Technologie Amont in Grenoble , for which a support from the  French RENATECH  network is acknowledged.
Griaule et ses collègues en fonction de leurs présupposés théoriques, de leurs méthodes et de leurs objectifs de recherche. […]  Il ne faut donc pas se tromper d'objectif: ces archives ethnographiques -qui appartiennent aux ethnologues qui les ont produites -n'ont pas à être "rendues" au Mali, mais d'un point de vue scientifique, il serait souhaitable qu'elles soient davantage accessibles aux chercheurs maliens  (Jolly 2001 (Jolly -2002: 24, 27: 24, 27 ). 10   The notion of "reusability" can be beneficial in this aspect. The concept of reusability is developed from the perspective of evaluating "how to implement" both the opening of data and the results of the research by utilizing electronic databases. Barwick and Thieberger (2005: 141). write that the concept of reuse derives from the fields of ecology and Introduction computer programming (reduce, reuse, recycle), and that it invites researchers "to work with field recordings in a way that allows their further use as archival objects", i.