dataesr/scientific-paragraphs-categorization huggingface dataset:
3 The output of the overlap-add process is visualised in Figure  10. Lastly, in order to verify this method on another dataset, a similar network was trained on the vowels dataset, adjusted to have the right size of input layer of 800, see Figure  9. The inferred space reflects the condition number (discrete, here), but the remaining parameter uniformly covers the range  [-1, 1]. In this case the extra variance beyond the conditional parameter consists only of small tonal changes in the recorded voice as well as microphone noise, and thus there is much more variance between vowels than within. It is apparent from Fig. 9 (c) that there is no "leakage" of these extra sources of variance to the vowel label parameter axis, as the vowel number adequately identifies the cluster. The clusters can therefore be nicely mapped to a desired space automatically by the conditioning and regularization.
Conclusions These experiments showed some modest success in copying the parameter-data relationship of a physical modeling synthesizer and fitting them into a desired configuration. Like many machine learning approaches, the quality of results depends strongly on the hyperparameters used: network size and architecture, learning rates, regularization weights, etc. , and these must be adapted to the dataset. Shown here are results from the best parameters found after some combination of automatic and manual optimisation on this specific dataset, which we use to demonstrate some principles of the design, however it should be noted that actual results varied sometimes unexpectedly with small changes to these parameters. This hyperparameter optimization is non-trivial, especially when it comes to audio where mean squared error may not reveal much about the perceptual quality of the results, and so a lot of trial and error is the game.
Nevertheless we have attempted to outline some potential for use of autoencoders and their latent spaces for audio analysis and synthesis based on a specific signal source. Only a very simple fully-connected single-layer architecture was used, and thus improvements should be explored, in particular the addition of convolutional layers. The advantages of time-and frequency-domain representations as learning targets should be characterised. More important than the quality of these specific results, we wish to point out the modular approach that autoencoders enable in modeling oscillator periods of known and unknown parameters, and that, in contrast to larger datasets covering many instruments (ENGEL, 2017), interesting insights and useful performance systems can be generated even from small data.
Motivation: Genome-wide datasets produced for association studies have dramatically increased in size over the past few years, with modern datasets commonly including millions of variants measured in dozens of thousands of individuals. This increase in data size is a major challenge severely slowing down genomic analyses, leading to some software becoming obsolete and researchers having limited access to diverse analysis tools. Results: Here we present two R packages, bigstatsr and bigsnpr, allowing for the analysis of large scale genomic data to be performed within R. To address large data size, the packages use memory-mapping for accessing data matrices stored on disk instead of in RAM. To perform data pre-processing and data analysis, the packages integrate most of the tools that are commonly used, either through transparent system calls to existing software, or through updated or improved implementation of existing methods.
In particular, the packages implement fast and accurate computations of principal component analysis and association studies, functions to remove single nucleotide polymorphisms in linkage disequilibrium and algorithms to learn polygenic risk scores on millions of single nucleotide polymorphisms. We illustrate applications of the two R packages by analyzing a case-control genomic dataset for celiac disease, performing an association study and computing polygenic risk scores. Finally, we demonstrate the scalability of the R packages by analyzing a simulated genome-wide dataset including 500 000 individuals and 1 million markers on a single desktop computer.
Introduction Genome-wide datasets produced for association studies have dramatically increased in size over the past few years, with modern datasets commonly including millions of variants measured in dozens of thousands of individuals. As a consequence, most existing software and algorithms have to be continuously optimized in order to avoid obsolescence. For computing principal component analysis (PCA), commonly performed to account for population stratification in association, a fast mode named FastPCA has been added to the software EIGENSOFT, and FlashPCA has been replaced by FlashPCA2  (Abraham and Inouye, 2014; Abraham et al. , 2016; Galinsky et al. , 2016; Price et al. , 2006). PLINK 1. 07, which has been a central tool in the analysis of genotype data, has been replaced by PLINK 1.
9 to speed-up computations, and there is also an alpha version of PLINK 2. 0 that will handle more data types  (Chang et al. , 2015; Purcell et al. , 2007). Increasing size of genetic datasets is a source of major computational challenges and many analytical tools would be restricted by the amount of memory (RAM) available on computers. This is particularly a burden for commonly used analysis languages such as R. For analyzing genotype datasets in R, a range of software are available, including for example the popular R packages GenABEL, SNPRelate and GWASTools  (Aulchenko et al. , 2007; Gogarten et al. , 2012; Zheng et al. , 2012b). Solving memory issues for languages such as R would give access to a broad range of already implemented tools for data analysis. Fortunately, strategies have been developed to avoid loading large datasets in RAM.
Approach In order to perform analyses of large-scale genomic data in R, we developed two R packages, bigstatsr and bigsnpr, that provide a widerange of building blocks which are parts of standard analyses. R is a programming language that makes it easy to tie together existing or new functions to be used as part of large, interactive and reproducible analyses  (R Core Team, 2017). We provide a similar format as filebacked 'big. matrix' objects that we called 'Filebacked Big Matrices (FBMs)'. Thanks to this matrix-like format, algorithms in R/Cþþ can be developed or adapted for large genotype data. This data format is a particularly good trade-off between easiness of use and computation efficiency, making our code both simple and fast. Package bigstatsr implements many statistical tools for several types of FBMs (unsigned char, unsigned short, integer and double).
This includes implementation of multivariate sparse linear models, PCA, association tests, matrix operations and numerical summaries. The statistical tools developed in bigstatsr can be used for other types of data as long as they can be represented as matrices. Package bigsnpr depends on bigstatsr, using a special type of filebacked big matrix (FBM) object to store the genotypes, called 'FBM. code256'. Package bigsnpr implements algorithms which are specific to the analysis of single nucleotide polymorphism (SNP) arrays, such as calls to external software for processing steps, Input/ Output (I/O) operations from binary PLINK files and data analysis operations on SNP data (thinning, testing, predicting and plotting). We use both a real case-control genomic dataset for celiac disease and large-scale simulated data to illustrate application of the two R packages, including two association studies and the computation of polygenic risk scores (PRS).
Memory-mapped files The two R packages do not use standard read operations on a file nor load the genotype matrix entirely in memory. They use a hybrid solution: memory-mapping. Memory-mapping is used to access data, possibly stored on disk, as if it were in memory. This solution is made available within R through the BH package, providing access to Boost Cþþ Header Files (http://www. boost. org/). We are aware of the software library SNPFile that uses memorymapped files to store and efficiently access genotype data, coded in Cþþ  (Nielsen and Mailund, 2008)  and of the R package BEDMatrix (https://github. com/QuantGen/BEDMatrix) which provides memory-mapping directly for binary PLINK files. With the two packages we developed, we made this solution available in R and in Cþþ via package Rcpp  (Eddelbuettel and Franc¸ois, 2011).
Data management, pre-processing and imputation We developed a special FBM object, called 'FBM. code256', that can be used to seamlessly store up to 256 arbitrary different values, while having a relatively efficient storage. Indeed, each element is stored in one byte which requires eight times less disk storage than double-precision numbers but four times more space than the binary PLINK format '. bed' which can store only genotype calls. With these 256 values, the matrix can store genotype calls and missing values (four values), best guess genotypes (three values) and genotype dosages (likelihoods) rounded to two decimal places (201 values). So, we use a single data format that can store both genotype calls and dosages. For pre-processing steps, PLINK is a widely-used software. For the sake of reproducibility, one could use PLINK directly from R via systems calls.
We therefore provide wrappers as R functions that use system calls to PLINK for conversion and quality control and a variety of formats can be used as input (e. g. vcf, bed/bim/fam, ped/map) and bed/bim/fam files as output (Supplementary Fig. S1 ). Package bigsnpr provides fast conversions between bed/bim/fam PLINK files and the 'bigSNP' object, which contains the genotype FBM (FBM. code256), a data frame with information on samples and another data frame with information on SNPs. We also provide another function which could be used to read from tabular-like text files in order to create a genotype in the format 'FBM'. Finally, we provide two methods for converting dosage data to the format 'bigSNP' (Supplementary notebook 'dosage'). Most modern SNP chips provide genotype data with large callrates.
Population structure and SNP thinning based on linkage disequilibrium For computing principal components (PCs) of a large-scale genotype matrix, we provide several functions related to SNP thinning and two functions for computing a partial singular value decomposition (SVD), one based on eigenvalue decomposition and the other one based on randomized projections, respectively named big_SVD and big_randomSVD (Fig. 1 ). While the function based on eigenvalue decomposition is at least quadratic in the smallest dimension, the function based on randomized projections runs in linear time in all dimensions  (Lehoucq and Sorensen, 1996). Package bigstatsr uses the same PCA algorithm as FlashPCA2 called implicitly restarted Arnoldi method (IRAM), which is implemented in R package RSpectra. The main difference between the two implementations is that FlashPCA2 computes vector-matrix multiplications with the genotype matrix based on the binary PLINK file whereas bigstatsr computes these multiplications based on the FBM format, which enables parallel computations and easier subsetting.
For PCA, the thinning procedure should remain unsupervised (no phenotype must be used) and we therefore propose to use the minor allele frequency (MAF) as the statistic of importance. This choice is consistent with the pruning algorithm of PLINK; when two nearby SNPs are correlated, PLINK keeps only the one with the highest MAF. Yet, in some worst-case scenario, the pruning algorithm can leave regions of the genome without any representative SNP at all (Supplementary notebook 'pruning-vs-clumping'). So, we suggest to use clumping instead of pruning, using the MAF as the statistic of importance, which is the default in function snp_clumping of package bigsnpr. In practice, for the three datasets we considered, the clumping algorithm with the MAF provides similar sets of SNPs as when using the pruning algorithm (results not shown).
The third approach, which is generally combined with pruning, consists of removing SNPs in long-range LD regions  (Price et al. , 2008). Long-range LD regions for the human genome are available as an online table (https://goo. gl/8TngVE) that package bigsnpr can use to discard SNPs in these regions before computing PCs. However, the pattern of LD might be population specific, so we developed an iterative algorithm that automatically detects these long-range LD regions and removes them. This algorithm consists in the following steps: first, PCA is performed using a subset of SNP remaining after clumping (with MAFs), then outliers SNPs are detected using the robust Mahalanobis distance as implemented in method pcadapt  (Luu et al. , 2017). Finally, the algorithm considers that consecutive outlier SNPs are in long-range LD regions.
, 2013; Dudbridge, 2013; Euesden et al. , 2015). On the contrary, the second approach does not use univariate summary statistics but instead train a multivariate model on all the SNPs and covariables at once, optimally accounting for correlation between predictors  (Abraham et al. , 2012). The currently available models are very fast sparse linear and logistic regressions. These models include lasso and elastic-net regularizations, which reduce the number of predictors (SNPs) included in the predictive models  (Friedman et al. , 2010; Tibshirani, 1996; Zou and Hastie, 2005). Package bigstatsr provides a fast implementation of these models by using efficient rules to discard most of the predictors  (Tibshirani et al. , 2012). The implementation of these algorithms is based on modified versions of functions available in the R package biglasso  (Zeng and Breheny, 2017).
Data analyzed In this paper, two datasets are analyzed: the celiac disease cohort and POPRES  (Dubois et al. , 2010; Nelson et al. , 2008). The celiac dataset is composed of 15 283 individuals of European ancestry genotyped on 295 453 SNPs. The POPRES dataset is composed of 1385 individuals of European ancestry genotyped on 447 245 SNPs. For computation time comparisons, we replicated individuals in the celiac dataset 5 and 10 times in order to increase sample size while keeping the same eigen decomposition (up to a constant) and pairwise SNP correlations as the original dataset. To assess scalability of the packages for a biobank-scale genotype dataset, we formed another dataset of 500 000 individuals and 1 million SNPs, also through replication of the celiac dataset.
Overview We present the results of four different analyses. First, we illustrate the application of R packages bigstatsr and bigsnpr. Second, by performing two GWAS, we compare the performance of bigstatsr and bigsnpr to the performance obtained with FastPCA (EIGENSOFT 6. 1. 4) and PLINK 1. 9, and also two R packages SNPRelate and GWASTools  (Chang et al. , 2015; Galinsky et al. , 2016; Gogarten et al. , 2012; Zheng et al. , 2012b). PCA is a computationally intensive step of the GWAS, so that we further compare PCA methods on larger datasets. Third, by performing a PRS analysis with summary statistics, we compare the performance of bigstatsr and bigsnpr to the performance obtained with PRSice-2  (Euesden et al. , 2015). Finally, we present results of the two new methods implemented in bigsnpr, one method for the automatic detection and removal of long-range LD regions in PCA and another for the in-sample imputation of missing genotypes (i.
Application The data were pre-processed following steps from Supplementary Figure  S1 , removing individuals and SNPs with more than 5% of missing values, non-autosomal SNPs, SNPs with a MAF lower than 0. 05 or a P-value for the Hardy-Weinberg exact test lower than 10 À10 , and finally, removing the first individual in each pair of individuals with a proportion of alleles shared IBD >0. 08  (Purcell et al. , 2007). For the POPRES dataset, this resulted in 1382 individuals and 344 614 SNPs with no missing value. For the celiac dataset, this resulted in 15 155 individuals and 281 122 SNPs with an overall genotyping rate of 99. 96%. The 0. 04% missing genotype values were imputed with the XGBoost method. If we would have used a standard R matrix to store the genotypes, this data would have required 32 GB of memory.
On the disk, the '. bed' file requires 1 GB and the '. bk' file (storing the FBM) requires 4 GB. We used bigstatsr and bigsnpr R functions to compute the first PCs of the celiac genotype matrix and to visualize them (Fig. 2 ). We then performed a GWAS investigating how SNPs are associated with celiac disease, while adjusting for PCs, and plotted the results as a Manhattan plot (Fig. 3 ). As illustrated in the Supplementary data, the whole pipeline is user-friendly, requires only 20 lines of R code and there is no need to write temporary files or objects because functions of packages bigstatsr and bigsnpr have parameters which enable subsetting of the genotype matrix without having to copy it. To illustrate the scalability of the two R packages, we performed a GWAS analysis on 500 K individuals and 1 M SNPs.
The GWAS analysis completed in $11 h using the aforementioned desktop computer. The GWAS analysis was composed of four main steps. First we converted binary PLINK files in the format 'bigSNP' in 1 h. Then, we removed SNPs in long-range LD regions and used SNP clumping, leaving 93 083 SNPs in 5. 4 h. Then, the 10 first PCs were computed on the 500 K individuals and these remaining SNPs in 1. 8 h. Finally, we performed a linear association test on the complete 500 K dataset for each of the 1 M SNPs, using the 10 first PCs as covariables in 2. 9 h.
Performance and precision comparisons First, we compared the GWAS computations obtained with bigstatsr and bigsnpr to the ones obtained with PLINK 1. 9 and EIGENSOFT 6. 1. 4, and also two R packages SNPRelate and GWASTools. For most functions, multithreading is not available yet in PLINK, nevertheless, PLINK-specific algorithms that use bitwise parallelism (e. g. pruning) are still faster than the parallel algorithms reimplemented in package bigsnpr (Table  1 ). Overall, performing a GWAS on a binary outcome with bigstatsr and bigsnpr is as fast as when using EIGENSOFT and PLINK, and 19-45 times faster than when using R packages SNPRelate and GWASTools. For performing an association study on a continuous outcome, we report a dramatic increase in performance by using bigstatsr and bigsnpr, making it possible to perform such analysis in <2 min for a relatively large dataset such as the celiac dataset.
This analysis was 7-19 times faster as compared to PLINK 1. 9 and 28-74 times faster as compared to SNPRelate and GWASTools (Table  1 ). Note that the PC scores obtained are more accurate as compared to PLINK (see the last paragraph of this subsection), which is also the case for the P-values computed for the two GWAS (see Supplementary notebook 'GWAS-comparison'). Second, we compared the PRS analysis performed with the R packages to the one using PRSice-2. There are five main steps in such an analysis (Table  2 ), including four steps handled with functions of packages bigstatsr and bigsnpr. The remaining step is the reading of summary statistics which can be performed with the widely used function fread of R package data. table. Using bigstatsr and bigsnpr results in an analysis as fast as with PRSice-2 when using our desktop computer, and three times slower when using our laptop (Table  2 ).
Finally, on our desktop computer, we compared the computation times of FastPCA (fast mode of EIGENSOFT), FlashPCA2 and PLINK 2. 0 (approx mode) to the similar function big_randomSVD implemented in bigstatsr. For each comparison, we used the 93 083 SNPs which were remaining after pruning and we computed 10 PCs. We used the datasets of growing size simulated from the celiac dataset (from 15 155 to 151 550 individuals). Overall, function big_randomSVD is almost twice as fast as FastPCA and FlashPCA2 and eight times as fast as when using parallelism with six cores, an option not currently available in either FastPCA or FlashPCA2 (Fig. 4 ). PLINK 2. 0 is faster than bigstatsr with a decrease in time of 20-40%. We also compared results in terms of precision by comparing squared correlation between approximated PCs and 'true' PCs provided by an exact eigen decomposition obtained with PLINK 2.
0 (exact mode). Package bigstatsr and FlashPCA2 (that use the same algorithm) infer all PCs with a squared correlation of more than Table  1. Execution times with bigstatsr and bigsnpr compared to PLINK 1. 9 and FastPCA (EIGENSOFT) and also to R packages SNPRelate and GWASTools for making a GWAS for the celiac dataset (15 155 individuals and 281 122 SNPs). The first execution time is with a desktop computer (6 cores used and 64 GB of RAM) and the second one is with a laptop (2 cores used and 8 GB of RAM).
Automatic detection of long-range LD regions For detecting long-range LD regions during the computation of PCA, we tested the function snp_autoSVD on both the celiac and POPRES datasets. For the POPRES dataset, the algorithm converged in two iterations. The first iterations found three long-range LD regions in chromosomes 2, 6 and 8 (Supplementary Table  S1 ). We compared the PCs of genotypes obtained after applying snp_autoSVD with the PCs obtained after removing pre-determined long-range LD regions (https://goo. gl/8TngVE) and found a mean correlation of 89. 6% between PCs, mainly due to a rotation of PC7 and PC8 (Supplementary Table  S2 ). For the celiac dataset, we found five long-range LD regions (Supplementary Table  S3 ) and a mean correlation of 98. 6% between PCs obtained with snp_autoSVD and the ones obtained by clumping and removing pre-determined longrange LD regions (Supplementary Table  S4 ).
For the celiac dataset, we further compared results of PCA obtained when using snp_autoSVD and when computing PCA without removing any long range LD region (only clumping at R 2 > 0. 2). When not removing any long range LD region, we show that PC4 and PC5 do not capture population structure and correspond to a long-range LD region in chromosome 8 (Supplementary Figs  S3  and S4 ). When automatically removing some long-range LD regions with snp_autoSVD, we show that PC4 and PC5 reflect population structure (Supplementary Fig. S3 ). Moreover, loadings are more equally distributed among SNPs after removal of long-range LD regions (Supplementary Fig. S4 ). This is confirmed by Gini coefficients (measure of dispersion) of each squared loadings that are significantly smaller when computing PCA with snp_autoSVD than when no long-range LD region is removed (Supplementary Fig.
Imputation of missing values for genotyped SNPs For the imputation method based on XGBoost, we compared the imputation accuracy and computation times with Beagle on the POPRES dataset (with no missing value). The histogram of the MAFs of this dataset is provided in Supplementary Figure  S6. We used a beta-binomial distribution to simulate the number of missing values by SNP and then randomly introduced missing values according to these numbers, resulting in $3% of missing values overall (Supplementary Fig. S7 ). Imputation was compared between function snp_fastImpute of package bigsnpr and Beagle 4. 1 (version of January 21, 2017) by counting the percentage of imputation errors (when the imputed genotype is different from the true genotype). Overall, in three runs, snp_fastImpute made only 4.