dataesr/scientific-paragraphs-categorization huggingface dataset:
Improving selection stability using bagging Since we need to compute block HSIC of the paired data ffðx ' i ; y ' i Þg B i¼1 g n=B '¼1 with a fixed partition, the performance can be highly affected by the partition. Thus, we propose to use a bagging version of the block HSIC estimator. Given M random permutations of the n samples, we define bagging block HSIC as HSIC bb ðf k ; yÞ ¼ 1 M X M m¼1 B n X n=B '¼1 HSIC v f ð';mÞ k ; y ð';mÞ ¼ u > k v; where f ð';mÞ k is the kth feature vector restricted to the 'th block as defined by the mth permutation, 2 R nB are the vectors of the mth block HSIC Lasso, respectively. u k ¼ ffiffiffiffiffi 1 M r u ð1Þ > k ;. ; u ðMÞ > k h i > 2 R nBM ; v ¼ ffiffiffiffiffi 1 M r v ð1Þ Hence, bagging block HSIC Lasso can be written as min a!0 jjv À U > ajj 2 2 þ kjjajj 1 ; where U ¼ ½u 1 ;.
Adjusting for covariates Data analysis tasks in bioinformatics can often be confounded by technical (e. g. batch) or biological variables (e. g. age), which might mask the relevant variables. To adjust for their effect, we consider the following variant of the block HSIC Lasso: min a!0 jjv À U > a À bzjj 2 2 þ kjjajj 1 ; where b ! 0 is a tuning parameter and z ¼ ffiffiffi ffi B n r vec K ð1Þ cov > ;. ; vec K ðn=BÞ cov > > 2 R nB contains the covariate information. K cov is the Gram matrix computed from the covariate input matrix X cov. Since for most purposes in bioinformatics we want to remove all information from the covariates, we set b to b ¼ HSIC b ðy; X cov Þ HSIC b ðX cov ; X cov Þ ¼ HSIC b ðy; X cov Þ; which is the solution of min b jjv À bzjj 2 2. Here, we used the property HSIC b ðX cov ; X cov Þ ¼ 1.
Feature selection methods HSIC Lasso and block HSIC Lasso: We used HSIC Lasso and block HSIC Lasso implemented in the Python 2/3 package pyHSICLasso. In block HSIC Lasso, M was set to 3 in all experimental settings; the block size B was set on an experiment-dependent fashion. In all the experiments, when we wanted to select k features, HSIC Lasso versions were required to first retrieve 50 features, and then the top k features were selected as the solution. In this article, we use the following kernels: • The RBF Gaussian kernel for pairs of continuous variables, of continuous outcomes, or one of each, and for pairs of a continuous variable and categorical outcome: K : x ðkÞ i ; x ðkÞ j 7 ! exp À jjx ðkÞ i À x ðkÞ j jj 2 2 2r 2 ! ; where r 2 > 0 is the bandwidth of the kernel; • The normalized Delta kernel for categorical variables (or outcomes): L : y i ; y j 7 ! 1 n c if y i ¼ y j ¼ c 0 otherwise; 8 < : where n c is the number of samples in class c.
Evaluation of the selected features Selection accuracy on simulated data: We simulated highdimensional data where only a few variables were truly related to the outcome. We used these datasets to evaluate the ability of the tested algorithms to find the true causal variables, instead of others, likely spuriously correlated to the outcome. To that end, we requested each algorithm to retrieve the known number of causal features. Then, we studied how many of them were actually causal. Classification with a random forest: In classification datasets, we evaluated the amount of information retained in the features selected by a given method by evaluating the performance of a random forest classifier based only on those features. We used random forests because of their ability to handle non-linearities.
Datasets We evaluated the performance of the different algorithms on synthetic data and four types of real-world high-dimensional datasets (Table  1 ). In our experiments on real-world datasets, we restricted ourselves to classification problems. All discussed methods can however handle regression problems (continuous-valued outcomes) as well, as we show on synthetic data. Synthetic data: We simulated random matrices of features X $ N ð0; 1Þ. A number of variables were selected as related to the phenotype, and functions that are non-linear in the data range were selected (cosine, sine and square) and combined additively to create the outcome vector y. Images: Facial recognition is a classification problem classically used to evaluate non-linear feature selection methods, as only a few of all features are expected to be relevant for the outcome, in a nonlinear fashion.
We used four face image datasets from the Arizona State University feature selection repository  (Li et al. , 2018) ): pixraw10P, warpAR10P, orlraws10P and warpPIE10P. Gene expression microarrays: We analyzed four gene expression microarray datasets from Arizona State University feature selection repository  (Li et al. , 2018). The phenotypes were subtypes of B-cell chronic lymphocytic leukemia (CLL-SUB-111), hepatocyte phenotypes under different diets (TOX-171), glioma (GLIOMA) and smoking-driven carcinogenesis (SMK-CAN-187). Single-cell RNA-seq: Single-cell RNA-seq (scRNA-seq) measures gene expression at cell resolution, allowing to characterize the diversity in a tissue. We performed feature selection on the three most popular datasets in the Broad Institute's Single Cell Portal, related to mouse small intestinal epithelium  (Haber et al.
, 2017) , mouse hippocampus  (Habib et al. , 2016)  and human blood cells  (Villani et al. , 2017). Missing gene expressions were imputed with  MAGIC (van Dijk et al. , 2018). GWA datasets: We studied the WTCCC1 datasets  (Burton et al. , 2007)  for rheumatoid arthritis (RA), type 1 diabetes (T1D) and type 2 diabetes (T2D) (2000 samples each), using the 1958BC cohort as control (1504 samples). Affymetrix 500K was used for genotyping. We removed the samples and the SNPs that did not pass WTCCC's quality controls, as well as SNPs in sex chromosomes and those that were not genotyped in both cases and controls. Missing genotypes were imputed with CHIAMO. Lastly, individuals with >10% genotype missing rate, and SNPs with >10% genotype missing rate, MAF < 5% or not in HWE (P-value < 0. 001) were removed.
Software availability and reproducibility Block HSIC Lasso was implemented in the Python 2/3 package pyHSICLasso. The source code is available on GitHub (https:// github. com/riken-aip/pyHSICLasso), and the package can be installed from PyPI (https://pypi. org/project/pyHSICLasso). All analyses in this article and the scripts needed to reproduce them are also available on GitHub (https://github. com/hclimente/nori).
In Figure  1 , we show the proportion of the causal features retrieved by each method. The different versions of HSIC Lasso outperform the other approaches in virtually all settings. Block HSIC Lasso with decreasing block sizes results in worse performances. As expected, vanilla HSIC Lasso outperforms the block versions in accuracy, but increases memory use. Crucially, block HSIC Lasso on a larger number of samples performs better than vanilla HSIC Lasso on fewer samples. Hence, when the number of samples is in the thousands, it is better to apply block HSIC Lasso on the whole dataset, than to apply vanilla HSIC Lasso on a subsample. We wanted to test these conclusions using a non-linear, real-world dataset. We selected four image-based face recognition tasks (Section 3. 3). In this case, we selected different numbers of features  (10, 20, 30, 40 and 50).
Adjusting by covariates improves feature selection To evaluate the impact of covariate adjustment, we worked on a synthetic dataset (Section 3. 2) with the following experimental  Lines are discontinued when the algorithm required more memory than the provided (50 GB). Note that in some conditions mRMR's line cannot be seen due to the overlap with LARS parameters: n ¼ 1 000; d ¼ f100; 2500; 5000; 10 000g features; seven causal features. Two covariates were generated by taking two causal features and adding Gaussian noise (mean ¼ 0; standard deviation ¼ 0. 5). In the experiment shown in Supplementary Figure  S2 , we tested the ability of (block) HSIC Lasso to retrieve exclusively the remaining five causal features adjusting for the covariates. We observe that block HSIC Lasso is able to find more relevant features when it adjusts for known covariates.
Block HSIC Lasso is computationally efficient In our experiments on synthetic data, vanilla HSIC Lasso runs into memory issues already with 1000 samples (Fig. 1 ). This experiment shows how block HSIC Lasso keeps the good properties of HSIC Lasso, while extending it to more experimental settings. Block HSIC Lasso with B ¼ 20 reaches the memory limit only at 10 000 samples, which is already sufficient for most common bioinformatics applications. If larger datasets need to be handled, it can be done by using smaller block sizes or a larger computer cluster. We next quantified the computational efficiency improvement the block HSIC estimator brings. We compared the runtime and the peak memory usage in the highest dimensional setting where all methods could run (n ¼ 1000; d ¼ 2500, 20 causal features) (Fig.
Block HSIC Lasso improves with more permutations We were interested in the trade-off between the block size and the number of permutations, which affect both the computation time and accuracy of the result. We tested the performance of block HSIC Lasso with B ¼ f5; 10; 15; 30; 60g and M ¼ f1; 2; 3; 5g in datasets of n ¼ 1000; d ¼ 2500 and 20 causal features. As expected, causal feature recovery increases with M and B (Fig. 2C ), as the HSIC estimator approaches its true value. The memory usage OðdnBMÞ of several of the conditions was the same, e. g. B ¼ 10, M ¼ 3 and B ¼ 30, M ¼ 1. Such conditions are indistinct from the points of view of both accuracy, and memory requirements. In practice, we found no major differences in runtime between different combinations of B and M. Hence, a reasonable strategy is to fix B to a given size, and tune the M to the available memory/desired amount of information.
Block HSIC Lasso finds more relevant features We tested the dimensionality reduction potential of different feature selectors. We selected a variable number of features from different multi-class biological datasets, then used a random forest classifier to retrieve the original classes (Section 3. 2). The underlying assumption is that only selected features which are biologically relevant will be useful to classify unseen data. To that end, we evaluated the classification ability of the biomarkers selected in four gene expression microarrays (Fig. 3 ) and three scRNA-seq experiments (Supplementary Fig. S3 ). Unsurprisingly, we observe that non-linear feature selectors perform notably better than linear selectors. Of the non-linear methods, in virtually all cases block HSIC Lasso showed similar or superior performance to mRMR.
Interestingly, as little as 20 selected genes retain enough information to achieve a plateau accuracy in most experiments. Surveying 10 5 À 10 6 SNPs in 10 3 À 10 4 patients, genome-wide association (GWA) datasets are among the most high dimensional in biology, an unbalance which worsens the statistical and computational challenges. We performed the same evaluation on three WTCCC1 phenotypes (Section 3. 3). As a baseline, we also computed the accuracy of a classifier trained on all the SNPs (Supplementary Table  S1 ). We observe that a feature selection prior step is not always favorable: LARS worsens the classification accuracy by 5-10%. On top of that, LARS could not select any SNP in 2 out of the 15 experimental settings. On the other hand, non-linear methods improve the classification accuracy by 10%, with mRMR and block HSIC Lasso achieving similar accuracies.
Block HSIC Lasso is robust to ill-conditioned problems Single-cell RNA-seq datasets differ from microarray datasets in two ways. First, the number of features is larger, equaling the number of genes in the annotation (> 20 000). Second, the expression matrices are very sparse, due to biological variability (genes actually not expressed in a particular cell) and dropouts (genes whose expression levels have not been measured, usually because they are low, i. e. technical zeroes). In summary, the problem is severely ill conditioned, and the feature selectors need to deal with this issue. We observed that block HSIC Lasso runs reliably when faced with variations in the data, even on ill-conditioned problems like scRNA-seq. In the different scRNA-seq datasets, LARS was unable to select the requested number of biomarkers in any of the cases, returning always a lower number (Supplementary Fig.
S4 ). mRMR did in all cases. However, the implementation of mRMR that we used crashed while selecting features on the full  Villani et al. (2017)  dataset. 4. 7 Block HSIC Lasso for biomarker discovery 4. 7. 1 New biomarkers in mouse hippocampus scRNA-seq To study the potential of block HSIC lasso for biomarker discovery in scRNA-seq data, we focused on the mouse hippocampus dataset from  Habib et al. (2016) , as a list of 1669 known biomarkers for the different cell types is also provided by the authors. We requested block HSIC Lasso, mRMR and LARS to select the best 20 genes for classification of 8 cell types (Supplementary Table  S2 ). The cell types were four different hippocampal anatomical subregions (DG, CA1, CA2 and CA3), glial cells, ependymal, GABAergic and unidentified cells. The overlap between the genes selected by different algorithms was empty.
We compared the selected genes to the known biomarkers. Out of the 20 genes selected by mRMR, 14 are known biomarkers, a number that goes down to 0 in the case of block HSIC Lasso (Supplementary Fig. S4A ). Hence, these 20 genes, which are sufficient for accurately separating the cell types, are potential novel biomarkers. However, we have no reason to believe that HSIC Lasso generally has a higher tendency to return novel genes than other approaches; we merely emphasize that it suggests alternative, statistically plausible biological hypotheses that can be worth investigating. We therefore evaluated whether the novel genes found by block HSIC Lasso participate in biological functions known to be different between the cell classes. To obtain the biological processes responsible for the differences between classes, we mapped the known biomarkers to GO Biological process categories using the GO2MSIG database  (Powell, 2014).
GWAS without assumptions on genetic architecture We applied block HSIC Lasso (B ¼ 60, M ¼ 1) to three GWA datasets (Section 3. 3). It is typical in GWAS to assume a genetic model before performing statistical testing of associations between SNPs and the phenotype. Two common, well-known models are the additive model-the minor allele in homozygosity has twice the effect as the minor allele in heterozygosity-and the dominant model-any number of copies of the minor allele have a phenotypic outcome. Using non-linear models such as block HSIC Lasso to explore the relationship between SNPs and outcome is attractive since no assumptions are needed on how individual SNPs affect the trait. The only assumption is that the phenotype can be explained by a combination of main effects, as block HSIC Lasso does not account for epistasis.
Discussion In this work, we presented block HSIC Lasso, a non-linear feature selector. Block HSIC Lasso retains the properties of HSIC Lasso while extending its applicability to larger datasets. Among the attractive properties of block HSIC Lasso we find, first, its ability to handle both linear and non-linear relationships between the variables and the outcome. Second, block HSIC Lasso has a convex formulation, ensuring that a global solution exists, and that it is accessible. Third, the HSIC score can be accurately estimated, as opposed to other measures of non-linearity like mutual information. Fourth, block HSIC Lasso's memory consumption scales linearly with respect to both the number of features and the number of samples. In addition, block HSIC Lasso can be easily adapted to different problems via different kernel functions that better capture similarities in new datasets.
Lastly, block HSIC Lasso can be adjusted for covariates known to affect the outcome, which helps removing confounding effects from the analysis. Due to all these properties, we show how block HSIC Lasso outperforms all other algorithms in the tested conditions. Block HSIC Lasso can be applied to different kinds of datasets. As other non-linear methods, block HSIC Lasso is particularly useful when we do not want to make strong assumptions about how the causal variables relate to the outcome. Thanks to the advantages mentioned above, HSIC Lasso and block HSIC Lasso tend to outperform other state-of-the-art approaches in terms of both causal features retrieval in simulated data, and classification accuracy on real-world datasets. Whereas the Lasso is limited to selecting at most as many features as there are available samples (n), for block HSIC Lasso the limitation is nBM.
Block HSIC Lasso's main drawback is the memory complexity, markedly lower than in vanilla HSIC Lasso but still OðdnBÞ. Memory issues might appear in low-memory servers in cases with a large number of samples n, of features d, or both. However, through our work on GWA datasets, the largest type of dataset in bioinformatics, we show that working on these datasets is feasible. Another drawback, which block HSIC Lasso shares with the other non-linear methods, is their black box nature. Block HSIC Lasso looks for biomarkers which, after an unknown, non-linear transformation, would allow a linear separation between the samples. Unfortunately, we cannot access this transformed space and explore it, which makes the results hard to interpret.
Funding Computational resources and support were provided by  RIKEN AIP. H. C-G. was funded by the  European Union 's  Horizon 2020 research and innovation program  under the  Marie Skłodowska-Curie  [666003]. S. K. was supported by the  Academy of Finland  ( 292334 ,  319264 ). M. Y. was supported by the  JST   PRESTO program  JPMJPR165A and partly supported by  MEXT   KAKENHI 16H06299  and the RIKEN engineering network funding.
clinicaltrials. gov/). Adipose-derived Stem Cells (ADSCs) are a subclass of mesenchymal stem/stromal cells initially derived from the bone marrow. However, to obtain the critical number of cells before transplantation, ADSCs must be expanded in vitro. This essential step has raised important concerns about the quality of adult stem cells, and interpatient variability is a challenge when seeking to define ADSCs. The development of uniform protocols for both preparation and characterization of MSCs, including standardized functional assays to assess their bio-logical potential, will be critical in contributing to their clinical utility. Here, we applied the SILAC method to engineered cell lines expressing either the standard 20S proteasome or the immunoproteasome to produce and purify isotope-labeled, endogenous versions of standard proteasome and immunoproteasome in vivo.
On the Q-Exactive Plus, the ten most intense ions per survey scan were selected for HCD fragmentation, and the resulting fragments were analyzed in the Orbitrap. Dynamic exclusion was used with a 60-s or a 30-s window (on the LTQ-Orbitrap Velos or the Q-Exactive Plus, respectively) to prevent repeated selection of peptides. Raw mass spectrometry files were processed using MaxQuant (version 1. 5. 5. 1) and Andromeda was used to match MS/MS spectra against the Human SwissProt database (March 2017 release -20,181 entries for human tissues analysis/release of August 2018 -20,386 entries for ADSCs analysis) and a list of potential contaminant sequences provided in MaxQuant1. 5. 5. 1. , with Carbamidomethylation of cysteines set as fixed modification. Oxidation of methionine and protein N-terminal acetylation were set as variable modifications.
The digestion specificity of trypsin was defined as cleavage after K or R, and up to two missed trypsin cleavage sites were allowed. The precursor mass tolerance was set to 20 ppm for the first search and 4. 5 ppm for the main search. The mass tolerance in MS/MS mode was set to 0. 8 Da for tissue analysis and to 20 ppm for ADSC analysis. Minimum peptide length was set to seven amino acids, and minimum number of unique peptides was set to 1. A target-decoy approach was used to validate hits, using a reverse database and applying a peptide and protein false-discovery rate of 1%. The "match between runs" option in MaxQuant was enabled, with a time window of 0. 7 min, to allow cross-assignment of MS features detected in different runs. Only unique peptides were used for quantification when analyzing human tissues, whereas both unique and razor peptides were used for the quantification when dealing with ADSCs data.
Proteins with fold-changes outside these thresholds were considered outliers from the global distribution. Outlier proteins with a p value of less than 0. 05 were considered as differentially expressed between normoxic and hypoxic conditions. GO terms enrichment analysis was performed on these proteins using GOrilla  (32). Multiple Reaction Monitoring to Quantify 20S Proteasome Subunits-Dried peptide samples were resuspended in a solution containing 2% acetonitrile, 0. 05% TFA, to obtain a final concentration of 1 g/l. To obtain data for the whole set of peptides/transitions, samples were injected twice as the list of transitions was split into two methods. The sample (2. 5 l, about 2. 5 g protein equivalent) was loaded onto the system and analyzed on a hybrid triple quadrupoleion trap mass spectrometer 6500 QTrap (AB Sciex, Framingham, MA) equipped with a nanoelectrospray ion source coupled to an Ultimate 3000 system (Dionex) for chromatographic peptide separation.
e. considering isoforms, variants, PTMs reported on protein sequence or possibly artifactually-induced by sample handling, or missed cleavages, observed in Protein (UniProt) or MS databases (PeptideAtlas)). To achieve maximal sensitivity, collision energies (CEs) were optimized within a 6 V window around the CE value recommended by Skyline software and based on the m/z of the precursor. Final SRM transitions are given in supplemental Data S3. Transitions could be unambiguously assigned thanks to the co-injected isotope-labeled peptides. Samples were run in a blinded fashion except for calibration curves, for which the lowest concentrations were injected first. To check system suitability and performance before injecting each batch of samples, Total Ion Current was tuned (chromatographic solvent ions of m/z 50 to 1000) at 2000 Da/s scan speed in Q1.
Quality controls (QC) (20 fmol of tryptic digest of betagalactosidase; transitions are given in supplemental Data S4) were injected between each sample. Carry-over was checked by injecting tryptic digests of 100 fmol, 500 fmol or 1000 fmol of isotopically labeled sP20S and iP20S spiked in 2. 5 g of HeLa, then one QC sample, and finally a blank sample (the maximum carry-over observed was less than 0. 01%). Quantitative reproducibility over time was checked based on signal intensities for heavy transitions (maximum deviation of 50% allowed); injection batches were generally carried out over periods of less than 1 week. Quantitative data analyses were performed using Skyline-Daily open-source software  (36). Area values for all transitions were first extracted automatically by the software, then checked manually and adjusted if necessary (i.
e. exclusion of data points if S/N Ͻ 10). Light transition (L) peak area signals were then normalized with respect to their labeled counterparts (H) (after correction by considering the 95% incorporation rate for R( 13 C 6 ) and K( 13 C 6 ) in labeled sP20S and iP20S). For each transition, technical replicates of injection (typically three) were averaged. Ratios (L/H) of all the transitions used to assay a given protein or for total 20S proteasome (details in supplemental Data S3) were then averaged. The amount of labeled reference mix (containing equimolar concentrations of isotope-labeled sP20S and iP20S) spiked into the biological sample before sample preparation was then used to determine the absolute quantity of each proteasome subunit or total 20S proteasome. For each transition, LOD and LLOQ were experimentally determined by injecting heavy-isotope-labeled sP20S and iP20S spiked at increasing concentrations in a HeLa protein lysate and processed by the same method as applied to the other biological samples.