dataesr/scientific-paragraphs-categorization huggingface dataset:
INTRODUCTION The recent development of high-throughput bio-technologies has rendered large-scale multi-omics datasets increasingly available. Biology has now entered the world of "big data", with a pressing need to manage, process and optimize the use of large-scale "-omics" sequencing measurements. In addition to the obvious challenge of having to deal with large volumes of information from widely heterogeneous sources, the underlying biological nature of such data poses the additional issue of high complexity, due to the multiple * To whom correspondence should be addressed. These authors contributed equally to this work. Email: {celine. brouard,jerome. mariette}@inrae. fr types of interactions existing within and across multiple levels in living organisms. This triggers the need to consider these multi-layered biological systems as a whole, and to develop accurate and innovative methods to integrate multiple and heterogeneous levels of information collected on the same individuals.
To address this challenge, kernel methods have proven useful and successful (1) because they offer a natural theoretical framework to handle the analysis of different types of data / features observed on the same individuals. They can also address the issue of the huge dimensionality by summarizing each level of information with a similarity matrix between individuals (whose number is generally small enough), providing a solution efficient in terms of computational cost and storage. Relevant kernels can embed expert knowledge and handle the high dimension much better than the Euclidean distance, notorious for behaving poorly when the number of features is large  (2). They have been successfully used in computational biology for exploratory  (3)  or prediction  (4, 5, 6)  purposes and to integrate datasets in a supervised  (7)  or unsupervised fashion  (8, 9).
Overview of feature selection. Feature selection has attracted a large amount of attention during the past years: (13) made a tentative comprehensive overview and classification of the most popular methods for feature selection, accompanied by an open-source repository and benchmark datasets http://featureselection. asu. edu. In addition, implementations of the methods are provided in the Python package scikit-feature. These methods are generally organized in three main families: embedded methods, where the selection is embedded in a prediction method, filter methods, where the selection is made independently of any prediction purpose and, wrapper methods, where the selection is made in relation with but not embedded into a prediction method. As visible through their description, embedded and wrapper methods are thus proposed in the framework of univariate supervised learning, where the selection is combined with the prediction objective to obtain a trade-off between number of selected features and accuracy of the prediction.
Most works on feature selection for kernel use a feature based kernel, i. e. , one kernel for each feature in the input dataset. The earliest work in this line is the Feature Vector Machine approach  (35) , where the feature kernel computes a similarity between a given kernel and all the other features. However, since the feature mapping maps each feature into a space having the same dimension than the original number of features, this approach is not adapted to cases where the number of features is larger than the number of observations. (36)  proposes a sparse additive model (SpAM) approach, which is a linear model using the feature kernels for predictors and a group-lasso based penalty that performs feature selection. One weakness of this approach is that it only targets additive models.
A 1 penalty is added in the learning process to obtain a sparse w = (w 1 ,. ,w p ), which corresponds to a feature selection. This allows to define a flexible framework for feature selection, which is also able to incorporate a priori latent cluster structure or known relations between descriptors if needed. Simulations on various types of problems (both supervised and unsupervised) and various types of output datasets (multivariate numeric outputs or time series) show the effectiveness of the proposed methods and its great stability compared to alternative approaches. In summary, the new approach for kernel method: • extends the supervised case to the unsupervised setting and to arbitrary kernel output, without the need of structural or a priori assumptions on the data; • accounts for colinearities between features to reduce redundancy in the selection; • is based on a general framework flexible enough to incorporate a priori knowledge on the data if available.
Our feature selection method for kernels is implemented in Python, using numpy https://numpy. org/ and autograd https://github. com/HIPS/autograd and inspired by the PyOptim https://github. com/rflamary/PyOptim library. It is embedded in the select. features function of the R package mixKernel version 0. 7 published on CRAN, using reticulate https://rstudio. github. io/reticulate.
Experimental evaluation setup. Benchmark datasets for the evaluation of the unsupervised feature selection. To evaluate the accuracy of the unsupervised version of the proposed approach, a benchmark of three biological datasets was analyzed: two microarray datasets, denoted in the sequel by "Carcinom" and "Glioma", provided in the Python package scikit-feature, and a DNA barcoding dataset, denoted by "Koren", available from the R package mixOmics  (28). "Carcinom" and "Glioma" datasets respectively contain the expression of 9,182 genes obtained from 174 samples and 4,434 genes from 50 samples. To perform the feature selection on these datasets, UKFS was used with the Gaussian kernel K x (x i ,x i ) = e -σ * x i -x i 2 with σ * chosen so as to minimize the reproduced inertia in the projection on the first two axes of the KPCA with kernel K x.
"Koren" includes the abundance of 973 operational taxonomic units (OTUs) collected from 43 samples. UKFS was used with the kernel induced by the Bray-Curtis dissimilarity between samples on raw abundances, d BC (x i ,x i ) = p s=1 |x is -x i s | p s=1 (x is +x i s ) (p is the number of observed OTUs). An interesting property of this dissimilarity is that it avoids the need to use one of the standard pre-processing steps to account for the compositional nature of this dataset (e. g. , total sum scaling normalization (TSS) and centered log ratio transformation (CLR)). UKFS was then compared to several alternatives: • two methods based on the computation of a score: the Laplacian score (denoted by lapl) (57) and SPEC (18); • three methods based on a learning approach constrained to a sparse representation.
These methods were mostly designed for clustering (or are based on the implicit assumption that samples are structured into subgroups) and require the a priori definition of a number of clusters: MCFS (19), NDFS (21) and UDFS (22); • one neural network: the concrete autoencoder, denoted Autoencoder (58). This method can handle supervised and unsupervised frameworks and the unsupervised setting is used for the purpose of comparison. Except for Autoencoder that requires large computational resources and CPU computing, simulations were all performed on the same 40-node computer without concurrent access. The Python implementation available from the scikit-feature package (13) https://github. com/jundongl/scikit-feature was used for lapl, SPEC, MCFS, NDFS and UDFS and a Python implementation based on Keras https://keras.
io/ was used for Autoencoder https://github. com/mfbalin/Concrete-Autoencoders. To address the underlying compositional structure of "Koren" with these methods, standard preprocessing steps, i. e. , total sum scaling normalization (TSS) and centered log ratio transformation (CLR), were applied before selecting the relevant features. No pre-processing was performed for the other two datasets. To evaluate the different methods, the following steps were used: 1. each method was run to select d features with increasing values of d ∈ {10, 20,. , 290, 300}, except for UKFS for which d is given by the number of selected features when increasing the regularization parameter, λ. The solutions for different values of λ were obtained using a warm restart strategy: the first solution was computed for a small λ and subsequent solutions were obtained using the previously obtained solution as an initial value for the method.
30 values of λ were used, with an exponential increase. Whereas this strategy shrinks the obtained solution around the values found for the initial λ, it has the advantage of providing a consistent and approximately smooth evaluation of the evolution of the quality criterion; 2. based on the selected features, the kernel k-means algorithm was repeated 20 times, using the Gaussian kernel for "Carcinom" and "Glioma" and the Bray-Curtis dissimilarity for "Koren". Data were clustered into C classes, in which C was chosen as the true value of the underlying clustering (C = 11 for "Carcinom", C = 4 for "Glioma" and C = 3 for "Koren"); 3. the relevance of the obtained feature selection was then obtained as its ability to recover the true underlying classification structure of the dataset. This true partition is used as ground truth to compute standard clustering performance metrics, i.
Evaluation of the structure based extension of unsupervised feature selection. The structure based version of the proposed approach (Equation (  12 )), was also evaluated in a similar fashion against alternative methods. Two datasets were used: the "Koren" dataset described in the previous section and another dataset included in the R package mixOmics, called "HMP". This latter dataset is a Human Microbiome 16S dataset, including OTU counts on the three most diverse bodysites: Subgingival plaque (Oral), Antecubital fossa (Skin) and Stool, sampled from 54 unique individuals for a total of 162 samples. The 1,674 OTUs processed and included in mixOmics are a subsample of the full dataset where OTU counts below 0. 01 percent compared to the total count were filtered out. The original dataset can be downloaded from the Human Microbiome Project (http://hmpdacc.
org/HMQCP/all/). Relations between features (OTUs for both cases) were obtained by computing the Pearson correlation matrix, which was used as the adjacency matrix of the graph (the method is denoted by UKFS-G in the sequel). Evaluation of the kernel output feature selection for multiple output regression problems. To evaluate the accuracy of KOKFS, it was first used to solve multiple output regression problems, i. e. , cases where Y = R q. This setting indeed provides an easy way to evaluate the method performance: more precisely, KOKFS was run with Gaussian input and output kernels and selected features were evaluated by assessment of their predictive power (this can be performed with any regression method). Note that contrary to standard multivariate linear regression incorporating feature selection (like multivariate lasso implemented in the R package glmnet (26)), the proposed approach can 1/ use a more adapted norm than the Euclidean norm for large dimensional input spaces and adapts the feature selection to this norm and 2/ does not directly aim at predicting Y but rather at selecting features that best explain the global resemblance between samples as described by the kernel K y.
The considered datasets were: 1. The "Nutrimouse" dataset  (60) , available in the R package mixOmics (28). This dataset contains the expressions of p = 120 genes (obtained by RT-qPCR) and the concentration of q = 21 hepatic fatty acids for n = 40 mice. 2. The "Diogenes" dataset, described in  (61, 62)  and available on GEO Gene Expression Omnibus (GEO repository, http://www. ncbi. nlm. nih. gov/geo/) under the accession number GSE95640. This dataset contains gene expression acquired by RNA-Seq on human adipose tissue at two different time steps (CID1 and CID2) of a dietary intervention (before and after a 8 week low calorie diet) for n = 167 individuals. The expression of p = q = 269 genes, respectively corresponding to expressions before and after the diet, were used in the experiments. These genes were genes of interest for the biologists and corresponded to genes listed in a parallel study on the same individuals (available on GEO under the accession number GPL19141).
The "TCGA" dataset based upon data generated by The Cancer Genome Atlas (TGCA) Research Network (https://www. cancer. gov/tcga). The subset is restricted to primary tumor samples of breast cancer for which mRNA and miRNA expressions were both available. Expression data were log-transformed and corrected for a batch effect (plate) prior analysis. The dataset was composed of n = 1194 individuals for which p = 655 miRNA and q = 9884 mRNA were available. Then, experiments were conducted in two steps: a first step for feature selection and a second one for performance assessment with nonparametric regressions based on the selected features. For the feature selection step, Gaussian kernels were used for the input and output kernels. The parameters of both Gaussian kernels were set to n(n-1) i =i z i -z i 2 2 , where z stands for either x or y.
Features were ordered by their order of appearance in the selection along the λ 2 paths. For the regression step, the features selected by KOKFS were then passed to a SVM regression to predict each of the q output features in Y. More precisely, for a number of selected features between 1 and 40 (ordered by the regularization path), regressions were performed by -regression SVM with Gaussian kernel and hyperparameters were tuned by crossvalidation as implemented in the R package e1071. For each output variable, pseudo-R 2 was computed as: pseudo-R 2 (y j ) = 1- n i=1 (y ij -ŷij ) 2 Var(y j )  (13)  in which y j stands for the jth variable and ŷij is the value predicted by the SVM from the selected features for the ith sample and the jth output variable. Since q = 9,884 for "TCGA", this regression step was restricted to q = 50 variables only.
To do so, a hierarchical clustering was performed, with Ward's linkage based on a distance obtained from the correlation between genes. The dendrogram resulting from the hierarchical clustering was cut at 50 clusters and, in each cluster, a single output variable was selected, which was the most correlated in average with the other variables from its cluster. The overall approach was compared to predictions made by several alternatives: • the multivariate Gaussian lasso method implemented in the R package glmnet  (26) , in which a group lasso penalty is used to select features common to all linear models fitted during the learning. The multivariate Gaussian lasso method was used in two different ways: either the direct predictions of the multivariate Gaussian lasso were used to compute a pseudo-R 2 as in Equation (  13 ) or the selected features were submitted to the same procedure than the features selected by KOKFS (i.
e. , SVM regression for each output variable, followed by pseudo-R 2 computation); • the multiple output regression Relief, as described in  (32)  and available through their Java program CLUS http://source. ijs. si/ktclus/clus-public/; • the multiple output random forest (RF), also described in  (32)  and available through their Java program CLUS http://source. ijs. si/ktclus/clus-public/; • the standard and block version of HSIC lasso  (34, 39) , available in the pyHSIClasso Python package https://github. com/riken-aip/pyHSIClasso. The three last methods were used in replacement to KOKFS in the feature selection step described above and subjected to the same type of regression step as the features selected by KOKFS. The number of features included in the prediction was varied from 1 to 40 by order of appearance along the regularization path (multivariate lasso, HSIC lasso) or by ranking of the features (Relief, RF).
In addition, for "Nutrimouse", the selected features were also compared to the set of features extracted by CCA in the seminal work  (29) , by submitting them to SVM regression. Finally, running times for every method were also obtained using a single core of the same computer. The provided running times were the ones needed to select 40 variables or, as Relief and RF are ranking methods, the running time for ranking all the variables. For multivariate lasso, the running time was measured for computing the regularization path for 100 values of λ (with default regularization path provided in the R package glmnet. Finally, to account for the need to tune the hyperparameter λ 2 in the running time of KOKFS, the space of λ 2 was explored in order to select the value for which 40 variables are selected.
Solutions to overcome these limitations include the use of dedicated mixed models (as for the extension of random forest in (63)) or the computation of a relevant similarity measure between time series. For instance, Fréchet distances (64) can provide informative insights on the resemblance and are increasingly used for the analysis of various type of non Euclidean data  (65). To illustrate the usefulness of the proposed approach when the resemblance between outputs is well embedded into Fréchet distance, the data on the impact of various interventions on Covid-19 epidemic was used. These data have previously been collected in  (66)  and are available in their github repository https://github. com/complexity-sciencehub/ranking npis. More precisely, their L2 version of nonpharmaceutical interventions (NPI) data was used, which consisted in the encoding of governmental interventions to face the Covid-19 pandemic into 46 themes for 79 countries during 261 days ranging from March to July 2020.
Input features X consisted of the activation of these various interventions (encoded as 0/1 to indicate if this intervention was activated or not) in every country at different days. The Gaussian kernel was used for computing a similarity between input features. To avoid redundancy in the dataset, only one day for each month was selected (the last day of the month). The corresponding output data, Y , consisted of the evolution of the reproduction number at date t, R t , and during the 20 days after the observation date, R t+1 ,. , R t+20. The Fréchet distances between time series were obtained and log-transformed to improve their discriminating power. An associated similarity matrix was obtained by subtracting the Fréchet distance values to their maximum. This matrix was used as the output kernel.
Removing all couples of (country, date) that had at least one missing value, the final dataset consisted of n = 365 different observations for p = 46 governmental interventions. As in the previous section, λ 2 was varied using a warm restart strategy. Given the shape of the λ 2 regularization path (see Figure  S1  of supplementary material), features were ranked by their maximum associated weight along the path, which was fairly similar to (but more discriminating than) their order of appearance in the selection. No ground truth is available for this problem so the ranking obtained by (  66 ) was used and a Spearman correlation test was performed to assess if the two rankings were significantly related.
RESULTS This section provides results obtained for the simulation setting described in the previous section. First, results obtained for the unsupervised framework are given with several benchmark datasets for the basic version and with two additional datasets for the structure based version. Then, results obtained for the kernel output framework are given, with several multiple output regression problems and then with time series output. Discussion on the obtained results is provided in the next section. Benchmark datasets for the evaluation of unsupervised feature selection. Table  2  presents the results obtained for UKFS and its competitors on the 3 benchmark datasets, in terms of ACC and NMI for d ∈ {10,300}, of average CPU time for one run over the range of d and of the average area under the curve (AUC) of NMI, ACC and COR over the range of d.
CPU time for Autoencoder for "Carcinom" is not given with the same exactness than for the other methods because this method was run in parallel contrary to the others. In addition, Figure  2  provides a comparison of the different approaches in terms of NMI and ACC evolution on "Carcinom" versus the number of selected features, d. Results demonstrate a high efficiency of the proposed approach to select relevant features with no a priori on the number of clusters present in the data. For the three tested datasets, UKFS is in the range of or surpasses results obtained with other methods. More precisely, Table  2  shows that UKFS, MCFS and UDFS respectively obtain the best results on "Carcinom", "Glioma" and "Koren" datasets. On the contrary, both score based methods exhibit poor performances, except on the "Glioma" dataset for which lapl ranks first when the clustering uses only 10 selected features.
The performances of this method are negatively impacted for an over-estimation of C (30 instead of 11) and strongly negatively impacted for an under-estimation of C (2 instead of 11). 3  the results of the comparison of UKFS and UKFS-G with all the other methods on "HMP". On "Koren", using the graph only gives a very slight improvement compared to the original method (average ACC AUC equal to 267. 96 (8. 03) compared to 242 (8. 71) without the constraint, which makes it the best method in terms of ACC AUC compared to results reported in Table  2  on "Koren"). For "HMP", the improvement is more important and allows UKFS-G to compete with the best approach on this  dataset (NDFS that uses the true number of clusters, C = 3, which is a strong advantage for this dataset). Overall, not only do these results show that adding the structure constraint does not alter the performance of UKFS but, in some cases, it can improve it.
Evaluation of kernel output feature selection for multiple output regression. Figure  6  provides the evolution of the average pseudo-R 2 (over the q outputs) for the different feature selection methods, when the number of selected features, d, used in the regression increases from 4 to 40. Note that it was not possible to obtain feature selection for multivariate lasso and RF for "TCGA" due to the large size of the dataset. On the contrary, block HSIC lasso selection was not computed for "Nutrimouse" since the dataset was very small (and the approach is thus not relevant in this case, compared to standard HSIC). All figures show the same tendencies: KOKFS selects features that, combined with SVM prediction, outperform the prediction performance of features selected with multivariate lasso, with HSIC lasso, with RF or with regularized CCA.
KOKFS is the slowest method on the smallest dataset ("Nutrimouse") but scales better for larger (input and output) dimensions ("Diogenes") and larger sample size ("TCGA") than the other two methods (RF and multivariate lasso). Note that, for these two methods, it was not possible to compute the solution for "TCGA" due to overloaded memory and that RF also overloaded the CPU for all simulations (so the reported running times are not really comparable with that of the other methods). Evaluation of kernel output feature selection with time series outputs. Table  5  provides the ranking of the government measures as provided by the feature selection approach KOKFS to explain the similarity between (R t ) t time series evolution during the 20 following days. Even though, contrary to (66), the purpose was not to directly find the government measures the most effective for reducing the reproduction number (R t ) t but merely to find the measures that most explain similarity in the reproduction number evolution, the found conclusions were very similar to the ones of (66): among important measures, border health check, individual movement restrictions, the "Nutrimouse" "Diogenes" "TCGA" Figure  6.