Robust Feature Selection using Sparse Centroid-Encoder

1 INTRODUCTION . Technological advancement has made high-dimensional data readily available . For example , in bioinformatics , the researchers seek to understand the gene expression level with microarray or next-generation sequencing techniques where each point consists of over 50,000 measurements ( Pease et al . ( 1994 ) ; Shalon et al . ( 1996 ) ; Metzker ( 2010 ) ; Reuter et al . ( 2015 ) ) . The abundance of features demands the development of feature selection algorithms to improve a Machine Learning task , e.g. , classification . Another important aspect of feature selection is knowledge discovery from data . Which biomarkers are important to characterize a biological process , e.g. , the immune response to infection by respiratory viruses such as influenza ( O ’ Hara et al . ( 2013 ) ) ? Additional benefits of feature selection include improved visualization and understanding of data , reducing storage requirements , and faster algorithm training times . Feature selection can be accomplished in various ways that can be broadly categorized into the filter , wrapper , and embedded methods . In a filter method , each variable is ordered based on a score . After that , a threshold is used to select the relevant features ( Lazar et al . ( 2012 ) ) . Variables are usually ranked using correlation ( Guyon & Elisseeff ( 2003 ) ; Yu & Liu ( 2003 ) ) , and mutual information ( Vergara & Estévez ( 2014 ) ; Fleuret ( 2004 ) ) . In contrast , a wrapper method uses a model and determines the importance of a feature or a group of features by the generalization performance of the predetermined model ( El Aboudi & Benhlima ( 2016 ) ; Hsu et al . ( 2002 ) ) . Since evaluating every possible combination of features becomes an NP-hard problem , heuristics are used to find a subset of features . Wrapper methods are computationally intensive for larger data sets , in which case search techniques like Genetic Algorithm ( GA ) ( Goldberg & Holland ( 1988 ) ) or Particle Swarm Optimization ( PSO ) ( Kennedy & Eberhart ( 1995 ) ) are used . In embedded methods , feature selection criteria are incorporated within the model , i.e. , the variables are picked during the training process ( Lal et al . ( 2006 ) ) . Iterative Feature Removal ( IFR ) uses the absolute weight of a Sparse SVM model as a criterion to extract features from the high dimensional biological data set ( O ’ Hara et al . ( 2013 ) ) . This paper proposes a new embedded variable selection approach called Sparse Centroid-Encoder ( SCE ) to extract features when class labels are available . Our method extends the Centroid-Encoder model ( Ghosh et al . ( 2018 ) ; Ghosh & Kirby ( 2020 ) ) , where we applied a l1 penalty to a sparsity promoting layer between the input and the first hidden layer . We evaluate this Sparse Centroid-Encoder on diverse data sets and show that the selected features produce better generalization than other state-of-the-art techniques . Our results showed that SCE picked fewer features to obtain high clas- sification accuracy . As a feature selection tool , SCE uses a single model for the multi-class problem without the need to create multiple one-against-one binary models typical of linear methods , e.g. , Lasso ( Tibshirani ( 1996 ) ) , or Sparse SVM ( Chepushtanova et al . ( 2014 ) ) . Although SCE can be used both in binary and multi-class problems , we focused on the multi-class feature selection problem in this paper . The work of Li et al . ( 2016 ) also uses a similar sparse layer between the input and the first hidden with an Elastic net penalty while minimizing the classification error with a softmax layer . The authors used Theano ’ s symbolic differentiation ( Bergstra et al . ( 2010 ) ) to impose sparsity . In contrast , our approach minimizes the centroid-encoder loss with an explicit differentiation of the l1 function using the sub-gradient . The article is organized as follows : In Section 2 we present the Sparse Centroid-Encoder algorithm . In Section 3 we apply SCE to a range of bench-marking data sets taken from the literature . In Section 4 , we review related work , for both linear and non-linear feature selection . In Section 5 , we present our discussion and conclusion . 2 SPARSE CENTROID-ENCODER . Centroid-encoder ( CE ) neural networks are the starting point of our approach ( Ghosh & Kirby ( 2020 ) ; Ghosh et al . ( 2018 ) ; Aminian et al . ( 2021 ) ) . We present a brief overview of CEs and demonstrate how they can be extended to perform non-linear feature selection . 2.1 CENTROID-ENCODER . The CE neural network is a variation of an autoencoder and can be used for both visualization and classification tasks . Consider a data set with N samples and M classes . The classes denoted Cj , j = 1 , . . . , M where the indices of the data associated with class Cj are denoted Ij . We define centroid of each class as cj = 1|Cj | ∑ i∈Ij x i where |Cj | is the cardinality of class Cj . Unlike autoencoder , which maps each point xi to itself , the CE maps each point xi to its class centroid cj by minimizing the following cost function over the parameter set θ : Lce ( θ ) = 1 2N M∑ j=1 ∑ i∈Ij ‖cj − f ( xi ; θ ) ) ‖22 ( 1 ) The mapping f is composed of a dimension reducing mapping g ( encoder ) followed by a dimension increasing reconstruction mapping h ( decoder ) . The output of the encoder is used as a supervised visualization tool ( Ghosh & Kirby ( 2020 ) ; Ghosh et al . ( 2018 ) ) , and attaching another layer to map to the one-hot encoded labels performs robust classification ( Aminian et al . ( 2021 ) ) . 2.2 SPARSE CENTROID-ENCODER FOR FEATURE SELECTION . The Sparse Centroid-encoder ( SCE ) is a modification to the centroid-encoder architecture as shown in Figure 1 . Unlike centroid-encoder , we haven ’ t used a bottleneck architecture as visualization is not our aim here . The input layer is connected to the first hidden layer via the sparsity promoting layer ( SPL ) . Each node of the input layer has a weighted one-to-one connection to each node of the SPL . The number of nodes in these two layer are the same . The nodes in SPL don ’ t have any bias or non-linearity . The SPL is fully connected to the first hidden layer , therefore the weighted input from the SPL will be passed to the hidden layer in the same way that of a standard feed forward network . During training , a l1 penalty will be applied to the weights connecting the input layer and SPL layer . The sparsity promoting l1 penalty will drive most of the weights to near zero and the corresponding input nodes/features can be discarded . Therefore , the purpose of the SPL is to select important features from the original input . Note we only apply the l1 penalty to the parameters of the SPL . Denote θspl to be the parameters ( weights ) of the SPL and θ to be the parameters of the rest of the network . The cost function of sparse centroid-encoder is given by Lsce ( θ ) = 1 2N M∑ j=1 ∑ i∈Ij ‖cj − f ( xi ; θ ) ) ‖22 + λ‖θspl‖1 ( 2 ) where λ is the hyper-parameter which controls the sparsity . A larger value of λ will promote higher sparsity resulting more near-zero weights in SPL . In other words , λ is a knob that controls the number of features selected from the input data . Like centroid-encoder , we trained sparse centroid-encoder using error backpropagation , which requires the gradient of the cost function of Equation 2 . As l1 function is not differentiable at 0 , we implement this term using the sub-gradient . 2.3 ITERATIVE FEATURE SELECTION USING SPARSE CENTROID-ENCODER . By design , sparse methods identify a small number of features that accomplish a classification task . If one is interested in all the discriminatory features that can be used to separate multiple classes , then one can repeat the process of removing good features . This section describes how sparse centroid-encoder ( SCE ) can be used iteratively to extract all discriminatory features from a data set ; see O ’ Hara et al . ( 2013 ) for an application of this approach to sparse support vector machines . SCE is a model based on neural network architecture ; hence , it ’ s a non-convex optimization . As a result , multiple runs will produce different solutions , i.e. , different feature sets on the same training set . These features may not be optimal given an unseen test set . To find out the robust features from a training set , we resort to frequency-based feature pruning . In this strategy , first , we divide the entire training set into k folds . On each of these folds , we ran the SCE and picked the top N ( user select ) number of features . We repeat the process T times to get k × T feature sets . Then we count the number of occurrences of each feature and call this number the frequency of a feature . We ordered the features based on the frequency and picked the optimum number from a validation set . We present the feature selection steps in Algorithm 1 . In Figure 2 , we plotted the magnitude of the feature weights for MNIST and GSE73072 in descending order to show the ability to promote sparsity of SCE . In both cases , the model ignored many features by setting their weight to near zero . 3 EXPERIMENTAL RESULTS . Here we present a range of comparative benchmarking experiments on a variety of data sets and feature selection models . We used five disparate data sets including single-cell data ( GM12878 ) , high dimensional infectious disease data set ( GSE73072 ) , vision letter data ( MNIST ) , hyperspectral imagery ( Indian Pines ) , and GIS data ( Forest Cover ) . 3.1 EXPERIMENTAL DETAILS . We did bench-marking experiments to compare the sparse centroid-encoder with other state-of-theart feature selection methods . To make the evaluation objective , we compared the classification accuracy on the unseen data using the selected features of different models . All experiments share the following workflow : • SCE is used to select an optimal number of features on the training samples . The l1 penalty parameter λ = 0.01 for all experiments save for MNIST where λ = 0.001 . • Build K classification models with these features on the training set . We used centroidencoder as the classification model ( Aminian et al . ( 2021 ) ) . • Compute the accuracy on the sequestered test set using theK trained models and report the mean accuracy with standard deviation . 3.2 QUANTITATIVE AND QUALITATIVE ANALYSIS . Now we present the results from a comprehensive analysis across five data sets . 3.2.1 SINGLE CELL DATA . GM12878 is a single cell data set that has been previously used to test multiclass feature selection algorithms ( Li et al . ( 2016 ) ) . The samples were collected from the annotated DNA region of lymphoblastoid cell line . Each sample is represented by a 93 dimensional features sampled from three classes : active enhancer ( AE ) , active promoter ( AP ) and background ( BG ) where each class contains 2 , 156 number of samples . The data set is split equally into a separate training , validation and test sets . We used the validation set to tune hyper-parameters and to pick the optimal number of features . After the feature selection step , we merged the training and validation set and trained K = 10 centroid-encoder classifiers with the selected features , and reported classification accuracy on the test set . We use the published results for deep feature selection ( DFS ) , shallow feature selection , LASSO , and Random Forest ( RF ) from the work of Li et al . to evaluate SCE as shown in Table 1 . To compare with Li et al. , we used the top 16 features to report the mean accuracy of the test samples . We also report the test accuracy using the top 48 features picked from the validation set , this was the best result in our experiment . When restricted to the top 16 , we see that the SCE features still outperform all the other models . Among all the models , LASSO features exhibit the worst performance with an accuracy of 81.86 % . This relatively low accuracy is not surprising , given LASSO is a linear model . The classification performance gives a quantitative measure that doesn ’ t reveal the biological significance of the selected genes . We did a literature survey of the top genes selected by sparse centroid-encoder and provided the detailed description in the appendix . Some of these genes play an essential role in transcriptional activation , e.g. , H4K20ME1 ( Barski et al . ( 2007 ) ) , TAF1 ( Wang et al . ( 2014 ) ) , H3K27ME3 ( Cai et al . ( 2021 ) ) , etc . Gene H3K27AC ( Creyghton et al . ( 2010 ) ) plays a vital role in separating active enhances from inactive ones . Besides that , many of these genes are related to the proliferation of the lymphoblastoid cancer cells , e.g. , POL2 ( Yamada et al . ( 2013 ) ) , NRSF/REST ( Kreisler et al . ( 2010 ) ) , GCN5 ( Yin et al . ( 2015 ) ) , PML ( Salomoni & Pandolfi ( 2002 ) ) , etc . This survey confirms the biological significance of the selected genes .

Summary: The authors propose a novel feature selection algorithm by introducing a L1-penalty over the Centroid-Encoder loss function. The experimental results shows the algorithm can outperform DFS algorithms in certain datasets.   The main contribution of this paper is the addition of the l1-regularization to the Centroid-Encoder loss, changing it to a feature selection algorithm.