Efficient Wrapper Feature Selection using Autoencoder and Model Based Elimination

1 INTRODUCTION . Feature selection is a preprocessing technique that ranks the significance of features to eliminate features that are insignificant to the task at hand . As examined by Yu and Liu ( 2003 ) , it is a powerful tool to alleviate the curse of dimensionality , reduce training time and increase the accuracy of learning algorithms , as well as to improve data comprehensibility . For classification problems , Weston et al . ( 2001 ) divide feature selection problems into two types : ( a ) given a fixed k d , where d is the total number of features , find the k features that lead to the least classification error and ( b ) given a maximum expected classification error , find the smallest possible k. In this paper , we will be focusing on problems of type ( a ) . Weston et al . ( 2001 ) formalize this type of feature selection problems as follows . Given a set of functions y = f ( x , α ) , find a mapping of data x 7→ ( x ∗ σ ) , σ ∈ { 0 , 1 } d , along with the parameters α for the function f that lead to the minimization of τ ( σ , α ) = ∫ V ( y , f ( ( x ∗ σ ) , α ) ) dP ( x , y ) , ( 1 ) subject to ‖σ‖0 = k , where the distribution P ( x , y ) - that determines how samples are generated - is unknown , and can be inferred only from the training set , x∗σ = ( x1σ1 , . . . , xdσd ) is an elementwise product , V ( · , · ) is a loss function and ‖ · ‖0 is the L0-norm . Feature selection algorithms are of 3 types : Filter , Wrapper , and Embedded methods . Filters rely on intrinsic characteristics of data to measure feature importance while wrappers iteratively measure the learning performance of a classifier to rank feature importance . Li et al . ( 2017 ) assert that although filters are more computationally efficient than wrappers , due to the absence of a learning algorithm that supervises the selection of features , the features selected by filters are not as good as those selected by wrappers . Embedded methods use the structure of learning algorithms to embed feature selection into the underlying model to reconcile the efficiency advantage of filters with the learning algorithm interaction advantage of wrappers . As examined by Saeys et al . ( 2007 ) , embedded methods are model dependent because they perform feature selection during the training of the learning algorithm . This serves as a motivation for the use of wrapper methods that are not model dependent . Weston et al . ( 2001 ) define wrapper methods as an exploration of the feature space , where the saliency of subsets of features are ranked using the estimated accuracy of a learning algorithm . Hence , τ ( σ , α ) in ( 1 ) can be approximated by minimizing τwrap ( σ , α ) = min σ τalg ( σ ) , ( 2 ) subject to σ ∈ { 0 , 1 } d , where τalg is a classifier having estimates of α. Wrappers methods can further be divided into three types : Exhaustive Search Wrappers , Random Search Wrappers , and Heuristic Search Wrappers . We will focus on Heuristic Search Wrappers that iteratively select or eliminate one feature at each iteration because unlike Exhaustive Search Wrappers , they are more computationally efficient and unlike Random Search Wrappers , they have deterministic guarantees on the set of selected salient features , as illustrated in Hira and Gillies ( 2015 ) . 1.1 MOTIVATION . 1.1.1 RELEVANCE AND REDUNDANCY . We hypothesize that the saliency of features is determined by two factors : Relevance and Redundancy . Irrelevant features are insignificant because their direct removal does not result in a drop in classification accuracy , while redundant features are insignificant because they are linearly or nonlinearly dependent on other features and can be inferred - or approximated - from them as long as these other features are not removed . As shown in Fig . 1 and detailed by Guyon et al . ( 2008 ) , one does not necessarily imply the other . Fig . 1 ( a ) shows 2 highly redundant features ( represented by x and y values ) that are both relevant , as removal of any of the features will lead to an inability to classify the data and ( b ) shows 2 features , where the removal of any one feature does not deter the classification ability , and thus any of them is irrelevant given the other . However , they are not highly redundant as the value of any of them can not be well approximated using the other . Filter methods are better at identifying redundant features while wrapper methods are better at identifying irrelevant features , and this highlights the power of embedded methods as they utilize aspects of both in feature selection as mentioned in Bolón-Canedo et al . ( 2013 ) . Since most wrapper methods do not take advantage of filter method based identification of redundant features , there is a need to incorporate a filter based technique to identify redundant features into wrapper methods , which we address using autoencoders . 1.1.2 TRAINING THE CLASSIFIER ONLY ONCE . Although wrapper methods often deliver higher classification accuracies compared to filter methods , their computational complexity is often significantly higher because the classifier needs to be trained for every considered feature set at every iteration . For greedy backward elimination wrappers , the removal of one out of d features requires removing each feature separately and training the classifier with the remaining d − 1 features and testing its performance on the validation set . The feature whose removal results in the highest classification accuracy is removed because its removal caused the least impact on performance . This is the procedure followed by most backward feature selection algorithms such as the Recursive Feature Elimination ( RFE ) method proposed by Guyon et al . ( 2002 ) . For iterative greedy elimination of k features from a set of d features , the classifier has to be trained for ∑k i=1 ( d− i+1 ) times , which poses a practical limitation when the number of features is large . Also , the saliency of the features selected is governed by how good the classifier that ranks the features is , and as such , we need to use state-of-the-art classifiers for ranking the features ( CNNs for image data , etc. ) . These models are often complex and thus , consume a lot of training time which implies a trade-off between speed and the saliency of selected features . We address this issue by training the the feature ranker model only once . 2 STATE OF THE ART . In this section , we describe top-notch fast/efficient feature selection methods that we will be comparing our proposed method to . With the exception of FQI , the implementations of these methods can be found in the scikit-feature package created by Li et al . ( 2017 ) . 2.1 FISHER SCORE . The Fisher Score encourages selection of features where feature values within the same class are similar and feature values belonging to different classes are distinct . Duda et al . ( 2012 ) define the Fisher Score for feature fi as FisherScore ( fi ) = ∑c j=1 nj ( µij − µi ) 2∑c j=1 njσ 2 ij , ( 3 ) where c is the number of classes , nj represents the number of training examples in class j , µi represents the mean value of feature fi , µij represents the mean value of feature fi for training examples in class j , and σ2ij represents the variance of feature fi for training examples in class j . 2.2 CMIM . Conditional Mutual Information Maximization ( CMIM ) is a fast feature selection method proposed in Vidal-Naquet and Ullman ( 2003 ) and Fleuret ( 2004 ) that iteratively selects features while maximizing the Shannon mutual information function between the feature being selected and class labels , given already selected features . Li et al . ( 2017 ) define the CMIM score for feature fi as JCMIM ( fi ) = min Xj∈S [ I ( Xi ; Y |Xj ) ] , ( 4 ) where S is the set of currently selected features , Xi is the random variable representing the value of feature fi , and I ( X ; Y |Z ) is the conditional mutual information between discrete random variables X and Y given a random variable Z . Also , we use empirical distributions to compute mutual information functions based on the training set . 2.3 RFS . Efficient and Robust Feature Selection ( RFS ) is an efficient feature selection method proposed by Nie et al . ( 2010 ) that exploits the noise robustness property of the joint ` 2,1-norm loss function , by applying the ` 2,1-norm minimization on both the loss function and its associated regularization function . Li et al . ( 2017 ) define RFS ’ s objective function as min W ‖XW − Y ‖2,1 + α‖W‖2,1 , ( 5 ) where X is the data matrix , Y is the one-hot label indicator matrix , W is a matrix indicating feature contributions to classes , and α is the regularization parameter . Features are then ranked by the ` 2 norm values of the corresponding row in the optimal matrix W . The value of α for our experiments was chosen by performing RFS on a wide range of values and picking the value that led to the highest accuracy on the validation set . 2.4 FQI . Feature Quality Index ( FQI ) is a feature selection method proposed by De Rajat et al . ( 1997 ) that utilizes the output sensitivity of a learning model to changes in the input , to rank features . FQI serves as the main inspiration for our proposed method and as elaborated in Verikas and Bacauskiene ( 2002 ) , the FQI of feature fi is computed as FQI ( fi ) = n∑ j=1 ‖oj − oij‖2 , ( 6 ) where n is the total number of training examples , oj is the output of the model when the jth training example is the input , and oij is the output of the neural network when the j th training example , with the value of the ith feature set to 0 , is the input . 3 AMBER . 3.1 SENSITIVITIES OF WEIGHTS TO FEATURES . During backpropagation , higher losses in the output layer tend to manifest as a result of larger changes , from the optimal , in the values of the weights in the neural network . Generally , the magnitudes of weights connected to the neurons in the input layer that correspond to more salient features tend to have larger magnitudes and this has been extensively documented by Bauer Jr et al . ( 2000 ) , Belue and Bauer Jr ( 1995 ) , and Priddy et al . ( 1993 ) . Similar to FQI , we measure the relevance of each feature by setting the input to the neuron corresponding to that feature to 0 . This essentially means that the input neuron is dead because all the weights/synapses from that neuron to the next layer will not have an impact on the output of the neural network which implies that the model that has been trained with all features will experience a degradation in its ability to classify the input data . Since more salient features possess weights of higher magnitude , these weights influence the output to a greater extent and setting the values of more salient features to 0 in the input will result in a greater degradation in the ability of the neural network to classify the input compared to when the same is done for less salient features . This can be measured by the loss given in the output layer where a greater loss corresponds to a greater degradation . This is the basis of the Weight Based Analysis feature selection methods outlined by Lal et al . ( 2006 ) . We further note that we normalize the training set before training by setting the mean of each feature to 0 and the variance to 1 , so that our simulation of feature removal is effectively setting the feature to its mean value for all training examples . To summarize , the pre-trained neural network ranker model prioritizes the removal of features that are non-relevant to the classification task by simulating the removal of a feature and computing the resulting loss of the model . Features whose removal results in a lower loss are less relevant and we will refer to the loss value of this model as a feature ’ s Relevance Score .

Summary: In this paper, the authors present an iterative approach for feature selection which selects features based both on the relevance and redundancy of each feature. The relevance of each feature is determined using a mild variant of the Feature Quality Index; essentially, the relevance is computed as the loss in model performance when setting each feature value to the mean and measuring the change in performance. Similarly, the redundancy of each feature is determined by comparing the reconstruction loss of an autoencoder when setting the feature value to its mean for all training samples. These two values are combined to give a single score for each feature at each iteration. The feature with the worst value is removed. A limited set of experiments suggests the proposed approach mildly outperforms other efficient feature selection methods.