Efficient Wrapper Feature Selection using Autoencoder and Model Based Elimination
Authors: Sharan Ramjee
Aly El Gamal
Venue:  (2020)
Keywords: Wrapper Feature Selection
AMBER
Ranker Model
Generative Training
Wireless Subsampling

Abstract: We propose a computationally efficient wrapper feature selection method - called Autoencoder and Model Based Elimination of features using Relevance and Redundancy scores (AMBER) - that uses a single ranker model along with autoencoders to perform greedy backward elimination of features. The ranker model is used to prioritize the removal of features that are not critical to the classification task, while the autoencoders are used to prioritize the elimination of correlated features. We demonstrate the superior feature selection ability of AMBER on 4 well known datasets corresponding to different domain applications via comparing the accuracies with other computationally efficient state-of-the-art feature selection techniques. Interestingly, we find that the ranker model that is used for feature selection does not necessarily have to be the same as the final classifier that is trained on the selected features. Finally, we hypothesize that overfitting the ranker model on the training set facilitates the selection of more salient features.

Under review as a conference paper at ICLR 2020
EFFICIENT WRAPPER FEATURE SELECTION USING AUTOENCODER AND MODEL BASED ELIMINATION
Anonymous authors Paper under double-blind review
ABSTRACT
We propose computationally efficient wrapper feature selection method called Autoencoder and Model Based Elimination of features using Relevance and Re- dundancy scores (AMBER)- that uses a single ranker model along with autoen- coders to perform greedy backward elimination of features. The ranker model is used to prioritize the removal of features that are not critical to the classification task. while the autoencoders are used to prioritize the elimination of correlated features. We demonstrate the superior feature selection ability of AMBER on 4 well known datasets corresponding to different domain applications via com- paring the accuracies with other computationally efficient state-of-the-art feature selection techniques. Interestingly, we find that the ranker model that is used for feature selection does not necessarily have to be the same as the final classifier that is trained on the selected features. Finally, hypothesize that overfitting the ranker model on the training set facilitates the selection more salient features.
INTRODUCTION
Feature selection is preprocessing technique that ranks the significance of features to eliminate features that are insignificant to the task at hand. As examined by and Liu (2003). it is powerful tool to alleviate the curse of dimensionality, reduce training time and increase the accuracy of learn- ing algorithms, as well as to improve data comprehensibility. For classification problems, Weston eet al. (2001) divide feature selection problems into two types: (a) given fixed < d, where d is the total number of features, find the & features that lead to the least classification error and (b) given a maximum expected classification error, find the smallest possible k. In this paper, we will be focusing on problems of type (a). Weston et al. (2001) formalize this type of feature selection problems as follows. Given set of functions y [xx ar), find mapping of data x + (xr r), {0, 1]d, along with the parameters for the funcctio ff that lead to the minimization of
T( V(y, f((x ())
subject to ||a||o k, where the distribution P(r, that determines how samples are generated - is unknown, and can be inferred only from the training set, x*0 = (Ij0)...., is an elementwise product, ((,-)) -) is loss function and || 1lo is the Lo-norm
Feature selection algorithms are of 3 types: Filter, Wrapper, and Embedded methods. Filters rely on intrinsic characteristics of data to measure feature importance while wrappers iteratively mea- classifier to rank feature importance. Li et al. (2017) assert that
sure the learning performance although filters are more computationally efficient than wrappers, due to the absence of a learning algorithm that supervises the selection of features, the features selected by filters are not as good as those selected by wrappers. Embedded methods feature selection into the underlying model to reconcile the efficiency advantage of filters with the learning algorithm interaction advantage of wrappers. As examined by [Sacys al. (2007). embed- ded methods are model dependent because they perform feature selection during the training of the learning algorithm. This serves as a motivation for the use of wrapper methods that are not model de- pendent. WWsttn et al. (2001) define wrapper methods as an exploration of the feature space, where the saliency of subsets of features are ranked using the estimated accuracy of a learning algorithm. Hence, 7 7(0,00 a)) in (T) can be approximated by minimizing
the structure of learning algorithms embed
Twrap (a, ax) = min Talg(a):

---

Under review as a conference paper at ICLR 2020
subject to E {0, 1) where Falg is classifier having estimates of a. Wrappers methods can further be divided into three types: Exhaustive Search Wrappers. Random Search Wrappers, and Heuristic Search Wrappers. We will focus on Heuristic Search Wrappers that iteratively select or eliminate one feature at each iteration because unlike Exhaustive Search Wrappers, they are more computationally efficient and unlike Random Search Wrappers, they have deterministic guarantees on the set of selected salient features, as illustrated in Hira and Gillies (2015).
1.1 MOTIVATION
1.1.1 RELEVANCE AND REDUNDANCY
We hypothesize that the saliency of features is determined by two factors: Relevance and Redun- dancy. Irrelevant features are insignificant because their direct removal does result in drop in classification accuracy, while redundant features are insignificant because they are linearly or non- linearly dependent on other features and can be inferred or approximated from them as long as these other features are not removed. As shown in Fig. and detailed by Guyon et al. (2008), one does not necessarily imply the other. Fig. [ (a) shows highly redundant features (represented by and y values) that are both relevant, removal of any of the features will lead to inability to classify the data and (b) shows 2 features, where the removal of any one feature does not deter the classification ability, and thus any of them is irrelevant given the other. However, they not highly redundant as the value of any of them cannot be well approximated using the other.
(a)
(b)
Figure 1: (a): Redundant but not irrelevant; (b): Irrelevant but not redundant
Filter methods are better at identifying redundant features while wrapper methods are better at iden- tifying irrelevant features, and this highlights the power of embedded methods as they utilize aspects of both in feature selection mentioned in BolónnCanned et al. (201)). Since most wrapper meth- ods do not take advantage of filter method based identification of redundant features, there is a need to incorporate filter based technique to identify redundant features into wrapper methods, which we address using autoencoders.
1.1.2 TRAINING THE CLASSIFIER ONLY ONCE
Although wrapper methods often deliver higher classification accuracies compared to filter methods, their computational complexity often significantly higher because the classifier needs to be trained for every considered feature set at every iteration. For greedy backward climination wrappers, the removal of one out of features requires removing each feature separately and training the classifier features and testing its performance
with the remaining whose removal results in the highest classification accuracy is removed because its removal caused the least impact on performance. This is the procedure followed by most backward feature selec- tion algorithms such as the Recursive Feature Elimination (RFE) method proposed by Guyon et al.) 20022). For iterative greedy elimination of features from a set of d features, the classifier has be trained for > (d i 1) times, which poses practical limitation when the number of features is large. Also, the saliency of the features selected is governed by how good the classifier that ranks the features is, and as such, we need to use state-of of-the-art classifiers for ranking the features (CNNs for image data, etc.). These models are often complex and thus, consume lot of training time which implies a trade-off between speed and the saliency of selected features. We address this issue by training the the feature ranker model only once.
the validation set. The feature
2

---

Under review as a conference paper at ICLR 2020
2 STATE OF THE ART
In this section, we describe top-notch fast/efficient feature selection methods that we will be com- paring our proposed method to. With the exception of FQL, the implementations of these methods can be found in the scikit-feature package created by et al (2017
2.1 FISHER SCORE
The Fisher Score encourages selection features where feature values within the same class are similar and feature values belonging to different classes are distinct. Duda et al. (2012) define the Fisher Score for feature f;
FisherScore(fi) Σÿ=1 nn - ma)
(3)
Ej=1 Mgg
where is the number of classes, ny represents the number of training examples in class j. μ represents the mean value of feature fi, Mij represents the mean value of feature fi for training examples in class j. and o? represents the variance of feature fo for training examples in class j.
2.2 CMIM
Conditional Mutual Information Maximization (CMIM) a fast feature selection method proposed in Vidal-Naquet and Ullman (2003) and Fleuret 2004) that iteratively selects features while maxi- mizing the Shannon mutual information function between the feature being selected and class labels, given already selected features. Li et al. 2017) define the CMIM score for feature fi
JCMIM (f) min [T(X,; YIX)), X,Es
(4)
where S is the set of currently selected features, X; is the random variable representing the value of feature fú. and T(X; Z|Z) is the conditional mutual information between discrete random variables empirical distributions compute mutual
X and Y given random variable Z. Also, information functions based on the training set.
2.3 RFS
Efficient and Robust Feature Selection (RFS) is an efficient feature selection method proposed by Nie et al. (2010) that exploits the noise robustness property of the joint mnrrm loss function, by applying the f2,1 -nom minimization on both the loss function and its associated regularization function. Li et al. (207)) define RFS's objective function as
min W
|XW 12,,, + a W 1
where is the data matrix, Y is the one-hot label indicator matrix, W matrix indicating feature contributions to classes, and is the regularization parameter. Features are then ranked by the & norm values of the corresponding row in the optimal matrix W. The value of for our experiments was chosen by performing RFS on a wide range of values and picking the value that led to the highest accuracy on the validation set.
2.4 FQI
Feature Quality Index (FQI) is feature selection method proposed by De Rajat et al. (197)) that utilizes the output sensitivity of a learning model to changes in the input, to rank features. FQI serves the main inspiration for our proposed method and elaborated in Verikas and Bacauskiene (2002), the FQI of feature fi is computed as
- 1
(6)
where n is the total number of training examples, 0j is the output of the model when the jth training example is the input, and of is the output of the neural network when the jth training example, with the value of the tth feature set to 0, is the input.
3

---

Under review as a conference paper at ICLR 2020
3 AMBER
3.1 SENSITIVITIES OF WEIGHTS TO FEATURES
During backpropagation, higher losses in the output layer tend to manifest result of larger changes, from the optimal, in the values of the weights in the neural network. Generally, the magni- tudes of weights connected the neurons in the input layer that correspond more salient features tend to have larger magnitudes and this has been extensively documented by |Bauer Jr et al. (2000), Belue and Bauer Jr (1955,, and Priddy al. (1993). Similar to FQI, we measure the relevance of each feature by setting the input to the neuron corresponding to that feature to 0. This essentially means that the imput neuron is dead because all the weights/synapses from that neuron to the next layer will not have an impact on the output of the neural network which implies that the model that has been trained with all features will experience a degradation its ability data. Since more salient features possess weights of higher magnitude, these weights influence the output to greater extent and setting the values of more salient features to 0 in the input will result in greater degradation in the ability of the neural network to classify the input compared to when the same done for less salient features. This can be measured by the loss given in the output layer where greater loss corresponds to greater degradation. This is the basis of the Weight Based Analysis feature selection methods outlined Ly[ et al. (2006). We further note that we normalize the training set before training by setting the mean of each feature to 0 and the variance 1, that our simulation of feature removal is effectively setting the feature to its mean value for all training examples. To summarize, the pre-trained neural network ranker model prioritizes the removal of features that are non-relevant to the classification task by simulating the removal of a feature and computing the resulting loss of the model. Features whose removal results in a lower loss are less relevant and we will refer to the loss value of this model as feature's 's Relevance Score.
classify the input
3.2 AUTOENCODERS REVEAL NON-LINEAR CORRELATIONS
It is possible that the weights connected to less salient features possess high magnitudes. This can take place as these features are redundant in presence of other salient features as described in Sec. 1.1.1 Hence, we a filter based technique that is independent of learning algorithm to detect these redundant features. We experimented with methods like PCA as detailed by Witten et al. detailed by Mitra et al. (2002) but these methods revealed only linear correlations in data, which is why we introduced autoencoders into the proposed method because they reveal non-linear correlations examined by Han al. (2018), [Balin al.) (2019). and (Sakurada and Yairi (2014). To eliminate one feature from set of k features, we train the autoencoder with one hidden layer consisting of k 1 hidden neurons illustrated in Fig. 2 ussing the normalized training set.
2009) and correlation coefficients
Iepur Lager 8*
H Hidden ge!
Outout Layor R
Figure 2: Autoencoder trained eliminate feature from a set of features
This hidden layer can either be dense, LSTM, or of other types depending on the data we are dealing with. To evaluate feature, we set its corresponding values in the training set to 0 and pass the set into the autoencoder. We then take the Mean Squared Error (MSE) between the output and the original input before the values corresponding to the evaluated feature were set to 0, and perform this for each of the & features separately. The feature with the lowest MSE is the least salient feature because the other features in the latent space consisting of k neurons were able to compensate with least reconstruction error for the loss of this feature. We refer to this MSE as a feature's Redundancy Score.
4

---

Under review as a conference paper at ICLR 2020
3.3 USING TRANSFER LEARNING TO PREVENT RETRAINING
To eliminate k out of d features, we pick state-of-the neural network model for the dataset and train it on the training set using part of it as the validation set. We call this model the Ranker Model (RM) as it allows us to rank the saliency of the features. Next, we set the input for each of the d features in all the examples of the training set to O one a time in a round-robin fashion to obtain a list of d Relevance Scores after evaluating the modified training sets on the RM. Additionally, we train the autoencoder with one hidden layer consisting of d 1 hidden neurons and pass the same modified training sets through the autoencoder to obtain d Redundancy Score for each of the features. We then divide the Relevance and Redundancy Scores by their corresponding ranges so that they both contribute equally to the final decision and add the corresponding Relevance and Redundancy Scores to obtain the Saliency Score. The feature with the lowest Saliency Score is then eliminated from the training set. In the context of the RM, elimination means that that feature is permanently set to O for all the examples in the training set. Thus, we can reuse the same RM on the next iteration of AMBER. In the context of the autoencoder, climination means that that feature is permanently removed from the training set for all the examples. This entire process is done iteratively k times to eliminate features. AMBER uses the RM and autoencoders to examine both relevance and redundancy relationships among features in the training data that they are already fit for, to iteratively eliminate features. The pseudocode for AMBER described Algorirthm D It is important to note that a feature selection algorithm can be implemented either to remove specified number of features or to stop when the accuracy of the learning algorithm ceases to increase depending the application. We chose the former method to specifically take into account the scenarios when this accuracy decreases before it increases later on in the feature selection process. Further, it is not difficult to implement variant of AMBER according to the latter objective.
Algorithm 1: AMBER Algorithm for Feature Selection
Inputs: k: Number of features to be eliminated; trainSet: Training Dataset; featList: List of l eliminated features
Outputs:
function AMBER(K, trainSet)
Train state of the art RM using trainSet Initialize featl .ist to empty list
for i 1 to k: do
Set TrtrainSet as trainSet where all features in featList are set to 0 Set TrainStt as trainSet where all features in feat are removed
Train autoencoder with one hidden layer containing d units using att
for j to d - +1 1 do
Record loss of RM when dm TrainSSet is evaluated after setting feature j Set c'TrainSet as autoTrainset where feature j is set to 0
Record MSE of auto araiTrainSet and output of autoencoder when e'raiiSet is evaluated
Normalize RM losses and MSEs and add corresponding values
Sort and add lowest scoring feature to featList return (featList)
Once the final set of l features to be eliminated are determined, they are completely removed from both the training and testing sets. The final architecture is then trained on the training set consisting of d features and tested the testing set also consisting of d k features.
4 RESULTS
4.1 EXPERIMENTAL SETUP
We used a GPU server equipped with Nvidia Tesla P100 GPUs, each with 16 GB of memory and used Keras with TensorFlow backend as the environment of choice. With the exception of the RadioML2016.1 10b dataset for which we used all GPUs, we only used 1 GPU for training. The experiments were performed times and the average accuracies were plotted each feature count in Fig. 3 The source code for AMBER, links to the datasets considered, and the error bars for the comparison plots are available at https: / com/ combe/mmerr-a. r- ic1r/AMBER /MMBER
5

---

Under review as a conference paper at ICLR 2020
4.2 DATASETS
Each dataset corresponds to a different domain of data and encompasses both low and high dimen- sional data to demonstrate the versatility of AMBER. The final models that are trained on the set of selected features are common across all the feature selection methods that are compared and are trained until early stopping is achieved with a patience value of 5 to ensure that the comparisons are fair. For all the datasets, the softmax activation function is applied to the output layer with the cross- entropy loss function. The test split used for the Reuters and the Wisconsin Breast Cancer dataset is 0.2 while the test split used for the RadioML 6. 10b dataset is 0.5. The plots in Fig. [3 were jagged when feature counts in decrements of 1 were plotted and thus, in the interest of readability, we plot- ted them in larger feature count decrements as specified for each dataset. Finally, to demonstrate that the final model does not necessarily have to be the same as the RM used by AMBER, we used different models as the final model and the RM for the MNIST and RadioML2016.10 60b datasets.
MNIST is an image dataset created by LeCun et al. 1998) consisting of 60000 28x28 grayscale images with 10 classes, each belonging one of the 10 digits, along with a test set that contains 10000 images of the same dimensions. The total number of features here is 784. The Ranker Model here is CNN consisting of 2 convolutional layers, max pooling layer, and 2 dense layers, in that order. The convolutional layers have 32 and 64 filters, in order of depth, with kernel sizes of (3x3) for both layers. The max pooling layer has a of pool size of (2x2) and the dense layers have 128 and 10 (output layer) neurons, with Softmax activation for the output layer. ReLU is applied to the remaining layers. The final model used is an MLP model consisting of 3 fully connected layers with 512, 512, and 10 (output layer) neurons. ReL.U is applied to each of the layers with 512 neurons and these layers are followed by dropout layers with a dropout rate of 0.2.
Reuters is text dataset from the Keras built-in datasets that consists of 11228 newswires from Reuters with 46 classes, each representing a different topic. Each wire is encoded as sequence of word indices, where the index corresponds to word's frequency in the dataset. For our demonstra- tion, the 1000 most frequent words will be used and thus, the total number of features is 1000. The Ranker Model and the final model are the same MLP model consisting of 2 fully connected layers with 512 and 46 (output layer) neurons with ReLU and Softmax activation functions, respectively.
Wisconsin Breast Cancer is a biological dataset created by Street et al. (1993) that consists of features that represent characteristics of cell nuclei that have been measured from an image of Fine Needle Aspirates (FNAs) of breast mass. The dataset consists of 569 examples that belong to 2 classes: malignant or benign. The total number of features here is 30. The Ranker Model and the final model are the same MLP model consisting of 4 fully connected layers with 16, 8, 6. and (output layer) neurons, in order of depth. Softmax is applied to the output layer while ReLU is applied to the remaining layers.
RadioML22016.100 is a datset of signal samples used by O' O'Shea et al. (016) that consists of 1200000 128-sample complex time-domain vectors with 10 classes, representing different modu- lation types. It consists of 20 Signal to Noise Ratio (SNR) values ranging from -20 dB to 18 dB in increments of dB; we only choose the results of the 18 dB data for better illustration. Each of the 128 samples consists of real part and complex part and thus, the input dimensions are 2x 1288, where the total number of features is 256. This dataset has the unique property that only pairs of features (belonging to the same sample) can be climinated. AMBER, like FQI. is powerful in such situations as the pairs of features can be set to 0 to evaluate their collective rank. This is also useful in the case of GANs, where sets of pixels/features in 2-D pool need to be evaluated craft adver- sarial attacks as elaborated by |Papernot et al. (2016). The other feature selection methods fail in this case because they account for feature interactions between the pairs of features as well, which is one reason for why AMBER outperforms them as it does not. For the other methods, to eliminate pairs of features belonging to the same sample, we simply added the scores belonging to the two features to obtain single score for each sample. The Ranker Model used here is a CLDNN while the final model used is a ResNet, both of which are described in Ramjee et al. (2019).
4.3 CLASSIFICATION ACCURACIES
The final models' classification accuracy plots using the selected features can be observed in Fig. 3 We observe the impressive performance delivered by AMBER that generally outperforms that of all 4 considered methods, particularly when the number of selected features becomes very low (about
6

---

Under review as a conference paper at ICLR 2020
99% average accuracy with 4 out of 30 features for the Cancer dataset and about 95% average accu- racy with 16 out of 128 samples for the RadioML dataset). The comparisons of the accuracies of the final models using the top 25% and 10% of features given in Table The results in the last two rows refer to using version AMBER without the Autoencoder's redundancy score, and another version of AMBER where the ranker model retrained in every iteration, respectively. Note from the depicted results (purple curve in the figures) how solely relying on the RM significantly reduces AMBER's performance, which validates our intuition about the benefit of using the Autoencoder to capture correlations to reduce the generalization error. Further, we observe how negligible gains are achieved when retraining the RM in every iteration, which comes at a significant computational cost, as demonstrated in Table 2 which validates our intuition about simulating the removal of features without retraining for computational efficiency while maintaining good performance.
Table 1: Accuracy Comparisons
Method
Avg. accuracy with top 25% features (%)
Avg. accuracy with top 10% features (%)
MNIST
Reuters
Cancer
RadioML
MNIST
Reuters
Cancer
RadioML
Fisher
96.28
68.00
96.78
82.96
88.37
51.21
92.40
73.42
CMTM
98.06
74.14
95.61
80.74
96.38
71.04
90.64
70.22
RFS
97.17
78.94
92.40
87.44
89.46
77.11
91.81
75.85
FQI
97.90
73.59
87.13
91.70
95.49
68.20
76.32
83.57
AMBER
98.08
79.84
97.37
98.78
97.21
77.55
96.78
95.15
Relevance
97.38
76.60
94.04
95.21
93.29
73.45
89.65
89.54
Retraining
98.37
81.25
98.25
99.73
97.21
78.11
97.37
97,49
CHIN
.000
50C
200
500
510
400
300
(a)
(b)
Pain
DMM
Runber df ofA)
c1.)
(c)
(d)
Accuracy Feature Count plots for the final models trained with the selected features for the (a) MNIST, (b) Reuters, (c) Wisconsin Breast Cancer, and (d) RadioML.2016 10b datasets.
Figure 3:
Table 2: Time needed rank all features in Seconds.
Method AMBER
MNIST
Reuters
Cancer
RadioML
10552.24
21710.78
40.04
26417.53
Retraining
24202.66
29005.08
739.01
42533.27
7

---

Under review as a conference paper at ICLR 2020
5 DISCUSSION
In Sec. and as evidenced by Wang et al. (2004), we elaborated on how more salient features possess higher magnitudes of weights in the input layer than features that are less salient, which is the property of neural networks that serves as the basis for AMBER. The performance of AMBER heavily depends on the performance of the RM that ranks the features. In some cases, however, even the state-of-the-an art models do not have high classification accuracies. In such cases, we can obtain better feature selection results with AMBER by overfitting the RM on the training set.
(b)
(a)
(c)
Figure 4: (a): Toy example from TensorFlow Playground; (b) Feature 1 and (c) Feature 2.
We will demonstrate this using the toy example illustrated in Fig. 4 that portrays the architecture used for the RM along with the corresponding hyperparameters. Feature is Ji and feature T22. Feature 2 is more salient than feature 1 as it is able to form a decision boundary that allows for better classification (see (b) and (c)). Each of these features have 3 weights in the input layer and as expected, the weights connected to feature 2 manifest into weights of higher average magnitude than those belonging to feature as shown in Fig. []] As the number of training epochs increases, the difference in the average magnitudes of the weights increases. This implies that the RM will be able to better rank the saliency of features as the difference between the Relevance Scores of more and less salient features increases. Thus, can overfit the RM on the training set by training it for a large number of epochs without regularization to enable better feature selection.
(a)
(b)
(c)
Figure 5: Input layer weight magnitudes after training for (a) 10, (b) 100, and (c) 1000 epochs.
6 CONCLUDING REMARKS
AMBER presents valuable balance in the trade-off between computational efficiency in feature selection, in which filter-based methods excel at, and performance (i.c. classification accuracy), in which traditional wrapper methods excel at. It is inspired by FQI with two major differences: 1- Instead of making the final selection of the desired feature set based on simulating the model's performance with elimination of only single feature, the final model's performance in AMBER is simulated with candidate combinations of selected features, 2- The autoencoder is used to capture redundant features; a property that is missing in FQI as well as most wrapper feature selection meth- ods. However, we found AMBER require slightly larger computational time than the considered state-of th-the-art methods, and we also found it to require far less time than tfate-ofthhe-e wrapper feature selection methods, it does not require retraining the RM in each iteration.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Muhammed Fatih Bahın, Abubakar Abid, and James Zou. Concrete autoencoders: Differentiable feature selec- tion and reconstruction. International Conference on Machine Learning, pages 4434453, 2019.
Kenneth W Bauer Jr, Stephen G Alsing, and Kelly A Greene. Feature screening using signal-to-noise ratios. Neurocomputing. 31(1-4):29-44, 2000.
Lisa M Belue and Kenneth W Bauer Jr. Determining input features for multilayer perceptrons. Neurocomputing, 7(2):111-121 1995.
Verónica Bolón-Canedo, Noelia Sánchez-Maroñoo, and Amparo Alonso-Betande A review of feature selection methods on synthetic data. Knowledge and information systems, 34(3):483-51) 2013.
Rajat, Nikhil R Pal, Sankar K Pal. Feature analysis: Neural network and fuzzy set theoretic ap- proaches. Patlern Recognition, 30( 10X100:1579-1990) 1997.
Richard Duda, Peter Hart, and David G Stork. Pattern classification. John Wiley & Sons, 2012.
François Fleuret. Fast binary feature selection with conditional mutual information. Journal of Machine learn- ing research, 5(Nov):1531 115515 2004.
Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. Gene selection for cancer classification using support vector machines. Machine learning, 46(1-3):389-44) 422,,2222222 2002.
Isabelle Guyon, Steve Gunn, Masoud Nikravesh, and Lofti A Zadeh. Feature extruction: foundations and applications, volume 207. Springer, 2008.
Kai Han, Yunhe Wang, Chao Zhang, Chao Li, and Chao Xu. Autoencoder inspired unsupervised feature selection. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2941-2945. IEFE, 2018.
Zena Hira and Duncan Gillies. A review of feature selection and feature extraction methods applied on microarray data. Advances bioinformatics, 2015, 2015.
Thomas Navin Lal, Olivier Chapelle, Jason Weston, and André Elissceff. Embedded methods. In Feature extraction, pages 137-165. Springer, 2006.
Yann LeCun, Corinna Cortes, and J.C. Burges, Christopher. The MNIST database of handwritten digits. http://yann. lecun. com/exdb/mnist// 1998.
Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P. Trevino, Jiliang Tang, and Huan Liu. Feature selection. ACM Computing Surveys, 50(6):145, Dec 2017. ISSN 0360-0300, doi: 10.1145/3136625.
hRtp:ttp://xx....
Pabitra Mitra, CA Murthy, and Sankar K. Pal. Unsupervised feature selection using feature similarity. IEEE
transactions on pattern analysis and machine intelligence, 24(3):01-312, 2002.
Feiping Nie, Heng Huang, Xiao Cai, and Chris Ding. Efficient and robust feature selection via joint 2, I-norms minimization. In Advances in neural information processing systems, pages 1813-1821, 2010.
Shea, Corgan, and Clancy. Convolutional radio modulation recognition networks, In Proc. Interna- tional conference on engineering applications of neural nenworks, 2016.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson. Z Berkay Celik, and Ananthram Swami. The limitations of deep learning adversarial settings. 2016 IEEE European Symposium on Security and Privacy (EuroS&P), pages 372-387. IEBE, 2016.
Kevin L. Priddy, Steven K Rogers, Dennis W Ruck, Gregory Tarr, and Matthew Kabrisky. Bayesian selection of important features feedforward neural networks. Neurocomputing, 5(2-3):91-100 1993.
Sharan Ramjee, Shengtai Ju. Diyu Yang. Xiaoyu Liu, Aly El Gamal, and Yonina C Eldar. Fast deep learning for automatic modulation classification. arXiv preprint arXiv:1901.0588. 2019.
Yvan Sacys, Iñaki Inza, and Pedro Larraaaga. A review of feature selection techniques in bioinformatics.
bioinformatics, 23 23(19):2507-251 2007.
Mayu Sakurada and Takehisa Yairi. Anomaly detection using autoencoders with nonlinear dimensionality reduction. In Proceedings of the MLSDA 2014 2Nd Workshop on Machine Learning for Sensory Data Analysis, MI 14, pages 4:4-4:11, New York, NY, USA, 2014. ACM. ISBN 978- 4503-331599- doi: 10. 10.1 00.1145/2689746.6668 URL htRt:ttp:://// / acm. org/ CC. 6 7

---

Under review as a conference paper at ICLR 2020
W Nick Street, William Н Wolberg, and Olvi L Mangasarian. Nuclear feature extraction for breast tumor diagnosis. Biomedical image processing and biomedical visualization, volume 1905, pages 861-871. International Society for Optics and Photonics, 1993.
Antanas Verikas and Marija Bacauskiene. Feature selection neural networks. Pattern Recognition Letters, 23(11)) (3(11):1323-133, 2002.
Michel Tidal-Naquet and Shimon Ullman. Object recognition with informative features and linear classifica- tion. In ICCV, volume 3, page 281, 2003.
Xizhao Wang, Yadong Wang, and Lijuan Wang. Improving fuzzy c-means clustering based on feature-weight learning. Pattern recognition letters, (2110)11133111 2004.
Jason Weston, Sayan Mukherjee, Olivier Chapelle, Massimiliano Pontil, Tomaso Poggio, Vladimir Vapnik. Feature selection for SVMs. In Advances in neural information processing systems, pages 0686674, 2001.
Daniela Witten, Robert Tibshirani, Trevor Hastie. A penalized matrix decomposition. with applications to sparse principal components and canonical correlation analysis. Biostatistics, 10(3):515 535444,,, 2009.
Lei and Huan Liu. Feature selection high-dimensional data: fast correlation based filter solution. In Proceedings of the 20th international conference on machine learning (ICML-03), pages 856-863, 2003.
10


Reviews:
[{"review_id": "r1g9DMyP9B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The article \"Efficient Wrapper Feature Selection using Autoencoder and Model Based Elimination\" considers the problem of feature selection for a broad class of machine learning models. The authors argue that it is important to consider the relevance of features for the considered supervised ML problem and redundancy of features. They propose the wrapper feature selection method based on this paradigm and report the results of the experimental comparison of the method with some approaches from the literature.\n\nThe proposed AMBER approach consists of 2 parts:\n1. The ranking of features based on the sensitivity of some supervised ML model with respect to the particular feature.\n2. The ranking of features with respect to their individual impact on the accuracy of the autoencoder trained on the features of the training data set.\n\nThe scores obtained on these 2 steps are added and the algorithm iteratively removes features with the lowest total score.\n\nI should note that the proposed approach is very general, but the paper gives very few details on the actual implementation. For example, it seems important to properly normalize relevance and redundancy scores before computing the final score but the paper doesn't discuss this issue. Also, there are many possible ways to compute losses. For example, one can use training or validation sets for that but the authors choose the training set without motivation. \n\nMost importantly, the experimental part of the paper considers just 4 datasets. I believe that the algorithms of such generality should be evaluated on a much broader selection of problems. The most important thing is whether it is possible to select hyperparameters of the method in a way that few human interventions are needed to achieve high-quality results.\n\nOverall, I am very concerned with making the particular instance of the proposed approach working on a vast selection of applied problems. The provided repository with code confirms my concerns as it doesn't provide the single algorithm but rather the collection of scripts tailored for particular problems considered.\n\nTo sum up, I think that while the motivation behind the paper is very natural, I am not convinced with experimental results and the overall applicability of the approach.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SylKDMA2FB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors present an iterative approach for feature selection which selects features based both on the relevance and redundancy of each feature. The relevance of each feature is determined using a mild variant of the Feature Quality Index; essentially, the relevance is computed as the loss in model performance when setting each feature value to the mean and measuring the change in performance. Similarly, the redundancy of each feature is determined by comparing the reconstruction loss of an autoencoder when setting the feature value to its mean for all training samples. These two values are combined to give a single score for each feature at each iteration. The feature with the worst value is removed. A limited set of experiments suggests the proposed approach mildly outperforms other efficient feature selection methods.\n\nMajor comments\n\nThe paper does not include relevant, recent work on using autoencoders for feature selection, such as [Han et al., ICASSP 2018; Balın et al., ICML 2019], among others. Thus, it is difficult to discern how this paper either theoretically or empirically advances the state of the art.\n\nI found the proposed approach to efficient feature selection reasonable. However, there is no theoretical justification for the approach. Thus, I would expect a thorough empirical analysis. Only a few limited experiments on toy datasets (and one slightly more challenging one) are given.\n\nThe paper is not well-written. For example, it seems as though the proposed approach is not applicable to datasets with categorical features. It is not obvious (and, presumably, would need to be shown empirically) if the mode could be used to replace categorical values analogously to how the mean is used for real-valued features. Alternatively, one could imagine one-hot encoding the categorical variables and grouping them in some manner similar to that used for the RadioML pairs (since the one-hot values are obviously highly correlated). However, the authors do not address these issues.\n\nSimilarly, the entire discussion in Section 3 seems to assume the ranker model will be some sort of neural network. However, as far as I can tell, the ranker model is treated as a black box, so it could easily be some random forest model, etc. If there are some implicit assumptions that the ranker model is a neural network, this should be made explicit; if not, the discussion should be revised (and, of course, non-neural models should be used in the experiments).\n\nThe approach seems to heavily depend on the ability of the autoencoder to reconstruct the input; however, it is unclear how the structure/capacity of the autoencoder affects the performance of the algorithm. For example, the authors propose a relatively simple structure, presumably to maintain computational efficiency. It would be interesting to explore more deeply how autoencoders with more capacity impact the results.\n\nIt is unclear why the autoencoder is retrained at each step compared to just setting the removed feature values to the respective means, as is done with the ranker model.\n\nClearly, the relevance and redundancy scores could be weighted unequally when selecting the feature to remove. It would be interesting to explore how different combinations affect the results.\n\nIt seems that the experiments only consider backward feature selection approaches. Including forward feature selection approaches would add useful context for how the proposed approach compares to other strategies.\n\nMinor comments\n\nThe cross-validation scheme used is not clear. While the authors mention that three runs are used to estimate performance variance, they do not describe if this is 3-fold cross validation, some Monte Carlo cross validation, or if the same splits are used all three times and just the random seeds are different.\n\nWhile methods like RFE have significantly higher computational cost than the methods considered here, it would be helpful to include it for at least one of the datasets to provide context on how much the less costly methods “lose”.\n\nWhat is the overlap in the selected features? both among the different methods and among the different folds for the same method.\n\nHow were the hyperparameters for the various models chosen?\n\nTypos, etc.\n\nThe references are not consistently formatted.\n\nThe Section 2 headers all have an unnecessary “0” in them (e.g., “2.0.1”).\n\nTable 1 should include the standard deviations.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gEqkE0qr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a wrapper feature selection method AMBER to use a single ranker model along with autoencoders to perform greedy backward elimination of features. Experimental results on various datasets show that their criterion outperforms other baseline methods. Generally, the paper is well written and easy to follow. However, the idea is simple and the originality seems incremental. \n\nFirst, although taking advantage of the power of neural network to help select better features sounds interesting, it is important for the author to discuss the benefit of doing it. As mentioned by the author, neural network essentially has the ability of selecting features. Instead of selecting features with AMBER explicitly, a more straightforward way is using all features as input and solving the downstream task with deep learning model. Feature selection will be automatically conducted during the learning process. It will be better if the author can explain more about the benefit and motivation of introducing feature selection explicitly in this scenario.\n\nSecond, the author proposed to use an autoencoder with one hidden layer consisting of d−1 hidden neurons to calculate feature’s redundancy score. Is there any specific reason for including d-1 hidden neurons. It will be better if the author can give some theoretical analysis of it.\n\nThird, the author calculates the redundancy score and relevance score independently and combine them together to obtain the saliency score. However, it seems unreasonable to regard relevance and redundancy as two independent factors and stiffly combine them. For example, a feature can be both relevant and redundant. Should we eliminate it? How the proposed method solve this case? \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper the authors propose a wrapper feature selection method that selects features based on 1) redundancy, i.e. the sensitivity of the downstream model to feature elimination, and 2) relevance, i.e. how the individual features impact the accuracy of the target task. The authors use a combination of the redundancy and relevance scores to eliminate the features. 

While acknowledging that the proposed model is potentially useful, the reviewers raised several important concerns that were viewed by AC as critical issues:
(1) all reviewers agreed that the proposed approach lacks theoretical justification or convincing empirical evaluations in order to show its effectiveness and general applicability -- see R1’s and R2’s requests for evaluation with more datasets/diverse tasks to assess the applicability and generality of the proposed model; see R1’s, R4’s concerns regarding theoretical analysis; 
(2) all reviewers expressed concerns regarding the technical issue of combining the redundancy and relevance scores -- see R4’s and R2’s concerns regarding the individual/disjoint calibration of scores; see R1’s suggestion to learn to reweigh the scores;
(3) experimental setup requires improvement both in terms of clarity of presentation and implementation -- see R1’s comment regarding the ranker model, see R4’s concern regarding comparison with a standard deep learning model that does feature learning for a downstream task; both reviewers also suggested to analyse how autoencoders with different capacity could impact the results.
Additionally R1 raised a concern regarding relevant recent works that were overlooked. 
The authors have tried to address some of these concerns during rebuttal, but an insufficient empirical evidence still remains a critical issue of this work. To conclude, the reviewers and AC suggest that in its current state the manuscript is not ready for a publication. We hope the reviews are useful for improving and revising the paper.

Decision: Reject