Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a wrapper feature selection method AMBER to use a single ranker model along with autoencoders to perform greedy backward elimination of features. Experimental results on various datasets show that their criterion outperforms other baseline methods. Generally, the paper is well written and easy to follow. However, the idea is simple and the originality seems incremental. 

First, although taking advantage of the power of neural network to help select better features sounds interesting, it is important for the author to discuss the benefit of doing it. As mentioned by the author, neural network essentially has the ability of selecting features. Instead of selecting features with AMBER explicitly, a more straightforward way is using all features as input and solving the downstream task with deep learning model. Feature selection will be automatically conducted during the learning process. It will be better if the author can explain more about the benefit and motivation of introducing feature selection explicitly in this scenario.

Second, the author proposed to use an autoencoder with one hidden layer consisting of d−1 hidden neurons to calculate feature’s redundancy score. Is there any specific reason for including d-1 hidden neurons. It will be better if the author can give some theoretical analysis of it.

Third, the author calculates the redundancy score and relevance score independently and combine them together to obtain the saliency score. However, it seems unreasonable to regard relevance and redundancy as two independent factors and stiffly combine them. For example, a feature can be both relevant and redundant. Should we eliminate it? How the proposed method solve this case? 


review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: The article "Efficient Wrapper Feature Selection using Autoencoder and Model Based Elimination" considers the problem of feature selection for a broad class of machine learning models. The authors argue that it is important to consider the relevance of features for the considered supervised ML problem and redundancy of features. They propose the wrapper feature selection method based on this paradigm and report the results of the experimental comparison of the method with some approaches from the literature.

The proposed AMBER approach consists of 2 parts:
1. The ranking of features based on the sensitivity of some supervised ML model with respect to the particular feature.
2. The ranking of features with respect to their individual impact on the accuracy of the autoencoder trained on the features of the training data set.

The scores obtained on these 2 steps are added and the algorithm iteratively removes features with the lowest total score.

I should note that the proposed approach is very general, but the paper gives very few details on the actual implementation. For example, it seems important to properly normalize relevance and redundancy scores before computing the final score but the paper doesn't discuss this issue. Also, there are many possible ways to compute losses. For example, one can use training or validation sets for that but the authors choose the training set without motivation. 

Most importantly, the experimental part of the paper considers just 4 datasets. I believe that the algorithms of such generality should be evaluated on a much broader selection of problems. The most important thing is whether it is possible to select hyperparameters of the method in a way that few human interventions are needed to achieve high-quality results.

Overall, I am very concerned with making the particular instance of the proposed approach working on a vast selection of applied problems. The provided repository with code confirms my concerns as it doesn't provide the single algorithm but rather the collection of scripts tailored for particular problems considered.

To sum up, I think that while the motivation behind the paper is very natural, I am not convinced with experimental results and the overall applicability of the approach.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: In this paper, the authors present an iterative approach for feature selection which selects features based both on the relevance and redundancy of each feature. The relevance of each feature is determined using a mild variant of the Feature Quality Index; essentially, the relevance is computed as the loss in model performance when setting each feature value to the mean and measuring the change in performance. Similarly, the redundancy of each feature is determined by comparing the reconstruction loss of an autoencoder when setting the feature value to its mean for all training samples. These two values are combined to give a single score for each feature at each iteration. The feature with the worst value is removed. A limited set of experiments suggests the proposed approach mildly outperforms other efficient feature selection methods.

Major comments

The paper does not include relevant, recent work on using autoencoders for feature selection, such as [Han et al., ICASSP 2018; Balın et al., ICML 2019], among others. Thus, it is difficult to discern how this paper either theoretically or empirically advances the state of the art.

I found the proposed approach to efficient feature selection reasonable. However, there is no theoretical justification for the approach. Thus, I would expect a thorough empirical analysis. Only a few limited experiments on toy datasets (and one slightly more challenging one) are given.

The paper is not well-written. For example, it seems as though the proposed approach is not applicable to datasets with categorical features. It is not obvious (and, presumably, would need to be shown empirically) if the mode could be used to replace categorical values analogously to how the mean is used for real-valued features. Alternatively, one could imagine one-hot encoding the categorical variables and grouping them in some manner similar to that used for the RadioML pairs (since the one-hot values are obviously highly correlated). However, the authors do not address these issues.

Similarly, the entire discussion in Section 3 seems to assume the ranker model will be some sort of neural network. However, as far as I can tell, the ranker model is treated as a black box, so it could easily be some random forest model, etc. If there are some implicit assumptions that the ranker model is a neural network, this should be made explicit; if not, the discussion should be revised (and, of course, non-neural models should be used in the experiments).

The approach seems to heavily depend on the ability of the autoencoder to reconstruct the input; however, it is unclear how the structure/capacity of the autoencoder affects the performance of the algorithm. For example, the authors propose a relatively simple structure, presumably to maintain computational efficiency. It would be interesting to explore more deeply how autoencoders with more capacity impact the results.

It is unclear why the autoencoder is retrained at each step compared to just setting the removed feature values to the respective means, as is done with the ranker model.

Clearly, the relevance and redundancy scores could be weighted unequally when selecting the feature to remove. It would be interesting to explore how different combinations affect the results.

It seems that the experiments only consider backward feature selection approaches. Including forward feature selection approaches would add useful context for how the proposed approach compares to other strategies.

Minor comments

The cross-validation scheme used is not clear. While the authors mention that three runs are used to estimate performance variance, they do not describe if this is 3-fold cross validation, some Monte Carlo cross validation, or if the same splits are used all three times and just the random seeds are different.

While methods like RFE have significantly higher computational cost than the methods considered here, it would be helpful to include it for at least one of the datasets to provide context on how much the less costly methods “lose”.

What is the overlap in the selected features? both among the different methods and among the different folds for the same method.

How were the hyperparameters for the various models chosen?

Typos, etc.

The references are not consistently formatted.

The Section 2 headers all have an unnecessary “0” in them (e.g., “2.0.1”).

Table 1 should include the standard deviations.

review_title: Official Blind Review #1

Meta-review: In this paper the authors propose a wrapper feature selection method that selects features based on 1) redundancy, i.e. the sensitivity of the downstream model to feature elimination, and 2) relevance, i.e. how the individual features impact the accuracy of the target task. The authors use a combination of the redundancy and relevance scores to eliminate the features. 

While acknowledging that the proposed model is potentially useful, the reviewers raised several important concerns that were viewed by AC as critical issues:
(1) all reviewers agreed that the proposed approach lacks theoretical justification or convincing empirical evaluations in order to show its effectiveness and general applicability -- see R1’s and R2’s requests for evaluation with more datasets/diverse tasks to assess the applicability and generality of the proposed model; see R1’s, R4’s concerns regarding theoretical analysis; 
(2) all reviewers expressed concerns regarding the technical issue of combining the redundancy and relevance scores -- see R4’s and R2’s concerns regarding the individual/disjoint calibration of scores; see R1’s suggestion to learn to reweigh the scores;
(3) experimental setup requires improvement both in terms of clarity of presentation and implementation -- see R1’s comment regarding the ranker model, see R4’s concern regarding comparison with a standard deep learning model that does feature learning for a downstream task; both reviewers also suggested to analyse how autoencoders with different capacity could impact the results.
Additionally R1 raised a concern regarding relevant recent works that were overlooked. 
The authors have tried to address some of these concerns during rebuttal, but an insufficient empirical evidence still remains a critical issue of this work. To conclude, the reviewers and AC suggest that in its current state the manuscript is not ready for a publication. We hope the reviews are useful for improving and revising the paper.

Decision: Reject