Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors propose a feature selection method for high-dimensional datasets that attempts to fit a model while selecting relevant features. 
The strategy they follow is below:

1. They formulate feature selection as an optimization problem by augmenting standard empirical risk minimization with zero-one variables associated with each feature representing the absence-presence, and adding a penalty proportional to the number of included features. They relax the discrete variables using a continuous relaxation and provide a simple unbiased estimator for the gradient of the relaxation. After training the relaxation is rounded to a zero-one solution by a simple scheme. 
2. They provide an information theoretic motivation for their formulation of feature selection
3. They exhibit the performance of their method on a number of synthetic and real data scenarios: (i) linear models with a true underlying sparse parameter, (ii) binary classification with a small number of true determining features, (iii) regression performance post-feature selection with synthetic non-linear models (with a few determining features) and two real datasets. They also use the method for a classification problem with RNA-seq data on T-cells and a survival analysis based on a breast-cancer dataset called METABRIC. 

Despite my recommendation, there are a number of things that I like about the paper that I list below, along with directions where I believe the article can be improved. 
1. At a certain abstraction, the main idea of the paper is to do feature selection at the same time as model fitting (as the LASSO for e.g. does) while ignoring constraints of convexity raised in the optimization, and simply using stochastic gradient with a reasonable unbiased estimate of the gradient. This is a reasonable idea, particularly if under some reasonable assumptions, the non-convex formulation that is obtained is expected to be computationally 'benign'. 
2. In a number of the experiments, and particularly 6.1 (sparse linear model) 6.2 (noisy XOR classification) I suspect the non-convex formulation is what is providing a lot of the improvement. This has been observed empirically in a number of other settings, for e.g. in matrix completion/factorization problems. Verifying this hypothesis in a simple, synthetic (and therefore controlled) dataset would be a good contribution for a future version.   
3. The authors have done a fairly good job of validating the method in a number of different settings, even if some of the presentation of their results can possibly be somewhat improved. For e.g. the median rank is better shown with box plots (as in the Chen et al 2018 paper cited by the authors).
4. There are a number of relaxations of discrete variables used in optimization and theoretical computer science literature. For instance, the approach of the authors is reminiscent to 'mean field' methods, or standard linear programming relaxation of combinatorial optimization problems (i.e. the first level of the Sherali-Adams LP hierarchy). On the other hand, naive versions of this are not likely to work well on (say) sparse linear regression. The current methods do which suggests that the continuous relaxation is useful. 

At an expository level, I also think the paper could do with quite a bit of improvement:
1. The introduction is sparse and hurried, and without providing sufficient motivation and intuition for the contributions of the article. 
2. In 6.4, 6.5, the introduction about RNA-seq or Cox models can be removed and relevant work cited instead. 
3.  Organizing the experiments as real data, and synthetic data might be semantically better, though that would necessitate splitting Table 1. I am also unclear on why the authors show  performance in Tables 1, 2 independent of the number of features selected, while for the experiment on RNA-seq data the full accuracy/#features tradeoff is given. The sparse explanation about using the Optuna paper is certainly not enough. 

Minor comments not related to decision:
1. The value for \alpha_N in synthetic sparse linear model experiment of 6.1 likely has an extraneous \sqrt \log k 

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: The paper is concerned with embedding a supervised feature selection within a classification setting. 
The originality is to use an L_0 regularization (counting the number of retained features), besides the classification loss; the authors leverage the ability to include boolean variables in a neural network and to optimize their value using gradient descent through the reparameterization trick.

I am mildly convinced by the paper:
* Out of the four contributions listed p. 2, STG is the most convincing one; still, the description thereof is not cristal clear: the reparametrization trick is not due to the authors. The discussion (section 5) needs be more detailed, adding the HC details (presently in appendix); could you comment upon the difference between the proposed STG and the Gumbel-Softmax due to Jang et al, cited ?
* Likewise the authors delve into details regarding the early state of the art, while omitting some key points. For instance, p. 3, the fact that many authors replaced an L_0 penalization with an L_1 one is rooted on the fact that, provided that the optimal L_0 solution is sparse enough, the L_0 and L_1 problems have same solutions. This section can be summarized;
* the sought sparsity is assumed to be known, which is bold; 
* Assumption 2 is debatable; one would like to find at most the Markov blanket of the label variable. See Markov Blanket Feature Selection for Support Vector Machines, AAAI 08.
* There are digressions in the paper which make it harder to follow the argumentation (section 6.1); section 6.2 is not at the state of the art; in Guyon et al's Feature Selection Challenge (2003), the Arcene artificial problem involves a XOR with 5 key features, and 15 additional features are functions of the key features.

Suggestion, you might compare with the L_0 inspired regularization setting used for unsupervised feature selection in Agnostic Feature Selection, Doquet et al, 2019.

Details: check the citation style: use \citep instead of \cite.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The author rebuttal sufficiently addresses my concerns, so I am upgrading my score.

***

The paper considers the problem of embedded feature selection for supervised learning with nonlinear functions. A feature subset is evaluated via the loss function in a "soft" manner: a fraction of an individual feature can be "selected". Sparsity in the feature selection is enforced via a relaxation of l0 regularization. The resulting objective function is differentiable both in the feature selection and learned function making (simultaneous) gradient-based optimization possible. A variety of experiments in several supervised learning tasks demonstrates that the proposed method has superior performance to other embedded and wrapper methods.

My decision is to reject, but I'm on the fence regarding this paper. I'm not clearly seeing the motivation for an embedded feature selection method for neural network models: for the datasets considered in the paper, it would seem that training a nonlinear model that used all the features would result in performance at least as good as training the nonlinear model with a prepended STG layer. Perhaps there is evidence that filtering features, e.g., irrelevant features, results in higher accuracy and that the prepended STG layer achieves this accuracy, but that evidence is missing from the paper. Also, there could be downstream computational savings, e.g., at prediction time, if the dimension was very large, but this is not the setting tested in the experiments. I suppose interpretability could be considered motivation, but, even so, isn't there at least one simpler, deterministic approach (described below) that also "solves" the problem? Finally, it isn't clear how the method scales with increasing sample size and dimension as all the datasets tested are relatively small in these respects.

***

Questions and suggestions related to decision:

* The performance values using all features should be included in the experimental results so that the value added by STG can be assessed.

* Why not use the simpler deterministic and differentiable relaxation z = \sigma(\mu), where \sigma() is a "squashing" function from the real numbers to [0,1] applied element-by-element to the vector \mu? What specifically is/are the advantage(s) that the randomness in the definition of z at the bottom of pg. 3 provide over this deterministic alternative?

* Though well-described and methodologically rigorous, the experimental comparison is none-the-less a little disappointing: one dataset for classification and half the datasets for regression are synthetic and low-dimensional. The remaining regression datasets are real but also low-dimensional. The survival analysis dataset is also low-dimensional (as described in the supplementary material). This leaves one real classification dataset which was on the order of 20,000 examples and 2500 features. Why were larger sample-size and dimensionality datasets not tested? These should be readily available. For example, the gisette dataset from the NIPS 2003 feature selection challenge has 5000 features. See "MISSION: Ultra Large-Scale Feature Selection using Count-Sketches" by Aghazadeh & Spring et al. (2018) for other high-dimensional datasets. Even a single run for each large dataset would have provided some evidence of scalability.

***

Other minor comments not related to decision:

* "Concrete Autoencoders for Differentiable Feature Selection and Reconstruction" by Abid et al. (2019) targets unsupervised feature selection but has enough similarities in the approach that it should be considered related work.

* [Typo?] The unnumbered equation after (5) should not have a sum over d in the second term. Perhaps a sum over k was intended? Also, in this equation, the gradient of the loss wrt/ z samples, average of gradients over z samples times..., does not seem to match what the gradient would be given the algorithmic description in the supplementary material, a gradient of the (sample) average z times...

* The abstract states the paper is proposing a method for high-dimensional feature selection, but all of the experiments have datasets with max. dimensionality 2538.

* Some discussion of how the regularization parameter can be selected by a user of the proposed method would be good to include.
review_title: Official Blind Review #1

Meta-review: The authors propose a method for feature selection in non linear models by using an appropriate continuous relaxation of binary feature selection variables. The reviewers found that the paper contains several interesting methodological contributions. However, they thought that the foundations of the methodology make very strong assumptions. Moreover the experimental evaluation is lacking comparison with other methods for non linear feature selection such as that of Doquet et al and Chang et al.
Decision: Reject