Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper considers a method of evaluating the importance of variables of a (possibly) non-convex function f under a Bayesian perspective. Given a Gaussian process over a function set (including true function f) as a prior distribution, the evaluation measure for variable j is the expectation of magnitude of the partial gradient of f w.r.t. j, where the expectation is taken w.r.t. some distribution from which instances are drawn. The paper proves convergence results that the estimate of the measure converges to the true value when the sample size increases up to infinity.
strengths_and_weaknesses: Strength:
- Consistent approach from a Bayesian perspective


Weakness:
-  Theoretical results would be more convincing if finite sample analyses are possible.


questions: - Do you have any assumption on the distribution from which instances are drawn?

-Do you assume that the function is fully differentiable?

- I don't understand how to compute AUCROC. Please explain.
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The importance of a variable in this paper is derived from the norm the corresponding partial derivative.
The authors use a Bayesian linear regression to capture the uncertainty of the model parameters.
Because of the form of feature represented method, the proposed method is independent to the model and thus can be applied to tree-based structures. 
strengths_and_weaknesses: Strength:
- The paper is well structured and written. 
- The uncertainty is counted into the calculation of variable selection. 
- Use the feature-based representation of Gaussian process to accelerate training and generalization on different machine learning models.
- The authors also added theoretical proofs to show the convergence and efficiency. 

Weakness:
- The experimental results does not fully reflect the profoundness of the theory. 
- The variable selection still relies on linear structure and could be hard to extend to more complex scenarios. 
- It will be better if the legends of Figure 1 are enlarged.
questions: - Even though the title uses the word "nonlinear", it seems that the variable selection part is still based on linear structure. If so, how could the model be extended to more complicated cases? For example, a multilayer neural network. 
- The above question also reflected in the current experiments. Except for the synthetic experiments, the results on the real dataset is not very consistent. Why does the performance of RFNN is not good? 
- Why does the points of GAM not show in the Figure 1? 


limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an uniﬁed framework for nonlinear variable selection that incorporates uncertainty in the prediction function and is compatible with a wide range of machine learning models (e.g., tree ensembles, kernel methods, neural networks, etc).  The authors (1) provide a principled approach for quantifying variable selection uncertainty by deriving its posterior distribution, and (2) show that the approach is generalizable even to non-differentiable models. Rigorous Bayesian nonparametric theorems are derived to guarantee the posterior consistency and asymptotic uncertainty of the proposed approach. Extensive simulations and experiments verify the effectiveness of the proposed method.
strengths_and_weaknesses: Strength:
+  The paper is well-written.

+ The proposed method is supported by theory and experiments.

Weakness:

The numerical experiments can be improved, e.g., considering larger input dimensions in the experiment.
questions: +  In line 76, the data-generating function f_0 is assumed as a ﬂexible nonlinear function that resides in a reproducing kernel Hilbert space (RKHS).  How restrictive is this assumption? in other words, does this assumption cover most real data generation functions.
+  In Theorem 1, what is the specific convergence rate of $\epsilon$ mentioned in condition (1) (line 203).
+  He et al. [1] also employ the partial derivative-based kernel method to realize variable selection. It would be appreciated if the authors could explain the differences between theirs and the advantages of this paper.

Ref:
[1] He et al. SCALABLE KERNEL-BASED VARIABLE SELECTION WITH SPARSISTENCY.
limitations: All my concerns are presented in “Weakness” and “Questions”. Moreover, I do not have any concern on negative social impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper develops a unified framework for uncertainty-aware nonlinear variable selection for machine learning models that can be represented by RKHS including linear models, neural networks, random forests, and tree ensembles. Two major theoretical results about the approach are derived, namely, posterior convergence of variable Importance and a Bernstein-von Mises Theorem verifying that posterior distribution of variance importance is statistically optimal.

The authors then apply and validate the approach on several synthetic and real datasets, and conclude that:

- there does not exist a model class that performs universally well across all data scenarios, highlighting the necessity of a unified framework
- good prediction translates to effective variable selection
strengths_and_weaknesses: ### Strengths

- Significance:
    - The paper considers an important problem in machine learning: uncertainty-aware nonlinear variable selections across multiple classes of models. The multiple-model-class aspect is particularly of great interest to the machine learning community since in many practical applications, it is often unknown a priori which specific class of model is more appropriate for the data.
- Quality and Clarity:
    - In general, the paper is well-written and well-organized. Discussions about pre-existing literature and related works are sufficient (with some caveats detailed below).
    - The theoretical analyses of the work are rigorous and of high quality.
    - One of the stronger points of the paper is that it provides a very careful treatment of the non-trivial gradient function assumption and studies in detail the complications arising when the features are discrete or the model is non-differentiable. These issues are often neglected in the field but are unavoidable in studying a unified framework, and the paper’s efforts in this direction should be appreciated.

### Weaknesses

- Literature: One concern I have with the paper is its lack of direct discussions/comparisons with the work

     > J. Liu (2019). Variable selection with rigorous uncertainty quantification using deep Bayesian neural networks: Posterior concentration and Bernstein-von Mises phenomenon.

   cited as [45] in the manuscript. [45] is mentioned in the paper with minor references (correlated features, simpler conditions), but I believe it’s hard to ignore the similarities between the two works (the same measure of feature importance, similar approaches, emphasis and questions, same theoretical results, even the titles are somewhat similar). 

  More importantly, the two papers lead to two contrasting conclusions:
    - the paper observes that good prediction translates to effective variable selection
    - [45] hinted that a powerful model alone is not sufficient to guarantee effective variable selection

    Given the connections (and the fact that the authors are aware of the work by Liu), a thorough comparison in both theoretical analyses and (perhaps) experiments would be more appropriate.
- Quality:
  - The paper conflates the two concepts of variable importance and variable selection, and this may lead to an incorrect interpretation of the results.
      - Mathematically, posterior concentration of variance important and Bernstein-von Mises phenomenon do not translate to efficient/consistent variable selection. For example, for standard settings of linear models, ordinary least square satisfies both of the above properties but is not selection consistent/efficient, thus necessitating the use of lasso-based methods. Thus Theorem 1 and 2 do not support the claim that “good prediction translates to effective variable selection” as the authors stated in Line 280.
      - In practice, to quantify the efficiency of variable selection procedures, a good measure of variable importance and a rigorous decision rule is equally important, and AUROC (which integrates over threshold) is not an appropriate measure in these comparisons. This might be the reason why the conclusions of the paper and [45] (which uses F1 scores on three different modes of thresholding) are pointing in two different directions.
  - Since the paper attempted to provide a framework that works across models, they seem to avoid directly stating the assumptions of the analyses. For example, the feature map could be differentiable, non-differentiable, or even discontinuous and there seem to be separate treatments for each case. However, this leads to a very difficult reading of the work, and it seems impossible to verify from the main text whether a general model would fit into the framework. I suggest adding a list of assumptions on the feature map, and the parameter spaces of the models in the mathematical descriptions.
  - The decisions of evaluating the methods on tabular data only, and thus remove neural network models from the comparisons are contradictory to the goal of providing a unified framework of variable selection. Some additional evaluations, even on a synthetic dataset, perhaps in comparison with Liu (2019) would strengthen the manuscript.
- Significance
    - The paper assumes that the number of significant features (d) is fixed while data (n) increases. This setting is significantly more restrictive than typical results about posterior concentrations in the field for both generalized linear models and neural network models.
    - In essence, the models considered in this work need to be compatible with an RHKS and have to be linear in terms of the parameters. As pointed out by the authors, this places a significant restriction on the application of the framework. For example, for neural network models, this means all layers of the network except the last (linear) one have to be fixed, and the same limit applies for tree-based models. I believe this should be acknowledged more directly in the title, the abstract, and the introduction.


#####Post-rebuttal comment####

The authors have addressed all of my comments with their extensive efforts with the revision.


questions: n/a
limitations: None noted
review_title: null

Meta-review: metareview: The authors propose a new method for quantifying variable importance (and thus, for variable selection) in a totally nonparametric setting. They use a Bayesian setting (GP) and quantify the variable importance based on the L2-norm of the corresponding partial derivative. The method comes with strong theoretical guarantees (rate of contraction, BvM theorem) and numerical experiments.

The reviewers initially agreed that the method proposed was nice and that the theoretical results were interesting enough to justify publication. However, some of them also pointed out various problems: weaknesses in the experiments [Reviewers AEuT, 2frY], problems in some figures [AEuT], missing references that would require further discussion [adBr], absence of a non-asymptotic (ie, d not fixed) analysis [smu5, adBr], assumptions to be clarified [adBr]. The authors addressed all these points, fixing the minor mistakes, adding new experiments and non-asymptotic results to the paper (on a personal note, I will add that I appreciate the summary of the changes provided by the authors, that make it easier to track the modifications in the paper). The reviewers agreed the new version of the paper is much better, and ready for publication in its current state.

confidence: Certain
Decision: Accept