Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors address a challenging and important problem of group feature selection. The uniqueness of their work is that they learn the grouping without any predefined knowledge. Feature selection is an important problem in computational biology, and quite often there are groups of features that act together in some biological/physical system. Finding the groups is important for understanding the driving factors behind the system that is being studied. The proposed method for solving this problem relies on an ensemble of feature selection models to find the informative features and groups. The authors demonstrate the effectiveness of their method using synthetic, semi-synthetic, and real data.
strengths_and_weaknesses: Overall the paper is well written, and I enjoyed reading it. The English level is satisfactory, and the method is motivated well and is scientifically sound. 
The proposed method relies on modern tools for feature selection, namely, probabilistic relaxation of Bernoulli variables, which have been found effective for this task in various applications. I would suggest trying a Gaussian distribution instead of the Concrete as shown in STG [40] but this is minor. 
One challenge of using the method is tuning the regularization parameters, which might be challenging since there are two in the proposed method.  Another weakness is that the authors don’t use any real-world data (MNIST is not really a dataset for feature selection). The semi-synthetic example is nice, but it seems that multiple methods lead to perfect accuracy, TPR, and FDR. 

questions: Some suggestions for improving the paper:
-Section 2- Knockoff procedure- > since these works typically have a slightly different goal in mind, it is worth explaining to the reader what are these methods trying to achieve. Just expand with a coupled of lines.
-Section 3- complex vs. simple interactions. These terms have not yet been defined or explained, so this mention is confusing for the reader. Please expand. 
-Missing comma after equations 1,2, 3.
-Section 3.2 line 106- when both are known-> do you mean when both are used by the model? The word known may be confusing since the features are always known (they are observed).
-Missing commas after many other equations.
-Why are the results of STG and CAE omitted from the main text? Clearly, these are the strongest competitors, both for the synthetic and for MNIST.
-P9 L327 “c.” not sure what this means.
-Why are you reporting only the average number of selected features? Isn’t the union more informative? Both for interpretability and computational aspects this is more important. Also, how can you compare this to STG and CAE if they select only a union of features? Shouldn’t those be in the same ball park?
I would be glad to consider raising my score if the authors would be able to provide a more challenging example with an evaluation of their method, since I think this is the major weakness of the paper.
limitations: The authors point out that the limitations appear in the conclusion, but I don’t see a real discussion about the limitations of the method there.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors of the paper propose a method for solving the problem of selecting groups of features in a supervised learning setting. The method is based on an ensemble of models where the number of models is the same as the expected number of groups. A new metric is also proposed to evaluate the quality of selected groups of features. 
strengths_and_weaknesses: The paper is written well, it is very clear and is easy to follow. The authors present the challenges of the group selection task, provide formal definition and a simple method to solve the task. The proposed formula for loss seems reasonable.

My main concern is about the problem the authors try to tackle. The incentives which are discussed in the introduction section for solving group feature selection problem are not convincing enough. I think it is better to provide an illustrative example that explains how groups of features could benefit the relevant stakeholders. One of the advantages of feature selection methods is to detect correlated features and exclude them by remaining a single representative for each “correlation group” (e.g. concrete autoencoders). So why should we care about these groups? The correlation could be based on linear and non linear interactions between features. E.g. in yout MNIST experiment, you got groups that look almost the same (1,3,4).
In addition the experimental part where only 3 datasets (Syn1-Syn4 could be labeled as a single dataset)  are used to evaluate the proposed method is not convincing. For datasets selected in the experiments p << number of samples, which is a non challenging setting for supervised feature selection task. I think it is better to justify the significance of your method on the more datasets including the real datasets.
An additional weakness of the proposed method is that it produces probability for each feature to be selected and not a binary vector. It requires an additional hyper parameter to be tuned manually.

Additional technical issues:
Line 67  - w_i is not defined before you mention it
Line 93 -  the space of functions F is not defined before it used here
Line 116 - rho function is not defined / described before it’s used here. If it’s the same as in the paper Deep Sets I think it is better to mention it.

questions: You claim that there is no source of dataset where the ground truth of groups is provided and that is the reason you use the Oracle method that selects a single group. Is it possible to manually construct such a dataset? 

Why do you use a small number of groups for synthetic datasets (up to 3) ? I think it is more convenient to check the method on a larger number of groups and also to evaluate it with varying the number of groups (is its performance degraded)?  

How is your method compared to the random initialization of any stochastic gates method? It is possible to train FS net and to get for each initialization different  selected features (assuming some stochasticity in the selection process).
limitations: The social impact is discussed.
In conclusion the authors describe some common limitations of the feature selection task but it is more reasonable to mention the limitations that are specific to your work: in the definition or in the method.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Most features of a sample are usually not independent but interdependent. In this paper, the author(s) introduced a method combining deep learning and ensemble learning to identify informative feature groups. Empirical experiments were implemented to validate the performance of the proposed method.
strengths_and_weaknesses: The topic of this paper looks interesting, I have the following comments:

### Strengths
---
1. The motivation of this paper looks interesting, as the authors illustrated at the beginning that "...in genomics, diseases might not be caused by any single mutation but require the presence of multiple mutations...". This paper attempted to detect groups of predictive features without the need for predefined groups.

2. Experiments on synthetic and semi-synthetic datasets showed some advantages to some extent.

---
---

Although the topic studied by this paper is interesting, I think the paper has the following deficiencies or unclear points (Please see the "Questions" section below for details).

### Weaknesses
---

1. Behind the group feature selection idea proposed in this paper is deep architecture plus ensemble learning, or more simply, ensemble STG in essence. Innovation seems limited.

2. The research review of traditional/standard feature selection methods is not sufficient. In fact, the traditional/standard feature selection methods have been developed even in the past two years, both in terms of method novelty and performance advantages.

3. Maybe the proposed method should be validated on genomics-related data.

---





questions: As for the Weaknesses mentioned above, to be specific, I have the following questions/comments which need further explanation from the author(s). Please correct me if I am wrong. Thanks.

1. In lines 10-11, it mentioned that "...we can rank the groups based on their predictive power for downstream decision making..." I didn't see an empirical example of this, or did I miss it?

2. In lines 46-47, "...why specific features are important than standard feature selection, with the goal of improving data-driven...". "specific features" or "specific feature selection"?

3. In lines 82-85, "complex interactions may happen within groups but only simple interactions between ... we do not enforce assumptions on the groups such as non-overlapping groups or every feature being...", I do not think it will hold for most cases due to "non-overlapping groups", for example, Group_1=\{x_1,x_2\} and Group_2=\{x_3,x_2\}, if there exist complex interactions between x_1 and x_2, then it will make the interactions between Group_1 and Group_2 become complex. From formula (4), I think it might be more accurate to say that the latent spaces from different groups are simple interactions, right?

4. For the implementation of STG and CAE, the author(s) just simply mentioned "Use the available Implementations". As far as I know, CAE in its original paper involved the use of different structures, such as one with no hidden layer, one with a hidden layer, and so on. So the author(s) should specify which specific structure is used. In addition, for a fair comparison, the compared methods should have similar or same structures.

5. Suggesting that for MNIST visualization examples, please (consider to) give more numerical results, so that the reader can see more clearly which model is really good.

6. At first, I was attracted by the motivation the author(s) demonstrated; that is, the example of genomics at the beginning, but after reading this paper, I was a little disappointed that I did not notice any genome-related data used in the analysis to illustrate the advantages of the proposed method. That's one of the reasons I have to give the current score. 

---

In addition, there are some tiny issues/typos:

(1) Journal/conference names are sometimes capitalized and sometimes not.

(2) The author's name in the reference is wrong, such as Ref. [23].

(3) In References, the format of conferences is inconsistent, sometimes abbreviations are given and sometimes not.

(4) In References, the format of authors' names is inconsistent, sometimes abbreviations are used and sometimes not.

I look forward to the response from the author(s), if the response adequately explains and can show the unique superiority of the proposed method in this respect, I will consider raising the score. Thanks.



-----------------------------------------------------------------------

The author(s) clarified most of my concerns. So I would like to raise my score from (4) Borderline reject to (5) Borderline accept. Thanks.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a composite feature selection, which aims to find groups of features with combined effects for prediction. More specifically, the authors proposed a deep learning architecture that uses an ensemble of feature selection models to find predictive groups. Experimental results on synthetic datasets, semi synthetic datasets and MNIST dataset demonstrated the effectiveness of the proposed method.
strengths_and_weaknesses: Pros
1. The paper proposed a new setting for feature selection problems, i.e., selecting feature groups that features in each group may interact with each other.
2. The experimental results seem promising. 
3. This paper is well-written and easy to follow.

Cons:
1. The problem setting seems to be related to some existing works. It seems to me that the setting in this paper is similar to the sub structures of the many machine learning tasks, e.g., there are several sub tasks which may or may not overlap with each other. In this paper, the authors proposed a solution for automatically discovering these sub tasks with the goals that all sub tasks can perform well and the ensemble of them can achieve optimal performance. This might be related to traditional ensemble learning methods, e.g., random forest or GBDT, in which many trees (upon a subset of features) are learned individually and then combined to achieve optimal accuracy.
2. In the abstract, it is mentioned that “For example, in genomics, diseases might not be caused by any single mutation but require the presence of multiple mutations”. This may not be supportive to the motivation of this paper. It might be better to show some examples of how different combinations of mutations can cause the same disease. But I am not sure how often it is in real diseases. 
3. Section 3 mentioned the challenge that different groups may have overlapping features, but I did not see how the proposed method can address such challenge.
4. The experiments on semi-synthetic datasets seem interesting, but it could be more convincing if there were some experiments on real medical/chemical datasets.

questions: 1. Can you use random forest or GBDT for feature grouping? For example, you can use the features shown in the same tree as a group.
2. For feature grouping, a trivial way is to cluster the features with clustering methods. How does the proposed method perform compared with clustering methods, e.g., spectrum clustering?

limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: ### Summary of paper
This work addresses the problem of grouped feature selection in the supervised learning setting. A new method based on ensemble of features is proposed as well as a new metric to evaluate the results on synthetic, semi-synthetic and real data. 

### Rebuttal
Authors were engaged and addressed all the reviewers' concerns and questions.  While reviewers did not engage in the rebuttal discussion nor post-rebuttal discussion, they significantly raised their scores, bringing the average score from 4.75 to 5.75, indicating that their main concerns were addressed.

### Acceptance
There are no major concerns that remain to be addressed and the paper is ready to be published. I hence recommend acceptance of the paper.  

confidence: Certain
Decision: Accept