Composite Feature Selection Using Deep Ensembles
Authors: Fergus Imrie
Alexander Luke Ian Norcliffe
Pietro Lio
Mihaela van der Schaar
Venue: NeurIPS 2022 Accept (2022)
Keywords: Feature Selection
Group Feature Selection

Abstract: In many real world problems, features do not act alone but in combination with each other. For example, in genomics, diseases might not be caused by any single mutation but require the presence of multiple mutations. Prior work on feature selection either seeks to identify individual features or can only determine relevant groups from a predefined set. We investigate the problem of discovering groups of predictive features without predefined grouping. To do so, we define predictive groups in terms of linear and non-linear interactions between features. We introduce a novel deep learning architecture that uses an ensemble of feature selection models to find predictive groups, without requiring candidate groups to be provided. The selected groups are sparse and exhibit minimum overlap. Furthermore, we propose a new metric to measure similarity between discovered groups and the ground truth. We demonstrate the utility our model on multiple synthetic tasks and semi-synthetic chemistry datasets, where the ground truth structure is known, as well as an image dataset and a real-world cancer dataset.

Composite Feature Selection Using Deep Ensembles
Fergus Imrie* University of California, Los Angeles imrie umrie@ucla.en
Alexander Norcliffe" University of Cambridge alin20cam. ac.uk
Pietro Liò
University of Cambridge p1219@cam ac uk
Mihaela van der Schaar University of Cambridge
The Alan Turing Institute University of California, Los Angeles mv4720cam.c ac
Abstract
In many real world problems, features do not act alone but combination with each other. For example, in genomics, diseases might not be caused by any single mutation but require the presence of multiple mutations. Prior work on feature selection either seeks to identify individual features can only determine relevant groups from a predefined set. We investigate the problem of discovering groups of predictive features without predefined grouping. To do so, we define predictive groups in terms of linear and non-linear interactions between features. We introduce novel deep learning architecture that uses an ensemble of feature selection models to find predictive groups, without requiring candidate groups to be provided. The selected groups are sparse and exhibit minimum overlap. Furthermore, we propose new metric to measure similarity between discovered groups and the ground truth. We demonstrate the utility of our model on multiple synthetic tasks and semi-synthetic chemistry datasets, where the ground truth structure is known, as well as an image dataset and a real-world cancer dataset.
Introduction
Feature selection is key problem permeating statistics, machine learning and broader science. Typically high-dimensional datasets, the majority of features will not responsible for the target response and thus important goal is to identify which variables truly predictive. For example, healthcare there may be many features (such as age, sex, medical history, etc.) that could be considered, while only small subset might in fact be relevant for predicting the likelihood of developing a specific disease. By eliminating irrelevant variables, feature selection algorithms can be used to drive discovery, improve model generalisstion//rbuutness and improve interpretability (18).
However, features often do not act alone but instead in combination. genetics, for instance, has been noted that understanding the origins of many diseases may require methods able to identify more complex genetic models than single variants (65). More generally, often there are multiple groups of variables which act (somewhat) independently of each other. For example, in medicine or biology, a number of discases can manifest from different mechanisms or pathways. Examples include cancer [63], amyotrophic lateral sclerosis (ALS) and frontotemporal dementia (FTD) (6). inflammatory bowel disease [3I], cardiovascular discase [42, and diabetes [64].
While feature selection might be able to identify a set of features associated with a particular response, the underlying structure of how features interact is not captured. Further, the resulting
Equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

predictive models can be complex, hard to interpret, and not amenable to the generation of hypotheses that can be experimentally tested [46]. This limits the impact such models can have in furthering scientific understanding across many domains where variables are known to interact, such as genetics medicine [8 and economics [8]
65 69
60]
Group feature selection is generalisation of standard feature selection, where instead of selecting individual features, groups of features are either entirely chosen entirely excluded. A primary application of group feature selection is when features are jointly measured, for example by different instruments. In such scenarios, groups are readily defined as features measured by the same instrument. A natural question is which instruments give the most meaningful measurements. Group feature selection has also been applied in situations where there is extensive domain knowledge regarding the group structure [72] or where groups are defined by the correlation structure between features (c.g. neighbouring pixels in images are highly correlated). The pervasive issue with current group feature selection methods is that predetermined grouping must be provided, and the groups are selected from the given candidates. In reality, we may not know how to group the variables.
In this paper we seek to solve related but ultimately different and more challenging problem, which we call Composite Feature Selection. We wish to find groups of variables without prior knowledge, where each group acts as a separate predictive subset of the features and the overall predictive power greatest when all groups are used in unison. We call each group of features a composite feature By imposing this structure on the discovered features, we attempt to isolate pathways from features to the response variable. Discovering groups of features offers deeper insights into why specific features are important than standard feature selection.
!
Contributions. (1) We formalise composite feature selection as extension of standard feature selection, defining composite features in terms of linear and non-linear interactions between variables (Sec. 3 (2) We propose a new deep learning architecture for composite feature selection using an bsemble-bbsed approach (Sec. 4h. (3) To assess our solution, we introduce a metric for assessing composite feature similarity based on Jaccard similarity (Sec. [5) (4) We demonstrate the utility of our model on range of synthetic and semi-synthetic tasks where the ground truth group features are known (Sec. (5). We see that our model not only frequently recovers the relevant features, but also often discovers the underlying group structure. We further illustrate our approach on an image dataset and a real-world cancer dataset, corroborating discovered features and feature interactions in the scientific literature.
2 Related Work
Significant attention has been placed on feature selection with a range of solutions including traditional
methods (e.g. [56] 47 34] and deep learning approaches [54] 91 7 49 (see Appendix B for further
discussion of standard feature selection). Several approaches have been extended to select predefined is a linear method
groups of variables instead of individual features. For example, LASSO [76
83
that uses an L1 penalty to impose sparsity among coefficients. Group LASSO allow predefined groups to be selected excluded jointly, rather than single features. by replacing the L1 penalty with L.2 penalties on each group. Other feature selection methods, such as SLOPE 12]. have been similarly extended to group feature selection give SOPEE 14] Further examples of group feature selection using adapted loss functions are SCAD-L2 995 and hicrarchical
93 generalises this to
LASSO [100]. Similarly, Bayesian approaches to feature selection (28 have also been generalised
to the group setting (35). Finally, the Knockoff procedure [
39
57
73
80 is a generative
procedure that creates fake covariates (knockoffs), obeying certain symmetries under permutations of real and knockoff features. By subsequently carrying out feature selection on the combined real
and knockoff data, it is possible to obtain guarantees on the False Discovery Rate of the selected features. Generalisations of the Knockoff procedure to the group setting also exist [3 I01,, where symmetries under permutations of entire groups must exist.
The key commonality is that none of these methods discover groups, but instead can only select groups from a set of predefined candidates. Therefore, while they may be applicable when we can split inputs into groups, they are not able to find groups of predictors on their own. Our work differs from these methods by considering the challenge of finding such groups in the absence of prior knowledge. Additionally, unlike prior work, we do not make assumptions about correlations
We will often refer composite features as groups for brevity; in this paper. they refer to the same thing.

---

between features or place restrictions on groups, such as requiring the candidate groups to partition the features.
3 Problem Description
Let X E X be p-dimensional signal (such as gene expressions or patient covariates) and Y E y be response (such as disease traits). Informally, wish to group features into the maximum number of subsets, 9i C [p), where the predictive power of any single group significantly decreases when any feature is removed, allowing response. Note that we do not enforce assumptions on the groups such as non-overlapping groups or every feature being in at least one group. In this section, begin with description of traditional feature selection before formalizing composite feature selection.
to separate the groups into different pathways from the signal the
3.1
Feature Selection
The goal of traditional feature selection is to select subset, S C [p], of features that are relevant for predicting the response variable. In particular, the case of embedded feature selection [32] this conducted jointly with the model selection process.
Let denote any point not in X' and define Xs- = UX {+)) Then, given X E XP, the selected subset of features can be denoted as Xs E X's where S = if k E S and JSS * if k: & S. Let f: Xss + y be a function in some space F (such as the space of neural networks) taking subset Xs as input to yield Y. Then, selecting relevant features for predicting a response can be achieved by solving the following optimization problem:
minimize Exps-xx ((xs))) subject to |S| s 6, feT, sclp)
where 8 constrains the number of selected features and ey (3, v/) is a task-specific loss function.
This can be solved by introducing selection vector M (M ... Mpp) {0, 133² consisting of binary random variables governed by distribution РM, with realization m indicating selection of the corresponding features. Then, the selected features given vector m can be written as
غ O x + (1 m) @ x,
where @ indicates element-wise multiplication and x are the values assigned to features that are not selected (typically x = 0 or x). Eq. ① can be (approximately) solved by jointly learning the model f and the selection vector distribution pM based on the following optimization problem:
minimize Ex,y<pxx Emppsr (y (y, J (x)) + 3 [mll]] PM
where 3 is balancing coefficient that controls the number of features be selected.
3.2 Composite Feature Selection
The goal of composite feature selection not only to find the predictive features, but also to group them based on how they are predictive. For example, assume features x1 and 12 are only predictive when both are known by the model, but have the same influence on the outcome independent of æz- Then wish to group I1, 22 separately from 23. In this section, we define the embedded composite feature selection problem; that is, we want to find a valid model f and groups ( 9N3 in parallel. A model is only valid when the group representations are combined in way where we can view each group as contributing an independent piece of information for the final prediction. A valid model acts Therefore, we must combine the representations using a permutation invariant aggregator.
set of groups (94), thus when combining groups, require order not to matter.
Let A (II, R²) + N be general permutation invariant aggregation function. It well established that for specific choice of o: R" + RMT and p : R" + RN, A can be decomposed as ,(E, ((-)) (see (94) for examples). This gives fx) = [ E, (xf((x)))) where f; encodes group p and give the permutation invariant aggregation, and g is any final non-linear function, for instance softmax. The function composition of & and fa can relabelled as f = de a fú. and the composition of g and can be relabelled as أ = go G p. This leads to f(x) = 7( E, f(xx,)) giving the following definition for valid model structure in composite feature selection.
3

---

Definition 3.1. The most general valid model for acting on N composite features is given by:
f(x) f(x)= = = (
(4)
That is, the groups must interact exactly once, all groups must be included, and the interaction is summation; all other interactions can (and often should) be non-linear.
Depending the task, specific permutation invariant aggregation may be chosen (e.g. Max ()). However, any permutation invariant aggregator can be (approximately) expressed in the form of Def. 3.1 thus, when learning from data, the general structure of Def. 3.1 means that this is not necessary.
The embedded composite feature selection problem can now be phrased in an analogous way to traditional feature selection. Let denote some point not in X' and define Xg. = (X' U {+)),. Then, given X E X", the selected group of features is denoted as Xg. E Xg. where øg..k 24 if 9 and nΛ if lo & G. Let Xg. + Z be function in F that takes input the subset Xg, and outputs a latent representation z,. Then, finding the groups of features can be achieved by solving the optimization problem:
Exyxpxy lx (r(B.pl E Ex(xx))))
|9 S 6 Vi, N 2 A,
minimize P.feEF ScClpl
subject
(5)
where å constrains the number of selected features in each group and Á gives the minimum number of groups. This objective leads to multiple smaller groups, rather than one group containing all features, which is consistent with our motivation of the problem.
Continuing to expand from traditional feature selection, we can also extend the solution to the composite setting. For N groups distribution Рм- For a realization M, the selected features from group i are given by
introduce selection matrix M {0, 1] xp, governed by
m, O x + (1 m:) @ X,
(6)
where m; is the Tth row of M. We can approximately solve Eq. (5) by solving the optimization problem:
minimize Ex.y<pxy EM--I [ex (v. + RMD)
Par
where /xx obeys Def. (4) and Rg is regularisation term which controls how features are selected in each group. R. should capture both group size (i.e. encourage as few features as possible to be selected) but also the relationships between groups (i.e. groups should be distinct and not redundant).
3.3 Challenges
There are various challenges in solving the composite feature selection problem. While the ultimate task is to find predictive groups of features, there first remains the necessity simply to identify predictive features, which is already an NP-hard problem [2]. Composite feature selection not only inherits this property but introduces additional complexity since can think of each group as solving a separate feature selection problem. Consider the number of potential solutions: in traditional feature selection (assuming not all features are selected), there are 2" 2 ways of selecting subset from features; even restricting to at most m << n quickly becomes unfeasible for even modest values of m. In composite feature selection, every group has the same number of solutions traditional feature selection, drastically increasing the total number of possible solutions. A challenge specific composite feature selection arises when the ground truth group structure contains groups with overlapping features (e.g. feature œ interacts independently with both 22 and 2z).. In this scenario, difficult separate these two effects while penalizing the inclusion of additional features.
4 Method: CompFS
In this section, we propose novel architecture for finding predictive groups of features, which we refer to as Composite Feature Selection (CompFS). order to discover groups of features, our model is composed of set of group selection models and aggregate predictor. Our approach resembles

---

Gate
X
Encoder (fa.)
Predictor (Myy)
y,
Sample
Predictor (0g)
Group Generator
Group Encoder (fay)
Group Predictoo (Phyy)
Gate
X,
")
Sample
Group Generator
Figure 1: An illustration of CompFS. We use an ensemble of group selection models to discover composite features and aggregate predictor combine these features when issuing predictions.
an ensemble of "weak" feature selection models, where each learner attempts to solve the task using sparse set of features (Figure []. These models are then trained in such way as discover distinct predictive groups. We first consider the group selection models in more detail before describing how the group selection models are combined and the training procedure.
4.1 Group Selection Models
CompFS is composed of set of group selection models, each of which primarily aims to solve the traditional feature selection problem specified by Eq. (T). We achieve this by solving Eq. ( using neural network-based approach with stochastic gating of the input features. Each group selection model consists of the following three components (Figure
Group Selection Probability, = (M1,5. Tp.i) E 0,, 1]" which is a trainable vector that governs the Bernoulli distribution used to generate the gate vector m,, Each element of the selection probability Th indicates the importance of the corresponding feature to the target.
Group Encoder, fe. X > Z, that takes as input the selected subset of features x; and outputs latent representations E Z.
Group Predictor, lt:; Z - that takes as input latent representations of the selected subset of features,
Jo. (x:). ) and outputs predictions on the target outcome.
Solving Eq. 3 directly not possible since the sampling step has no differentiable inverse. Instead, we use the relaxed Bernoulli distribution [59 38 and apply the reparameterization trick as follows.
(T.,... Tp) and independent Uniform(0, 1) random variables .Ui,... ... Up). we can generate relaxed gate vector m = (ma,. ",pp) E (0, 1)P based on the following reparameterization trick [59]:
Formally, given selection probability
me ( (log log Th log(1 Tx) log Uk log(1 Ui)))
(8)
where or (xx (1 + exp(-x)) is the sigmoid function. This relaxation is parameterized by and temperature (0, oo). Further, as T + the gate vectors mig converge Bernoulli(T)) random variables. Crucially, this is differentiable with respect to π.
Given group selection probability π., we first sample relaxed Bernoulli random variable m, according to Eq. (8) and then use m, in gating procedure to select the group of features. The output of the gate is:
X, gate, (x) m,
m, O
where we replace the variables that were not selected by their mean value X. The mean is used because certain tasks a feature having value 0 may particularly meaningful. However, any (arbitrary) value could be used for non-selected features. The gate output x; is then fed into the group encoder fo, to yield representation z; So, (x;). This representation is finally passed to the group predictor he, produce the prediction for individual learner, ÿ: hz,
5

---

4.2 Group Aggregation
The final component necessary for CompFS is way to aggregate the individual group selection models. This is achieved via an overall predictor, h& Z + y, that takes input the set of latent representations .zi...... Zy] produced by the individual learners and outputs predictions on the target outcome. For simplicity, we apply linear prediction head to the latent representations and use element-wise summation to aggregate. Thus, the prediction of the ensemble is given by:
hf(1........ ZN =
Σ W.z + bb.)
(10)
where N is the number of members of the ensemble (i.e. the number of groups) and is suitable transformation (e.g. softmax). Note that by using element-wise summation, our model satisfies Def.
(4) for acting composite features.
4.3 Loss Functions
Group Selection Models. The individual learners can be trained perform (traditional) feature selection (Eq. (I)) by minimizing the following loss function:
Cg. = Exxxxxx by (v. ha, (fe,
+ ((T)))
(11)
where ly is a suitable loss function for the prediction task (e.g. cross-entropy for classification tasks and MSE for regression tasks) and 3 2 O balances the two terms. Note the selections probabilities n, not regularized with the typical L.1 penalty. Instead, we apply an 1.2 penalty to the mean selection probability (T&) for each individual learner. This is justified as follows. Recall the optimization problem given by Eq. (5). We desire solution with the maximal number of predictive groups N while minimizing the number of selected features per group Σ Gil. The standard LI penalty term does not achieve this goal since adding an additional feature to either group G. Gj incurs the same penalty. In contrast, the L2 penalty imposed on (T,,) penalizes adding extra features to already large groups, favoring the construction of smaller groups over larger ones.
Aggregate Predictor. The aggregate predictor can be trained jointly with the group selection models by minimizing a standard prediction loss (where yy is the same in Eq.
LE = Exyypxx ey (y. ho ............)))..)..
2n11)
(12)
Additional Regularization. If we simply apply the losses given by Eqs. limited (or even no) differentiation among the individual learners and the optimal solution would be for each learner to simply solve the traditional feature selection problem (Eq. (I) This results in all learners selecting the same features, which does not achieve our aim of discovering groups of predictive features. In order encourage differentiation between the models, introduce additional loss that penalizes the selection of the same features in multiple groups:
12 there will be
N π.
LR = Ex.y<xx
ΣΣ
T
(13)
Overall Loss.
Combining the above, our overall loss function therefore can be written as follows:
Σ Lg. + BELE + BuCn,
(14)
where Bg,BR Bn are hyperparameters to balance the losses.
Training CompFS with the loss given by Eq. (4] is designed to achieve the following: (1) The overall ensemble network should be a good predictor (Cx). (2) Each individual learner should solve the traditional feature selection problem (L which requires the group predictor to be accurate while selecting minimal features. However, the individual learners should not be maximally predictive by

---

definition (hence why we compare individual group feature selection models to weak learners). (3) Finally, we want the groups to distinct and thus discourage highly similar groups (CR). However, note that we do not exclude the possibility of some overlap of features between groups. The model is end-to-end differentiable, so we train with gradient descent.
Evaluation. During evaluation, only the gating procedure changes. The way features can be selected chosen by the user. A standard solution which we adopt in this paper is using threshold À and computing gate vectors m, as follows: mi,k 1, Ti,k À and 0 otherwise.
5 Experiments
We evaluate CompFS using several synthetic and semi-synthetic datasets where ground truth feature importances and group structure are known. In addition, we illustrate our method on an image dataset (MNIST) and real-world cancer dataset (METABRIC). Specific architectural details are given in App. Additional information regarding experiments, benchmarks, and datasets can be found in
App. Additional ablations and sensitivity analysis in App. A The code for our method and
experiments is available on Github.
Benchmarks. The primary goal of our experiments is demonstrate the utility of discovering composite features over traditional feature selection. Our main benchmark is an oracle feature selection method ("Oracle") that perfectly selects the ground truth features but provides no structure, giving all features as one group. By definition, this is the strongest standard feature selection baseline for the scenarios where the ground truth features are known. We also include comparisons to a linear feature selection method (LASSO) { and two non-linear, state of the art approaches, Stochastic Gates (STG) 9T and Supervised Concrete Autoencoder (Sup-CAE) 77. Finally, we compare with Group LASSO 193), where we enumerate all groups with I or 2 features predefined groups. Note this represents a significant simplification of the task for Group Lasso. We include additional baselines in App. G
Metrics. When the ground truth feature groups G1, Gy are known, we use True Positive Rate (TPR) and False Discovery Rate (FDR) to assess the discovered features against the ground truth. To assess composite features, i.e. grouping, we define the Group Similarity (Gsim) as the normalized Jaccard similarity between ground truth feature groups and the most similar proposed group:
N
Gsim max N. K)
Σ
max J(G., JE(K)
(15)
where I is the Jaccard index [37 and 91,... , 9x are the discovered groups Gam [0, 1]. where Gsi corresponds to perfect recovery of the ground truth groups, while Gsim= 0 when none of the correct features are discovered (see App. for additional details together with examples). We assess the models by seeing if the ground truth features have been correctly discovered, using TPR and FDR. We then see if the underlying grouping has been uncovered (and correct features) using Gaim Finally, we assess the predictive power of the discovered features using accuracy or area under the receiver operating curve (AUROC).
5.1 Synthetic Experiments.
Dataset Description. We begin by evaluating our method on range of synthetic datasets where the ground truth feature importance is known (Table []. We generate synthetic datasets by sampling from the Gaussian distribution with initially no correlations among features (X ~ AO, T)). We construct binary classification tasks, where the class y is determined by the following decision rules:
(Syn1) y = 1 if x1 > 0.55 or 22 > 0.55, 0 otherwise. The ground truth groups are {{1), {2))- This task assesses whether the model can separate two features rather than group them together. (Syn2) y = if 2 > 0.30 or Z3 > 0,30, otherwise. The ground truth groups are {[1, 2), (3,4)] This task requires identifying groups consisting of more than one variable.
(Syn3) y = if 21 > 0.30 or 21 > 0.30, otherwise. The ground truth groups are ([1,2), (1,3)}. This task investigates whether a model can split the features into two overlapping
groups of two, rather than one group with all three features.
http://gttta.akaoaaeeeerreaaaaae Sellectioon

---

Table I: Performance on Synthetic Datasets, values are recorded with their standard deviations.
DATASET
MODRI.
TPR
FDR
Gun
No. GROUPS
ACCURACY (%)
COMPFS(S)
100.0 = 0.0
0,0 - 0.0
0,91 1.1.4
2.2 0.44
98.9 0.5
ORACLE
100.0 0.0
0.0 0.00
0.50 1:000
1.000.0
100,0 1: 0.0
LASSO
100.0
0.0
0.50
1,0000
81.8
SYNI
GROUP LASSO
100.0 0.0
0.0
0.67 0:00
3.0
83.8 1.4
STG
100.0 0.0
0.0 =
0.50 +: 0.00
4.0
97.8 :: 1.4
SUP-CAE
100.0
0.0
0.50
97.8
COMPES(S)
ORACLE
=
0.0
0.50 0.00
1.0
100.0 0.0
LASSO
0.0.
0.0
0.00 0.000
0.0.00..0
52.6 2.9
SYN2
GROUP LASSO
0.0 0.000
0.0
0.00 4: 0.00
0.0
52.2 #
STG
100.0
0.0
0.50
1.0000
93.9
SUP-CAE
37.5 31.7
42.5
0.24 + 0.20
1,0
61.9 12.8
COMPFS(S)
100.0 = 0.0
0.0
0.68 1: 0.05
1.3
97.4 1:
ORACLE
100.0 0.0
0.0
0.67 1 0.000
1.000.0
100 0 0.0
SYN3
LASSO
0.0
0.0
0.00 0:00
0.0
GROUP LASSO
0.0
0.0
0.00
0.0
54.6
STG
=.0
0.0
0.67 1: 0.00
1.0
95.3 +: 1.7
SUP-CAE
23.3 31.6
1:7.1
0.23
1.000.
62.6 12.6
COMPES(5)
90.0 12.2
11.8
0.47 # 0.20
2.5 0.7
95.8 =
ORACLE
100.0
0.0
0.50
100.0
LASSO
0,0
0.0
0.00 1 0.00
0.0
51.8 3.2
SYN4
GROUP LASSO
0.0 0.0
10.0
0.00 0:00
0.1 2:33
53.0
STG
100.0 0.0
66.7 0.0
0.17 0:00
1.0
94.2
SUP-CAE
72.5
16.7 14.7
0.39 0:008
1,0
72.2 13.2
(Syn4) y = if 2144 0.30 or 37310 > 0.30, 0 otherwise. The ground truth groups are [[1,4), (7, 103). This task is equivalent to Syn2, however, here the features exhibit strong correlation in collections of 3, i.c. features 1, 2. and 3 are highly correlated, features 4, 5, and 6 are highly correlated, and feature selection (and indeed standard feature selection) when the features are highly correlated.
This task demonstrates the difficulty of carrying out group
The decision rules are created such that there is minimal class imbalance. We use signals with 500 dimensions to demonstrate the utility in the high dimensional regime. We use 20,000 samples to train and 200 to test. Each experiment repeated 10 times.
Analysis. On both Synl and Syn2, CompIS achieves high TPR with false discoveries (0% FDR) and significantly higher Gsim than the Oracle. Despite allowing CompF'S to discover up to groups, CompFS typically finds the correct number of groups (2), demonstrating that it is not necessary for the number of potential composite features to match the ground truth, which is vital in real-world use cases where this is unknown. Syn3 is significantly more challenging due to the overlapping structure and we observe essentially the same performance as Oracle. Despite finding all the correct features and no false discoveries, CompFS typically finds the union {1, 2,3) rather than the underlying group structure {{1,2). {1,3}}. Finally, for Syn4, while CompFS has relatively high FDR, frequently finds the ground truth relevant features and groups with similar Gsim to Oracle, This is a challenging task with significant correlation between features. Despite this, CompFSS is able to uncover the underlying group structure, providing additional insight over traditional feature selection. STG typically performs reasonably in terms of traditional feature selection, but scores poorly in terms of Gsim due to not providing any group information.
5.2 Semi-Synthetic Experiments.
Dataset Description. Next, assess ability to identify composite features using semi-synthetic molecular datasets. These tasks are analogs of real-world problems, such as identifying biologically active chemical groups; however, the labels determined by a synthetic "binding logic" that the ground truth feature relevance is known. We use several of the datasets constructed by [62], some of which were also used by (75 The synthetic "binding logics" are expressed as a combination of molecular fragments that must either be present or absent for binding to occur and are used to label molecules from the ZINC database (36). Each logic includes up to four functional groups (Table 6).
*Data from htapl//e/s/.hhhooooggeeeeeeeeeee 16_ [6_logics_ train_ and test.zis

---

Molecules are featurized using set of 84 functional groups, where feature = 1 if the molecule contains functional group and otherwise. The specific binding logics are given App. F
Table 2: Performance Chemistry Datasets, values recorded with their standard deviations.
DATASET
MODEL
TPR
FDR
Gun
No. GROUPS
ACCURACY (9)
COMPFS(5)
100.0 0.0
0.0 1.00
0.82 1 0.20
1,9 0.0.5
100.0 1.0
ORACLE
100.0=0.0
0.0 =00:00
0.50 0:00
0.00
100.0 0.00
LASSO
=0
0.50
1,0000
75.8
CHEMI
GROUP LASSO
100.0
0.0
0.67
3.0
100.0
STG
100.0 = 0.D
0.0 1.0
0.50 = 0.00
1.0
100.0 0.D.
SUP-CAE
62.5 13:2
23.3 17.5
0.37
1.000..
77.8 11.0
COMPFS(S)
100.0 =
0.0 u.0
0.72 # 0.24
0.6
0.8
ORACLE
100.0
0.50 0:00
100.0
CHEM2
LASSO
100.0 =00
0.0
0,50 1 0.00
1.0
81.6 1:
GROUP LASSO
100.0
0.0
0.40 0.00
5.0
81.6 0.0
STG
1.0
SUP-CAE
66.7 000
0.0
0.42
1,0000
80.9
COMPFS(S)
100.0
11.7
0.62
2.4
100.0
ORACLE
100.0
0.0
0.50 0:00
1.0
100.0 0.0
LASSO
100.0 =00 0.0
0.0
0.50 + 0.00
1.0
87.4
CиЕM3
GROUP LASSO
=
STG
100.0
0.0
0.50
1.0
100.0 0.0
SUP-CAE
62.5 1 13.2
23.3 17.5
0.37 0.0.0
1.0
77.8 11.0
Analysis. All methods are able to identify the ground truth relevant features; however, only CompFS provides deeper insights. Unlike for Synl-4, LASSO correctly selects the ground truth features since the dataset consists of binary variables and thus it is possible to find performant linear models. However, while discovering the correct features, Group LASSO selects all possible combinations of these features, adding no benefit over standard feature selection.
For Cheml-2, CompFS perfectly recovers the group structure in the majority of experiments, leading to high Gsim far exceeding traditional feature selection. On Chem3, we occasionally discover part of the binding logic. However, number of molecular fragments are strongly correlated with the binding logic, even though they are not themselves included. In fact, some features contain information about multiple functional groups. For example, esters contain a carbonyl and an ether; both are in the binding logic for Chem3, while ester is not, despite being highly informative, and thus occasionally CompFS incorrectly selects this feature. spite of this, CompFS achieves significantly higher Gsim than even Oracle. This demonstrates the benefit of the grouping discovered by CompFS, even with a modest number of false discoveries. As before, CompFS typically finds the correct number of groups (2), despite being able to discover up to groups, further demonstrating that the number of composite features need not be known priori, which is the case in real-world applications.
additional features that
5.3 MNIST
Dataset Description. We investigate CompFS on the MNIST dataset [48]. While this well-known dataset consists of 28x28 images and typically fixed pixel lo-
Group
Group 2
Group
Oroup 4
cations do not have specific meaning, it has been extensively used in the feature selec-
All Gooags
S7C
Sup CAE
Unsup CAE
tion literature due to the handwritten digits being centered and scaled, thus each of the 784 pixels can (somewhat) meaningfully treated as separate feature. While the ground truth group structure is unknown, benefit of MNIST is that can readily visualise selected features.
Figure 2: Pixels selected by CompFS and baselines.
Analysis. The features discovered by CompFS (using 4 groups), STG, Sup-CAE and Unsup- are shown in Figure 2 expected, all selected pixels are central and relatively spread out. However, the four groups discovered by CompFS appear to have slightly different focus, in particular Group 3.

---

To investigate the impact of these differences, we evaluate the predictive power of each of the groups. We find that the individual groups have relatively low accuracies (72%-81 81%). in part due only using 15 pixels (<2% of total). However, the union of these features achieve significantly greater accuracy of 95%, equalling the performance of STG and (Sup- This illustrates that the grouping does not seem to drastically affect performance, despite enforcing constraints on how the features are used by the model.
Finally, we consider the per-class accuracy of each of these groups, Interestingly, the variance in performance between classes is significantly higher for the groups than any of the overall methods (Fig. 4). For example, Group 2 struggles to identify digits 4 and 5, while Group performs poorly on digits 2, 3, and 8. This highlights the distinct information contained in each group. Further details and analysis is provided in App. G.3
5,4 Real-World Data: METABRIC
Table 3: METABRIC performance. We compare CompFS and STG using 25 fea- tures to an MLP using all 489 features.
Dataset Description. Finally, we assess CompFS on real-world dataset, METABRIC [2] 68), where the ground truth group structure is unknown. METABRIC contains gene expression, mutation, and clinical data for 1.980 primary breast cancer samples. We evaluated the ability to predict the progesterone receptor (PR) status of the tissue based on the gene expression data, which consists of measurements for 489 genes.
Model
AUROC
MLP (All features)
0,869
CompFS(5)
0.830
STG 0.843
Analysis. CompFS suffers limited performance degradation compared to using all features, despite only using 5% of the features (Table 3). Despite imposing a more rigid structural form on how features can interact in the predictive model, STG only had marginally greater predictive power than CompFS. However, CompFS provides greater insight into how the features interact than STG.
We found supporting evidence in the scientific literature for all but 1 of the genes discovered by CompFS (Table In addition, within each group, we found further evidence of the interactions between genes, demonstrating the ability for CompFS to learn informative groups of features. For example, in Group 1, CXCR1 and PFN-2 (the protein encoded by PSENEN) are known to interact 5 In Group 2, BMP6 encodes member of the TGF-3 superfamily of proteins, and TGF-8 triggers activation of SMAD3 [1]. In the same group, MAPKI activity is dependent on the activity of PRKCQ in breast cancer cells (15), while MAPKI is also known interact with MAPT (5]).
SMAD3 266. and BMP6 [96]. Additional supporting evidence can be found in Appendix H
6 Conclusion
In this paper, we introduced CompFS, an ensemble-based approach that tackles the newly proposed challenge of composite feature selection. Using synthetic and semi-synthetic data, we assess our ability to go beyond traditional feature selection and recover deeper underlying connections between variables. CompFS is not without limitations: as with other methods, points of difficulty arise when features are highly correlated, or if predictive composites contain overlapping features. Future work may overcome this by using correlated gates. Further, as with many traditional feature selection methods, there no guarantees false discovery rate. This could be tackled by first proposing candidate composite features, and then using the Group Knockoff procedure. Additionally, to discover groups, CompFS requires the introduction of additional hyperparameters which could be challenging to tune in practice. More broadly, as with standard feature selection, groups found under composite feature selection must be verified by domain experts (both features but additionally interactions). However, we believe the additional structure provided by composite feature selection could be of significant benefit wide variety of practitioners.
Acknowledgements
We thank the anonymous reviewers for their comments and suggestions. We also thank Bogdan Cebere and Evgeny Saveliev for reviewing our public code. Fergus Imrie and Mihaela van der Schaar are supported by the National Science Foundation (NSF, grant number 1722516) Mihaela van der Schaar additionally supported by the Office of Naval Research (ONR). Alexander Norcliffe supported by GlaxoSmith grant.
10

---

References
[1] Wail Al Sarakbi, Sara Reefy, Wen G. Jiang, Terry Roberts, Robert F. Newbold, and Kefah Mokbel. Evidence of tumour suppressor function for DLECI in human breast cancer. Anticancer Research, 30(4):077--1082 2010.
[2] Edoardo Amaldi and Viggo Kann. On the approximability of minimizing nonzero variables or unsatisfied relations in linear systems. Theoretical Computer Science, 209()):337-660, 1998.
[3] M. Ampuja, E.L. Alarmo, P. Owens, R. Havumen, A.E. Gorska, H.L. Moses, and A. Kallioniemi. The impact of bone morphogenetic protein 4 (BMP4) on breast cancer metastasis in mouse xenograft model. Cancer Letters, 375(2):238-24) 244,, 2016.
[4] Sharon Arcuri, Georgia Pennarossa, Fulvio Gandolfi, and Tiziana A. L. Brevini. Generation trophoblast-like cells from hypomethylated porcine adult dermal fibroblasts. Frontiers in Veterinary Science, 8, 2021.
[5] Martina Bakele, Amelie S. Lotz-Havla, Anja Jakowetz, Melanie Carevic, Veronica Marcos, Ania C. Muntau, and Dominik Gersting, Soeren Wland Hartl. An interactive network of elastase, secretases, and PAR-2 protein regulates CXCRI receptor surface expression neutrophils. Journal of Biological Chemistry, 289(3)):0016620522) 2014.
[6] Rubika Balendra and Adrian M. Isaacs. C9orf72-mediated AL.S and FTD: multiple pathways disease. Nature Reviews Neurology, 14(9):544 558, Sep 2018.
Muhammed Fatih Balın, Abubakar Abid, and James Zou. Concrete autoencoders: Differen-
tiable feature selection and reconstruction. In International Conference on Machine Learning (ICML), 2019.
[8] Hatice Ozer Balli and Bent E. Sørensen. Interaction effects in econometrics. Empirical Economics, 45(1):583-603, 2013.
[9] Rina Foygel Barber and Emmanuel Candès. Controlling the false discovery rate via knockoffs. The Annals of Statistics, 23555:2055 2015.
[10] Amir Beck and Marc Teboulle. fast iterative thieshigg- algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2(1):183-202, 2009.
[1]] Alexis Bellot and Mihaela van der Schaar. Conditional independence testing using generative adversarial networks. In Advances Neural Information Processing Systems (NeurIPS), 2019.
[12] Malgorzata Bogdan, Ewout van den Berg, Weijie Su, and Emmanuel Candes. Statistical estimation and testing via the sorted L1 norm. arXiv preprint arXiv: 2013.
[13] Terry Brown. Silica exposure, smoking, silicosis and lung cancer-comples interactions. Occupational Medicine, 59(2):99-95, 03 2009.
[14] Damian Brzyski, Alexej Gossmann, Weijie Su. and Malgorzata Bogdan. Group SLOPE adaptive selection of groups of predictors. Journal of the American Statistical Association. 114(525):419-4333 2019.
[15] Jessica Byerly, Gwyneth Halstead-Nuseee Koichi Ito, Igor Katsyv, and Hanna Y. Irie. PRKCQ promotes oncogenic growth and anoikis resistance of subset of triple-negatin breast cancer cells. Breast Cancer Reseurch, 18(1):95, 2016
[16] Jessica H. Byerly, Elisa Port, and Hanna Y. Irie. PRKCQ inhibition enhances chemosensi- tivity of triple-negative breast cancer by regulating Bim. Breast Cancer Research, 22(1):72, 2020.
[17] Emmanuel Candes, Yingying Fan, Lucas Janson, and Jinchi Lv. Panning for gold: 'model-X' knockoffs for high dimensional controlled variable selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 80(3)5551-577, 2018
[18] Girish Chandrashekar and Ferat Sahin. A survey on feature selection methods. Computers Electrical Engineering, 40(1):16-28, 2014.

---

[19] Bijun Chen, Ruoshui Li, Silvia C. Hernandez. Anis Hanna, Kai Su, Arti V. Shinde, and Niko- laos Frangogiannis. Differential effects of smad2 and smad3 in regulation of macrophage phenotype and function in the infarcted myocardium. Journal of Molecular and Cellular Cardiology, 171:1-15, 2022.
[20] Tianyi Cheng, Peiying Chen, Jingyi Chen, Yingtong Deng, and Chen Huang. Landscape analysis of matrix metalloproteinases unveils key prognostic markers for patients with breast cancer. Frontiers in Genetics, 12, 2022.
[21] Christina Curtis, Sohrab P. Shah, Suet-Feung Chin, Gulisa Turashvili, Oscar M. Rueda, Mark J. Dunning, Doug Speed, Andy G. Lynch, Shamith Samarajiwa, Yinyin Yuan, Stefan Gräf, Gavin Ha, Gholamreza Haffari, Ali Bashashati, Roslin Russell, Steven McKinney, METABRIC Group, Anita Langerød, Andrew Green, Elena Provenzano, Gordon Wishart, Sarah Pinder, Peter Watson, Florian Markowetz, Leigh Murphy, Ian Ellis, Arnie Purushotham, Anne-Lise Borresen-Dale, James D. Brenton, Simon Tavaré, Carlos Caldas, and Samuel Aparicio. The genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups. Nature, 486(7403):346 352, 2012.
[22] Kun Dai, Hong-Yi Yu, and Qing Li. A semisupervised feature selection with support vector machine. Journal of Applied Mathematics, 64: 14:141,158, 2013.
[23] Ran Dai and Rina Barber. The knockoff filter for FDR control in group-sparse and multitask regression. In International Conference on Machine Learning (ICML), 2016.
[24] Mei Dong, Tam How, Kellye C. Kirkbride, Kelly J. Gordon, Jason D. Lee, Nadine Hempel, Patrick Kelly, Benjamin J. Moeller, Jeffrey R. Marks, and Gerard C. Blobe. The type III TGF-8 receptor suppresses breast cancer progression. The Journal of Clinical Investigation, 117(():206-21) 2007.
[25] Gary Doran, Krikamol Muandet, Kun Zhang, and Bernhard Schölkopf. A permutation-baser kernel conditional independence test. In Uncertainty in Artificial Intelligence (UAI), pages 132-141, 2014.
[26] Wei Bin Fang, Iman Jokar, Zou, Diana Lambert, Prasanthi Dendukuri, and Nikki Cheng. CCL2/CCR2 chemokine signaling coordinates survival and motility of breast cancer cells through smad3 protein- and p42/44 mitogen-aattivate protein kinase -dePenddeeendent mechanisms. Journal of Biological Chemistry, 287(4)) 2012.
[27] Michael Y. Fessing, Ruzanna Atoyan, Ben Shander, Andrei N. Mardaryev, Vladimir V. Botchkarev Jr., Krzysztof Poterlowicz, Yonghong Peng, Tatiana Efimova, and Vladimir A. Botchkarev. BMP signaling induces cell-type-specffic changes in gene expression programs of human keratinocytes and fibroblasts. Journal of Investigative Dermatology, 130(2):398-40) 2010.
[28] Edward I. George and Robert E. McCulloch. Approaches for bayesian variable selection. Statistica Sinica, pages 339-373, 1997.
[29] Christophe Ginestier, Suling Liu, Mark E. Diebel, Hasan Korkaya, Ming Luo, Marty Brown, Julien Wicinski, Olivier Cabaud, Emmanuelle Charafe-Jauffret, Daniel Birnbaum, Jun-Lin Guan, Gabriela Dontu, and Max S. Wicha. CXCRI blockade selectively targets human breast cancer stem cells in vitro and in xenografts. The Journal of Clinical Investigation. (20(2):485-497, 2010.
[30] Santiago M. Gómez Bergna, Abril Marchesini, Leslie C. Amorós Morales, Paula N. Arrías, Hernán G. Farina, Víctor Romanowski, M. Florencia Gottardo, and Matias L. Pidre. Exploring the metastatic role of the inhibitor of apoptosis BIRC6 in breast cancer. bioRxiv, 2021.
[31] Daniel B. Graham and Ramnik J. Xavier. Pathway paradigms revealed from the genetics of inflammatory bowel disease. Nature, 578(7796):527-53, Feb 2020.
[32] Isabelle Guyon and André Elisseeff. An introduction to variable and feature selection. Journal of Machine Learning Research, 117 2003.
12

---

[33] Kai Han, Yunhe Wang, Chao Zhang, Chao Li, and Chao Xu. Autoencoder inspired unsuper- vised feature selection. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018.
[34] Xiaofei He, Deng Cai, and Partha Niyogi. Laplacian score for feature selection. In Advances in Neural Information Processing Systems (NewrIPS), 2005.
[35] Daniel Hernndezz spike-and and-slab priors for bayesian group feature selection using expectation propagation. Journal of Machine Learning Research, 14(7), 2013.
José Miguel Hernández-Lobate and Pierre Dupont. Generalized
[36] John J. Irwin, Teague Sterling, Michael M. Mysinger, Erin S. Bolstad, and Ryan G. Coleman. ZINC: A free tool to discover chemistry for biology. Journal of Chemical Information and Modeling, 12(7):1577-1768, 2012.
[37] Paul Jaccard. The distribution of the flora in the alpine zone. The New Phytologist, 11(2):37-50, 1912.
[38] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizatioo with Gumbel-Softmee In International Conference on Learning Representations (ICLR), 2017.
[39] James Jordon, Jinsung Yoon, and Mihaela van der Schaar. KnockofIGAN: Generating knock- offs for feature selection using generative adversarial networks. In International Conference on Learning Representations (ICLR), 2018.
[40] Päivi Järvensivu, Taija Heinosalo, Janne Hakkarainen, Pauliina Kronqvist, Niina Saarinen, and Matti Poutanen. HSD17B1 expression induces inflammation-aide aided rupture of mammary gland myoepithelium. Endocrine- Cancer, 25(4):393 - 406, 2018.
[41] Tadashi Kato, Atsushi Yamada, Mikiko Ikehata, Yuko Yoshida, Kiyohito Sasa, Naoko Morimura, Akiko Sakashita. Takehiko lijima, Daichi Chikazu, Hiroaki Ogata, and Ryu- Kamijo. FGF-2 suppresses expression of nephronectin via JNK and PI3K pathways. FEBS Open Bio, 8(5):836 (00)8336-42.200 2018.
[42] Matthew Kelly and Christopher Semsarian. Multiple mutations in genetic cardiovascular disease. Circulation: Cardiovascular Genetics, 2(2):182-190, 2009.
[43] Pora Kim, Feixiong Cheng, Junfei Zhao, and Zhongming Zhao. ccmGDB: database for cancer cell metabolism genes. Nucleic Acids Research, 44(D)):9999-D968, 2015.
[44] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 141..690. 2014.
[45] Kenji Kira and Larry A. Rendell. A practical approach to feature selection. Machine Learning Proceedings, pages 246-256. 1992.
[46] Theo A. Knijnenburg, Gunnar W. Klau, Francesco Iorio, Mathew J. Garnett, Ultan McDermott, Ilya Shmulevich, and Lodewyk F. A. Wessels. Logic models to predict continuous outputs based on binary inputs with an application to personalized cancer therapy. Scientific Reports, 6(1):36812, 2016.
[47] Ron Kohavi and George H. John. Wrappers for feature subset selection. Artificial Intelligence, -2244,, 1997.
[48] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. bradient-based learning applied to document recognition. Proceedings of the IEEE,
1998.
[49] Changhee Lee, Fergus Imrie, and Mihacla van der Schaar. Self-supervision enhanced feature selection with correlated gates. In International Conference on Learning Representations (ICLR), 2022.
[50] Ismael Lemhadri, Feng Ruan, and Rob Tibshirani. LassoNet: Neural networks with feature sparsity. In International Conference Artificial Intelligence and Statistics (AISTATS), 2021.
13

---

[51] Chad Leugers, Ju Yong Koh, Willis Hong, and Gloria Lee. Tau in MAPK activation. Frontiers in Neurology, 2013.
[52] Xinghua Li. Weijiang Liang, Junling Liu, Chuyong Lin, Shu Wu, Libing Song, and Zhongyu Weeniiiinn Yuan. Transducin (5)-like X-linked receptor promotes proliferation and tumorigenicity in human breast cancer via activation of beta-catenin signaling. Breast Cancer Research, 16(5):465, 2014.
[53] Yifeng Li Li, Chih-Y Yu Chen, and Wyeth W. Wasserman. Deep feature selection: theory and application to identify enhancers and promoters. Journal of Computational Biology, 23(5):322-3366 -36, 2016.
[54] Faming Liang, Qizhai Li, and Lei Zhou. Bayesian neural networks for selection of drug sensitive genes. Journal of the American Statistical Association, 11((523)955--972, 2018.
[55] Ofir Lindenbaum, Uri Shaham, Erez Peterfreund, Jonathan Svirsky, Nicolas Casey, and Yuval Kluger. Differentiable unsupervised feature selection based on a gated laplacian. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[56] Huan Liu and Rudy Setiono. A probabilisticc approach to feature selection filter solution. In International Conference on Machine Leurning (ICML), 1996.
[57] Ying Liu and Cheng Zheng. Deep latent variable models for generating knockoffs. 8(1 8(1):e260, 2019.
Stat,
[58] Huanyu Lu, Yue Guo, Gaurav Gupta, and Xingsong Tian. Mitogen-activanee protein kinase (MAPK): New insights in breast cancer. Journal of Environmental Pathology, Toxicology and Oncology, 38(1):51-59, 2019.
[59] Chris J. Maddison, Andriy Mnih, and Yee Whye Tch. The Concrete distribution: A con- tinuous relaxation of discrete random variables. International Conference on Learning Representations (ICLR), 2017.
[60] Ramamurthy Mani, Robert P. St.Onge, John L. Hartman, Guri Giaever, and Frederick P. Roth. Defining genetie interaction. Proceedings of the National Academy of Sciences, 105(9):3461- 3466, 2008
[61] Pulak R. Manna, Ahsen U. Ahmed, Shengping Yang, Madhusudhanan Narasimhan, Joëlle Johen-Tannoudie Andrzej T. Slominski, and Kevin Pruitt. Genomic profiling of the steroido- genic acute regulatory protein in breast cancer: In silico assessments and a mechanistic perspective. Cancers, 1(5), 2019.
[62] Kevin McCloskey, Ankur Taly, Federico Monti, Michael P. Brenner, and Lucy J. Colwell. Using attribution to decode binding mechanism in neural network models for chemistry. Proceedings of the National Academy of Sciences, 116(24): 1624-11629, 2019.
[63] Lisiane B. Meira, Antonio M.C. Reis, David I.. Cheo, Dorit Nahari, Dennis Burns, and Errol C. Friedberg. Cancer predisposition in mutant mice defective in multiple genetic pathways: uncovering important genetic interactions. Mutation hanturcc//Fudamttaal and Molecular Mechanisms of Mutagenesis, 477(1):51-58. 2001.
[64] Jordi Merino, Marta Guasch-Ferré, Jun Li, Wonil Chung, Yang Hu, Baoshan Ma, Yanping Li, Jae H. Kang, Peter Kraft, Liming Liang, Qi Sun, Paul W. Franks, JoAnn E. Manson, Walter C. Willet, Jose C. Florez, and Frank B. Hu. Polygenic scores, diet quality, and type 2 diabetes risk: An observational study among 35,759 adults from 3 US cohorts. PLOS Medicine, 19(4):1-20. 04 2022.
[65] Sofa Papadimitriou, Andrea Gazzo, Nassim Versbraegen, Charlotte Nachtegael, Jan Aerts, Yves Moreau, Sonia Van Dooren, Ann Nowé, Guillaume Smits, and Tom Lenaerts. Predicting disease-causing variant combinations. Proceedings of the National Academy of Sciences, 1 116724) 2019.
14

---

[66] Ui-Hyun Park, Mi Ran Kang, Eun-Joo Kim, Young-Sor Soo Kwon, Wooyoung Hur. Seung Kew Yoon. Byoung-Joon Song, Jin Hwan Park, Jin-Tack Hwang. Ji- Ji-Cheon Jeong, and Soo-Jong Um. ASXI. promotes proliferation of breast cancer cells by linking ERa to histone methylation. Oncogene, 35(28):3722-3752, 2016.
[67] Hanna M. Peltonen, Annakaisa Haapasalo, Mikko Hiltunen, Vesa Kataja, Veli-Matti Kosma, and Arto Mannermaa. r-secretase components as predictors of breast cancer outcome. PLOS ONE, ((11), 2013.
[68] Bernard Pereira, Suet-Feung Chin, Oscar M. Rueda, Hans-Kristian Moen Vollan, Elena Provenzano, Helen A. Bardwell, Michelle Pugh, Linda Jones, Roslin Russell, Stephen-John Sammut, Dana W. Y. Tsui, Bin Liu, Sarah-Jane Dawson, Jean Abraham, Helen Northen, John F. Peden, Abhik Mukherjee, Gulisa Turashvili. Andrew R. Green, Steve McKinney. Arusha Oloumi, Sohrab Shah, Nitzan Rosenfeld, Leigh Murphy, David R. Bentley, Ian O. Ellis, Arnie Purushotham, Sarah F. Pinder, Anne-Lise Syprresen-aake Helena M. Earl, Paul D. Pharoah, Mark T. Ross, Samuel Aparicio, and Carlos Caldas. The somatic mutation profiles of 2,433 breast cancers refine their genomic and transcriptomic landscapes. Nature Communications, 7(1):11479, 2016.
[69] Patrick C. Phillips. Epistasis - the essential role of gene interactions in the structure and evolution of genetic systems. Nature Reviews Genetics, 9(1 ((11))855--67, 2008.
[70] Michael W. Pickup, Laura D. Hover, Eleanor R. Polikowsky, Anna Chytil, Agnieszka E. Gorska, Sergey V. Novitskiy, Harold L. Moses, and Philip Owens. BMPR2 loss in fibroblasts promotes mammary carcinoma metastasis via increased inflammation. Molecular Oncology, 91):1199111, 2015.
[71] Barbara Maria Piskór, Andrzej Przylipiak, Emilia Dąbrowska, Iwona Sidorkiewiez, Marek Niczyporuk, Maciej Szmitkowski, and Stawomir Lawicki. Plasma level of MMP-10 may be prognostic marker early stages of breast cancer. Journal of Clinical Medicine, 9(2),, 2020.
[72] Franck Rapaport, Emmanuel Barillot, and lean-Philippe Vert. Classification of arraYGGH data using fused SVM. Bioinformatics, 24(13):1375-i82, 2008.
[73] Yaniv Romano, Matteo Sesia, and Emmanuel Candès. Deep knockoffs. Journal of the American Statistical Association, 115(532):8811-1777)) 2020.
[74] Jakob Runge. Conditional independence testing based nearest-neighbor estimator of conditional mutual information. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 938-947, 2018.
[75] Benjamin Sanchez-LLengllinggg Jennifer Wei, Brian Lee, Emily Reif, Peter Wang, Wesley Qian, Kevin McCloskey, Lucy Colwell, and Alexander Wiltschko. Evaluating attribution for graph neural networks. In Advances Neural Information Processing Systems (NeurIPS), 2020.
[76] Fadil Santosa and William W. Symes. Linear inversion of band-limited reflection seismograms. SIAM Journal on Scientific and Statistical Computing, 7(4):1307-1330, 1986.
[77] Rajat Sen, Ananda Theertha Suresh, Karthikeyan Shanmugam, Alexandros G Dimakis, and Sanjay Shakkottai. Model-powered conditional independence test. In Advances in neural information processing systems (NeurIPS), 2017.
[78] Razich Sheikhpour, Mehdi Agha Sarram, Sajjad Gharaghani, and Mohammad Ali Zare Cha- hooki. A survey semi-supervised feature selection methods. Pattern Recognition, 64:141- 158, 2017.
[79] Prajjal Singha, Srilakshmi Pandeswara, Hui Geng, Rongpei Lan, Manjeri Venkatachalam, Albert Dobi, Shiv Srivastava, and Pothana Saikumar. Increased smad3 and reduced smad2 levels mediate the functional switch TGF-3 from growth suppressor to growth and metastasis promoter through TMEPAIPMEPAI in triple negative breast cancer. Genes & cancer, 10(5- 6):134, 2019.
[80] Mukund Sudarshan, Wesley Tansey, and Rajesh Ranganath. Deep direct likelihood knockoffs. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
15

---

[81] Mina Takahashi, Fumio Otsuka, Tomoko Miyoshi. Hiroyuki Otani, Junko Goto, Misuzu Yamashita, Toshio Ogura, Hirofumi Makino, and Hiroyoshi Doihara. Bone morphogenetic protein 6 (BMP6) and BMP7 inhibit estrogen-induced proliferation of breast cancer cells by suppressing p38 mitogen-activatee protein kinase activation. Journal of Endocrinology, 199(3):445 455, 2008.
[82] Jiliang Tang, Salem Alelyani, and Huan Liu. Feature selection for classification: A review. Data classification: Algorithms and applications, page 37, 2014.
[83] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B B (Methodological), 58(1):267-288, 1996.
[84] Nicholas Turner, Alex Pearson, Rachel Sharpe, Maryou Lambros, Felipe Geyer, Maria A. Lopez-Garcia, Rachael Natrajan, Caterina Marchio, Elizabeth Iorns, Alan Mackay, Cheryl Gillett, Anita Grigoriadis, Andrew Tutt, Jorge S. Reis-Filho, and Alan Ashworth. FGFRI amplification drives endocrine therapy resistance and is a therapeutic target in breast cancer. Cancer Research, 70(5):2085-2099) 2010.
[85] Dongfeng Wang. Jian Li, Fengling Cai, Zhi Xu. Li Li. Huanfeng Zhu, Wei Liu, Qingyu Xu, Jian Cao, Jingfeng Sun, and Jinhai Tang. Overexpression of MAPT-ASI is associated with better patient survival in breast cancer. Biochemistry and Cell Biology, 97(2):158- -64, 2019.
[86] Dongsheng Wang, Chenglong Zhao, Liangliang Gao, Yao Wang, Xin Gao, Liang Tang, Kun Zhang, Zhenxi Li, Jing Han, and Jianru Xiao. NPNT promotes early-stage bone metastases in breast cancer by regulation of the osteogenic niche. Journal of Bone Oncology, 13:91-96. 2018.
[87] Chang- Yuan Wei, Qi-Xing Tan, Xiao Zhu, Qing-Hong Qin, Fei-Bai Zhu, Qin-Guo Mo, and Wei-Ping Yang. Expression of C/221A/p21 and TGFBR2 in breast cancer and their prognos- tic significance. International Journal of Clinical and Experimental Pathology, 8(1 111111441,, 2015.
[88] Michael K. Wendt, Molly A. Taylor, Barbara J. Schiemann, Khalid Sossey-Alaoe Alaoui, and William P. Schiemann. Fibroblast growth factor receptor splice variants are stable markers oncogenic transforming growth factor 31 signaling metastatic breast cancers. Breast Cancer Research, 16(2):R24, 2014.
[89] Tim Wilson, Tim Holt, and Trisha Greenhalgh. Complexity and clinical care. BMJ, 323(7314):685-688 2001.
[90] Xinxing Wu and Qiang Cheng, Algorithmic stability and generalization of an unsupervised feature selection algorithm. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[91] Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, and Yuval Kluger. Feature selection using stochastic gates. In International Conference on Machine Learning (ICML), 2020.
[92] Jinsung Yoon. James Jordon, and Mihaela van der Schaar. Invase: Instance-wise variable selection using neural networks. In International Conference on Learning Representations (ICIR), 2019.
[93] Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series (Statistical Methodology). 68(1):49-67, 2006.
[94] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexander Smola. Deep sets. In Advances in Neural Information Processing Systems (NeurIPS). 2017.
[95] Lingmin Zeng and Jun Xie. Group variable selection via SCAD-L2, Statistics, 48(1):49 66. 2014.
[96] Xin- Yue Zhang, Hsun-Ming Chang, Elizabeth L. Taylor, Rui-Zhi Liu, and Peter C. K. Leung. BMP6 downregulates GDNF expression through SMADI/5 and ERK signaling pathways in human granulosa-lutee cells. Endocrinology, 159(8):2926-999 2018.
16

---

[97] Yong-ping Zhang, Wen-ting Na, Xiao-qiang Dai. Ruo-fei Li, Jian-xiong Wang, Ting Gao, Wei-bo Zhang, and Cheng Xiang, Over-expression SRDSA3 and its prognostic significance in breast cancer. World Journal of Surgical Oncology, 19(1):260, 2021.
[98] Jidong Zhao, Ke Lu, and Xiaofei He. Locality sensitive semi-supervised feature selection. Neurocomputing, 71:1842 1849, 2008
[99] Ting Zhong, Feifei Xu, Jinhui Xu, Liang Liu, and Yun Chen. Aldo-keto reductase 1C3 (AKRIC3) associated with the doxorubicin resistance in human breast cancer via PTEN loss. Biomedicine & Pharnacotherapy, 69:317-325, 2015.
[100] Nengfeng Zhou and Ji Zhu. Group variable selection via hierarchical lasso and its oracle property, arXiv preprint arXiv 2871, 2010.
[101] Guangyu Zhu and Tingting Zhao. Deep-gKnock: Nonlinear group-feature selection with deep neural networks. Neural Networks, 135,139-147,,000
17

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Conclusion for details.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] In the Conclusion, we caution that as with any feature selection method, discovered features must be verified evaluated by domain experts. This verification evaluation might be costly, and should the method perform poorly, could result in wasted resources. In addition, without additional oversight (primarily dataset construction but also when validating features), features that contain bias could remain and be identified by feature selection algorithms.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proof' of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes] Code is available at either of the following GitHub repositories: com/
Peeteesteeeeeattree-eeeeeeiiiiee https //github. cwm/ http:///hww..oooooocoooooo/oo/ta
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Hyperparameters for each experiment are provided in Table 4 Architecture details are provided in Appendix Caand further experimental details
provided in Appendix D
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] All experiments are repeated 10 times and results are reported along with standard deviations.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] All experiments can be run easily commereially-availabee laptop. We provide further details of the compute resources used in Appendix D
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]] We used several existing methods and datasets (see Experiments). All benchmark methods and datasets are clearly cited.
(b) Did you mention the license of the assets? [Yes] Licenses of assets (benchmark methods and datasets) is provided in Appendix[Dannd
(c) Did you include any new assets either in the supplemental material or as URL? [Ys]] Code is available at either of the following GitHub repositories: httpse///gccaab...aaaaarroriiiie
https
//github.
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [Yes] We only use publicly available, anonymized datasets. See Broader Impact- Datasets.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] We only use publicly available, anonymized datasets. See Broader Impact- Datasets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
18

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
19


Reviews:
[{"review_id": "pseP8B6kpzc", "reviewer": "Reviewer_WTbm", "summary": "The authors of the paper propose a method for solving the problem of selecting groups of features in a supervised learning setting. The method is based on an ensemble of models where the number of models is the same as the expected number of groups. A new metric is also proposed to evaluate the quality of selected groups of features. ", "questions": "You claim that there is no source of dataset where the ground truth of groups is provided and that is the reason you use the Oracle method that selects a single group. Is it possible to manually construct such a dataset? \n\nWhy do you use a small number of groups for synthetic datasets (up to 3) ? I think it is more convenient to check the method on a larger number of groups and also to evaluate it with varying the number of groups (is its performance degraded)?  \n\nHow is your method compared to the random initialization of any stochastic gates method? It is possible to train FS net and to get for each initialization different  selected features (assuming some stochasticity in the selection process).", "limitations": "The social impact is discussed.\nIn conclusion the authors describe some common limitations of the feature selection task but it is more reasonable to mention the limitations that are specific to your work: in the definition or in the method.\n\n", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is written well, it is very clear and is easy to follow. The authors present the challenges of the group selection task, provide formal definition and a simple method to solve the task. The proposed formula for loss seems reasonable.\n\nMy main concern is about the problem the authors try to tackle. The incentives which are discussed in the introduction section for solving group feature selection problem are not convincing enough. I think it is better to provide an illustrative example that explains how groups of features could benefit the relevant stakeholders. One of the advantages of feature selection methods is to detect correlated features and exclude them by remaining a single representative for each “correlation group” (e.g. concrete autoencoders). So why should we care about these groups? The correlation could be based on linear and non linear interactions between features. E.g. in yout MNIST experiment, you got groups that look almost the same (1,3,4).\nIn addition the experimental part where only 3 datasets (Syn1-Syn4 could be labeled as a single dataset)  are used to evaluate the proposed method is not convincing. For datasets selected in the experiments p << number of samples, which is a non challenging setting for supervised feature selection task. I think it is better to justify the significance of your method on the more datasets including the real datasets.\nAn additional weakness of the proposed method is that it produces probability for each feature to be selected and not a binary vector. It requires an additional hyper parameter to be tuned manually.\n\nAdditional technical issues:\nLine 67  - w_i is not defined before you mention it\nLine 93 -  the space of functions F is not defined before it used here\nLine 116 - rho function is not defined / described before it’s used here. If it’s the same as in the paper Deep Sets I think it is better to mention it.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "eG7ONpyow4k", "reviewer": "Reviewer_iDHM", "summary": "This paper proposed a composite feature selection, which aims to find groups of features with combined effects for prediction. More specifically, the authors proposed a deep learning architecture that uses an ensemble of feature selection models to find predictive groups. Experimental results on synthetic datasets, semi synthetic datasets and MNIST dataset demonstrated the effectiveness of the proposed method.", "questions": "1. Can you use random forest or GBDT for feature grouping? For example, you can use the features shown in the same tree as a group.\n2. For feature grouping, a trivial way is to cluster the features with clustering methods. How does the proposed method perform compared with clustering methods, e.g., spectrum clustering?\n", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n1. The paper proposed a new setting for feature selection problems, i.e., selecting feature groups that features in each group may interact with each other.\n2. The experimental results seem promising. \n3. This paper is well-written and easy to follow.\n\nCons:\n1. The problem setting seems to be related to some existing works. It seems to me that the setting in this paper is similar to the sub structures of the many machine learning tasks, e.g., there are several sub tasks which may or may not overlap with each other. In this paper, the authors proposed a solution for automatically discovering these sub tasks with the goals that all sub tasks can perform well and the ensemble of them can achieve optimal performance. This might be related to traditional ensemble learning methods, e.g., random forest or GBDT, in which many trees (upon a subset of features) are learned individually and then combined to achieve optimal accuracy.\n2. In the abstract, it is mentioned that “For example, in genomics, diseases might not be caused by any single mutation but require the presence of multiple mutations”. This may not be supportive to the motivation of this paper. It might be better to show some examples of how different combinations of mutations can cause the same disease. But I am not sure how often it is in real diseases. \n3. Section 3 mentioned the challenge that different groups may have overlapping features, but I did not see how the proposed method can address such challenge.\n4. The experiments on semi-synthetic datasets seem interesting, but it could be more convincing if there were some experiments on real medical/chemical datasets.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "S8oeHlBpFST", "reviewer": "Reviewer_LHmD", "summary": "Most features of a sample are usually not independent but interdependent. In this paper, the author(s) introduced a method combining deep learning and ensemble learning to identify informative feature groups. Empirical experiments were implemented to validate the performance of the proposed method.", "questions": "As for the Weaknesses mentioned above, to be specific, I have the following questions/comments which need further explanation from the author(s). Please correct me if I am wrong. Thanks.\n\n1. In lines 10-11, it mentioned that \"...we can rank the groups based on their predictive power for downstream decision making...\" I didn't see an empirical example of this, or did I miss it?\n\n2. In lines 46-47, \"...why specific features are important than standard feature selection, with the goal of improving data-driven...\". \"specific features\" or \"specific feature selection\"?\n\n3. In lines 82-85, \"complex interactions may happen within groups but only simple interactions between ... we do not enforce assumptions on the groups such as non-overlapping groups or every feature being...\", I do not think it will hold for most cases due to \"non-overlapping groups\", for example, Group_1=\\{x_1,x_2\\} and Group_2=\\{x_3,x_2\\}, if there exist complex interactions between x_1 and x_2, then it will make the interactions between Group_1 and Group_2 become complex. From formula (4), I think it might be more accurate to say that the latent spaces from different groups are simple interactions, right?\n\n4. For the implementation of STG and CAE, the author(s) just simply mentioned \"Use the available Implementations\". As far as I know, CAE in its original paper involved the use of different structures, such as one with no hidden layer, one with a hidden layer, and so on. So the author(s) should specify which specific structure is used. In addition, for a fair comparison, the compared methods should have similar or same structures.\n\n5. Suggesting that for MNIST visualization examples, please (consider to) give more numerical results, so that the reader can see more clearly which model is really good.\n\n6. At first, I was attracted by the motivation the author(s) demonstrated; that is, the example of genomics at the beginning, but after reading this paper, I was a little disappointed that I did not notice any genome-related data used in the analysis to illustrate the advantages of the proposed method. That's one of the reasons I have to give the current score. \n\n---\n\nIn addition, there are some tiny issues/typos:\n\n(1) Journal/conference names are sometimes capitalized and sometimes not.\n\n(2) The author's name in the reference is wrong, such as Ref. [23].\n\n(3) In References, the format of conferences is inconsistent, sometimes abbreviations are given and sometimes not.\n\n(4) In References, the format of authors' names is inconsistent, sometimes abbreviations are used and sometimes not.\n\nI look forward to the response from the author(s), if the response adequately explains and can show the unique superiority of the proposed method in this respect, I will consider raising the score. Thanks.\n\n\n\n-----------------------------------------------------------------------\n\nThe author(s) clarified most of my concerns. So I would like to raise my score from (4) Borderline reject to (5) Borderline accept. Thanks.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The topic of this paper looks interesting, I have the following comments:\n\n### Strengths\n---\n1. The motivation of this paper looks interesting, as the authors illustrated at the beginning that \"...in genomics, diseases might not be caused by any single mutation but require the presence of multiple mutations...\". This paper attempted to detect groups of predictive features without the need for predefined groups.\n\n2. Experiments on synthetic and semi-synthetic datasets showed some advantages to some extent.\n\n---\n---\n\nAlthough the topic studied by this paper is interesting, I think the paper has the following deficiencies or unclear points (Please see the \"Questions\" section below for details).\n\n### Weaknesses\n---\n\n1. Behind the group feature selection idea proposed in this paper is deep architecture plus ensemble learning, or more simply, ensemble STG in essence. Innovation seems limited.\n\n2. The research review of traditional/standard feature selection methods is not sufficient. In fact, the traditional/standard feature selection methods have been developed even in the past two years, both in terms of method novelty and performance advantages.\n\n3. Maybe the proposed method should be validated on genomics-related data.\n\n---\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Bq6XN5pzKY", "reviewer": "Reviewer_jAMg", "summary": "The authors address a challenging and important problem of group feature selection. The uniqueness of their work is that they learn the grouping without any predefined knowledge. Feature selection is an important problem in computational biology, and quite often there are groups of features that act together in some biological/physical system. Finding the groups is important for understanding the driving factors behind the system that is being studied. The proposed method for solving this problem relies on an ensemble of feature selection models to find the informative features and groups. The authors demonstrate the effectiveness of their method using synthetic, semi-synthetic, and real data.", "questions": "Some suggestions for improving the paper:\n-Section 2- Knockoff procedure- > since these works typically have a slightly different goal in mind, it is worth explaining to the reader what are these methods trying to achieve. Just expand with a coupled of lines.\n-Section 3- complex vs. simple interactions. These terms have not yet been defined or explained, so this mention is confusing for the reader. Please expand. \n-Missing comma after equations 1,2, 3.\n-Section 3.2 line 106- when both are known-> do you mean when both are used by the model? The word known may be confusing since the features are always known (they are observed).\n-Missing commas after many other equations.\n-Why are the results of STG and CAE omitted from the main text? Clearly, these are the strongest competitors, both for the synthetic and for MNIST.\n-P9 L327 “c.” not sure what this means.\n-Why are you reporting only the average number of selected features? Isn’t the union more informative? Both for interpretability and computational aspects this is more important. Also, how can you compare this to STG and CAE if they select only a union of features? Shouldn’t those be in the same ball park?\nI would be glad to consider raising my score if the authors would be able to provide a more challenging example with an evaluation of their method, since I think this is the major weakness of the paper.", "limitations": "The authors point out that the limitations appear in the conclusion, but I don’t see a real discussion about the limitations of the method there. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall the paper is well written, and I enjoyed reading it. The English level is satisfactory, and the method is motivated well and is scientifically sound. \nThe proposed method relies on modern tools for feature selection, namely, probabilistic relaxation of Bernoulli variables, which have been found effective for this task in various applications. I would suggest trying a Gaussian distribution instead of the Concrete as shown in STG [40] but this is minor. \nOne challenge of using the method is tuning the regularization parameters, which might be challenging since there are two in the proposed method.  Another weakness is that the authors don’t use any real-world data (MNIST is not really a dataset for feature selection). The semi-synthetic example is nice, but it seems that multiple methods lead to perfect accuracy, TPR, and FDR. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept